You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The test suite takes about 30 minutes to run on circle CI and is unstable due to transient issues associated requesting data from external APIs.
Specifically, one of the main issues is that RAiDER tests download data from various weather model APIs during runtime. These "integration" tests can fail due to transient connectivity issues, not the RAiDER code itself, and the entire test suite has to be re-run. In theory, downloads or API testing should be isolated into a single test or better still, mocked for correctness. Additionally, algorithmic tests should not require downloading historical weather model data and generally should be requesting any data from external APIs.
That said, it requires more work to expand the test suite to this end. Also, the current test suite has excellent coverage for RAiDER's applicability. We have to figure out how to ensure how the package remains reliable (via testing) while improving the efficiency and reliability of the test suite itself. This is particularly evident with GMAO downloads failing on some random nontrivial fraction of the tests.
Possible options are
staging data in S3 (still requests data from the internet),
making unit tests smaller with simple embedded array tests
sharing weather model data downloadedacross tests (minimizes internet requests but does not necessarily solve the issue) - i.e. through pytest fixtures
The text was updated successfully, but these errors were encountered:
The test suite takes about 30 minutes to run on circle CI and is unstable due to transient issues associated requesting data from external APIs.
Specifically, one of the main issues is that RAiDER tests download data from various weather model APIs during runtime. These "integration" tests can fail due to transient connectivity issues, not the RAiDER code itself, and the entire test suite has to be re-run. In theory, downloads or API testing should be isolated into a single test or better still, mocked for correctness. Additionally, algorithmic tests should not require downloading historical weather model data and generally should be requesting any data from external APIs.
That said, it requires more work to expand the test suite to this end. Also, the current test suite has excellent coverage for RAiDER's applicability. We have to figure out how to ensure how the package remains reliable (via testing) while improving the efficiency and reliability of the test suite itself. This is particularly evident with GMAO downloads failing on some random nontrivial fraction of the tests.
Possible options are
The text was updated successfully, but these errors were encountered: