Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shorten runtime and improve reliability of test suite #535

Open
bbuzz31 opened this issue May 11, 2023 · 0 comments
Open

Shorten runtime and improve reliability of test suite #535

bbuzz31 opened this issue May 11, 2023 · 0 comments
Labels
enhancement New feature or request optimization

Comments

@bbuzz31
Copy link
Collaborator

bbuzz31 commented May 11, 2023

The test suite takes about 30 minutes to run on circle CI and is unstable due to transient issues associated requesting data from external APIs.

Specifically, one of the main issues is that RAiDER tests download data from various weather model APIs during runtime. These "integration" tests can fail due to transient connectivity issues, not the RAiDER code itself, and the entire test suite has to be re-run. In theory, downloads or API testing should be isolated into a single test or better still, mocked for correctness. Additionally, algorithmic tests should not require downloading historical weather model data and generally should be requesting any data from external APIs.

That said, it requires more work to expand the test suite to this end. Also, the current test suite has excellent coverage for RAiDER's applicability. We have to figure out how to ensure how the package remains reliable (via testing) while improving the efficiency and reliability of the test suite itself. This is particularly evident with GMAO downloads failing on some random nontrivial fraction of the tests.

Possible options are

  • staging data in S3 (still requests data from the internet),
  • making unit tests smaller with simple embedded array tests
  • staging/mocking data, possibly leveraging https://git-lfs.com/ for large files
  • sharing weather model data downloadedacross tests (minimizes internet requests but does not necessarily solve the issue) - i.e. through pytest fixtures
@bbuzz31 bbuzz31 added enhancement New feature or request optimization labels May 11, 2023
@cmarshak cmarshak changed the title Remove test dependency on GMAO Improve runtime of test suite May 11, 2023
@cmarshak cmarshak changed the title Improve runtime of test suite Shorten runtime and reliability of test suite May 11, 2023
@cmarshak cmarshak changed the title Shorten runtime and reliability of test suite Shorten runtime and improve reliability of test suite May 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request optimization
Projects
None yet
Development

No branches or pull requests

1 participant