Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move large test datasets out of git repo #1132

Open
atravitz opened this issue Feb 14, 2025 · 1 comment
Open

Move large test datasets out of git repo #1132

atravitz opened this issue Feb 14, 2025 · 1 comment
Assignees
Labels

Comments

@atravitz
Copy link
Contributor

Is your feature request related to a problem? Please describe.
We are currently storing an increasing amount of data (even though it's zipped) in this git repo.

Describe the solution you'd like
We should switch to storing data on figshare or zenodo.

Additional context
One complication is that openfe test won't be able to be run on hpcs that restrict web access. We should think carefully about how to have reasonable test coverage without needing full test data.

@atravitz atravitz self-assigned this Feb 14, 2025
@mikemhenry
Copy link
Contributor

One complication is that openfe test won't be able to be run on hpcs that restrict web access. We should think carefully about how to have reasonable test coverage without needing full test data.

I think pooch would work if you drop the data it needs in the right spot since then it would be a cache hit -- so I think we just need a guide "how to download test data on an air-gapped network" that will go something like "download all these files, and put them in .pooch/openfe/"

Then that should work, but we will then need a flag that disables the "do I have internet test" since that will skip the tests and it won't even check the cache. I think making a OFE_USE_LOCAL_TEST_DATA envar that a user would export would be best which would then disable the internet check but still run the tests the "need" internet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants