Add caching and create outputs as symlinks #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Alter the download_cds command to download files to a user specified cache directory where files are named after `get_json_sem_hash(request)`. The `RequestEntry` class is introduce to keep the data necessary for producing the hash collected. Output according to `filename_spec` are created using symlinks to the cache. Main motivation for this is to allow multiple projects that might have overlapping data to re-use previously downloaded requests.
The mutually exclusive options to download_cds are split into click subcommands: - download - list-files - list-dangling
|
Hey @petermcevoy ! Remembered that I should look at this 😄 Overall it looks really nice, and the core functionality should remain the same. I guess the bump to python 3.11 is because of using built-in tomli? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi, here is a larger PR for changes to make cdsapi_helper download files into a central cache and then symlink outputs according to the input specification files.
The individual commits in this PR hopefully introduce the changes in smaller and more clear steps.
Let me know what you think and if it makes sense to include.
MVH
Peter