Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to retrieve data into user specified target files without any parsing #338

Open
sandorkertesz opened this issue Mar 8, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@sandorkertesz
Copy link
Collaborator

sandorkertesz commented Mar 8, 2024

Is your feature request related to a problem? Please describe.

Currently we cannot retrieve data directly into a user defined file, because we cannot pass a file target to from_source, but we have to use e.g.

ds = eathkit.data.from_source("cds", ....)
ds.save("my_file")

Here, the data is first downloaded into a tmp/cache, unpacked when needed, then parsed/interpreted. This all happens before calling save(), which actually performs a file copy operation.

Describe the solution you'd like

A possible API:

ds = eathkit.data.from_source("cds", ...., target="my_file")

where the data will be save into file my_file.

Some questions immediately arise:

  • what would ds be? (it has to be a source)
  • what if multiple files are retrieved? (with the cds source it is possible)
  • what if the user wants to preserve the remote file name?
  • sometimes the type of the retrieved file is not know in advance e.g. it can be a "zip" file. It would be nice to have an option to automatically add the right suffix to the target file. E.g. when this option is used we just need to specify target="my_file" and the data would be saved into my_file.zip.

Describe alternatives you've considered

No response

Additional context

No response

Organisation

ECMWF

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant