Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consistent pandas_read_csv_kwargs for file and CDS sources #211

Conversation

EddyCMWF
Copy link
Contributor

@EddyCMWF EddyCMWF commented Oct 2, 2023

See #180 for full details and background.

CDS and file sources now have consistent pandas_read_csv_sources such that a resource from theCDS has the same behaviour whether opened automatically or saved for a later date. Several other small refactorisations also included.

I have not yet modified the ecmwf_api default kwargs as that has more bespoke requirements. But this is due to them being TAB sperated values, not COMMA seperated values.

@EddyCMWF
Copy link
Contributor Author

EddyCMWF commented Oct 3, 2023

Latest commit is following an offline discussion to improve the visibility and explicitness of the kwarg handling in the read_csv method

@EddyCMWF
Copy link
Contributor Author

Can this be merged?

@sandorkertesz sandorkertesz merged commit 6845996 into develop Oct 11, 2023
11 checks passed
@EddyCMWF EddyCMWF deleted the 180-the-file-source-is-not-able-to-open-with-pandas-cds-files-previously-downloaded branch October 12, 2023 15:21
malmans2 pushed a commit to malmans2/earthkit-data that referenced this pull request Oct 12, 2023
* Providing different defaults to different sources is confusing

* need to specify compression method used by source

* read_csv kwargs consistent for cds and file sources

* improved kwarg handling of read_csv

* test demonstrating that default parse_date kwargs is not a good idea

* Tests for other formats

* Compression removed from CSV.to_pandas kwargs, special treatment not required

---------

Co-authored-by: Sandor Kertesz <Sandor.Kertesz@ecmwf.int>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

The "file" source is not able to open with pandas CDS files previously downloaded
2 participants