Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create tests for reading compressed tables from S3 URLs #14875

Closed
dhimmel opened this issue Dec 13, 2016 · 3 comments
Closed

Create tests for reading compressed tables from S3 URLs #14875

dhimmel opened this issue Dec 13, 2016 · 3 comments
Labels
IO Network Local or Cloud (AWS, GCS, etc.) IO Issues Testing pandas testing functions or related to the test suite

Comments

@dhimmel
Copy link
Contributor

dhimmel commented Dec 13, 2016

In #14576, we consolidated compression code and expanded support for compressed URLs. While we added tests for compressed URLs, we did not add tests for compressed S3 files. First, someone will have to upload the compressed tables to Amazon: see #14587 (comment).

Also tagging #13137, a pull request which may want to also add these tests.
@dhimmel dhimmel mentioned this issue Dec 13, 2016
Closed
@jreback jreback added Difficulty Intermediate IO CSV read_csv, to_csv Testing pandas testing functions or related to the test suite labels Dec 13, 2016
@jreback jreback added this to the Next Major Release milestone Dec 13, 2016
@TomAugspurger
Copy link
Contributor

👍 I can add those in #13137.

@dhimmel
Copy link
Contributor Author

dhimmel commented Dec 13, 2016

@TomAugspurger great. I put the compressed URL tests in pandas/io/tests/parser/test_network.py. Adding the S3 tests there would probably not take too much additional code.

@mroeschke mroeschke added IO Network Local or Cloud (AWS, GCS, etc.) IO Issues and removed IO CSV read_csv, to_csv labels May 22, 2020
@mroeschke
Copy link
Member

Looks like test_network.py has some tests that read S3 files that are compressed (gz and bz2), so I think that should suffice the spirit of this issue. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
IO Network Local or Cloud (AWS, GCS, etc.) IO Issues Testing pandas testing functions or related to the test suite
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jreback @dhimmel @TomAugspurger @jbrockmendel @mroeschke