Add s3 bucket support #70
Conversation
Codecov ReportBase: 100.0% // Head: 100.0% // No change to project coverage 👍
Additional details and impacted files@@ Coverage Diff @@
## dev #70 +/- ##
=======================================
Coverage 100.0% 100.0%
=======================================
Files 2 2
Lines 44 44
=======================================
Hits 44 44
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
For more information, see https://pre-commit.ci
|
I think the failing tests here are because we were using an old version of the EPA CEMS outputs previously, which had a few additional columns that have since been removed by @aesharpe, so I am updating the expectations in the integration tests now. |
This way people can get access to the dev data by installing the pudl_catalog package from git. This value will be updated for tagged releases so people can access the release data from pypi and conda.
…/pudl-data-catalog into aws-opendata-bucket
|
Oh thank goodness nobody will have to authenticate. That was gonna be annoying. |
I removed caching disabling because fsspec starting throwing unexpected keyword argument errors when making requests to s3 with caching disabled. See intake/intake-parquet#26 for the full explanation.
For more information, see https://pre-commit.ci
…/pudl-data-catalog into aws-opendata-bucket
|
If the CI passes this PR should be good to go @zaneselvans |
| @@ -25,10 +28,45 @@ | |||
| "hourly_emissions_epacems/epacems-2020-FL.parquet", | |||
| ], | |||
| ) | |||
| def test_file_exists(filename: str) -> None: | |||
| def test_gcs_file_exists(filename: str) -> None: | |||
There was a problem hiding this comment.
So we're going to keep testing both S3 and GCS, which seems like a good idea if we actually want GCS to keep working as a fallback.
This PR adds the new
s3://intake.catalyst.coopbucket and sets it as the default. I also removed the requester pays documentation, given we don't want users to be using the GCS requester pays bucket.