Use gcs cache in ci

[zaneselvans]

Tell Tox to tell pytest to default to pulling data from the GCS Cache in the Github CI workflow, to increase the reliability of the tests on here.

[codecov]

Codecov Report

Merging #1858 (10641af) into dev (a4284fc) will increase coverage by 0.0%.
The diff coverage is 57.1%.

@@          Coverage Diff          @@
##             dev   #1858   +/-   ##
=====================================
  Coverage   83.0%   83.1%           
=====================================
  Files         65      65           
  Lines       7327    7332    +5     
=====================================
+ Hits        6088    6093    +5     
  Misses      1239    1239           

Impacted Files	Coverage Δ
src/pudl/workspace/datastore.py	67.9% <57.1%> (-1.8%)	⬇️
src/pudl/workspace/resource_cache.py	88.4% <0.0%> (+5.3%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[zaneselvans]

@bendnorman do you know off the top of your head what the right way is for us to give the tox-pytest workflow permission to read data from the GCS Cache? It seems like we have to both give it some kind of cloud permissions e.g. via the GCE_SA_KEY and then also tell it what account to bill any egress fees to on the zenodo-cache bucket, by setting the default project to GCP_BILLING_PROJECT. Do we use GitHub actions for these things? Or should it just be shell commands?

[bendnorman]

GoogleCloudStorageCache finds the billing project and credentials using google.auth.default(). I've been using the google auth github action to set the default credentials in our github actions, though I haven't tried it with tox. I can add it to the pytest workflow to test it out.

[zaneselvans]

Should it use the GCE_SA_KEY credentials? Or should it have some other more restricted credentials?

The GOOGLE_APPLICATION_CREDENTIALS environment variable is passed in to Tox if it's set, so if running the google auth action sets that variable I think it should work.

[bendnorman]

I pushed some code that adds the cloud auth step to tox-pytest. The GCE_SA_KEY is associated with the deploy-pudl-github-action service account, which has some bonus permissions, so I created a new service account called tox-pytest-github-action that only has the Service Usage Consumer role. I also changed GCE_SA_KEY to DEPLOY_PUDL_SA_KEY to be more specific.

We'll see if the credentials get passed through to tox...

[zaneselvans]

It looks like it worked! There was no logging output from the try/except where it falls back to Zenodo.

[bendnorman]

Ok! It looks like the most recent checks passed and didn't raise a DefaultCredentialsError. We could set the logging level to DEBUG to confirm the GCS cache is being used. Changing the log level could help debug future CI failures too. What do you think @zaneselvans?

[zaneselvans]

In CI the more logs the merrier!

[zaneselvans]

Alas, there is no --loglevel option in our pytest. But I think you can set the pytest loglevel in the configuration within tox.ini, which I've just changed.

[bendnorman]

This PR closes #1679.

Github didn't let me connect an issue probably because this PR is closed.

[zaneselvans]

I connected it in GitHub (from the Development sidebar). Zenhub doesn't seem to allow the connection though.