How do I successfully generate signed URLs to cloud storage objects when using workload identity federation?

[cortadocodes]

TL;DR

• Some of my tests generate signed URLs to Google Cloud Storage objects, but this does not work with workload identity federation
• My GitHub CI workflow uses workload identity federation to authenticate, which has worked fine with all other cloud operations carried out in the tests
• The tests pass when using an actual service account file locally

Expected behavior

The signed URLs to be generated normally

Observed behavior

As seen here, I get the following failed test

ERROR: test_create_signed_url (tests.cloud.storage.test_client.TestGoogleCloudStorageClient)
Test that a signed URL to a cloud object can be created and used to access the file.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/octue-sdk-python/octue-sdk-python/tests/cloud/storage/test_client.py", line 323, in test_create_signed_url
    url = self.storage_client.generate_signed_url(
  File "/home/runner/work/octue-sdk-python/octue-sdk-python/octue/cloud/storage/client.py", line 280, in generate_signed_url
    return blob.generate_signed_url(expiration=expiration, **api_access_endpoint)
  File "/home/runner/work/octue-sdk-python/octue-sdk-python/.tox/py/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 621, in generate_signed_url
    return helper(
  File "/home/runner/work/octue-sdk-python/octue-sdk-python/.tox/py/lib/python3.8/site-packages/google/cloud/storage/_signing.py", line 396, in generate_signed_url_v2
    signed_query_params = get_signed_query_params_v2(
  File "/home/runner/work/octue-sdk-python/octue-sdk-python/.tox/py/lib/python3.8/site-packages/google/cloud/storage/_signing.py", line 80, in get_signed_query_params_v2
    ensure_signed_credentials(credentials)
  File "/home/runner/work/octue-sdk-python/octue-sdk-python/.tox/py/lib/python3.8/site-packages/google/cloud/storage/_signing.py", line 52, in ensure_signed_credentials
    raise AttributeError(
AttributeError: you need a private key to sign credentials.the credentials you are currently using <class 'google.auth.identity_pool.Credentials'> just contains a token. see https://googleapis.dev/python/google-api-core/latest/auth.html#setting-up-a-service-account for more details.

Action YAML

https://github.com/octue/octue-sdk-python/blob/main/.github/workflows/python-ci.yml

Log output

https://github.com/octue/octue-sdk-python/runs/6006955054?check_suite_focus=true

Additional information

I found this similar issue for the google-cloud-go package and attempted some of the solutions, but they didn't work.

[sethvargo]

Hi @cortadocodes

Thank you for opening an issue. When using an exported service account key, the URL is signed with the private key embedded in the service account (there's no API call to GCP). When using WIF, there is no private key material stored locally, so you need to use the iamcredentials API's signBlob function to generate the signature.

This process is documented in the GCS documentation for signing:

Your programming language should have a library for performing RSA signatures. Alternatively, you can use the IAM signBlob method provided by Google Cloud if your expiration time is 12 hours or less.

So the algorithm is:

1. Build the canonical request
2. Build the string to sign
3. Sign it with signBlob
4. Concatenate to the URL

Unfortunately it doesn't look like there's a native function in the Python SDK to do this yet.

Does that make sense?

[cortadocodes]

Thank you for your quick reply @sethvargo! That makes sense. Do you know if there will be a native function in the python SDK anytime soon to do this?

[sethvargo]

Hi @cortadocodes - I'm not sure. It would be best to file an issue against the Python library directly: https://github.com/googleapis/python-storage

[sethvargo]

Hi @cortadocodes

I'm going to close this out. Let me know if you need any more assistance!