Usage report upload fails with GCS 403

[easyCZ]

When the usage component attempts to use the content service to upload a usage report, it fails with the following error:

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"failed to upload usage report: failed to obtain signed upload URL: failed RPC to content service: rpc error: code = NotFound desc = cannot ensure storage exists: googleapi: Error 403: stag-meta-eu02-storage@gitpod-staging.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket., forbidden","level":"error","message":"Reconciliation run failed.","serviceContext":{"service":"usage","version":"commit-66578f1793ce828eaa28c604c087036411aeb646"},"severity":"ERROR","time":"2022-07-27T10:53:13Z"}

Investigate why, and what needs to happen to fix it.

[andrew-farries]

Copying in my message from Slack:

There's two parts to this:

• GCloud storage bucket names have to be globally unique - we were using a hard-coded bucket name of usage-reports that is not unique. The bucket name will also need to vary between staging and production. I'm working on a PR against the installer and content service to do this.
• A unique bucket name allows the content-service to ensure the bucket exists, creating it if needed, but uploading a file to the bucket still fails with a 403 Forbidden . Once I've made the bucket name configurable I'll look into why that's happening.

[andrew-farries]

The error from gcloud when trying to use the signed upload URL is:

HTTP/1.1 403 Forbidden
Content-Length: 226
Content-Type: application/xml; charset=UTF-8
Date: Wed, 27 Jul 2022 13:59:44 GMT
Server: UploadServer
Vary: Origin
X-GUploader-UploadID: ADPycdvIzwf09QkyLRewUuK3kZag-1uXMv4-mQW_HcbVGPZ0lX9cx19lUkjTorUP6r6-V-B-vbtvr3A6VYDyBu2VVjdd

<?xml version='1.0' encoding='UTF-8'?>
<Error>
  <Code>AccessDenied</Code>
  <Message>Access denied.</Message>
  <Details>Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object.</Details>
</Error>

The creator of the signed upload URL needs to have permission to create objects otherwise clients that try to use the URL will also not be able to create objects [1].

This suggests that content-service (as the creator of the upload URL) does not have permission to create objects.

The service account that's mounted into the pod has "role": "roles/storage.admin" which is sufficient to create objects [2].

I noticed that the signing method content-service uses defaults to V2 signing, which is legacy. I'll try to use V4 signing to see if that makes a difference.