GCS permission denied 'storage.buckets.get' when using 'open'

[baptistejub]

Hello,

After updating to 6.3.0 (from 6.2.0), we encounter a permission denied on storage.buckets.get when uploading a file to GCS.

Raised by https://github.com/RaRe-Technologies/smart_open/blob/v6.3.0/smart_open/gcs.py#L156 when checking for the bucket existence.

from google.cloud import storage
from smart_open import open

client = storage.Client()
file = open('gs://my-bucket/my_file.ext', 'wb', transport_param=dict(client=client))

File "/home/app/.local/lib/python3.10/site-packages/smart_open/smart_open_lib.py", line 224, in open
  binary = _open_binary_stream(uri, binary_mode, transport_params)
File "/home/app/.local/lib/python3.10/site-packages/smart_open/smart_open_lib.py", line 400, in _open_binary_stream
  fobj = submodule.open_uri(uri, mode, transport_params)
File "/home/app/.local/lib/python3.10/site-packages/smart_open/gcs.py", line 47, in open_uri
  return open(parsed_uri['bucket_id'], parsed_uri['blob_id'], mode, **kwargs)
File "/home/app/.local/lib/python3.10/site-packages/smart_open/gcs.py", line 101, in open
  _blob = Writer(bucket=bucket_id,
File "/home/app/.local/lib/python3.10/site-packages/smart_open/gcs.py", line 156, in Writer
  if not g_bucket.exists():
File "/home/app/.local/lib/python3.10/site-packages/google/cloud/storage/bucket.py", line 900, in exists
  client._get_resource(
File "/home/app/.local/lib/python3.10/site-packages/google/cloud/storage/client.py", line 372, in _get_resource
  return self._connection.api_request(
File "/home/app/.local/lib/python3.10/site-packages/google/cloud/storage/_http.py", line 72, in api_request
  return call()
File "/home/app/.local/lib/python3.10/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
  return retry_target(
File "/home/app/.local/lib/python3.10/site-packages/google/api_core/retry.py", line 191, in retry_target
  return target()
File "/home/app/.local/lib/python3.10/site-packages/google/cloud/_http/__init__.py", line 494, in api_request
  raise exceptions.from_http_response(response)
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/my-bucket?fields=name&prettyPrint=false: my-sa@my-org.iam.gserviceaccount.com does not have storage.buckets.get access to the Google Cloud Storage bucket. Permission 'storage.buckets.get' denied on resource (or it may not exist).

The requirement for this permission was removed with #516 but reintroduced by #744.

We've updated our SA permissions for now, but it would be nice if we could avoid it, for the same reasons as presented in #516

I could submit a PR to remove the check altogether if that's fine (trying to upload to a not found bucket should already raise a not found error).

[ddelange]

remove the check altogether if that's fine (trying to upload to a not found bucket should already raise a not found error).

you're right! PR is up ^ cc @mpenkov

[mpenkov]

Argh, we need to sync up @ddelange , I just made a new bugfix release a few minutes ago :)