-
-
Notifications
You must be signed in to change notification settings - Fork 382
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AttributeError: 'Blob' object has no attribute 'download_as_bytes' #601
Comments
What version of google-cloud-storage are you using? |
Whatever gets install when I install smart-open[gcs]. This is a clean docker that's created specifically for this job. Does it lock in a version? |
Looked up the docker build: |
Very odd, the download_as_bytes method is still definitely there: @petedannemann Can you please investigate?
No, we don't pin any versions for this package. |
I can't reproduce this. @Cartman75 can you post exactly what code you are running? What is your |
Essentially I am taking mysqldump .sql files from gcp and loading them in a side table:
|
Ok I still can't reproduce this at all and still have no idea what your Can you try a minimally reproducible example like this? This is a publically available file in GCS. path = "gs://tensorflow-nightly/prod/tensorflow/release/ubuntu_16/gpu_py37_full/nightly_release/18/20190813-010608/github/tensorflow/pip_pkg/tf_nightly_gpu-1.15.0.dev20190813-cp37-cp37m-linux_x86_64.whl"
import smart_open
import google.cloud.storage
client = google.cloud.storage.Client.create_anonymous_client()
with smart_open.open(path, transport_params=dict(client=client)) as f:
for line in f:
print(line) |
I don't have the transport_params part on my open statement. Is that maybe why I am seeing this? What's that doing? I have to log in to get access to the gs buckets. Its via a server-side service account.. So how your doing it is much different than what I am doing. Not sure if it matters. |
I just downgraded to smart_open[gcp]==2.2.1 and its working again. The plot thickens. The latest version I can use is 4.0.1. All newer version break with the above error. |
Had the same issue. Pinning to the latest version of It would be nice to pin |
@gelioz Sounds reasonable. Are you interested in making a PR? |
* Pin google-cloud-storage to >=1.31.1 in extras Fixes #601 * Update CHANGELOG.md Co-authored-by: Michael Penkov <m@penkov.dev>
Problem description
Be sure your description clearly answers the following questions:
What are you trying to achieve?
Trying to pull a file from gcp storage
What is the expected result?
I can read a file from gcp storage
What are you seeing instead?
Error: AttributeError: 'Blob' object has no attribute 'download_as_bytes'
Steps/code to reproduce the problem
Failed on this line:
for line in open(file, 'r'):
I think it gets the file open and begins reading for the loop when it fails.
In order for us to be able to solve your problem, we have to be able to reproduce it on our end.
Without reproducing the problem, it is unlikely that we'll be able to help you.
Include full tracebacks, logs and datasets if necessary.
Please keep the examples minimal (minimal reproducible example).
Apr 05 16:19:30 docker[2234166]: File "/usr/local/lib/python3.7/site-packages/smart_open/gcs.py", line 325, in read1
Apr 05 16:19:30 docker[2234166]: return self.read(size=size)
Apr 05 16:19:30 docker[2234166]: File "/usr/local/lib/python3.7/site-packages/smart_open/gcs.py", line 320, in read
Apr 05 16:19:30 docker[2234166]: self._fill_buffer(size)
Apr 05 16:19:30 docker[2234166]: File "/usr/local/lib/python3.7/site-packages/smart_open/gcs.py", line 374, in _fill_buffer
Apr 05 16:19:30 docker[2234166]: bytes_read = self._current_part.fill(self._raw_reader)
Apr 05 16:19:30 docker[2234166]: File "/usr/local/lib/python3.7/site-packages/smart_open/bytebuffer.py", line 152, in fill
Apr 05 16:19:30 docker[2234166]: new_bytes = source.read(size)
Apr 05 16:19:30 docker[2234166]: File "/usr/local/lib/python3.7/site-packages/smart_open/gcs.py", line 178, in read
Apr 05 16:19:30 docker[2234166]: binary = self._download_blob_chunk(size)
Apr 05 16:19:30 docker[2234166]: File "/usr/local/lib/python3.7/site-packages/smart_open/gcs.py", line 194, in _download_blob_chunk
Apr 05 16:19:30 i docker[2234166]: binary = self._blob.download_as_bytes(start=start, end=end)
Apr 05 16:19:30 docker[2234166]: AttributeError: 'Blob' object has no attribute 'download_as_bytes'
Apr 05 16:19:30 docker[2234166]: I0405 16:19:30.721222 139676694796096 gcs.py:241] close: called
Versions
Please provide the output of:
Python 3.7.10 (default, Apr 2 2021, 22:24:51)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Tried on the latest 4 version too, same thing.
Checklist
Before you create the issue, please make sure you have:
The text was updated successfully, but these errors were encountered: