Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor Google Cloud Storage to use blob.open #599

Closed
1 of 3 tasks
petedannemann opened this issue Mar 24, 2021 · 0 comments · Fixed by #744
Closed
1 of 3 tasks

Refactor Google Cloud Storage to use blob.open #599

petedannemann opened this issue Mar 24, 2021 · 0 comments · Fixed by #744

Comments

@petedannemann
Copy link
Contributor

petedannemann commented Mar 24, 2021 •
edited
Loading

Problem description

python-storage has recently added an open function to blobs that allows for file like IO. This would greatly reduce the amount of code we have to maintain for Google Cloud Storage and maybe solve #555. Note this hasn't been released yet so this should be a future task.

Before you create the issue, please make sure you have:

  • Described the problem clearly
  • Provided a minimal reproducible example, including any required data
  • Provided the version numbers of the relevant software
cadnce added a commit to cadnce/smart_open that referenced this issue Oct 1, 2022
@cadnce
Fixes piskvorky#599
6cf3625 
Swap to using GCS native blob open under the hood.
Reduces code maintenence overhead.
cadnce added a commit to cadnce/smart_open that referenced this issue Oct 1, 2022
@cadnce
Fixes piskvorky#599
1b36f9f 
Swap to using GCS native blob open under the hood.
Reduces code maintenence overhead.
@cadnce cadnce mentioned this issue Oct 1, 2022
Closed
@ddelange ddelange mentioned this issue Dec 5, 2022
Merged
5 tasks
mpenkov pushed a commit that referenced this issue Dec 9, 2022
@ddelange @cadnce
Refactor Google Cloud Storage to use blob.open (#744)
50bbbea 
* Fixes #599
Swap to using GCS native blob open under the hood.
Reduces code maintenence overhead.

* Use the underlying google-storage-blob instead of writing a proxy.
Breaking changes:
* Removed gcs.Reader/gcs.Writer classes
* No Reader/Writer.terminate()
* The buffer size can no-longer be controlled independently of chunk_size
* calling close twice on a gcs file object will now throw an exception

* Fix lint issues

* Restore original apis

* Maintain all original interfaces

* Add deprecation warning to un-used parameters

* Fix linting

* Bump gcs to 2.6.0

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>

* Fix linter

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>

* Remove test for deprecated terminate method

* Amend open docstring

Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>
Co-authored-by: cadnce <cadnce@users.noreply.github.com>
@q-aaronzolnailucas q-aaronzolnailucas mentioned this issue Dec 14, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@petedannemann