Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blob.upload_from_file doesn't work for file-like objects without size parameter #900

Closed
theacodes opened this issue May 28, 2015 · 4 comments
Assignees
Labels
api: storage Issues related to the Cloud Storage API.

Comments

@theacodes
Copy link
Contributor

When using StringIO, io.BytesIO or other file-like objects (such as Flask/werkzeug's file upload object), trying to upload via Blob.upload_from_file doesn't work because it attempts to find the size of the object using os.stat throwing a AttributeError: StringIO instance has no attribute 'fileno'. Passing in the size manually fixes this, but imo this inconsistency shouldn't burden the user as there are other ways than fileno to determine the size of a StringIO or io.BytesIO instance (seek, tell).

At any rate, the error message could be more helpful.

@dhermes
Copy link
Contributor

dhermes commented May 29, 2015

Can you give a stacktrace of the error message?

The offending line is

        # Get the basic stats about the file.
        total_bytes = size or os.fstat(file_obj.fileno()).st_size

which as @jonparrott points out, assumes the file-like object is a file (from the stdlib).

@theacodes
Copy link
Contributor Author

Traceback (most recent call last):
  File "/Users/jonwayne/workspace/getting-started-python/env/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/jonwayne/workspace/getting-started-python/env/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/Users/jonwayne/workspace/getting-started-python/env/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/jonwayne/workspace/getting-started-python/env/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/jonwayne/workspace/getting-started-python/env/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/jonwayne/workspace/getting-started-python/env/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/jonwayne/workspace/getting-started-python/env/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/jonwayne/workspace/getting-started-python/env/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/jonwayne/workspace/getting-started-python/oauth2/__init__.py", line 33, in decr
    return f(*args, **kwargs)
  File "/Users/jonwayne/workspace/getting-started-python/bookshelf/crud.py", line 57, in add
    image_url = upload_image_file(request.files.get('image'))
  File "/Users/jonwayne/workspace/getting-started-python/bookshelf/crud.py", line 25, in upload_image_file
    current_app.config['ALLOWED_EXTENSIONS']
  File "/Users/jonwayne/workspace/getting-started-python/storage/__init__.py", line 52, in upload_file
    blob.upload_from_file(file)
  File "/Users/jonwayne/workspace/getting-started-python/env/src/gcloud/gcloud/storage/blob.py", line 395, in upload_from_file
    total_bytes = size or os.fstat(file_obj.fileno()).st_size
UnsupportedOperation: fileno

@dhermes
Copy link
Contributor

dhermes commented May 30, 2015

Ha, ok. Doesn't provide anything we didn't know.

@theacodes
Copy link
Contributor Author

Heh, let me know if there's anything else I can provide to help here. The file-like object is a flask file upload (werkzeug.FileStorage). It fails likewise when using blob.upload_from_file(file) and blob.upload_from_file(file.stream).

@dhermes dhermes added the api: storage Issues related to the Cloud Storage API. label Jun 5, 2015
@dhermes dhermes modified the milestone: Storage Stable Jun 5, 2015
@dhermes dhermes self-assigned this Jun 6, 2015
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Jun 6, 2015
In storage, file upload will now fail if no size is
passed in and if the file object does not have a fileno
which can be stat'ed.

Fixes googleapis#900.
gcf-owl-bot bot added a commit that referenced this issue May 7, 2024
Source-Link: googleapis/googleapis@3820562

Source-Link: googleapis/googleapis-gen@a2cd2c3
Copy-Tag: eyJwIjoicGFja2FnZXMvZ29vZ2xlLWNsb3VkLWNvbXB1dGUvLk93bEJvdC55YW1sIiwiaCI6ImEyY2QyYzNiYWNiZjlhZWNmN2RkMWY4ODQ1MGE2YTE0NTdlMjQzOGEifQ==
parthea added a commit that referenced this issue May 11, 2024
…240430 (#900) (#12648)

- [ ] Regenerate this pull request now.

Source-Link:
googleapis/googleapis@3820562

Source-Link:
googleapis/googleapis-gen@a2cd2c3
Copy-Tag:
eyJwIjoicGFja2FnZXMvZ29vZ2xlLWNsb3VkLWNvbXB1dGUvLk93bEJvdC55YW1sIiwiaCI6ImEyY2QyYzNiYWNiZjlhZWNmN2RkMWY4ODQ1MGE2YTE0NTdlMjQzOGEifQ==

BEGIN_NESTED_COMMIT
feat: [google-cloud-compute] Update Compute Engine API to revision
20240421 (#899)
Source-Link:
googleapis/googleapis@55499b5

Source-Link:
googleapis/googleapis-gen@cf5049b
Copy-Tag:
eyJwIjoicGFja2FnZXMvZ29vZ2xlLWNsb3VkLWNvbXB1dGUvLk93bEJvdC55YW1sIiwiaCI6ImNmNTA0OWI3MDc5MjgyMDA2NWRiMzhlNzEyN2YzMmVhYjc3MDU5NDQifQ==
END_NESTED_COMMIT
BEGIN_NESTED_COMMIT
feat: [google-cloud-compute] Update Compute Engine API to revision
20240407 (#890)
Source-Link:
googleapis/googleapis@018c0e3

Source-Link:
googleapis/googleapis-gen@dad8476
Copy-Tag:
eyJwIjoicGFja2FnZXMvZ29vZ2xlLWNsb3VkLWNvbXB1dGUvLk93bEJvdC55YW1sIiwiaCI6ImRhZDg0NzZjOTRlYjFlY2IyZmQ3MmYyOTY4MjVlOTYyMWVhOGI0NWQifQ==
END_NESTED_COMMIT

---------

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Co-authored-by: Anthonios Partheniou <partheniou@google.com>
parthea pushed a commit that referenced this issue May 16, 2024
🤖 I have created a release *beep* *boop*
---


<details><summary>google-apps-chat: 0.1.6</summary>

##
[0.1.6](google-apps-chat-v0.1.5...google-apps-chat-v0.1.6)
(2024-05-16)


### Documentation

* [google-apps-chat] update Chat API comments
([#12694](#12694))
([f66440d](f66440d))
</details>

<details><summary>google-cloud-batch: 0.17.20</summary>

##
[0.17.20](google-cloud-batch-v0.17.19...google-cloud-batch-v0.17.20)
(2024-05-16)


### Features

* Add UpdateJob API to update the job spec, only task_count is supported
now
([b855629](b855629))
* Update description on allowed_locations in LocationPolicy field
([b855629](b855629))


### Documentation

* [google-cloud-batch] Refine description for field `task_execution`
([f149fb8](f149fb8))
* [google-cloud-batch] Refine description for field `task_execution`
([#12693](#12693))
([f149fb8](f149fb8))
* updated comments
([b855629](b855629))
</details>

<details><summary>google-cloud-compute: 1.19.0</summary>

##
[1.19.0](google-cloud-compute-v1.18.0...google-cloud-compute-v1.19.0)
(2024-05-16)


### Features

* [google-cloud-compute] Update Compute Engine API to revision
([f515bc6](f515bc6))
* [google-cloud-compute] Update Compute Engine API to revision
([f515bc6](f515bc6))
* [google-cloud-compute] Update Compute Engine API to revision 20240430
([#900](#900))
([#12648](#12648))
([f515bc6](f515bc6))
</details>

<details><summary>google-cloud-dataplex: 2.0.0</summary>

##
[2.0.0](google-cloud-dataplex-v1.13.0...google-cloud-dataplex-v2.0.0)
(2024-05-16)


### ⚠ BREAKING CHANGES

* An existing field `entry` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
* An existing field `display_name` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
* An existing field `entry_type` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
* An existing field `modify_time` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
* An existing field `fully_qualified_name` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
* An existing field `description` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
* An existing field `relative_resource` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`

### Features

* updated client libraries for Dataplex Catalog: removed deprecated
fields, updated comments
([fd6e39c](fd6e39c))


### Bug Fixes

* An existing field `description` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
([fd6e39c](fd6e39c))
* An existing field `display_name` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
([fd6e39c](fd6e39c))
* An existing field `entry_type` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
([fd6e39c](fd6e39c))
* An existing field `entry` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
([fd6e39c](fd6e39c))
* An existing field `fully_qualified_name` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
([fd6e39c](fd6e39c))
* An existing field `modify_time` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
([fd6e39c](fd6e39c))
* An existing field `relative_resource` is removed from message
`.google.cloud.dataplex.v1.SearchEntriesResult`
([fd6e39c](fd6e39c))


### Documentation

* A comment for field `aspects` in message
`.google.cloud.dataplex.v1.Entry` is changed
([fd6e39c](fd6e39c))
* A comment for field `filter` in message
`.google.cloud.dataplex.v1.ListEntriesRequest` is changed
([fd6e39c](fd6e39c))
</details>

<details><summary>google-cloud-deploy: 1.19.0</summary>

##
[1.19.0](google-cloud-deploy-v1.18.1...google-cloud-deploy-v1.19.0)
(2024-05-16)


### Features

* [google-cloud-deploy] add Skaffold verbose support to Execution
Environment properties
([#12701](#12701))
([03c7b0c](03c7b0c))
</details>

<details><summary>google-cloud-documentai: 2.27.1</summary>

##
[2.27.1](google-cloud-documentai-v2.27.0...google-cloud-documentai-v2.27.1)
(2024-05-16)


### Documentation

* [google-cloud-documentai] clarify the unavailability of some features
([#12699](#12699))
([3ec5a2c](3ec5a2c))
* updated comments
([9722673](9722673))
</details>

<details><summary>google-cloud-gke-multicloud: 0.6.10</summary>

##
[0.6.10](google-cloud-gke-multicloud-v0.6.9...google-cloud-gke-multicloud-v0.6.10)
(2024-05-16)


### Features

* [google-cloud-gke-multicloud] option to ignore_errors while deleting
Azure clusters / nodepools
([#12686](#12686))
([3d1901e](3d1901e))
</details>

<details><summary>google-cloud-kms: 2.23.0</summary>

##
[2.23.0](google-cloud-kms-v2.22.0...google-cloud-kms-v2.23.0)
(2024-05-16)


### Features

* add client library for KMS Autokey service, which enables automated
KMS key provision and management
([b74c6c2](b74c6c2))
</details>

<details><summary>google-cloud-parallelstore: 0.2.0</summary>

##
[0.2.0](google-cloud-parallelstore-v0.1.2...google-cloud-parallelstore-v0.2.0)
(2024-05-16)


### ⚠ BREAKING CHANGES

* An existing field `end_time` is removed from message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
* An existing field `source` is removed from message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
* An existing field `destination` is removed from message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
* [google-cloud-parallelstore] An existing field `create_time` is
removed from message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
* An existing field `destination_path` is renamed to
`destination_parallelstore` in message
`.google.cloud.parallelstore.v1beta.ImportDataRequest`
* An existing field `source_path` is renamed to `source_parallelstore`
in message `.google.cloud.parallelstore.v1beta.ExportDataRequest`
* An existing field `destination_gcs_uri` is renamed to
`destination_gcs_bucket` in message
`.google.cloud.parallelstore.v1beta.ExportDataRequest`
* An existing field `source_gcs_uri` is renamed to `source_gcs_bucket`
in message `.google.cloud.parallelstore.v1beta.ImportDataRequest`

### Features

* A new field `api_version` is added to message
`.google.cloud.parallelstore.v1beta.ExportDataMetadata`
([2f57a44](2f57a44))
* A new field `api_version` is added to message
`.google.cloud.parallelstore.v1beta.ImportDataMetadata`
([2f57a44](2f57a44))
* A new field `create_time` is added to message
`.google.cloud.parallelstore.v1beta.ExportDataMetadata`
([2f57a44](2f57a44))
* A new field `create_time` is added to message
`.google.cloud.parallelstore.v1beta.ImportDataMetadata`
([2f57a44](2f57a44))
* A new field `destination_gcs_bucket` is added to message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
([2f57a44](2f57a44))
* A new field `destination_parallelstore` is added to message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
([2f57a44](2f57a44))
* A new field `end_time` is added to message
`.google.cloud.parallelstore.v1beta.ExportDataMetadata`
([2f57a44](2f57a44))
* A new field `end_time` is added to message
`.google.cloud.parallelstore.v1beta.ImportDataMetadata`
([2f57a44](2f57a44))
* A new field `requested_cancellation` is added to message
`.google.cloud.parallelstore.v1beta.ExportDataMetadata`
([2f57a44](2f57a44))
* A new field `requested_cancellation` is added to message
`.google.cloud.parallelstore.v1beta.ImportDataMetadata`
([2f57a44](2f57a44))
* A new field `source_gcs_bucket` is added to message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
([2f57a44](2f57a44))
* A new field `source_parallelstore` is added to message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
([2f57a44](2f57a44))
* A new field `status_message` is added to message
`.google.cloud.parallelstore.v1beta.ExportDataMetadata`
([2f57a44](2f57a44))
* A new field `status_message` is added to message
`.google.cloud.parallelstore.v1beta.ImportDataMetadata`
([2f57a44](2f57a44))
* A new field `target` is added to message
`.google.cloud.parallelstore.v1beta.ExportDataMetadata`
([2f57a44](2f57a44))
* A new field `target` is added to message
`.google.cloud.parallelstore.v1beta.ImportDataMetadata`
([2f57a44](2f57a44))
* A new field `verb` is added to message
`.google.cloud.parallelstore.v1beta.ExportDataMetadata`
([2f57a44](2f57a44))
* A new field `verb` is added to message
`.google.cloud.parallelstore.v1beta.ImportDataMetadata`
([2f57a44](2f57a44))
* A new message `DestinationGcsBucket` is added
([e3c48d7](e3c48d7))
* A new message `DestinationParallelstore` is added
([e3c48d7](e3c48d7))
* A new message `SourceGcsBucket` is added
([e3c48d7](e3c48d7))
* A new message `SourceParallelstore` is added
([e3c48d7](e3c48d7))


### Bug Fixes

* [google-cloud-parallelstore] An existing field `create_time` is
removed from message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
([2f57a44](2f57a44))
* An existing field `destination_gcs_uri` is renamed to
`destination_gcs_bucket` in message
`.google.cloud.parallelstore.v1beta.ExportDataRequest`
([e3c48d7](e3c48d7))
* An existing field `destination_path` is renamed to
`destination_parallelstore` in message
`.google.cloud.parallelstore.v1beta.ImportDataRequest`
([e3c48d7](e3c48d7))
* An existing field `destination` is removed from message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
([2f57a44](2f57a44))
* An existing field `end_time` is removed from message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
([2f57a44](2f57a44))
* An existing field `source_gcs_uri` is renamed to `source_gcs_bucket`
in message `.google.cloud.parallelstore.v1beta.ImportDataRequest`
([e3c48d7](e3c48d7))
* An existing field `source_path` is renamed to `source_parallelstore`
in message `.google.cloud.parallelstore.v1beta.ExportDataRequest`
([e3c48d7](e3c48d7))
* An existing field `source` is removed from message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata`
([2f57a44](2f57a44))


### Documentation

* A comment for field `counters` in message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata` is
changed
([2f57a44](2f57a44))
* A comment for field `transfer_type` in message
`.google.cloud.parallelstore.v1beta.TransferOperationMetadata` is
changed
([2f57a44](2f57a44))
</details>

<details><summary>google-cloud-public-ca: 0.3.10</summary>

##
[0.3.10](google-cloud-public-ca-v0.3.9...google-cloud-public-ca-v0.3.10)
(2024-05-16)


### Features

* added protos for publicca v1
([f38aae4](f38aae4))
</details>

<details><summary>google-cloud-service-directory: 1.11.4</summary>

##
[1.11.4](google-cloud-service-directory-v1.11.3...google-cloud-service-directory-v1.11.4)
(2024-05-16)


### Documentation

* add maximum page_size to ListEndpoint API documentation
([f8ff81b](f8ff81b))
* add maximum page_size to ListNamespace API documentation
([f8ff81b](f8ff81b))
* add maximum page_size to ListService API documentation
([f8ff81b](f8ff81b))
</details>

<details><summary>google-cloud-visionai: 0.1.0</summary>

## 0.1.0 (2024-05-16)


### Features

* add initial files for google.cloud.visionai.v1
([#12669](#12669))
([33215ca](33215ca))
</details>

<details><summary>google-maps-fleetengine: 0.2.0</summary>

##
[0.2.0](google-maps-fleetengine-v0.1.9...google-maps-fleetengine-v0.2.0)
(2024-05-16)


### ⚠ BREAKING CHANGES

* An existing method `SearchFuzzedVehicles` is removed from service
`VehicleService`
* An existing message `UpdateVehicleLocationRequest` is removed
* An existing method `UpdateVehicleLocation` is removed from service
`VehicleService`

### Bug Fixes

* An existing message `UpdateVehicleLocationRequest` is removed
([e6969d5](e6969d5))
* An existing method `SearchFuzzedVehicles` is removed from service
`VehicleService`
([e6969d5](e6969d5))
* An existing method `UpdateVehicleLocation` is removed from service
`VehicleService`
([e6969d5](e6969d5))


### Documentation

* [google-maps-fleetengine] mark TerminalPointId as deprecated
([#12698](#12698))
([262ef80](262ef80))
</details>

<details><summary>google-maps-fleetengine-delivery: 0.2.0</summary>

##
[0.2.0](google-maps-fleetengine-delivery-v0.1.8...google-maps-fleetengine-delivery-v0.2.0)
(2024-05-16)


### ⚠ BREAKING CHANGES

* An existing method `SearchFuzzedVehicles` is removed from service
`VehicleService`
* An existing message `UpdateVehicleLocationRequest` is removed
* An existing method `UpdateVehicleLocation` is removed from service
`VehicleService`

### Bug Fixes

* An existing message `UpdateVehicleLocationRequest` is removed
([6c696a2](6c696a2))
* An existing method `SearchFuzzedVehicles` is removed from service
`VehicleService`
([6c696a2](6c696a2))
* An existing method `UpdateVehicleLocation` is removed from service
`VehicleService`
([6c696a2](6c696a2))


### Documentation

* remove comment about deleted SearchTasks method
([11fb7ce](11fb7ce))
</details>

<details><summary>google-maps-routeoptimization: 0.1.0</summary>

## 0.1.0 (2024-05-16)


### Features

* add initial files for google.maps.routeoptimization.v1
([#12670](#12670))
([524cd1e](524cd1e))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the Cloud Storage API.
Projects
None yet
Development

No branches or pull requests

2 participants