GCS Blob does not support the io.IOBase interface

[eap]

I'm branching this issue from the (currently resolved) issue #2871. The title covers the request, but to add some context see below.

Many python libraries (pandas, biopython, pillow, etc.) support reading from a 'python file object' which canonically are objects that implement the io.IOBase interface. If we could open a blob as a file object, we could do operations as shown in the example below where I'm just reading a few lines from a (possibly very large) file.

from Bio import SeqIO
from google.cloud.storage import blob

with blob.Blob('/data/example.fastq', 'mybucket').open('rb') as fileobj:
    for i, rec in enumerate(SeqIO.parse(fileobj, 'fastq')):
        print(rec.seq)
        print('  name=%s\n  annotations=%r' % (rec.name, rec.annotations))
        if i > 5:
            break

google-resumable-media covers some of the need expressed in this bug, but it does not satisfy users who need blobs to be parsed by libraries expecting standard file objects. The standard advice might be to download files first but that advice ignores the expense and time of downloading large files when indexed operations are available.

Some work has been done towards this end - the following two repositories have solutions. That said, I think most people would be far more happy to adopt a solution from this project since it has a far larger community and more active maintenance / governance.

• https://github.com/Akrog/gcs-client
• https://github.com/martindurant/gcsfs

[lukesneeringer]

@dhermes Any thoughts on how long it would take to implement a file-like interface on Blob?

[dhermes]

We already have Blob.download_to_file that takes a write-able stream. google-resumable-media allows manual chunking during downloads:

>>> download = ChunkedDownload(
...     media_url, chunk_size, stream)
>>> response = download.consume_next_chunk(transport)

It might be easiest to support this if ChunkedDownload.consume_next_chunk just took an optional chunk_size argument that over-rides the instance's chunk size. Then fake_file.read(chunk_size) could be implemented in some wrapper.

I don't think it makes sense to add support for this in google-cloud-storage, and I'm 50-50 on whether we should actually provide the file-like wrapper. I am totally fine adding support for the per-chunk over-ride on chunk size to make it possible.

[eap]

@dhermes
would the implementation you described be able to provide a seek operation? The most commonly used methods for reading Python file objects are read, seek, tell, readlines, and readline. The first three alone are incredibly important for programs that read indexed files.

As for the larger question of supporting file-like interface: I have additional notes on the topic -

1. File interfaces are core to many Python libraries and programs. Failing to provide a file interface in a google-endorsed library simply forces developers to re-invent this interface over and over again, or it forces them to rely on poorly maintained community contributions that may or may-not support new functionality like requester-pays buckets, and may also block or slow upgrades when security patches are needed.

2. Supporting file objects would enable appengine standard environment to appengine flexible environment migrations. If you recall, the standard environment has a cloudstorage library that gives file handles for GCS blobs (see their docs here). Large applications with this dependency will need extensive rewrites if no equivalent functionality can be added to the non-appengine GCS interface.

[yan-hic]

I second this request, not only for reading but writing too.

Why is it that only App Engine Standard has cloudstorage.open(<path_to_blob>,'w') ?

Because not supported elsewhere, one needs to provision a VM or similar with sufficient disk or memory to buffer before writing.
Once Google Functions supports Python (?), I assume - hope - a file-like storage object will exist there too.

Could boto not help in the meantime ?

[martindurant]

Thank @yiga2 for the pointer.

Indeed, anyone that wants pythonic file-like access to GCS is welcome to use gcsfs.

[eap]

I think that writable file objects is better left for a different discussion since it seems like seekable writers would conflict with core GCS concepts thanks to object-immutability. I could imagine a case being made for a non-seekable writer; many Python programs that accept file handles only require the availability of write or writelines methods.

Aside from general GCS concept difficulties, even a non-seekable writer creates caching and failure-handling headaches that aren't shared on the read-side of the IOBase interface implementation.

[chemelnucfin]

Hello, feature requests will now be tracked in the project Feature Requests. I will close this issue now, please feel free to continue to address any issues/concerns here.