Provide a way to stream read/write requests using a single RPC call

[aozarov]

This topic was also discussed in #61. Current implementation of the streaming API for both reads and writes is done via multiple RPC calls. Writes are done in chucks (resumable-writes) which is good for error handling (save-points) but at a cost of multiple RPC instead of one. Reads are also segmented (via range requests) instead one RPC for the whole content.

Things to consider:

1. some platforms may have restrictions on RPC requests (e.g. AE 10MB upstream and 32MB downstream, as well as time-to-live restrictions).
2. Current implementation of the apiary client does not expose the underlying stream and will copy its content to the caller.
3. Rate of errors and cost-benefit evaluation (if the recommended size per one RPC is small enough it might be that the option of create with content or load all is sufficient).

[aozarov]

PR #534 was taking a stub at it for BigQuery. It would be great if we resurrect it when possible and also apply it for Storage.

[mziccard]

I started having a look at this lately. My only concern is how to expose this functionality to users. Now in Storage we have:

Blob create(BlobInfo blobInfo, byte[] content, BlobTargetOption... options);
Blob create(BlobInfo blobInfo, InputStream content, BlobWriteOption... options);

Both method use non-resumable upload.

As far as I know when resumable upload is used to create a object the RPC call does not return the object metadata in the response body. So methods that use resumable upload would not be able to offer a return type. Should we call such methods write rather than create?

Also, resumable upload as implemented in #534 needs a SeekableByteChannel to function (in case of failed upload we need to seek latest upload position through the channel) and not an InputStream. Would adding the following two methods look odd to you?

void write(BlobInfo blobInfo, byte[] content, BlobTargetOption... options);
void write(BlobInfo blobInfo, SeekableByteChannel content, BlobWriteOption... options);

/cc @jgeewax for opinions

[mziccard]

@Capstan Is there a way to make resumable upload return the information on the blob just created (as it happens for direct upload)? Just to be sure I am not missing something here.

If not, would it be possible in your opinion to have the "closing" call to a resumable upload URL return the blob's information in the response body?

[Capstan]

@mziccard I would be surprised to find that it does not return that information. What's the behavior you're seeing? What are you using for your alt parameter?

[mziccard]

I am requesting a resumable upload URL with:

POST https://www.googleapis.com/upload/storage/v1/b/test-bucket/o?uploadType=resumable&name=test-file
(blob information in the body)

Which returns the resumable URL in the body:

https://www.googleapis.com/upload/storage/v1/b/test-bucket/o?uploadType=resumable&name=test-blob&upload_id=some-long-upload-id

I upload the file with:

POST https://www.googleapis.com/upload/storage/v1/b/test-bucket/o?uploadType=resumable&name=test-blob&upload_id=some-long-upload-id

The blob is correctly created, upload is successful but this latest request does not return blob information back.

[mziccard]

@Capstan any update on this?

[Capstan]

None, though I'm used to seeing PUT in the last request. See https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload for examples.

[garrettjonesgoogle]

This has been added to our feature backlog: https://github.com/GoogleCloudPlatform/google-cloud-java/wiki/Feature-backlog . This issue will be closed but is linked in the backlog and can continue to be used for comment and discussion.