Sequential requests when downloading file

[mattzeunert]

I'm using the File.download method and was wondering if its performance could be improved.

Currently it makes two requests sequentially:

1. /storage/.../key?alt=media to download the file
2. /storage/.../key to download file metadata

Would it be possible to make those requests in parallel? The second request adds another 20 - 250ms to the download.

[stephenplusplus]

I believe this is happening because of our check for data integrity after the download is completed. With the latest metadata, we get the most recent hash of the remote object. It does indeed seem that we could make those requests in parallel, getting the metadata at the same time the download request starts. Thank you for the suggestion, I'll take a look at this soon!

[danvk]

Even better, could we make this second request optional? When you download a file, you just get the Buffer, you don't get the metadata. I'd rather not pay the cost of getting the metadata just on the off chance I might use it.

[stephenplusplus]

@danvk we do the metadata request for the sake of data integrity-- did we give you the exact file the server is storing. However, we should definitely stop making that request if you disable validation (file.createReadStream({validation: false}).

[danvk]

Yes, I appreciate your concern for my data integrity :) I'd just like an option to let you play fast and loose.

Here's my hack to disable this second RPC. ⚠️ Use at your own peril! ⚠️

/**
 * This is a hack to avoid issuing a second RPC to cloud storage for file metadata that we don't
 * use. This RPC is done in serial after the content is downloaded and is unconditional.
 * Whatever you do, don't use the file object after you patch it, other than to call
 * createReadStream or download!
 */
export function patchFileForSpeedyDownload(file: File) {
  // See https://github.com/googleapis/nodejs-storage/issues/1338
  (file.getMetadata as any) = function (
    this: File,
    options: any,
    callback?: (...args: any[]) => void,
  ) {
    this.metadata = {};
    if (callback) {
      callback(null, null, null);
    } else {
      return Promise.resolve([null, null]);
    }
  };
}

Make sure to set {validation: false} if you use this.

[danvk]

I was following through some @google-cloud/storage code in my debugger today and noticed that the ?alt=media request already returns the crc/md5 hashes in response headers (x-goog-hash):

So even when you are doing validation, the second metadata request isn't needed.

[stephenplusplus]

Great find. PR sent: #1419

[danvk]

And this is now released in 5.8.2. Thanks for pushing this through @stephenplusplus, I'm very excited to delete my patchFileForSpeedyDownload hack :)

[mattzeunert]

Thanks @stephenplusplus for implementing this, and thanks @danvk for pointing out the second request isn't necessary!

Just deployed this earlier and reads are now 50% faster.

Also seems like some metadata requests were really slow, and now the max duration is down significantly.

[danvk]

@mattzeunert what monitoring tool is that? I'd like something similar for my site!

[mattzeunert]

@danvk It's a Grafana dashboard running on Hosted Metrics. I think Grafana also has their own Cloud offering too.