-
Notifications
You must be signed in to change notification settings - Fork 370
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(perf): pull hashes without refreshing metadata #1419
Conversation
9f35847
to
809acc2
Compare
Codecov Report
@@ Coverage Diff @@
## master #1419 +/- ##
=======================================
Coverage 99.10% 99.10%
=======================================
Files 14 14
Lines 12145 12155 +10
Branches 529 530 +1
=======================================
+ Hits 12036 12046 +10
Misses 109 109
Continue to review full report at Codecov.
|
809acc2
to
2ce0764
Compare
const delimiterIndex = hashKeyValPair.indexOf('='); | ||
const hashType = hashKeyValPair.substr(0, delimiterIndex); | ||
const hashValue = hashKeyValPair.substr(delimiterIndex + 1); | ||
hashes[hashType as 'crc32c' | 'md5'] = hashValue; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed this PR with @shaffeeullah on md5 hashes she'll follow-up on remaining Node.js specific changes:
- Composed objects will not support md5. Could you add a warning / message to users that it can't be validated because it's a composed object (only md5: https://cloud.google.com/storage/docs/composite-objects#metadata).
- Could you also add a unit test for this case?
Example of a non-composed GCS Object response headers:
HTTP/1.1 200 OK
X-goog-metageneration: 53
Content-location: https://storage.googleapis.com/storage/v1/b/anima-frank/o/coderfrank?alt=media
Content-disposition: attachment
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Vary: Origin,X-Origin
Content-length: 0
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Etag: CNerzaGOku8CEDU=
X-goog-generation: 1614705373435351
Pragma: no-cache
X-guploader-uploadid: ABg5-UzkhLZtBh_nh_MyJJUrxf2cptQxl5InjXd8jYRhUN31wS_DHiHBctrKCu7RLyi0q_5cZvAdHpj1D7nS9FsgsbI
Date: Mon, 15 Mar 2021 20:48:06 GMT
X-goog-hash: crc32c=AAAAAA==,md5=1B2M2Y8AsgTpgAmY7PhCfg==
Content-type: application/octet-stream
X-goog-storage-class: MULTI_REGIONAL
Example of a composed GCS Object response headers:
HTTP/1.1 200 OK
X-goog-metageneration: 1
Content-location: https://storage.googleapis.com/storage/v1/b/anima-frank/o/coderfrank-composed?alt=media
Content-disposition: attachment
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Vary: Origin,X-Origin
Content-length: 0
Cache-control: no-cache, no-store, max-age=0, must-revalidate
Etag: CNr8/deVs+8CEAE=
X-goog-generation: 1615841237892698
Pragma: no-cache
X-guploader-uploadid: ABg5-UzkOS0YokMzs44P_reufnM5l2idHXaVhKKbsjyQr1OzvgqZtOUIEkahxuFplJfkcylpFQ_4GyEq98JbWocEvt4
Date: Mon, 15 Mar 2021 20:48:59 GMT
X-goog-hash: crc32c=AAAAAA==
Content-type: application/octet-stream
X-goog-storage-class: MULTI_REGIONAL
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wouldn’t mind tackling that in a follow-up PR, unless you feel strongly it should be included here. This change shouldn’t alter the behavior we currently have, as far as how that situation is handled.
I will add a test for if just one algorithm comes back.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Test added.
2ce0764
to
1d79a1b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @stephenplusplus, LGTM!
Pending @shaffeeullah approval as well.
* build: add sync-repo-settings and change branch protection Source-Link: googleapis/synthtool@ed8079c Post-Processor: gcr.io/cloud-devrel-public-resources/owlbot-nodejs:latest@sha256:80bfa0c67226453b37b501be7748b2fa2a2676cfeec0012e79e3a1a8f1cbe6a3
Fixes #1338
This PR narrows the conditions for when a metadata refresh is required during a file download.
Before:
file.getMetadata()
to get the remote file's hashes, even if the user disabled data integrity (validation:false
)After:
x-goog-hash
)So, one less API call.