-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[6.4.0] Add support for the BLAKE3 digest function #19191
[6.4.0] Add support for the BLAKE3 digest function #19191
Commits on Aug 7, 2023
-
Add BLAKE3 digest function to remote_execution.proto
(cherry picked from commit e382abf)
Configuration menu - View commit details
-
Copy full SHA for 33a71e5 - Browse repository at this point
Copy the full SHA 33a71e5View commit details -
Add BLAKE3 source code to third_party
(cherry picked from commit 0ff4f6d)
Configuration menu - View commit details
-
Copy full SHA for 67faa67 - Browse repository at this point
Copy the full SHA 67faa67View commit details -
Add BLAKE3 source code to third_party
This PR adds the BLAKE3 C and asm sources to third_party, and includes a BUILD file to build them. PiperOrigin-RevId: 541539341 Change-Id: I49b1edce20a7d0f986e29712e6050e4e0b9c1d44 (cherry picked from commit a3a569e)
Configuration menu - View commit details
-
Copy full SHA for a5329b3 - Browse repository at this point
Copy the full SHA a5329b3View commit details -
Support new-style digest functions
Support new-style digest functions. This PR adds support for new-style digest functions to the remote execution library code. The remote-apis spec says: ``` // * `digest_function` is a lowercase string form of a `DigestFunction.Value` // enum, indicating which digest function was used to compute `hash`. If the // digest function used is one of MD5, MURMUR3, SHA1, SHA256, SHA384, SHA512, // or VSO, this component MUST be omitted. In that case the server SHOULD // infer the digest function using the length of the `hash` and the digest // functions announced in the server's capabilities. ``` PiperOrigin-RevId: 543691155 Change-Id: If8c386d923db1b24dff6054c8ab3f783409b7f13 (cherry picked from commit 88412ce)
Configuration menu - View commit details
-
Copy full SHA for 62463fb - Browse repository at this point
Copy the full SHA 62463fbView commit details -
This PR adds the Blake3Hasher and Blake3HashFunction classes to vfs and makes them available under the flag --digest_function=BLAKE3. PiperOrigin-RevId: 550525978 Change-Id: Iedc0886c51755585d56b4d8f47676d3be5bbedba (cherry picked from commit cc49d68)
Configuration menu - View commit details
-
Copy full SHA for 9a5f88a - Browse repository at this point
Copy the full SHA 9a5f88aView commit details -
Simplify blake3 hasher and improve performance
Improve BLAKE3 performance by reducing copies. Rather than buffering bytes, it's faster to just update the blake3 hasher as bytes are hashed. Performance with buffering: ``` Benchmark (inputSize) Mode Cnt Score Error Units Blake3Benchmark.blake3Hash 1 avgt 10 1766.697 ± 709.515 ns/op Blake3Benchmark.blake3Hash 10 avgt 10 1466.253 ± 19.474 ns/op Blake3Benchmark.blake3Hash 100 avgt 10 1522.845 ± 15.480 ns/op Blake3Benchmark.blake3Hash 1000 avgt 10 2254.156 ± 8.588 ns/op Blake3Benchmark.blake3Hash 10000 avgt 10 4660.881 ± 28.637 ns/op Blake3Benchmark.blake3Hash 100000 avgt 10 24283.191 ± 32.754 ns/op Blake3Benchmark.sha256Hash 1 avgt 10 742.091 ± 6.307 ns/op Blake3Benchmark.sha256Hash 10 avgt 10 757.844 ± 12.042 ns/op Blake3Benchmark.sha256Hash 100 avgt 10 942.902 ± 555.874 ns/op Blake3Benchmark.sha256Hash 1000 avgt 10 1208.336 ± 508.392 ns/op Blake3Benchmark.sha256Hash 10000 avgt 10 4871.231 ± 494.507 ns/op Blake3Benchmark.sha256Hash 100000 avgt 10 40686.231 ± 63.814 ns/op ``` Performance without buffering (after this CL): ``` Benchmark (inputSize) Mode Cnt Score Error Units Blake3Benchmark.blake3Hash 1 avgt 10 1021.075 ± 11.640 ns/op Blake3Benchmark.blake3Hash 10 avgt 10 1029.561 ± 19.850 ns/op Blake3Benchmark.blake3Hash 100 avgt 10 1070.509 ± 12.140 ns/op Blake3Benchmark.blake3Hash 1000 avgt 10 1685.043 ± 13.963 ns/op Blake3Benchmark.blake3Hash 10000 avgt 10 3939.516 ± 13.212 ns/op Blake3Benchmark.blake3Hash 100000 avgt 10 21730.550 ± 22.976 ns/op Blake3Benchmark.sha256Hash 1 avgt 10 745.943 ± 9.853 ns/op Blake3Benchmark.sha256Hash 10 avgt 10 747.649 ± 17.381 ns/op Blake3Benchmark.sha256Hash 100 avgt 10 962.802 ± 590.879 ns/op Blake3Benchmark.sha256Hash 1000 avgt 10 1189.069 ± 412.327 ns/op Blake3Benchmark.sha256Hash 10000 avgt 10 4594.978 ± 21.833 ns/op Blake3Benchmark.sha256Hash 100000 avgt 10 39224.248 ± 229.265 ns/op ``` PiperOrigin-RevId: 551225043 Change-Id: I57dc0700b2f44d6faf75ffbd29f1607544e54f29 (cherry picked from commit d922ab3)
Configuration menu - View commit details
-
Copy full SHA for ee243f2 - Browse repository at this point
Copy the full SHA ee243f2View commit details -
Ensure namedSetOfFiles URIs specify blob type correctly
I noticed when testing the BLAKE3 digest function that uploaded files were being referenced incorrectly in the BES because they were missing the digest type. This CL fixes that. Before: https://github.com/bazelbuild/bazel/assets/141737/52781f1b-b897-48f0-8956-f63c57b59436 After: https://github.com/bazelbuild/bazel/assets/141737/01ebc61b-3512-4ca5-8e2d-f47ad5f086b7 PiperOrigin-RevId: 553405394 Change-Id: Ic976e5a58f80ab8b5270b4aedc6204c22562533a (cherry picked from commit 1929367) Signed-off-by: Brentley Jones <github@brentleyjones.com>
Configuration menu - View commit details
-
Copy full SHA for 80c1df2 - Browse repository at this point
Copy the full SHA 80c1df2View commit details -
Separate new-style-hash content in DiskCache
DiskCache always stores files in /root/{cas/ac}/digestHash. This change keeps the current behavior, but for new style digest functions inserts a directory between /root/ and {cas/ac} to disambiguate the digest function type. This prevents issues that could theoretically happen if there were hash collisions between two digest functions sharing the same cache directory. PiperOrigin-RevId: 554477775 Change-Id: Ibef994e432764c260a3cab821ca6583c231c5b50 (cherry picked from commit f17b280) Signed-off-by: Brentley Jones <github@brentleyjones.com>
Configuration menu - View commit details
-
Copy full SHA for 1286538 - Browse repository at this point
Copy the full SHA 1286538View commit details
Commits on Aug 8, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 9393dca - Browse repository at this point
Copy the full SHA 9393dcaView commit details