Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[6.4.0] Add support for the BLAKE3 digest function #19191

Merged
merged 9 commits into from
Aug 8, 2023

Commits on Aug 7, 2023

  1. Add BLAKE3 digest function to remote_execution.proto

    (cherry picked from commit e382abf)
    tylerwilliams authored and brentleyjones committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    33a71e5 View commit details
    Browse the repository at this point in the history
  2. Add BLAKE3 source code to third_party

    (cherry picked from commit 0ff4f6d)
    tylerwilliams authored and brentleyjones committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    67faa67 View commit details
    Browse the repository at this point in the history
  3. Add BLAKE3 source code to third_party

    This PR adds the BLAKE3 C and asm sources to third_party, and includes a BUILD
    file to build them.
    
    PiperOrigin-RevId: 541539341
    Change-Id: I49b1edce20a7d0f986e29712e6050e4e0b9c1d44
    
    (cherry picked from commit a3a569e)
    tylerwilliams authored and brentleyjones committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    a5329b3 View commit details
    Browse the repository at this point in the history
  4. Support new-style digest functions

    Support new-style digest functions.
    
    This PR adds support for new-style digest functions to the remote execution
    library code. The remote-apis spec says:
    
    ```
    // * `digest_function` is a lowercase string form of a `DigestFunction.Value`
    //   enum, indicating which digest function was used to compute `hash`. If the
    //   digest function used is one of MD5, MURMUR3, SHA1, SHA256, SHA384, SHA512,
    //   or VSO, this component MUST be omitted. In that case the server SHOULD
    //   infer the digest function using the length of the `hash` and the digest
    //   functions announced in the server's capabilities.
    ```
    
    PiperOrigin-RevId: 543691155
    Change-Id: If8c386d923db1b24dff6054c8ab3f783409b7f13
    
    (cherry picked from commit 88412ce)
    tylerwilliams authored and brentleyjones committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    62463fb View commit details
    Browse the repository at this point in the history
  5. Add BLAKE3 hasher to vfs

    This PR adds the Blake3Hasher and Blake3HashFunction classes to vfs and
    makes them available under the flag --digest_function=BLAKE3.
    
    PiperOrigin-RevId: 550525978
    Change-Id: Iedc0886c51755585d56b4d8f47676d3be5bbedba
    
    (cherry picked from commit cc49d68)
    tylerwilliams authored and brentleyjones committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    9a5f88a View commit details
    Browse the repository at this point in the history
  6. Simplify blake3 hasher and improve performance

    Improve BLAKE3 performance by reducing copies.
    
    Rather than buffering bytes, it's faster to just update the blake3 hasher as bytes are hashed.
    
    Performance with buffering:
    ```
     Benchmark                   (inputSize)  Mode  Cnt      Score     Error  Units
     Blake3Benchmark.blake3Hash            1  avgt   10   1766.697 ± 709.515  ns/op
     Blake3Benchmark.blake3Hash           10  avgt   10   1466.253 ±  19.474  ns/op
     Blake3Benchmark.blake3Hash          100  avgt   10   1522.845 ±  15.480  ns/op
     Blake3Benchmark.blake3Hash         1000  avgt   10   2254.156 ±   8.588  ns/op
     Blake3Benchmark.blake3Hash        10000  avgt   10   4660.881 ±  28.637  ns/op
     Blake3Benchmark.blake3Hash       100000  avgt   10  24283.191 ±  32.754  ns/op
     Blake3Benchmark.sha256Hash            1  avgt   10    742.091 ±   6.307  ns/op
     Blake3Benchmark.sha256Hash           10  avgt   10    757.844 ±  12.042  ns/op
     Blake3Benchmark.sha256Hash          100  avgt   10    942.902 ± 555.874  ns/op
     Blake3Benchmark.sha256Hash         1000  avgt   10   1208.336 ± 508.392  ns/op
     Blake3Benchmark.sha256Hash        10000  avgt   10   4871.231 ± 494.507  ns/op
     Blake3Benchmark.sha256Hash       100000  avgt   10  40686.231 ±  63.814  ns/op
    ```
    
    Performance without buffering (after this CL):
    
    ```
     Benchmark                   (inputSize)  Mode  Cnt      Score     Error  Units
     Blake3Benchmark.blake3Hash            1  avgt   10   1021.075 ±  11.640  ns/op
     Blake3Benchmark.blake3Hash           10  avgt   10   1029.561 ±  19.850  ns/op
     Blake3Benchmark.blake3Hash          100  avgt   10   1070.509 ±  12.140  ns/op
     Blake3Benchmark.blake3Hash         1000  avgt   10   1685.043 ±  13.963  ns/op
     Blake3Benchmark.blake3Hash        10000  avgt   10   3939.516 ±  13.212  ns/op
     Blake3Benchmark.blake3Hash       100000  avgt   10  21730.550 ±  22.976  ns/op
     Blake3Benchmark.sha256Hash            1  avgt   10    745.943 ±   9.853  ns/op
     Blake3Benchmark.sha256Hash           10  avgt   10    747.649 ±  17.381  ns/op
     Blake3Benchmark.sha256Hash          100  avgt   10    962.802 ± 590.879  ns/op
     Blake3Benchmark.sha256Hash         1000  avgt   10   1189.069 ± 412.327  ns/op
     Blake3Benchmark.sha256Hash        10000  avgt   10   4594.978 ±  21.833  ns/op
     Blake3Benchmark.sha256Hash       100000  avgt   10  39224.248 ± 229.265  ns/op
    ```
    
    PiperOrigin-RevId: 551225043
    Change-Id: I57dc0700b2f44d6faf75ffbd29f1607544e54f29
    
    (cherry picked from commit d922ab3)
    tylerwilliams authored and brentleyjones committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    ee243f2 View commit details
    Browse the repository at this point in the history
  7. Ensure namedSetOfFiles URIs specify blob type correctly

    I noticed when testing the BLAKE3 digest function that uploaded files were being referenced incorrectly in the BES because they were missing the digest type. This CL fixes that.
    
    Before: https://github.com/bazelbuild/bazel/assets/141737/52781f1b-b897-48f0-8956-f63c57b59436
    
    After: https://github.com/bazelbuild/bazel/assets/141737/01ebc61b-3512-4ca5-8e2d-f47ad5f086b7
    
    PiperOrigin-RevId: 553405394
    Change-Id: Ic976e5a58f80ab8b5270b4aedc6204c22562533a
    
    (cherry picked from commit 1929367)
    Signed-off-by: Brentley Jones <github@brentleyjones.com>
    tylerwilliams authored and brentleyjones committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    80c1df2 View commit details
    Browse the repository at this point in the history
  8. Separate new-style-hash content in DiskCache

    DiskCache always stores files in /root/{cas/ac}/digestHash. This change keeps the current behavior, but for new style digest functions inserts a directory between /root/ and {cas/ac} to disambiguate the digest function type.
    
    This prevents issues that could theoretically happen if there were hash collisions between two digest functions sharing the same cache directory.
    
    PiperOrigin-RevId: 554477775
    Change-Id: Ibef994e432764c260a3cab821ca6583c231c5b50
    
    (cherry picked from commit f17b280)
    Signed-off-by: Brentley Jones <github@brentleyjones.com>
    tylerwilliams authored and brentleyjones committed Aug 7, 2023
    Configuration menu
    Copy the full SHA
    1286538 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. Configuration menu
    Copy the full SHA
    9393dca View commit details
    Browse the repository at this point in the history