Add BLAKE3CONCAT: a version of BLAKE3 that supports chunking

In PR #233 I proposed the addition of two new ContentAddressableStorage methods (ConcatenateBlobs and SplitBlobs) that allow one to gain random access it large CAS objects, while still providing a way to very data integrity. As part of that change, I added a new digest function to help with that, named BLAKE3CONCAT. This PR adds just this digest function, without bringing in any support for chunking. This will be done separately, as it was requested that both these features landed independently. I have also included test vectors for the BLAKE3CONCAT digest function. I have derived these by modifying the BLAKE3 reference implementation written in Rust, and rerunning the tool that emits the official test vectors: https://github.com/BLAKE3-team/BLAKE3/blob/master/test_vectors/test_vectors.json Furthermore, I have been able to validate the newly obtained test vectors using a custom BLAKE3CONCAT implementation that I have written in Go, which will become part of Buildbarn.
bazelbuild · Nov 24, 2022 · 55dd91f · 55dd91f
1 parent 3a21dee
commit 55dd91f
Show file tree

Hide file tree

Showing 3 changed files with 453 additions and 252 deletions.
diff --git a/build/bazel/remote/execution/v2/blake3concat_test_vectors.json b/build/bazel/remote/execution/v2/blake3concat_test_vectors.json
@@ -0,0 +1,145 @@
+{
+ "_comment": "Each test is an input length and one hash output. The input in each case is filled with a repeating sequence of 251 bytes: 0, 1, 2, ..., 249, 250, 0, 1, ..., and so on.",
+ "cases": [
+ {
+ "input_len": 0,
+ "hash": "b4686b3c8d953f4d185d51bc9cd7bce6d9782d7659f8c06068d4c3ffa6e56841"
+ },
+ {
+ "input_len": 1,
+ "hash": "0df1a7881d71d287d0afc2fcd7d23d28f12e401ab858ca267f11c5f1710df315"
+ },
+ {
+ "input_len": 2,
+ "hash": "b2530fd8d3fa7ca531d3e03d3b44f434354af89b59c14123d1d1ecc664b243a6"
+ },
+ {
+ "input_len": 3,
+ "hash": "8e2451ec9f2ae6282e10a52c09d3bf7856236cfa990b5f1aaf0b3943d5b72d18"
+ },
+ {
+ "input_len": 4,
+ "hash": "826e2530317a718add9a012ad6cf1c2cd298c6f7fa09022dd7f1847cfd03ddb8"
+ },
+ {
+ "input_len": 5,
+ "hash": "c6e98dc8c0e86be0a25d8e4a4ac951f5bf4efa111d8a332843aabe723fab3452"
+ },
+ {
+ "input_len": 6,
+ "hash": "9532a581f853cd63594625ddd0bdcac2edaf9cf436b0b42af77938ed5cfc5181"
+ },
+ {
+ "input_len": 7,
+ "hash": "1d2cfb6af10502c1b7067f6e4515dac354cb4f6c9def44704e9a79e32f41688b"
+ },
+ {
+ "input_len": 8,
+ "hash": "ffa72fdba626d7aa0d1ad38bb7f289c91386ae4d2929fa093ccc5d188433e906"
+ },
+ {
+ "input_len": 63,
+ "hash": "eefdb8f62d0cb234d94b16a2837eb72665018861396a89ef9f28d1fb190fca24"
+ },
+ {
+ "input_len": 64,
+ "hash": "6dde41c9d05a39b09b486620f2c3cf7652fde7f3eb412353d95734294c5d348e"
+ },
+ {
+ "input_len": 65,
+ "hash": "5bf4b8a5dba62b01ebd11264f3a724638e3f0ef274f253dfa41c0a657db3ab3e"
+ },
+ {
+ "input_len": 127,
+ "hash": "cdfdc645bd59bb2415df258b0d97a1f07e68710f67e6e61e78aa15f4703d53a2"
+ },
+ {
+ "input_len": 128,
+ "hash": "d41a4a8d07d439dc3892f45a296b934f2fbbd966f29f8640e88f15dde20015c7"
+ },
+ {
+ "input_len": 129,
+ "hash": "7a0235d0b622ed10c03a5c9f7fcfb36b2fc8aefb50e3a95ded8add3e06126214"
+ },
+ {
+ "input_len": 1023,
+ "hash": "ad61f1f5c9f2a65c7b96152b47321c7d4320e2c2719e14bee6640815e4182ddb"
+ },
+ {
+ "input_len": 1024,
+ "hash": "5c9e654411e393d1f4bec710ccd5bc5669ab177d610a0eb691fcfee92fb4e8b1"
+ },
+ {
+ "input_len": 1025,
+ "hash": "ee286548954cbdc4bf9b5fef0555f1b7d813edbe2bc7f3693126d5825999139d"
+ },
+ {
+ "input_len": 2048,
+ "hash": "80c9b515da8b7fcabfefcc4b049b7cbcf16983c2cd1448e7717168f279b4d7ad"
+ },
+ {
+ "input_len": 2049,
+ "hash": "051bad3acf63a2d27429524232b86b8230234e0091f38e0686faf79df77e1b30"
+ },
+ {
+ "input_len": 3072,
+ "hash": "cd28dcf373e40cc36aa4a7349687d3805babb92229c0cf28d903ee2d08e1194c"
+ },
+ {
+ "input_len": 3073,
+ "hash": "fe07c25f6c1bc98d104ed1ca7971177e8515b28f6287e35569098b448f31f39a"
+ },
+ {
+ "input_len": 4096,
+ "hash": "65e4ec2f94073d6eb00a9bf5c085cde958ec73ed3baaf2b80d5e6f90dc868e94"
+ },
+ {
+ "input_len": 4097,
+ "hash": "7f59a96c5b78c65db13b981c327ebc170bcdd155123857dfb8720349b0ae7699"
+ },
+ {
+ "input_len": 5120,
+ "hash": "88cf814865f6b6f1488e3c8dd14ebab858659459e8c1552a26bcfd81f75465c7"
+ },
+ {
+ "input_len": 5121,
+ "hash": "01f181c74b1a8284101fa35c8082734f167fcc7f6ac75c2547ecfb0c765a1073"
+ },
+ {
+ "input_len": 6144,
+ "hash": "b739390236994f666fd8c0a6d57e3a6192034a43ca990906f2abbad1509c719f"
+ },
+ {
+ "input_len": 6145,
+ "hash": "a0e15c6796ed7da911d75d545ee14c960ff1e9dd37ecfbd9101c25f9e4d073f9"
+ },
+ {
+ "input_len": 7168,
+ "hash": "b55ab1c0b44dab6c0e25839c33198f3f76a688f31ec4a8d109f190befb3634f2"
+ },
+ {
+ "input_len": 7169,
+ "hash": "23cb0c89436728d339dae6771e9078109df0fdbdee617d6e49d615c5d5212ef2"
+ },
+ {
+ "input_len": 8192,
+ "hash": "fda9453f03d5f5080d3c41126fddc294c0d8f48aa3fdbcdd63f9cda4d33fc211"
+ },
+ {
+ "input_len": 8193,
+ "hash": "5da258b0a71add844c953fd1b3c74a99f3af9011baeaedae7962f4dde67e5cf9"
+ },
+ {
+ "input_len": 16384,
+ "hash": "222b977bbd1d5dc6529f424379ac4e987f927a10411f52ffce160e7e1f4c06b8"
+ },
+ {
+ "input_len": 31744,
+ "hash": "f1ae00b44cb183227d4296ad578af3c2b27132dcf3ee94e7731acc198fdeb114"
+ },
+ {
+ "input_len": 102400,
+ "hash": "9dd1fb18c727e9bfc58c9ff5487ac5b08e6b1f32d65348a68dc4131e967cd2ce"
+ }
+ ]
+}