Introduce Blake3

[chiefbiiko]

Adds Blake3 to Substrate primitives and FRAME.

---

On paper Blake3 is fast (er than Blake2b/s, almost x2 for short inputs). That makes it an attractive alternative to Blake2 as a runtime's generic hasher (BlakeThree256) as well as for storage hashing (Blake3_128Concat). Mostly interesting for new parachains but theoretically Blake3 could substitute Blake2 throughout the stack.

Other projects that have adopted Blake3 so far include IPFS, Solana, and LLVM.

---

Below storage benchmarks comparing Blake version 2 vs 3 show no diff in reads but a considerable speedup writing. On my machine (passes benchmark check) I see write time decrease by 20%-33%.

git clone --depth 1 --branch blake3 https://github.com/chiefbiiko/substrate substrate-blake3 && cd substrate-blake3

# bench blake2
cargo run --manifest-path bin/node/cli/Cargo.toml --release --features runtime-benchmarks -- benchmark storage --dev --database paritydb --state-version 1 --weight-path blake2_storage_weights.rs

# apply blake3
grep -Prl 'blake2(?!b)' bin/node client test-utils primitives/runtime/src/generic primitives/state-machine primitives/transaction-storage-proof | xargs sed -i 's/blake2/blake3/g'
grep -Prl 'Blake2(?!b)' bin/node client test-utils primitives/runtime/src/generic primitives/state-machine primitives/transaction-storage-proof | xargs sed -i 's/Blake2/Blake3/g'
grep -rl BlakeTwo bin/node client test-utils primitives/runtime/src/generic primitives/state-machine primitives/transaction-storage-proof | xargs sed -i 's/BlakeTwo/BlakeThree/g'
backup=$(mktemp -d)
mkdir -p $backup/support
cp -r frame/support $backup
grep -Prl 'blake2(?!b)' frame | xargs sed -i 's/blake2/blake3/g'
grep -Prl 'Blake2(?!b)' frame | xargs sed -i 's/Blake2/Blake3/g'
grep -rl BlakeTwo frame | xargs sed -i 's/BlakeTwo/BlakeThree/g'
rm -rf frame/support
mv $backup/support frame

# bench blake3
cargo run --manifest-path bin/node/cli/Cargo.toml --release --features runtime-benchmarks -- benchmark storage --dev --database paritydb --state-version 1 --weight-path blake3_storage_weights.rs

cat blake*_storage_weights.rs

[chiefbiiko]

Would open a frame-metadata companion to expand its StorageHasher enum if thumbs-up.

[ggwpez]

How do the outputted weight files compare?
I will also try it out with the frame-system/frame-benchmarking baseline benchmarks.

[davxy]

Some benches over the same 3x4K input

https://github.com/davxy/crypto-benches/tree/main/hash#blake3

https://github.com/davxy/crypto-benches/tree/main/hash#blake2

[bkchr]

This pull request doesn't contain any reasoning on why we should add Blake3. Yes there is some mentioning of some faster benchmarks, but there are no numbers in this pr. Why do you need this? Do you want to use this on some Parachain?
Please give more context and reasoning.

[chiefbiiko]

Thanks for your feedback! Will run some more benchmarks later today.

[ggwpez]

I am only seeing a difference for large key types (will push the code). So it really depends on your use-case.
But please also provide some context, as Basti pointed out.

[cheme]

Small question, in the benches, does blake2-rfc run with simd activated?

[davxy]

Small question, in the benches, does blake2-rfc run with simd activated?

@cheme #12266 (comment)

[cheme]

Small question, in the benches, does blake2-rfc run with simd activated?

@cheme #12266 (comment)

Thanks, I remember seeing good improvement for blake2 with simd (but it was a long time ago so probably on blake2-rfc), I am a bit surprise the blake3 perf are that much higher (it is just a bit less round from blake2 and we probably don't want parallelism for most of our content). On a side note I would be interested to use blake3 for some blob storage (to be able to use the binary tree hashing for recalculation on change and eventually proofs (but at this time blob storage is only something "in progress" and non persistent)).

[burdges]

We should overall the blake3 interface to make blake3 Vec<u8>s act like child trees, so we could've efficient edit proofs into large blake3 Vec<u8>s. If done right then large blake3 Vec<u8>s would actually produce smaller and faster PoVs than our current radix 16 trie, but they're obviously not real Vec<u8>s under the hood, just views into the parachain's Vec<u8>s.

A minor concern, blake3 might not work so well on really really tiny signer devices. It needs maybe 32 * (5 + log_2 n) bytes to hash an n byte string, but the rust implementations sets log_2 n = 55, so each blak3 hasher needs 16kb. https://github.com/BLAKE3-team/BLAKE3/blob/master/src/lib.rs#L947

An idea, one host call that creates an opaque blake3 hasher in the host, and then update and finalize hostcalls that manipulates it there, including building the copath elements that go into the PoV.

None of what I'm saying here prevents adding a blake3 via the normal interface of course. Just fyi here really.

[cheme]

one host call that creates an opaque blake3 hasher in the host

I have such hashers for the existing ones implemented in https://github.com/cheme/substrate/blob/2278e8c761cede6eadb6e71a910fd2d414241d35/primitives/io/src/lib.rs#L1428 (my branch with the transient storage), it requires to store the hashers state in the state machine changes overlay, not finished/fully tested though.
Would need a specific tree hashing api though (but the point is that it already introduce the opaque (or to be opaque) handle on a hasher state). Still I am not too sure about this design yet (make sense to me but I feel like it may be something that will not reach consensus).

[burdges]

I opened a discussion on the blake3 repo BLAKE3-team/BLAKE3#279 which may tell us if many people think all that would be useful.

[cheme]

I opened a discussion on the blake3 repo BLAKE3-team/BLAKE3#279 which may tell us if many people think all that would be useful.

From the heading comment in https://github.com/BLAKE3-team/BLAKE3/blob/master/src/guts.rs they already expose some internal for the bao project, which makes it possible to implement it ourselve, even if a bit of work that could be shared at the crate level indeed.

[burdges]

It's also unclear how much we care, like how often someone needs a big bunch of bytes

[stale]

Hey, is anyone still working on this? Due to the inactivity this issue has been automatically marked as stale. It will be closed if no further activity occurs. Thank you for your contributions.

[nazar-pc]

We will use Blake3 for other purposes in our protocol and from our tests it does provide the lowest latency comparing to Sha256 and Blake3, so we'd be happy to utilize it across the board instead of Blake2 (so we have less variation in the protocol).