-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement BLAKE2b checksums #2166
Comments
I would like to send a patch if this is approved (no new dependencies). |
Well, we already have SHA-512, which is much faster than SHA-256 (at least in the current implementation). |
BLAKE2b is about 1.5x faster than SHA-512 (similar to margin between SHA-256 and SHA-512), while also being way simpler (compare https://github.com/openssl/openssl/blob/master/crypto/sha/sha512.c at 694 SLOC with https://github.com/openssl/openssl/blob/master/crypto/blake2/blake2b.c at 241 SLOC). See https://blake2.net/. It was also introduced in coreutils, see (For anyone who wants to study the function, here's cleaned up source code: https://github.com/yegortimoshenko/sylpheed/blob/d7a91f82e5785992e3716529b63d8530560a5479/src/blake2b.c) |
Another promising candidate is BLAKE3 (which appeared recently in Jan 2020). It is 6x faster than sha512 on my machine on some random 91M file from Benchmark$ du -sh /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi
91M /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi
$ hyperfine 'b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi' 'sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi'
Benchmark #1: b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi
Time (mean ± σ): 45.8 ms ± 4.4 ms [User: 142.5 ms, System: 10.4 ms]
Range (min … max): 41.4 ms … 58.9 ms 62 runs
Benchmark #2: sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi
Time (mean ± σ): 274.6 ms ± 19.2 ms [User: 258.6 ms, System: 15.8 ms]
Range (min … max): 254.9 ms … 309.2 ms 11 runs
Summary
'b3sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi' ran
5.99 ± 0.72 times faster than 'sha512sum /nix/store/j434ylwqjl06w3qzcis73500ks82msb7-wine-mono-4.9.3.msi'
$ b3sum --version
b3sum 0.1.1
$ sha512sum --version
sha512sum (GNU coreutils) 8.31
Packaged by https://NixOS.org
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Ulrich Drepper, Scott Miller, and David Madore.
$ grep 'model name' /proc/cpuinfo | head -n1
model name : Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz |
Probably duplicate of #1133. Gentoo uses blake2b since end of 2017. (source: wiki page on manifest files and mailing list annoucement) Archlinux has blake2 support. (source: arch wiki PKGBUILD and b2sums in PKGBUILD(5)) |
I marked this as stale due to inactivity. → More info |
blake3 seems like a good candidate |
I marked this as stale due to inactivity. → More info |
SHA256 is rather slow at checksumming large files (tangentially related to #619), would be nice to have something that is both fast and future-proof.
Use case would be to
nix-hash
large (> 4GB) files to check if already present in Nix store. This takes a while with SHA256, and would probably take at least 2x less time with BLAKE2b.The text was updated successfully, but these errors were encountered: