You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you check the entire file's contents after comparing hashes, then it is probably worthwhile to explore faster hashes such as xxhash or to use hardware-accelerated checksums such as CRC32 on x86_64 if it's available, as a first pass before more expensive hashes or block by block file comparisons.
I would add an option to disable hashes altogether, too.
For example;
If i back up some files using rsync -a , i already know they are identical if the sizes are the same. If i later use rdfind to change the backup files to hard links, hashing is a waste of time and resources.
If you check the entire file's contents after comparing hashes, then it is probably worthwhile to explore faster hashes such as xxhash or to use hardware-accelerated checksums such as CRC32 on x86_64 if it's available, as a first pass before more expensive hashes or block by block file comparisons.
Here's a benchmark of fast hashes: https://github.com/rurban/smhasher#summary
Here's the crc32c Intel intrinsics: https://www.intel.com/content/www/us/en/docs/intrinsics-guide/index.html#ig_expand=1494,1494&text=crc32
The text was updated successfully, but these errors were encountered: