[Auditbeat] Add xxhash as a hash option for faster file integrity checks on large files

[geekpete]

xxhash is one of the best performing hashes for computing a hash against large files.

See benchmarks by @jpountz:
https://jpountz.github.io/lz4-java/1.2.0/xxhash-benchmark/
there's many other benchmarks out there if more evidence is needed.

Including xxhash in the list of available hashes that can be used with auditbeat would allow larger files to be hashed much much faster.
This might then provide the ability to increase the default max file size that is hashed, which currently defaults to excluding files over 100MiB being hashed at all, by either using xxhash for larger files and whatever other hash for smaller files or the ability to realise better overall audit hashing performance for larger files generally when using xxhash for greater visibility and auditing for use cases that deal with larger files.

[andrewkroh]

So far we have not made available in Auditbeat any non-cryptographic hash functions. I was mostly thinking of security use cases, but for purely doing integrity checks I see no issues with adding non-cryptographic hash functions. We can differentiate them in the docs.

At the moment Auditbeat's most performant hash (at least on amd64 on my machine) is blake2b. Even though it is probably the fastest it is not the default. I wanted to keep SHA1 as the default because it's relatively fast and it common in data feeds from threat exchanges.

I found a pure Go implementation that we can try out. https://github.com/OneOfOne/xxhash

[andrewkroh]

For anyone that's interested in contributing this feature: take a look at the touch points for SHA3 when it was added. Each of those same places will need updated for xxhash. The you can run the benchmark test (go test -bench BenchmarkHashFile -benchtime 10s -run XXX -v).

[andrewkroh]

Feature has been added in #7311 to master and will be included in the next release - v6.4.0.