[RFC] Support references to multiple DDicts

[senhuang42]

This PR adds a new experimental param ZSTD_d_refMultipleDDicts which controls the behavior of ZSTD_refDDict().

If enabled, then multiple calls to ZSTD_refDDict() will persist the multiple referenced ZSTD_DDict* in memory rather than overwriting the last reference. Under the hood, we use a hash set w/ linear probing to keep track of the ZSTD_DDict*. Example usage is in the included unit test in fuzzer.c, but is basically as simple as:

ZSTD_DCtx* dctx = ZSTD_createDCtx();
ZSTD_DCtx_setParameter(dctx, ZSTD_d_refMultipleDDicts, ZSTD_rmd_refMultipleDDicts);
ZSTD_DCtx_refDDict(dctx, ddict1);
ZSTD_DCtx_refDDict(dctx, ddict2);
ZSTD_DCtx_refDDict(dctx, ddict3);
...
ZSTD_decompress...

The goal is no changes to the existing API, leaving existing behavior/usages of ZSTD_refDDict() unchanged.
Only when we call setParameter() with the new experimental param can behavior change.

There are still issues with this PR, but I wanted to put this up to get some initial comments to see if this was roughly the direction we wanted to head in.

Open questions/followups:

• Hash table allocation entry point (is refDDict() best?)
• Performance (hash table param tuning/hash function)
• API Extensions? (un-reference a dictionary)
• Add streaming unit test
• Still need to test more extensively
• Some code cleanup

[senhuang42]

One thing I can't quite figure out is why on certain builds/CI tests, zstd_decompress.c can't see XXH64, but can see XXH64_digest(), etc.? As far as I can tell, xxhash.h is within the graph of dependencies, and locally, it seems fine. Is there an additional macro I need to invoke?

[Cyan4973]

Is this related to the linux kernel test ?

[senhuang42]

Yeah, I do see a line about Replacing XXH64 prefix with xxh64 in the logs, and I assume it might be related.

[terrelln]

You should just need to change this line:

zstd/contrib/freestanding_lib/freestanding.py

Line 579 in bc0a1e4

(re.compile(r"([^\w]|^)(?P<orig>XXH64)_"), self._xxh64_prefix)

Something like: re.compile(r"([^\w]|^)(?P<orig>XXH64)[\(_]")

[Cyan4973]

minor (potential suggestion) : could also use a mask if ddictPtrTableSize is a clean power of 2.