Fix dictionary training huffman segfault and small speed improvement

[senhuang42]

Since dictionary training can result in very large counts for a single symbol, we increase the number of log2 buckets from 17 to 32, and increase the sorting scratch space to 192 elements.

This actually increases speed since we have more distinct count buckets too, increased from 114 to 166.
So now:

gcc
4KB blocks: enwik7: 153MB/s -> 159MB/s
4KB blocks: silesia: 194MB/s -> 200MB/s

Test: Also add a test to check that we can train on a high and low compressibility large corpus without failing. (This test fails on dev)

[terrelln]

Why does this change?

[senhuang42]

Isn't level 0 == level 3, and therefore uses huffman?

[Cyan4973]

Indeed, level 0 == level 3, and level 3 changes by the same amount.

Though it's somewhat unclear why this PR would introduce a (very minor) 1-byte regression on multiple samples.
The expectation is that it does impact huffman sorting speed only, but not the outcome of the sort operation,
which means the output should be identical ?

[senhuang42]

Oh, I see now that the question was probably referring to the change itself rather than level 0 specifically.

The changes do affect the outcome of the sort operation. Because we now accept a larger range [0, 166] (previously [0, 114]) of symbols that get distinct bucketing, any symbols with counts in [115, 165] will not be sorted via quicksort, but directly inserted into buckets. So they can end up in a different ordering than before, which can affect the header generation.