ENH: Offer zstd encoding for pareto superior compression

[corneliusroemer]

Recently, zstd has become supported in Chromium alongside br, gzip and deflate as a supported compression method.

It might be worth looking into using it for on the fly compression for LAPIS as that's exactly the domain where zstd is absolutely amazing: fast compression on server side, good compression ratios, fast decompression in browser.

I had that idea when I noticed you use gzip for the many requests for covSpectrum:

https://caniuse.com/mdn-http_headers_content-encoding_zstd

[fengelniederhammer]

If at all, this should be implemented in LAPIS, so I transferred the issue.

[fengelniederhammer]

Actually this is more or less implemented, we should simply prefer zstd over gzip: #739

Note that this doesn't guarantee that CoV-Spectrum gets zstd compressed data, because this is part of the content negotiation and proxies play their part in that game. It might be that our proxies apply their own compression. Locally, without a proxy, Chrome got zstd compressed data.

[corneliusroemer]

I did a quick test including/excluding gzip from the content-encoding accept header using requestly and these are the results (caching disabled):

Zstd:

Gzip:

Note that zstd is slightly less efficient in compressing, but server responses are faster. I wonder if zstd compression strength can be increased.