Feature request: add zstd compression support

[aborruso]

Miller is the data tool I use the most. Another tool that I use a lot is duckdb.

It supports zstd (and gzip) compressed csv. ZSTD compression and decompression can be extremely fast. I compress a 4.5 GB CSV file in 3 seconds (I have 16 GB of ram and 12th Gen Intel(R) Core(TM) i7-1280P 2.00 GHz).
The output is a 160 MB compressed csv file.
And it's possible to run a duckdb SUMMARIZE on it in 8.5 seconds.
The CSV has 1745439 rows and 199 columns.

A big credit goes to duckdb, but part of the credit goes to this compression format.

This issue to ask enable it in Miller compressed data.

Thank you

[aborruso]

What do you think about this @johnkerl ?

Thank you

[johnkerl]

@aborruso for comparison let's first look at gzip. There are two ways to get gzip: --prepipe gunzip and --gzin. The first one is flexible: you get to specify the executable. The second one is done in-process and it requires support from the Go library: https://pkg.go.dev/compress/gzip

Now for zstd. If there is an executable for that, you can do --prepipe zstd. To implement --zstdin we'd need a Go library for handling zstd data. But https://pkg.go.dev/compress does not have one. There may be some other place to get a Go library that does zstd: for example https://pkg.go.dev/github.com/klauspost/compress/zstd.

[johnkerl]

@aborruso can you check out head and try this?
#1360

No worries if not; please let me know ...

Also you can take a peek at head docs here:
https://miller.readthedocs.io/en/main/reference-main-compressed-data/#compressed-data

[aborruso]

@aborruso can you check out head and try this?

Wow, it works great. I was already using zstd with the prepipe, but it seemed very convenient and important for Miller to support it natively and directly.
I think it's becoming a "standard" in the context of compressed structured text data.

Thank you very much