Tuning knobs to tradeoff CPU and compression in parquet

[alamb]

Moving a PR discussion to a ticket so it doesn't get lost.

At a high level, the observation that @JigaoLuo and @mapleFU made is that there is a tradeoff when writing parquet between file size and encode/decode speed (typically the smaller the size, the slower the file is to decode)

Currently the arrow-rs parquet reader allows some configuration here:

1. Setting the block compression to something like zstd decreases file size, but increases encode/decode speed

However there are other settings that we might consider:

1. Automatically turning off block compression if the size savings isn't "good enough" (as described in Parquet: Do not compress v2 data page when compress is bad quality #8257)
2. Adjusting the RLE lengths as suggested by @jhorstmann in Parquet LevelEncoder is much too eager to write short rle runs #7739

This ticket tracks potentially improving the ability of users to tune this tradeoff

Hello everyone,

I just came across this PR and noticed that most of the discussion is happening here, so I’d like to continue the conversation in this thread rather than on the issue page.

I believe the direction of this PR aligns well with a previous issue we discussed in XiangpengHao/liquid-cache#227. I’ve been working on my own parquet-rewrite tool that touches on similar ideas, particularly with the score metric—a kind of breakeven point to decide whether compression should be applied. The goal of this tool is to help the reader skip unnecessary compression that adds overhead without delivering meaningful size reduction, ultimately improving the reader's reading performance.

Setting this score is quite tricky and empirical. For now, I’ve set it at 10%, mainly to catch cases where compression offers no size benefit at all. Here is an example of this case (in the level of full column):

---

As a side note, I’ve also made some patches to Xiangpeng’s viewer tool, which I use to inspect my generated Parquet files. This has been instrumental in iterating on my reader implementation.

Originally posted by @JigaoLuo in #8257 (comment)

[JigaoLuo]

Thanks as always—I really enjoy these topics and often think about them (when working with the cuDF reader), especially around the tradeoff between computation and I/O.

Just a quick comment: I had some dummy PR to the parquet-rewriter CLI, but it didn’t scale well to add a lot features such as this score. So I ended up building my own separate Parquet rewriter, where I implemented the scoring logic and a few other custom features.

[JigaoLuo]

One thing I still don’t fully understand is when and why to apply specific compression algorithms.

• For example, could it be that Snappy doesn’t compress the encoded data much beyond what the encoding already achieves? At the same time, other schemes like ZSTD or GZIP in the search space might offer a higher compression ratio than Snappy?

I’ve also noticed something at my side that is also related to this umbrella issue: when I sort my Parquet file by some columns, Snappy compression could not offer a nice compression ratio.

(I have a solid grasp of encodings. But when it comes to compression, my understanding is still limited.)

[JigaoLuo]

I’ll continue collecting observations here gradually.

One thing I’ve noticed after sorting a column for stat-pruning is that the sorted integer keys become quite dense—often resulting in an encoded size under 1MB. In such cases, compression doesn’t seem to provide much benefit.

• This leads me to consider a potential heuristic: if the encoded size falls below a certain threshold, it might be better to skip compression altogether. Applying compression in these scenarios could add overhead without meaningful space savings.

---

Example:

[mapleFU]

Another issue: #8378

Personally I think an encoding selector is good. Seems that deciding encoding "perfectly" is hard, but we can choose dictionary using some smart way here