executor logs ton of `INFO CodecPool: Got brand-new decompressor [.zstd]`

[camper42]

Apache Iceberg version

1.1.0 (latest release)

Query engine

Spark

Please describe the bug 🐞

call rewrite_data_files on a zstd+parquet table (feed by stream)

executor logs too many INFO CodecPool: Got brand-new decompressor [.zstd] and I have to set log level to WARN or my pod log will fill up ephemeral storage

Is this a side effect of #5681 ?

[nastra]

I've looked at the code and this happened even before #5681. The decompressors are actually being re-used, but not across different Parquet Files. So this means in your case you have lots of Parquet files that are being read, and thus you're seeing this log output from Hadoop's CodecPool

[camper42]

thx, so for my scenario, I may need to compact more promptly and/or set the log level of the CodecPool to WARN