Provide mechanism to cache manifest file content across several planFiles call

[rizaon]

Iceberg planFiles API has benefit of filtering out unnecessary files given a set of filter predicates. During query planning, a query engine might need to call planFiles against the same table multiple times but with slightly different filter predicates for each scan. This lead to repeated manifest files reading over the same table. It is often expensive, especially if it involve remote reads through network. Certain stream operation such as backward seek and small reads characteristics by Avro library can have high latency depending on the target file system / cloud connector.

Having ability to cache manifest content in memory for several planFiles calls can improve performance. One idea is to add method in TableScanContext.java to supply a caffeine cache, say Cache<String, byte[]>, and pass it further to AvroIO.java. In turn, AvroIO can consult with this cache and use the cached manifest file content if it is available.

[rajarshisarkar]

It is often expensive, especially if it involve remote reads through network. Certain stream operation such as backward seek and small reads characteristics by Avro library can have high latency depending on the target file system / cloud connector.

Thanks, for bringing this up. We also recently noticed this while using S3 as the target file system. The use case was for high throughput streaming jobs with a lot of snapshots.

Do let me know if you would like to collaborate on this work.

[rizaon]

Hello @rajarshisarkar , thanks for your interest!
I'm happy to collaborate. I filed PR #4518 as a draft for this improvement idea.