Crazy Ideas: OpenDAL Cache

[Xuanwo]

In the past, we attempted to implement a cache layer in OpenDAL, but it did not function effectively. Therefore, it is now necessary for us to revisit this area and identify potential improvements.

For example, can we port https://github.com/facebook/CacheLib to pure rust with opendal?

[Xuanwo]

Maybe we can try cache for parquet/orc/json directly instead a general cache lib.

[Ted-Jiang]

Ask a question, not read the code overall, what the unit of the cache of parquet (whole or page) ? 🤔

[Ted-Jiang]

Sorry, another question : Is there any case combine opendal with datafusion parquetExec 🤔

[Xuanwo]

Is there any case combine opendal with datafusion parquet

Yes! We have a project to make opendal work with object_store: https://github.com/apache/incubator-opendal/tree/main/integrations/object_store. And we are working with upstream to implement some services via OpenDAL.

object_store_opendal is nearly ready for using 😆 (some edge case not passed yet)

[Ted-Jiang]

Oh thanks! Nice to here this !

[Xuanwo]

Please let me know if there are any issues with integrating OpenDAL.

[Ted-Jiang]

Sure , if have a chance i will have a try!

[MrCroxx]

Hi, @Xuanwo . Could I have a try with foyer? Is there a guide to do it?

[Xuanwo]

Perfect! I think we can add a FoyerLayer directly.

The only thing we need to do is implement a Layer in opendal. You can refer to LoggingLayer as an example.