bug: Disable `DataFrame.cache()` for ballista

[milenkovicm]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Ballista at the moment does not support cache functionality, df.cache() does not produce expected behaviour, so until we come up with the caching strategy for ballista I would suggest disabling the functionality

Describe the solution you'd like

DataFusion 52 comes with method to override cache factory SessionState::with_cache_factory we should use it to override cache behaviour when SessionContext is created, and return error if df.cache is called.

EDIT: maybe we could introduce logical plan extension which represents distributed cache, and then users can plug-in their behaviour on scheduler side.

Describe alternatives you've considered

Alternative would be to implement cache behaviour, which should be done, but as it is non trivial task, having dependencies on few other things we would have incorrect behaviour.

Additional context

• This task depends on Update to DataFusion v.52 #1357
• Also there is example how to create cache factory in doc: add example for cache factory datafusion#19139

[milenkovicm]

maybe we could introduce logical plan extension which represents distributed cache, and then users can plug-in their behaviour on scheduler side.

[killzoner]

maybe we could introduce logical plan extension which represents distributed cache, and then users can plug-in their behaviour on scheduler side.

Are you thinking to allow usage of https://github.com/XiangpengHao/liquid-cache with ballista distributed ?

[milenkovicm]

no, my idea is to provide same functionality like spark https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.cache.html

but if this could allow use of it, great

[killzoner]

Should we still allow it for standalone Ballista or do we make an error for both modes?

[milenkovicm]

it should work the same in both modes