keeping datafusion versions aligned

[kevinjqliu]

Apache Iceberg Rust version

None

Describe the bug

When using datafusion in rust and python, Fatal Python error: Bus error usually means there is an issue with the FFI boundary. Also see apache/datafusion-python#1217

For example, datafusion 48 -> 49 has a breaking change. Thus mixing code built with datafusion 48 and datafusion 49 will throw the above error.

The issue is with newer versions of datafusion, older versions are still compatible.

This is mostly presented in the python ecosystem since installing the datafusion package usually picks the latest version. This is what happened in #1635 and pinning the datafusion version to 47 resolved the issue.

Going forward, we have to be mindful of which datafusion version we use in the repo and also to build pyiceberg-core. This will determine which datafusion python version we can use.

To Reproduce

No response

Expected behavior

No response

Willingness to contribute

None

[kevinjqliu]

No action here, really.

I just want to write this down as notes so it is searchable

[kevinjqliu]

I just added #1648 as a follow up to #1635

pyiceberg-core works with datafusion (python) library 45, 46, 47, and 48. just not 49, due to the breaking change.

[kevinjqliu]

pyiceberg-core 0.4.0 doesnt include datafusion dep, https://github.com/apache/iceberg-rust/blob/v0.4.0/bindings/python/Cargo.toml

pyiceberg-core 0.5 include datafusion ffi 47, https://github.com/apache/iceberg-rust/blob/v0.5.1/bindings/python/Cargo.toml

• this is compatible with datafusion 45, 46, 47, and 48. but not 49

pyiceberg-core 0.6 also includes datafusion ffi 47, https://github.com/apache/iceberg-rust/blob/v0.6.0/bindings/python/Cargo.toml#L38

in the upcoming version, we can either upgrade to datafusion 48 or 49. But be mindful datafusion 49 will be backwards incompatible. So if we release pyiceberg-core with datafusion 49 FFI, downstream would have to use datafusion (python) 49 or newer

[kevinjqliu]

#1649 to bump datafusion from 47 to 48. 48 is still backwards compatible.

we should be intentional when upgrading to datafusion 49.