Expose datafusion table provider as python binding

[kevinjqliu]

Which issue does this PR close?

• Closes [pyiceberg_core] Expose IcebergTableProvider to python #865

What changes are included in this PR?

This PR creates a new IcebergDataFusionTable python class and exposes it through the new pyiceberg_core.datafusion module.

from pyiceberg_core.datafusion import IcebergDataFusionTable

The goal of exposing IcebergDataFusionTable is to be able to register the Iceberg table provider to datafusion-python, using the register_table_provider API.
See the usage example in bindings/python/tests/test_datafusion_table_provider.py

The integration relies on the FFI_TableProvider API as described in https://datafusion.apache.org/python/user-guide/io/table_provider.html

Note that this integration only works for datafusion >= 45 due to this issue apache/datafusion#13851

Are these changes tested?

Yes, unit tests.

To build and test locally:

cd bindings/python
hatch run dev:develop
hatch run dev:test

[kevinjqliu]

cc @timsaucer for helping me with the integration, finally got around to this

[Xuanwo]

Do we need to make sure that rust and python are using the same version of datafusion?

cc @alamb for ideas.

[timsaucer]

They do not need to strictly be the same version, but there have been a couple of small ffi api changes that were breaking. I need to track down exactly which versions were compatible. I will try to get to this soon.

Once the ffi api is stable you can use different versions of datafusion rust and python. I have demonstrated this in other projects.

[Xuanwo]

That's really great! Thank you @timsaucer for the help.

[Xuanwo]

Mostly LGTM, cc @liurenjie1024 for another look.

[sdd]

LGTM now, thanks @kevinjqliu 🙌🏼

[kevinjqliu]

Thanks everyone!