Rust <> Python integration point

[kevinjqliu]

After establishing #518, I want to start the conversation to create the first integration between PyIceberg and iceberg-rust.
As discussed in the dev list, we want to create an integration based on pluggable FileIO.

I'm wondering if there's also a way to create an integration for a pluggable catalog, based on the in-memory catalog implementation in #475.

I'm not familiar with the rust ecosystem, so would appreciate any pointers

[Xuanwo]

I'm wondering if there's also a way to create an integration for a pluggable catalog, based on the in-memory catalog implementation in #475.

I believe this should also be possible. So, the pyiceberg community wants to have an in-memory catalog based on iceberg-rust. Does pyiceberg provide an interface that we can integrate with?

The in-memory catalog depends on FileIO, so we might need to build FileIO first. However, it also makes sense to expose a purely in-memory catalog (memory FileIO and memory catalog) to pyiceberg initially.

[liurenjie1024]

I think it's definitely possible since PyIceberg is Catalog interface is extensible. I think you need to start with pyo3 first to understand how it works.

[kevinjqliu]

Does pyiceberg provide an interface that we can integrate with?

Yes, there is a py-catalog-impl configuration that will try to load a given classpath. (documentation, implementation, test)

The in-memory catalog depends on FileIO, so we might need to build FileIO first. However, it also makes sense to expose a purely in-memory catalog (memory FileIO and memory catalog) to pyiceberg initially.

I'm bringing up this issue because I want the simplest way to integrate iceberg-python and iceberg-rust. If FileIO integration is a prerequisite, we can start there instead.

[Xuanwo]

Hi, @kevinjqliu, I'm sorry for blocking your innovation this way.

I've been a bit busy recently, but I plan to create something that really works next week. For instance, reading data from PyIceberg using pyiceberg-core. This will enable our community to build more cool things based on that.

[kevinjqliu]

@Xuanwo very cool! looking forward to it.

[kevinjqliu]

Looks like @sungwy already started by exposing Transforms in #556

I'll take a stab at exposing the Catalogs, see #534 (comment)