FFI Execution Plans that spawn threads panic

[timsaucer]

Describe the bug

If you have an FFI Execution Plan that spawns it's own tasks, it has no reactor and causes a panic.

To Reproduce

This is most easily demonstrated in datafusion-python. Just create a table provider that spawns a thread. I have examples on my computer but they are non-trivial (and it's probably me working on this issue).

Expected behavior

We should never cause a panic across the ffi boundary.

Additional context

No response

[alamb]

Seems like there are two problems:

1. The panic is going across FFI boundaries
2. The FFI code doesn't have a tokio rutime setup

Maybe we need to have some way to pass in a handle to the tokio runtime 🤔 Handle across the FFI boundaries

[timsaucer]

There is only one point where we have ffi calls that are async, currently. This happens in the record batch stream. So I think what needs to happen is that within the private data for the RBS we also include an Option<Arc<Runtime>> and if it exists, we enter the runtime during the async calls.

We do pass a context. I worry about a possible problem where we could have deadlocks if we have two different runtimes each doing a blocking call. The only place that is immediately jumping out to me as a potential problem spot is RecordBatchReceiverStreamBuilder, but I haven't had time to thoroughly look through this.

I think the immediate next thing to do is to build up a good reproducible test, ideally that I could include as a unit test.

[timsaucer]

Initial testing of passing in an Option<Arc<Runtime>> to the table provider and feeding it down through the execution plan and record batch stream was successful.

[kevinjqliu]

Hi @timsaucer thanks for creating this issue!
I'm looking to expose table provider in iceberg-rust to python using FFI_TableProvider. I think I ran into the issue you described above.

This is the current WIP PR https://github.com/kevinjqliu/iceberg-rust/pull/2/files
and here's the error log with backtrace after running hatch run dev:test https://gist.github.com/kevinjqliu/0ab4597427d196c906f279242294d051

I've been looking at reasons why I'm see the error message: there is no reactor running, must be called from the context of a Tokio 1.x runtime. And this issue makes the most sense.

Would love to get your thoughts on this and see if its the same issue

[kevinjqliu]

I tried to incorporate #13937 in kevinjqliu/iceberg-rust#3 and ran into a different error https://gist.github.com/kevinjqliu/cba636cfd4a2a0c8acbba060768e33c9

tests/test_table_provider.py::test_iceberg_table_provider Fatal Python error: Segmentation fault

I'm not sure where the error occurs, but i think integrating the iceberg table provider will be a good test for this issue.

This is my first time working with rust/pyo3/datafusion so there might be something obvious I'm not seeing.

[timsaucer]

I can definitely help and I have a PR to fix the spawn issue. I will be back to work on this next week if you can give me a few dats

[kevinjqliu]

Thank you! Feel free to ping me if you need any help setting up the env for iceberg-rust.