Rust: Split adbc_core into several crates

[eitsupi]

What feature or improvement would you like to see?

As pointed out by @lidavidm¹, splitting the adbc_core crate seems worthwhile for a better separation of concerns.

Considering the use cases for our current Rust drivers, I believe splitting the crate into the following three components would clarify dependencies and effectively isolate unsafe code.

1. adbc_core: This crate would represent the C API specification (The current rust/core/src/ffi/constants.rs) and the ADBC behavior purely in Rust. For crates written purely in Rust, like adbc_datafusion, this would be sufficient, excluding the export_driver macro.
2. adbc_ffi: This crate would depend on adbc_core and implement the unsafe functions. The export_driver macro definition is also included.
3. adbc_driver_manager: This crate would depend on adbc_ffi and provide the functionality currently found in rust/core/src/driver_manager.rs. Rust-written drivers like adbc_datafusion would not need to depend on this crate.

I'd appreciate your feedback on this proposal.

Footnotes

1. https://github.com/apache/arrow-adbc/issues/2739#issuecomment-2976726395 ↩

[lidavidm]

CC @mbrobbel @felipecrv @zeroshade as potentially interested

[felipecrv]

Yes! I'm in favor of more modularization.

For crates written purely in Rust, like adbc_datafusion

You mean "drivers written in Rust", right?

Does adbc_core and adbc_ffi really need to be separate?

adbc_driver_manager could be simplified to be called adbc. Our "marketing team" could then say: adbc is the crate you use when you want to use adbc drivers from Rust. This is similar to odbc32.dll and libunixODBC being the libraries you use when you want to use ODBC drivers.

[lidavidm]

IIRC, some Rust users are very concerned about the presence of unsafe code, so splitting out everything to do with FFI might be 'safer'

[felipecrv]

IIRC, some Rust users are very concerned about the presence of unsafe code, so splitting out everything to do with FFI might be 'safer'

But there is no way to use ADBC without going through the C FFI. It's the most justified use of unsafe there is.

[lidavidm]

Well, there are drivers like DataFusion that are in pure Rust. So you don't need FFI for those. Hence splitting out unsafe from the "core" crate is worthwhile, I think. Also, while currently these are managed with features, I think that complicates (transitive) dependency management, vs just having separate crates.

[lidavidm]

In the future there may be more pure-Rust drivers. So I'd like the (presumably small but rather opinionated) subset of Rust users who are wary of FFI/unsafe to feel...safe...using ADBC traits knowing they aren't indirectly taking a dependency on unsafe code (even if they aren't actually using it).

[eitsupi]

Thank you all for the discussion!

It's true that even drivers written in Rust like adbc_datafusion currently use export_driver and therefore rely on FFI.
However, I wonder separating the crates ensures that if someone in the community wishes to use a driver in a pure-Rust context without the FFI in the future, adbc_core without FFI is available for them to build upon.

[felipecrv]

I think supporting ADBC-driver-as-rust-crate instead of only dynamically loadable shared objects confuses the messaging about ADBC (a decoupled standard interface), but I can see how open-source projects tend towards maximum flexibility.

[lidavidm]

It's the same situation in Go and C++: you can directly use the drivers without the driver manager or FFI if the drivers are "native". In Rust, you would still use the traits, you just wouldn't bounce through FFI, so the interface would still be the same.

[felipecrv]

My observation applies to any language. I'm not expecting a change in direction, just lamenting and pointing out that this makes it harder to explain the value proposition of ADBC.

[lidavidm]

Fair enough. In practice, I expect people will use the driver manager, not least because existing drivers are already written in a hodgepodge of languages. But the intent has always been to at least enable direct usage; that's part of why the driver manager tries its best to present itself as "just another driver" (at least in C++) in the first place.