Feature Request: Don't require PyArrow for `Connection.adbc_get_info()`

[henryharbeck]

What feature or improvement would you like to see?

Feature Request: Don't require PyArrow for Connection.adbc_get_info()

This allows Polars to introspect the connection passed without requiring PyArrow.

Related to pola-rs/polars#22745

[henryharbeck]

Hi @lidavidm, I had a crack at implementing this, but I think I am blocked by the lack of union types in Polars. It would be good if you could confirm. Here is a simple reproducer.

import adbc_driver_sqlite.dbapi

import polars  # noqa: F401

# Ensure no PyArrow
try:
    import pyarrow
except ImportError:
    pass
else:
    raise RuntimeError("Uninstall PyArrow")

conn = adbc_driver_sqlite.dbapi.connect()

# print(conn._backend)  # <adbc_driver_manager._dbapi_backend._PolarsBackend object...>

handle = conn._conn.get_info()

# print(type(handle))  # <class 'adbc_driver_manager._lib.ArrowArrayStreamHandle'>

conn._backend.import_array_stream(handle)  # Panic

# Try direct constructors as well
# polars.from_arrow(handle)  # Panic
# polars.DataFrame(handle)  # Panic (also supports PyCapsule interface)

All panics are

thread '<unnamed>' panicked at crates/polars-core/src/datatypes/field.rs:256:19:
Arrow datatype Union(UnionType { fields: [Field { name: "string_value", dtype: Utf8, is_nullable: true, metadata: None }, Field { name: "bool_value", dtype: Boolean, is_nullable: true, metadata: None }, Field { name: "int64_value", dtype: Int64, is_nullable: true, metadata: None }, Field { name: "int32_bitmask", dtype: Int32, is_nullable: true, metadata: None }, Field { name: "string_list", dtype: List(Field { name: "item", dtype: Utf8, is_nullable: true, metadata: None }), is_nullable: true, metadata: None }, Field { name: "int32_to_int32_list_map", dtype: Map(Field { name: "entries", dtype: Struct([Field { name: "key", dtype: Int32, is_nullable: false, metadata: None }, Field { name: "value", dtype: List(Field { name: "item", dtype: Int32, is_nullable: true, metadata: None }), is_nullable: true, metadata: None }]), is_nullable: false, metadata: None }, false), is_nullable: true, metadata: None }], ids: Some([0, 1, 2, 3, 4, 5]), mode: Dense }) not supported by Polars. You probably need to activate that data-type feature.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/home/henry/development/temp/repro.py", line 22, in <module>
    conn._backend.import_array_stream(handle)  # Panic
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/henry/development/temp/.venv/lib/python3.11/site-packages/adbc_driver_manager/_dbapi_backend.py", line 147, in import_array_stream
    return polars.from_arrow(handle)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/henry/development/temp/.venv/lib/python3.11/site-packages/polars/convert/general.py", line 536, in from_arrow
    return pycapsule_to_frame(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/henry/development/temp/.venv/lib/python3.11/site-packages/polars/_utils/pycapsule.py", line 41, in pycapsule_to_frame
    s = wrap_s(PySeries.from_arrow_c_stream(obj))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: Arrow datatype Union(UnionType { fields: [Field { name: "string_value", dtype: Utf8, is_nullable: true, metadata: None }, Field { name: "bool_value", dtype: Boolean, is_nullable: true, metadata: None }, Field { name: "int64_value", dtype: Int64, is_nullable: true, metadata: None }, Field { name: "int32_bitmask", dtype: Int32, is_nullable: true, metadata: None }, Field { name: "string_list", dtype: List(Field { name: "item", dtype: Utf8, is_nullable: true, metadata: None }), is_nullable: true, metadata: None }, Field { name: "int32_to_int32_list_map", dtype: Map(Field { name: "entries", dtype: Struct([Field { name: "key", dtype: Int32, is_nullable: false, metadata: None }, Field { name: "value", dtype: List(Field { name: "item", dtype: Int32, is_nullable: true, metadata: None }), is_nullable: true, metadata: None }]), is_nullable: false, metadata: None }, false), is_nullable: true, metadata: None }], ids: Some([0, 1, 2, 3, 4, 5]), mode: Dense }) not supported by Polars. You probably need to activate that data-type feature.

Looking with the PyArrow backend gives a bit more context

# conn as above, but this time with PyArrow backend
handle = conn._conn.get_info()

reader = conn._backend.import_array_stream(handle)
tbl = reader.read_all()
print(tbl.schema)
# info_name: uint32 not null
# info_value: dense_union<string_value: string=0, bool_value: bool=1, int64_value: int64=2, int32_bitmask: int32=3 (... 94 chars omitted)
#   child 0, string_value: string
#   child 1, bool_value: bool
#   child 2, int64_value: int64
#   child 3, int32_bitmask: int32
#   child 4, string_list: list<item: string>
#       child 0, item: string
#   child 5, int32_to_int32_list_map: map<int32, list<item: int32>>
#       child 0, entries: struct<key: int32 not null, value: list<item: int32>> not null
#           child 0, key: int32 not null
#           child 1, value: list<item: int32>
#               child 0, item: int32

print(tbl.to_pylist())
# [
#     {'info_name': 0, 'info_value': 'SQLite'},
#     {'info_name': 1, 'info_value': '3.45.3'},
#     {'info_name': 100, 'info_value': 'ADBC SQLite Driver'},
#     {'info_name': 101, 'info_value': '(unknown)'},
#     {'info_name': 102, 'info_value': '0.6.0'}
# ]

and for reference

_KNOWN_INFO_VALUES = {
    0: "vendor_name",
    1: "vendor_version",
    2: "vendor_arrow_version",
    100: "driver_name",
    101: "driver_version",
    102: "driver_arrow_version",
    103: "driver_adbc_version",
}

Is there a way around the union type? Perhaps exposing the "info_value" field to Python as a string instead? I do note that "driver_adbc_version" has an (albeit inconsistent with other version info values) int value (per the postgres test), which would need to be cast afterwards.

Apologies if these questions are a bit naïve given I am only looking at Python here.

I am keen to do this (or have it picked up by yourself or another dev), as it is the final upstream piece of the related Polars issue mentioned in the description. FWIW, union types may come in Polars, but not yet (pola-rs/polars#9112 (comment))

Keen to hear your thoughts. Thanks

[lidavidm]

The type is dictated by the API standard. The best way would be for Polars to add native support. (I'm surprised Polars panics in this case...)

[henryharbeck]

Thanks for the quick response (and the work already done to remove the PyArrow requirement for a lot of the API)

The type is dictated by the API standard

Makes sense. Apologies

The best way would be for Polars to add native support

Yes, I agree.

So I take it there isn't any workaround on the ADBC side? And this is blocked for Polars (but perhaps not for any other backends implementing the union type)?

Thanks again and apologies for the naivety

[lidavidm]

Um, I suppose you could use ctypes to manipulate the schema and turn the union into a struct (and hope polars doesn't notice the discrepancy), and then read the buffers yourself to figure out the type discriminator when looking at the struct. (I wouldn't really recommend that.)

[henryharbeck]

I think I will indeed avoid that.

Is there any other way (in Python) to get the vendor_name, driver_name or parent driver after instantiating an adbc_driver_manager.AdbcDatabase, adbc_driver_manager.AdbcConnection or adbc_driver_manager.dbapi.Connection?

For example if doing

db = adbc_driver_snowflake.connect(uri)

can I introspect db and see with which driver it was instantiated?

[lidavidm]

The information isn't exposed any other way, unfortunately

[henryharbeck]

Would you consider adding a adbc_driver_manager.DatabaseOptions option for it?
Or exposing it some other way?
If so, I would be happy to create a new feature request

The goal being the same as the description of this issue

[lidavidm]

In the C API, this is the only place it's exposed - so someone at some point has to be able to interpret the Union type.