Support for Arrow PyCapsule interface

[kylebarron]

Description

Looking at the code below, the write_deltalake function is hard-coded to support only specific pyarrow and pandas objects.

delta-rs/python/deltalake/writer.py

Lines 283 to 306 in 6f81b80

	if isinstance(data, RecordBatchReader):
	data = convert_pyarrow_recordbatchreader(data, large_dtypes)
	elif isinstance(data, pa.RecordBatch):
	data = convert_pyarrow_recordbatch(data, large_dtypes)
	elif isinstance(data, pa.Table):
	data = convert_pyarrow_table(data, large_dtypes)
	elif isinstance(data, ds.Dataset):
	data = convert_pyarrow_dataset(data, large_dtypes)
	elif _has_pandas and isinstance(data, pd.DataFrame):
	if schema is not None:
	data = convert_pyarrow_table(
	pa.Table.from_pandas(data, schema=schema), large_dtypes=large_dtypes
	)
	else:
	data = convert_pyarrow_table(
	pa.Table.from_pandas(data), large_dtypes=large_dtypes
	)
	elif isinstance(data, Iterable):
	if schema is None:
	raise ValueError("You must provide schema if data is Iterable")
	else:
	raise TypeError(
	f"{type(data).__name__} is not a valid input. Only PyArrow RecordBatchReader, RecordBatch, Iterable[RecordBatch], Table, Dataset or Pandas DataFrame are valid inputs for source."
	)

With the Arrow PyCapsule interface, you can support any Arrow-based Python input, regardless of the library implementation.

Use Case

I'm working on a Python library for Arrow-backed geospatial operations, and support for the PyCapsule interface would allow interfacing with my or any other Python Arrow library without needing hard-coded support.

Related Issue(s)

[ion-elgreco]

@kylebarron hey Kyle, are you willing to make this contribution here?

[kylebarron]

I put up a short initial PR for the write side to get things moving. I think we may be able to wrap some additional classes to export through the pycapsule interface as well (deferring to to_pyarrow under the hood)