BUG: interchange protocol with nullable pyarrow datatypes a non-null validity provides nonsense results

[MarcoGorelli]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pyarrow.interchange as pai
import pandas as pd
import polars as pl

data = pd.DataFrame(
    {
        "b1": pd.Series([1, 2, None], dtype='Int64[pyarrow]'),
    }
)
print(pd.api.interchange.from_dataframe(data.__dataframe__()))
print(pai.from_dataframe(data))
print(pl.from_dataframe(data))

Issue Description

Result:

                    b1
0  4607182418800017408
1  4611686018427387904
2  9221120237041090560
pyarrow.Table
b1: int64
----
b1: [[4607182418800017408,4611686018427387904,9221120237041090560]]
shape: (3, 1)
┌─────────────────────┐
│ b1                  │
│ ---                 │
│ i64                 │
╞═════════════════════╡
│ 4607182418800017408 │
│ 4611686018427387904 │
│ 9221120237041090560 │
└─────────────────────┘

Expected Behavior

[1, 2, None]

Installed Versions

INSTALLED VERSIONS

commit : 92a52e2
python : 3.10.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.133.1-microsoft-standard-WSL2
Version : #1 SMP Thu Oct 5 21:02:42 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 3.0.0.dev0+355.g92a52e2315
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : 59.6.0
pip : 24.0
Cython : 3.0.8
pytest : 8.0.0
hypothesis : 6.98.6
sphinx : 7.2.6
blosc : None
feather : None
xlsxwriter : 3.1.9
lxml.etree : 5.1.0
html5lib : 1.1
pymysql : 1.4.6
psycopg2 : 2.9.9
jinja2 : 3.1.3
IPython : 8.21.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : 1.3.7
fastparquet : 2024.2.0
fsspec : 2024.2.0
gcsfs : 2024.2.0
matplotlib : 3.8.3
numba : 0.59.0
numexpr : 2.9.0
odfpy : None
openpyxl : 3.1.2
pyarrow : 15.0.0
pyreadstat : 1.2.6
python-calamine : None
pyxlsb : 1.0.10
s3fs : 2024.2.0
scipy : 1.12.0
sqlalchemy : 2.0.27
tables : 3.9.2
tabulate : 0.9.0
xarray : 2024.1.1
xlrd : 2.0.1
zstandard : 0.22.0
tzdata : 2024.1
qtpy : 2.4.1
pyqt5 : None

[MarcoGorelli]

same as #57643, but for pyarrow nullable

[WillAyd]

Oof...I think this is a lifecycle issue similar to data-apis/dataframe-api#351

Looking at the pandas implementation, we are doing:

        elif self.dtype[0] in (
            DtypeKind.INT,
            DtypeKind.UINT,
            DtypeKind.FLOAT,
            DtypeKind.BOOL,
        ):
            np_arr = self._col.to_numpy()
            buffer = PandasBuffer(np_arr, allow_copy=self._allow_copy)
            dtype = self.dtype

For non-null arrow arrays this "works" by pure coincidence. The self._col.to_numpy() call is zero-copy from Arrow -> NumPy, and creates a view that does not own the data. When that NumPy array goes out of scope the memory address it stored is still valid as long as the original PyArrow array is:

By contrast, when there are null values present it is no longer a zero-copy exchange, so when buffer and np_arr get garbage collected the heap storage np_arr pointed to originally is free'd. Someone somewhere else still has a buffer that points to this location, but it is garbage.

I think this is a rather large deficiency in the protocol

[WillAyd]

As you have already mentioned self._col.to_numpy() is also not really viable if it performs any kind of conversions, which in this case it does. There are any number of workarounds to that problem although I think we are best served in the long term if we stop trying to fit everything through a numpy lens and create the buffer managing objects we need at a lower level