ENH: DatetimeTZDtype is missing `__from_arrow__`

[tswast]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import pyarrow as pa

data = [0, 123456789, None, 2**63 - 1, -123456789]
pa_type = pa.timestamp('ns', tz='UTC')
arr = pa.array(data, type=pa_type)
table = pa.Table.from_arrays([arr], ['ts_col'])

dtype = pd.DatetimeTZDtype(unit='ns', tz='UTC')
table.to_pandas(types_mapper=lambda _: dtype)

Output:

File /opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pyarrow/pandas_compat.py:778, in _reconstruct_block(item, columns, extension_columns)
    776 pandas_dtype = extension_columns[name]
    777 if not hasattr(pandas_dtype, '__from_arrow__'):
--> 778     raise ValueError("This column does not support to be converted "
    779                      "to a pandas ExtensionArray")
    780 pd_ext_arr = pandas_dtype.__from_arrow__(arr)
    781 block = _int.make_block(pd_ext_arr, placement=placement)

ValueError: This column does not support to be converted to a pandas ExtensionArray

Issue Description

Since DatetimeTZDtype does not support __from_arrow__ this extension dtype can't be used in pyarrow's to_pandas method.

Expected Behavior

The example should return a pandas DataFrame with a DatetimeTZDtype column.

Installed Versions

In [4]: pandas.show_versions()
/opt/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/_distutils_hack/init.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS

commit : 6c50f70
python : 3.8.16.final.0
python-bits : 64
OS : Darwin
OS-release : 22.3.0
Version : Darwin Kernel Version 22.3.0: Mon Jan 30 20:42:11 PST 2023; root:xnu-8792.81.3~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.0.dev0+310.g6c50f70650
numpy : 1.23.5
pytz : 2022.7.1
dateutil : 2.8.2
setuptools : 67.6.0
pip : 23.0.1
Cython : 0.29.33
pytest : 7.2.2
hypothesis : 6.70.0
sphinx : 4.5.0
blosc : None
feather : None
xlsxwriter : 3.0.9
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : 1.0.2
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.11.0
pandas_datareader: None
bs4 : 4.12.0
bottleneck : 1.3.7
brotli :
fastparquet : 2023.2.0
fsspec : 2023.3.0
gcsfs : 2023.3.0
matplotlib : 3.6.3
numba : 0.56.4
numexpr : 2.8.3
odfpy : None
openpyxl : 3.1.0
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : 1.2.1
pyxlsb : 1.0.10
s3fs : 2023.3.0
scipy : 1.10.1
snappy :
sqlalchemy : 2.0.7
tables : 3.7.0
tabulate : 0.9.0
xarray : 2023.1.0
xlrd : 2.0.1
zstandard : 0.19.0
tzdata : 2022.7
qtpy : None
pyqt5 : None

[jbrockmendel]

Shouldn't users use a ArrowDtype instead?

[tswast]

@jbrockmendel Some additional context: I'm working on some improvements to interoperability between BigQuery and pandas. BigQuery represents TIMESTAMP data as microseconds UTC, so corresponds most closely to DatetimeTZDtype(unit='us', tz='UTC').

BigQuery can send query response data back as arrow record batches, so going direct from pyarrow to a corresponding pandas dtype would be ideal.

Indeed we could use the arrow dtype for everything, but I presume the Series.dt accessor and datetime indexes would be more cumbersome to use at the moment with an arrow dtype wrapping an arrow timestamp, right?

[jbrockmendel]

Indeed we could use the arrow dtype for everything, but I presume the Series.dt accessor and datetime indexes would be more cumbersome to use at the moment with an arrow dtype wrapping an arrow timestamp, right?

@mroeschke recently implemented Series.dt with timestamp-ArrowDtype. You wouldn't get a DatetimeIndex, but you should have Index.dt available

[tswast]

Series.dt with timestamp-ArrowDtype

Very cool! I'll have to try that out.

I think it still makes sense to have DatetimeTZDtype.__from_arrow__ available for consistency with the other extension dtypes, but this does unblock me.

[jorisvandenbossche]

For historical context, the reason that DatetimeTZDtype is missing __from_arrow__ is because this is a data type that has always already been handled natively by pyarrow itself in its to_pandas conversion, and __from_arrow__ was introduced for custom dtypes. Now, it can certainly be added for consistency, and to enable the use case you bring up (although pyarrow will change behaviour here to preserve the unit in the near future, so this specific example should no longer need to use types_mapper)