BUG: AttributeError: 'NoneType' object has no attribute 'total_seconds' using ZoneInfo

[taosharma]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from datetime import datetime as dt
from zoneinfo import ZoneInfo

tz = ZoneInfo("Europe/London")
my_date = dt(2022, 10, 10, tzinfo=tz)
data = {"foo": {my_date: 50}}

pd.DataFrame(data)  # AttributeError: 'NoneType' object has no attribute 'total_seconds'

Issue Description

The above error is thrown when creating a DataFrame with a datetime object that uses a ZoneInfo timezone for its tzinfo. I am in the process of migrating our project from pytz to zoneinfo. The code works perfectly if you replace the ZoneInfo timezone with a pytz timezone.

Thanks for reading and looking into this!

Expected Behavior

For pd.DataFrame(data) to create a dataframe something like:

                           foo
2022-10-10 01:01:00+01:00   50

Stack Trace

In [11]: pd.DataFrame(data)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [11], in <cell line: 1>()
----> 1 pd.DataFrame(data)

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/core/frame.py:636, in DataFrame.__init__(self, data, index, columns, dtype, copy)
    630     mgr = self._init_mgr(
    631         data, axes={"index": index, "columns": columns}, dtype=dtype, copy=copy
    632     )
    634 elif isinstance(data, dict):
    635     # GH#38939 de facto copy defaults to False only in non-dict cases
--> 636     mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
    637 elif isinstance(data, ma.MaskedArray):
    638     import numpy.ma.mrecords as mrecords

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/core/internals/construction.py:502, in dict_to_mgr(data, index, columns, dtype, typ, copy)
    494     arrays = [
    495         x
    496         if not hasattr(x, "dtype") or not isinstance(x.dtype, ExtensionDtype)
    497         else x.copy()
    498         for x in arrays
    499     ]
    500     # TODO: can we get rid of the dt64tz special case above?
--> 502 return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/core/internals/construction.py:125, in arrays_to_mgr(arrays, columns, index, dtype, verify_integrity, typ, consolidate)
    122         index = ensure_index(index)
    124     # don't force copy because getting jammed in an ndarray anyway
--> 125     arrays = _homogenize(arrays, index, dtype)
    126     # _homogenize ensures
    127     #  - all(len(x) == len(index) for x in arrays)
    128     #  - all(x.ndim == 1 for x in arrays)
   (...)
    131 
    132 else:
    133     index = ensure_index(index)

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/core/internals/construction.py:615, in _homogenize(data, index, dtype)
    611 if isinstance(val, dict):
    612     # GH#41785 this _should_ be equivalent to (but faster than)
    613     #  val = create_series_with_explicit_dtype(val, index=index)._values
    614     if oindex is None:
--> 615         oindex = index.astype("O")
    617     if isinstance(index, (DatetimeIndex, TimedeltaIndex)):
    618         # see test_constructor_dict_datetime64_index
    619         val = dict_compat(val)

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/core/indexes/base.py:1049, in Index.astype(self, dtype, copy)
   1047 if isinstance(values, ExtensionArray):
   1048     with rewrite_exception(type(values).__name__, type(self).__name__):
-> 1049         new_values = values.astype(dtype, copy=copy)
   1051 elif isinstance(dtype, ExtensionDtype):
   1052     cls = dtype.construct_array_type()

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/core/arrays/datetimes.py:666, in DatetimeArray.astype(self, dtype, copy)
    664 elif is_period_dtype(dtype):
    665     return self.to_period(freq=dtype.freq)
--> 666 return dtl.DatetimeLikeArrayMixin.astype(self, dtype, copy)

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/core/arrays/datetimelike.py:415, in DatetimeLikeArrayMixin.astype(self, dtype, copy)
    411 if self.dtype.kind == "M":
    412     # *much* faster than self._box_values
    413     #  for e.g. test_get_loc_tuple_monotonic_above_size_cutoff
    414     i8data = self.asi8.ravel()
--> 415     converted = ints_to_pydatetime(
    416         i8data,
    417         # error: "DatetimeLikeArrayMixin" has no attribute "tz"
    418         tz=self.tz,  # type: ignore[attr-defined]
    419         freq=self.freq,
    420         box="timestamp",
    421     )
    422     return converted.reshape(self.shape)
    424 return self._box_values(self.asi8.ravel()).reshape(self.shape)

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/_libs/tslibs/vectorized.pyx:158, in pandas._libs.tslibs.vectorized.ints_to_pydatetime()

File ~/.cache/pypoetry/virtualenvs/vidacycle_webapps-foslGWoH-py3.9/lib/python3.9/site-packages/pandas/_libs/tslibs/timezones.pyx:266, in pandas._libs.tslibs.timezones.get_dst_info()

AttributeError: 'NoneType' object has no attribute 'total_seconds'

Installed Versions

INSTALLED VERSIONS

commit : e8093ba
python : 3.9.1.final.0
python-bits : 64
OS : Linux
OS-release : 4.19.128-microsoft-standard
Version : #1 SMP Tue Jun 23 12:58:10 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.3
numpy : 1.23.2
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 58.5.3
pip : 21.3.1
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.3
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : 2.1.1
matplotlib : 3.5.3
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.0
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
zstandard : None

[simonjayhawkins]

can you try on main (or the release candidate) xref #46425

[mroeschke]

Yeah I can validate this works on the 1.5.0rc and main as Zoneinfo support was added in 1.5.0 so closing.

>>> import pandas as pd
>>> pd.__version__
'1.5.0rc0'
>>> from datetime import datetime as dt
>>> from zoneinfo import ZoneInfo
>>> tz = ZoneInfo("Europe/London")
>>> my_date = dt(2022, 10, 10, tzinfo=tz)
>>> data = {"foo": {my_date: 50}}
>>>
>>> pd.DataFrame(data)
                           foo
2022-10-10 00:00:00+01:00   50

[taosharma]

Thanks so much for your help! We'll eagerly await the next pandas release :)