Skip to content

BUG: to_datetime incorrectly parsing UNIX timestamps to *wrong* timezone #45054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
vgoklani opened this issue Dec 24, 2021 · 1 comment
Closed
2 of 3 tasks
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@vgoklani
Copy link

vgoklani commented Dec 24, 2021

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

    d = datetime.datetime.utcnow()
    t = d.timestamp()
    s = d.isoformat()

    d
    datetime.datetime(2021, 12, 24, 17, 30, 51, 507737)

    s
    '2021-12-24T17:30:51.507737'

    t
    1640385051.507737

    pd.to_datetime(t, unit="s", utc=True)
    Timestamp('2021-12-24 22:30:51.507736832+0000', tz='UTC')

This is incorrect! It appears that pd.to_datetime is shifting the timezone internally. There are lots of hacks on stackoverflow for fixing this, but this looks like a glitch.

Issue Description

It appears that pd.to_datetime is incorrectly parsing the timezone when passed in a UNIX timestamp that is inherently in UTC.

Expected Behavior

pd.to_datetime should manage the timezones correctly, and when passed in a UNIX timestamp, it should not try to shift the timezone to a local timezone.

Thanks!

Installed Versions

INSTALLED VERSIONS

commit : 945c9ed
python : 3.9.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-91-generic
Version : #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8

pandas : 1.3.4
numpy : 1.21.2
pytz : 2021.3
dateutil : 2.8.2
pip : 21.3.1
setuptools : 58.0.4
Cython : 0.29.24
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.2
lxml.etree : 4.6.4
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.2
IPython : 7.29.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : 1.3.2
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.5.0
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 3.0.0
pyxlsb : None
s3fs : None
scipy : 1.7.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
numba : None

@vgoklani vgoklani added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Dec 24, 2021
@jbrockmendel
Copy link
Member

In this context t is not actually a unix timestamp. Compare it to time.time() and (in your locale) the two will be about 5*3600 apart.

The cause here is a combination of datetime.timestamp() being weird for naive datetimes and datetime.utcnow now returning a tzaware datetime. xref #22451, https://mail.python.org/archives/list/datetime-sig@python.org/thread/PT4JWJLYBE5R2QASVBPZLHH37ULJQR43/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
Development

No branches or pull requests

2 participants