Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python] Support non-nanosecond units in method to_pandas_dtype for timestamp and duration types #34462

Closed
lukemanley opened this issue Mar 6, 2023 · 3 comments
Closed

[Python] Support non-nanosecond units in method to_pandas_dtype for timestamp and duration types #34462

lukemanley opened this issue Mar 6, 2023 · 3 comments
Labels
Component: Python Type: enhancement

Comments

@lukemanley
Copy link

Describe the enhancement requested

Pyarrow timestamp(unit).to_pandas_dtype() and duration(unit).to_pandas_dtype are currently returning dtype('<M8[ns]') with nanosecond units regardless of the pyarrow units. Pandas has recently added support for non-nanosecond units so I believe pyarrow can now preserve the units here.

>>> import pyarrow as pa

>>> pa.timestamp("us").to_pandas_dtype()
dtype('<M8[ns]')

>>> pa.duration("ms").to_pandas_dtype()
dtype('<m8[ns]')

Component(s)

Python
@lukemanley lukemanley mentioned this issue Mar 6, 2023
Merged
4 tasks
@AlenkaF AlenkaF changed the title Support non-nanosecond units in method to_pandas_dtype for timestamp and duration types [Python] Support non-nanosecond units in method to_pandas_dtype for timestamp and duration types Mar 6, 2023
@jorisvandenbossche
Copy link
Member

Related: #33321 (for the actual data conversions)

This was referenced Apr 20, 2023
Closed
Closed
@AlenkaF
Copy link
Member

AlenkaF commented Apr 20, 2023

This issue also came up after pandas implemented __from_pyarrow__ support to DatetimeTZDtype. See:

@jorisvandenbossche jorisvandenbossche added this to the 13.0.0 milestone Apr 20, 2023
@AlenkaF AlenkaF removed this from the 13.0.0 milestone Jun 20, 2023
@AlenkaF
Copy link
Member

AlenkaF commented Jun 20, 2023

Duplicate of #33321

@AlenkaF AlenkaF marked this as a duplicate of #33321 Jun 20, 2023
@AlenkaF AlenkaF closed this as not planned Won't fix, can't repro, duplicate, stale Jun 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Python Type: enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jorisvandenbossche @lukemanley @AlenkaF