Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PERF: read_csv should check if column is already a datetime column before initiating the conversion #52546

Closed
2 of 3 tasks
phofl opened this issue Apr 8, 2023 · 0 comments · Fixed by #52548
Closed
2 of 3 tasks
Labels
Arrow pyarrow functionality IO CSV read_csv, to_csv Performance Memory or execution speed performance
Milestone

Comments

@phofl
Copy link
Member

phofl commented Apr 8, 2023

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this issue exists on the latest version of pandas.

  • I have confirmed this issue exists on the main branch of pandas.

Reproducible Example

dr = pd.Series(pd.date_range("2019-12-31", periods=1_000_000, freq="s").astype(pd.ArrowDtype(pa.timestamp(unit="ns"))), name="a")
dr.to_csv("tmp.csv")
pd.read_csv("tmp.csv", engine="pyarrow", dtype_backend="pyarrow", parse_dates=["a"])

The read call takes 1.6 seconds, without parse dates it's down to 0.01 and pyarrow already enforces timestamp

            int64[pyarrow]
a    timestamp[s][pyarrow]
dtype: object

This was introduced by the dtype backend I guess, so would like to fix soonish

Installed Versions

main

Prior Performance

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Arrow pyarrow functionality IO CSV read_csv, to_csv Performance Memory or execution speed performance
Projects
None yet
1 participant