Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOCS: series.astype return error when a datetime64 (unitless) series astype numpy.datetime64 #47844

Closed
3 tasks done
bzhaoopenstack opened this issue Jul 25, 2022 · 7 comments · Fixed by #48555
Closed
3 tasks done
Labels
Blocker Blocking issue or pull request for an upcoming release Docs Dtype Conversions Unexpected or buggy dtype conversions Non-Nano datetime64/timedelta64 with non-nanosecond resolution
Milestone

Comments

@bzhaoopenstack
Copy link

bzhaoopenstack commented Jul 25, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

pandas master branch:

import pandas as pd
import numpy
>>> pd.Series(['1970-01-01', '1970-01-01', '1970-01-01'], dtype="datetime64[ns]").astype(numpy.datetime64, copy=False)
*** TypeError: Cannot cast DatetimeArray to dtype datetime64

Issue Description

I'm not sure whether it is expect behavior. I can repro on main branch, but can not repro on 1.4.x.
I know pandas community introduce a new feature about ENH: DTI/DTA.astype support non-nano . But the new feature disable the original code tree.
In this case, the original code tree will locate on

        elif self.tz is None and is_datetime64_dtype(dtype) and dtype != self.dtype:
            # unit conversion e.g. datetime64[s]
            return self._ndarray.astype(dtype)

But now, it is

        return dtl.DatetimeLikeArrayMixin.astype(self, dtype, copy)

And raise an error.

Expected Behavior

>>> import pandas as pd
>>> import numpy
>>> pd.Series(['1970-01-01', '1970-01-01', '1970-01-01'], dtype="datetime64[ns]").astype(numpy.datetime64, copy=False)
0   1970-01-01
1   1970-01-01
2   1970-01-01
dtype: datetime64[ns]

Installed Versions

pandas-1.5.0.dev0+1135.gc711be009b

@bzhaoopenstack bzhaoopenstack added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 25, 2022
@mroeschke mroeschke added Dtype Conversions Unexpected or buggy dtype conversions Non-Nano datetime64/timedelta64 with non-nanosecond resolution and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 25, 2022
@mroeschke
Copy link
Member

cc @jbrockmendel

@jbrockmendel
Copy link
Member

Looks like passing dtype=np.datetime64 is equivalent to passing dtype = "datetime64", which we don't accept bc it is unit-less, but numpy accepts when we pass it to self._ndarray.astype(dtype). So the new behavior is "more correct"

@mroeschke
Copy link
Member

Is it worth defaulting this case to nanosecond or just documenting this change?

@jbrockmendel
Copy link
Member

i think just document, otherwise we're special-casing in this one place and will be inconsistent elsewhere

@mroeschke mroeschke added Docs and removed Bug labels Jul 26, 2022
@mroeschke mroeschke changed the title BUG: series.astype return error when a datetime64 series astype numpy.datetime64 DOCS: series.astype return error when a datetime64 (unitless) series astype numpy.datetime64 Jul 26, 2022
@jorisvandenbossche jorisvandenbossche added this to the 1.5 milestone Sep 6, 2022
@jorisvandenbossche jorisvandenbossche added the Blocker Blocking issue or pull request for an upcoming release label Sep 6, 2022
@jorisvandenbossche
Copy link
Member

I ran into this issue as well in the dask test suite, this change is giving some failures there (dask/dask#9465).

So this worked before:

In [1]: s = pd.Series(pd.date_range("2012-01-01", periods=3))

In [2]: s.astype(np.dtype("M8"))
Out[2]: 
0   2012-01-01
1   2012-01-02
2   2012-01-03
dtype: datetime64[ns]

and is now indeed raising.

I agree that we should ideally raise in this case, but given this has been working fine, I think we should consider deprecating this first.

@jesusch
Copy link

jesusch commented Apr 15, 2023

Seems this was reintroduced in pandas 2.0.0?

@maxschulz-COL
Copy link

Seem to have the same issue, worked fine in pandas == 1.5.3 and not breaks in 2.0.0

@jesusch jesusch mentioned this issue May 1, 2023
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocker Blocking issue or pull request for an upcoming release Docs Dtype Conversions Unexpected or buggy dtype conversions Non-Nano datetime64/timedelta64 with non-nanosecond resolution
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants