Skip to content

BUG: Fix inserting of wrong-dtyped NaT, closes #27297 #27311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Jul 22, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ee426ab
BUG: fix+test setitem datetime64(NaT) into Series[timedelta64]
jbrockmendel Jul 9, 2019
511650c
BUG: fix+test setting datetime64(NaT) in TimedeltaArray
jbrockmendel Jul 9, 2019
3c916c1
TST: inserting timedelta64(NaT) into DatetimeArray, fixed in previous…
jbrockmendel Jul 9, 2019
d39503d
BUG: fix+test assigning timedelta64(NaT) to datetime series
jbrockmendel Jul 9, 2019
b047cce
BUG: fix+test assignment of wrong nat to DataFrame
jbrockmendel Jul 9, 2019
7399ef0
Merge branch 'master' of https://github.com/pandas-dev/pandas into wr…
jbrockmendel Jul 10, 2019
5e0004c
Merge branch 'master' of https://github.com/pandas-dev/pandas into wr…
jbrockmendel Jul 11, 2019
7721404
Merge branch 'master' of https://github.com/pandas-dev/pandas into wr…
jbrockmendel Jul 15, 2019
217ce63
pre-rebase
jbrockmendel Jul 15, 2019
bca4e2a
Merge branch 'master' of https://github.com/pandas-dev/pandas into wr…
jbrockmendel Jul 15, 2019
d32c1bf
typo
jbrockmendel Jul 15, 2019
715002f
Merge branch 'master' of https://github.com/pandas-dev/pandas into wr…
jbrockmendel Jul 17, 2019
c001aca
update test for datetime series
jbrockmendel Jul 17, 2019
49eec2e
fixups
jbrockmendel Jul 17, 2019
96bda53
Merge branch 'master' of https://github.com/pandas-dev/pandas into wr…
jbrockmendel Jul 19, 2019
42183e9
Merge branch 'master' of https://github.com/pandas-dev/pandas into wr…
jbrockmendel Jul 20, 2019
7c260d5
remove apparently unnecessary reshape
jbrockmendel Jul 21, 2019
b035e4e
Merge branch 'master' of https://github.com/pandas-dev/pandas into wr…
jbrockmendel Jul 21, 2019
fbc1836
whatsnew
jbrockmendel Jul 22, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v1.0.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Categorical

Datetimelike
^^^^^^^^^^^^

- Bug in :meth:`Series.__setitem__` incorrectly casting ``np.timedelta64("NaT")`` to ``np.datetime64("NaT")`` when inserting into a :class:`Series` with datetime64 dtype (:issue:`27311`)
-
-

Expand Down
3 changes: 3 additions & 0 deletions pandas/_libs/index.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -533,6 +533,9 @@ cpdef convert_scalar(ndarray arr, object value):
pass
elif isinstance(value, (datetime, np.datetime64, date)):
return Timestamp(value).value
elif util.is_timedelta64_object(value):
# exclude np.timedelta64("NaT") from value != value below
pass
elif value is None or value != value:
return NPY_NAT
elif isinstance(value, str):
Expand Down
3 changes: 2 additions & 1 deletion pandas/core/internals/blocks.py
Original file line number Diff line number Diff line change
Expand Up @@ -2589,8 +2589,9 @@ def setitem(self, indexer, value):
try:
return super().setitem(indexer, value)
except (ValueError, TypeError):
obj_vals = self.values.astype(object)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you really need to do this?

if you actually do, then .reshape(1, -1) should just work (or use np.atleast_2d)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this really is needed.

reshape(1, -1) is what is already here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you give an example that specific needs this 2d reshape?

this just is a completely different pattern than anywhere else. Why for example would the ObjectBlock construct not simply reshape (if needed); it already knows the dim & the values shape?

newb = make_block(
self.values.astype(object), placement=self.mgr_locs, klass=ObjectBlock
obj_vals, placement=self.mgr_locs, klass=ObjectBlock, ndim=self.ndim
)
return newb.setitem(indexer, value)

Expand Down
32 changes: 32 additions & 0 deletions pandas/tests/series/indexing/test_indexing.py
Original file line number Diff line number Diff line change
Expand Up @@ -654,6 +654,38 @@ def test_timedelta_assignment():
tm.assert_series_equal(s, expected)


@pytest.mark.parametrize(
"nat_val,should_cast",
[
(pd.NaT, True),
(np.timedelta64("NaT", "ns"), False),
(np.datetime64("NaT", "ns"), True),
],
)
@pytest.mark.parametrize("tz", [None, "UTC"])
def test_dt64_series_assign_nat(nat_val, should_cast, tz):
# some nat-like values should be cast to datetime64 when inserting
# into a datetime64 series. Others should coerce to object
# and retain their dtypes.
dti = pd.date_range("2016-01-01", periods=3, tz=tz)
base = pd.Series(dti)
expected = pd.Series([pd.NaT] + list(dti[1:]), dtype=dti.dtype)
if not should_cast:
expected = expected.astype(object)

ser = base.copy(deep=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these need to be deep? (no big deal though)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

ser[0] = nat_val
tm.assert_series_equal(ser, expected)

ser = base.copy(deep=True)
ser.loc[0] = nat_val
tm.assert_series_equal(ser, expected)

ser = base.copy(deep=True)
ser.iloc[0] = nat_val
tm.assert_series_equal(ser, expected)


@pytest.mark.parametrize(
"nat_val,should_cast",
[
Expand Down