Skip to content

Commit

Permalink
Backport PR #46636 on branch 1.4.x (REGR: Replace changes the dtype o…
Browse files Browse the repository at this point in the history
…f other columns) (#47123)

Backport PR #46636: REGR: Replace changes the dtype of other columns

Co-authored-by: Simon Hawkins <simonjayhawkins@gmail.com>
  • Loading branch information
meeseeksmachine and simonjayhawkins authored May 26, 2022
1 parent 4bb454f commit 3b3162b
Show file tree
Hide file tree
Showing 3 changed files with 28 additions and 6 deletions.
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.4.3.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ including other versions of pandas.

Fixed regressions
~~~~~~~~~~~~~~~~~
- Fixed regression in :meth:`DataFrame.replace` when the replacement value was explicitly ``None`` when passed in a dictionary to ``to_replace`` also casting other columns to object dtype even when there were no values to replace (:issue:`46634`)
- Fixed regression in :meth:`DataFrame.nsmallest` led to wrong results when ``np.nan`` in the sorting column (:issue:`46589`)
- Fixed regression in :func:`read_fwf` raising ``ValueError`` when ``widths`` was specified with ``usecols`` (:issue:`46580`)
- Fixed regression in :meth:`.Groupby.transform` and :meth:`.Groupby.agg` failing with ``engine="numba"`` when the index was a :class:`MultiIndex` (:issue:`46867`)
Expand Down
14 changes: 8 additions & 6 deletions pandas/core/internals/blocks.py
Original file line number Diff line number Diff line change
Expand Up @@ -875,12 +875,14 @@ def _replace_coerce(
)
else:
if value is None:
# gh-45601, gh-45836
nb = self.astype(np.dtype(object), copy=False)
if nb is self and not inplace:
nb = nb.copy()
putmask_inplace(nb.values, mask, value)
return [nb]
# gh-45601, gh-45836, gh-46634
if mask.any():
nb = self.astype(np.dtype(object), copy=False)
if nb is self and not inplace:
nb = nb.copy()
putmask_inplace(nb.values, mask, value)
return [nb]
return [self] if inplace else [self.copy()]
return self.replace(
to_replace=to_replace, value=value, inplace=inplace, mask=mask
)
Expand Down
19 changes: 19 additions & 0 deletions pandas/tests/frame/methods/test_replace.py
Original file line number Diff line number Diff line change
Expand Up @@ -675,6 +675,25 @@ def test_replace_NAT_with_None(self):
expected = DataFrame([None, None])
tm.assert_frame_equal(result, expected)

def test_replace_with_None_keeps_categorical(self):
# gh-46634
cat_series = Series(["b", "b", "b", "d"], dtype="category")
df = DataFrame(
{
"id": Series([5, 4, 3, 2], dtype="float64"),
"col": cat_series,
}
)
result = df.replace({3: None})

expected = DataFrame(
{
"id": Series([5.0, 4.0, None, 2.0], dtype="object"),
"col": cat_series,
}
)
tm.assert_frame_equal(result, expected)

def test_replace_value_is_none(self, datetime_frame):
orig_value = datetime_frame.iloc[0, 0]
orig2 = datetime_frame.iloc[1, 0]
Expand Down

0 comments on commit 3b3162b

Please sign in to comment.