BUG: DataFrame.replace() is inconsistent between float64 and Float64

[jeckst]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
pd.DataFrame({"a": 0.0, "b": 0.0}, index=[0]).astype(
    {"a": "Float64", "b": "float64"}
).replace(False, 1)

Issue Description

Depending on the column dtype, zero values are picked up when replacing False values in the DataFrame, so the column a ends up with 1 and column b ends up with 0.

For the record, other forms of .replace() seem to have the same issue:

pd.DataFrame({"a": 0.0, "b": 0.0}, index=[0]).astype(
    {"a": "Float64", "b": "float64"}
).replace({"a": {False: 1}})

pd.DataFrame({"a": 0.0, "b": 0.0}, index=[0]).astype(
    {"a": "Float64", "b": "float64"}
).replace({"a": {False: 1}, "b": {False: 1}})

There is also a possibly unrelated issue of infering the resulting dtype. The following will work, converting the column a to object dtype:

pd.DataFrame({"a": [0, 1, 2]}, dtype="float64").replace({1: "A"})

while this next one will end up with a TypeError: Invalid value 'A' for dtype Float64

pd.DataFrame({"a": [0, 1, 2]}, dtype="Float64").replace({1: "A"})

I am not sure if this behavior is an intended property of Float64 or not and whether it is a separate issue or not.

Expected Behavior

Results for both dtypes (in both columns) should be the same. I would expect it to not treat 0 as False, but either way both replaced values should be the same.

Installed Versions

INSTALLED VERSIONS ------------------ commit : e86ed37 python : 3.11.5.final.0 python-bits : 64 OS : Linux OS-release : 6.2.0-33-generic Version : #33~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep 7 10:33:52 UTC 2 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 2.1.1
numpy : 1.25.2
pytz : 2023.3
dateutil : 2.8.2
setuptools : 65.5.0
pip : 23.2.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.1.2
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.16.0
pandas_datareader : None
bs4 : 4.12.2
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.7.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[paulreece]

pd.DataFrame({"a": [0, 1, 2]}, dtype="Float64").replace({1: "A"})

In regards to this please see here:

pandas/pandas/core/array_algos/putmask.py

Lines 29 to 59 in a76b3f4

	def putmask_inplace(values: ArrayLike, mask: npt.NDArray[np.bool_], value: Any) -> None:
	"""
	ExtensionArray-compatible implementation of np.putmask. The main
	difference is we do not handle repeating or truncating like numpy.
	Parameters
	----------
	values: np.ndarray or ExtensionArray
	mask : np.ndarray[bool]
	We assume extract_bool_array has already been called.
	value : Any
	"""
	if (
	not isinstance(values, np.ndarray)
	or (values.dtype == object and not lib.is_scalar(value))
	# GH#43424: np.putmask raises TypeError if we cannot cast between types with
	# rule = "safe", a stricter guarantee we may not have here
	or (
	isinstance(value, np.ndarray) and not np.can_cast(value.dtype, values.dtype)
	)
	):
	# GH#19266 using np.putmask gives unexpected results with listlike value
	# along with object dtype
	if is_list_like(value) and len(value) == len(values):
	values[mask] = value[mask]
	else:
	values[mask] = value
	else:
	# GH#37833 np.putmask is more performant than __setitem__
	np.putmask(values, mask, value)

I guess I'm trying to understand where you would be using Float points as Boolean values when the convention is to use Integers. If you had an entire Boolean column or Series of 1.0's and 0.0's could you not just convert them to int and then run replace?

>>> pd.DataFrame({"a": 0.0, "b": 0.0}, index=[0]).astype(
...     {"a": "Float64", "b": "float64"}
... ).astype(
...     {"a": "Int64", "b": "Int64"}
... ).replace(False, 1)
   a  b
0  1  1

[jeckst]

So the actual context is having a larger DataFrame with different dtypes in columns. So there are some boolean columns where I am replacing True/False to "✔️"/"✖️" (chechmark/cross symbols) for nice ouput as well as some other things. And I do it over the whole DataFrame, which was OK since the replaced values are exclusive to their columns. And while that is not the most explicit thing to do, it saves some unnecessary selecting of what replacements are to be done for which columns. Indeed, the workaround/fix on my side is just specifying the columns in .replace().

But the issue is not how to do the replacements, but the fact that it works inconsistently between float64 and Float64. And indeed even between int64 and Int64. If I alter your example slighlty

>>> pd.DataFrame({"a": 0.0, "b": 0.0}, index=[0]).astype(
...     {"a": "Float64", "b": "float64"}
... ).astype(
...     {"a": "Int64", "b": "int64"}
... ).replace(False, 1)
   a  b
0  1  0

it still ends up inconsistent. As I wrote above, I do not expect the .replace() to pick up the 0 value at all, since that is the "old" behavior and I haven't found any explicit (documented) reason for why this should be different with the extension arrays (Float, Int).

Regarding the putmask code you linked, I'm sorry but I don't know which bit am I supposed to look at. I do understand that it behaves differently and I guess the code shows that no dtype changes are attempted, but I don't understand if that is an intended behavior change. Is it documented somewhere?

[paulreece]

pd.DataFrame({"a": 0.0, "b": 0.0}, index=[0]).astype(
... {"a": "Float64", "b": "float64"}
... ).astype(
... {"a": "Int64", "b": "int64"}
... ).replace(False, 1)
a b
0 1 0
it still ends up inconsistent. As I wrote above, I do not expect the .replace() to pick up the 0 value at all, since that is the "old" behavior and I haven't found any explicit (documented) reason for why this should be different with the extension arrays (Float, Int).

This makes sense. I agree that we should have consistent behavior between the two. A maintainer can chime in in regards to which way will be correct..ignoring zeros or reading them as False.

Regarding the putmask code you linked, I'm sorry but I don't know which bit am I supposed to look at. I do understand that it behaves differently and I guess the code shows that no dtype changes are attempted, but I don't understand if that is an intended behavior change. Is it documented somewhere?

This is referenced to show why a TypeError is being called.

[lithomas1]

Not too familiar with the replace code, but I think the issue might be with the detection of whether we can hold False in the Float64 array.

We have

>>> a._mgr.blocks[0]._can_hold_element(False) # for Float64
True
>>> a._mgr.blocks[1]._can_hold_element(False) # for float64
False

This block looks kind of suspect.
https://github.com/pandas-dev/pandas/blame/main/pandas/core/dtypes/cast.py#L1708-L1723

Maybe @jbrockmendel would know more?

[jbrockmendel]

can_hold_element is indeed wrong for general EAs. Long ago there was a discussion of fixing it so that EAs would upcast on setitem-like operations like non-EAs (i.e. the opposite change of PDEP6), but we decided against fixing it in anticipation of doing PDEP 6 instead.

i suspect that a patch in mask_missing may be the right approach.