BUG: Inconsistent behavior between None and pd.NA

[jeff-hernandez]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> import pandas as pd
>>> x = pd.Series(['90210', '60018-0123', '10010', 'text', '6.7', "<class 'object'>", '123456'], dtype="string")
>>> invalid = pd.Series([False, False, False, True, True, True, True])
>>> x[invalid] = None
>>> x[6] = pd.NA
>>> x[2] = None
>>> x
0         90210
1    60018-0123
2          <NA>
3          None
4          None
5          None
6          <NA>
dtype: string

Issue Description

There is inconsistent behavior when assigning missing values to a string series. As shown in the reproducible example, using an integer to assign None will result in a conversion to pd.NA, whereas using a boolean series to assign None will result in None with no conversion.

Expected Behavior

I'd expect there to be a single representation for missing values, so assigning x[invalid] = None would result in a conversion to pd.NA.

>>> import pandas as pd
>>> x = pd.Series(['90210', '60018-0123', '10010', 'text', '6.7', "<class 'object'>", '123456'], dtype="string")
>>> invalid = pd.Series([False, False, False, True, True, True, True])
>>> x[invalid] = None
>>> x
0         90210
1    60018-0123
2         10010
3          <NA>
4          <NA>
5          <NA>
6          <NA>
dtype: string

Installed Versions

INSTALLED VERSIONS

commit : e8093ba
python : 3.9.7.final.0
python-bits : 64
OS : Darwin
OS-release : 21.3.0
Version : Darwin Kernel Version 21.3.0: Wed Jan 5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.3
numpy : 1.22.1
pytz : 2021.3
dateutil : 2.8.2
setuptools : 58.0.4
pip : 22.1
Cython : None
pytest : 7.0.1
hypothesis : None
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 7.31.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2022.01.0
gcsfs : None
markupsafe : 2.0.1
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 6.0.1
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.7.3
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[JMBurley]

I'd expect there to be a single representation for missing values, so assigning x[invalid] = None would result in a conversion to pd.NA.

Agree that there is a problem, but I disagree with the diagnosis & expected behaviour. I think pandas should follow the principle of "least surprise"

x[invalid] = None  # inserts None.  Good behaviour
x[6] = pd.NA  # inserts NA.  Good behaviour
x[2] = None  # inserts NA.  Bad behaviour

[jeff-hernandez]

I understand how the conversion can come as a surprise, but isn't one of the reasons for introducing pd.NA to have a single representation for missing values? Other dtypes like boolean and Float64 already insert pd.NA when x[invalid] = None.

[phofl]

I think the conversion was an intentional decision, we are doing the same with np.nan, but not completely sure. You would have to check the issue tracker for information about this

Edit: https://pandas.pydata.org/docs/user_guide/integer_na.html#scalar-na-value

[jorisvandenbossche]

but isn't one of the reasons for introducing pd.NA to have a single representation for missing values?

Indeed, it is the design goal that the StringArray only uses NA as missing value. And for example upon construction this also happens (conversion from None to NA), like with the scalar setitem:

In [14]: pd.Series(["a", None], dtype="string")
Out[14]: 
0       a
1    <NA>
dtype: string

This is the documented behaviour (see https://pandas.pydata.org/docs/reference/api/pandas.arrays.StringArray.html in addition to the link @phofl provided), and intentional:

pandas/pandas/core/arrays/string_.py

Lines 309 to 323 in 2eca7e1

	def _validate(self):
	"""Validate that we only store NA or strings."""
	if len(self._ndarray) and not lib.is_string_array(self._ndarray, skipna=True):
	raise ValueError("StringArray requires a sequence of strings or pandas.NA")
	if self._ndarray.dtype != "object":
	raise ValueError(
	"StringArray requires a sequence of strings or pandas.NA. Got "
	f"'{self._ndarray.dtype}' dtype instead."
	)
	# Check to see if need to convert Na values to pd.NA
	if self._ndarray.ndim > 2:
	# Ravel if ndims > 2 b/c no cythonized version available
	lib.convert_nans_to_NA(self._ndarray.ravel("K"))
	else:
	lib.convert_nans_to_NA(self._ndarray)

But so it seems that for the case of __setitem__ with a mask, we don't do this validation on the input, and that is certainly a bug.

Thanks for the report!

[jorisvandenbossche]

It seems this is a regression compared to pandas 1.3 (but already buggy in 1.4 as well).

The StringArray itself seems to do this correctly:

In [17]: arr = pd.array(['a', 'b', 'c'], dtype="string")

In [19]: arr[arr == 'a'] = None

In [20]: arr
Out[20]: 
<StringArray>
[<NA>, 'b', 'c']
Length: 3, dtype: string

But the strange thing is that doing this on the Series level doesn't end up calling StringArray.__setitem__, it seems to go through Series._where and eventually BlockManager.putmask, and ExtensionArray._putmask, and that last one is not correctly implemented for StringArray.

[jorisvandenbossche]

I opened a PR with a potential fix -> #47763

[simonjayhawkins]

It seems this is a regression compared to pandas 1.3 (but already buggy in 1.4 as well).

first bad commit: [164aa28] ENH: implement EA._putmask (#44387)

But the strange thing is that doing this on the Series level doesn't end up calling StringArray.__setitem__, it seems to go through Series._where and eventually BlockManager.putmask, and ExtensionArray._putmask, and that last one is not correctly implemented for StringArray.

Is this correct to bypass a EA __setitem__ method or could this also affect 3rd party EAs