BUG: series.where(boolean) works but df.where(boolean) does not

[dycw]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd

x = pd.Series(range(3), dtype=float)
mask = pd.Series([True, False, nan], dtype=pd.BooleanDtype())
print(f"x=\n{x}\n\nmask=\n{mask}\n\nx.where(mask)=\n{x.where(mask)}")

x=
0   0.00000
1   1.00000
2   2.00000
dtype: float64

mask=
0     True
1    False
2     <NA>
dtype: boolean

x.where(mask)=
0   0.00000
1       nan
2       nan
dtype: float64

x_df = x.to_frame()
mask_df = mask.to_frame()
print(f"x_df=\n{x_df}\n\nmask_df=\n{mask_df}")
x_df.where(mask_df)

x_df=
        0
0 0.00000
1 1.00000
2 2.00000

mask_df=
       0
0   True
1  False
2   <NA>

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-9-7e122bb223a4> in <module>
      2 mask_df = mask.to_frame()
      3 print(f"x_df=\n{x_df}\n\nmask_df=\n{mask_df}")
----> 4 x_df.where(mask_df)

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/generic.py in where(self, cond, other, inplace, axis, level, errors, try_cast)
   8990         """
   8991         other = com.apply_if_callable(other, self)
-> 8992         return self._where(
   8993             cond, other, inplace, axis, level, errors=errors, try_cast=try_cast
   8994         )

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/generic.py in _where(self, cond, other, inplace, axis, level, errors, try_cast)
   8847 
   8848         else:
-> 8849             new_data = self._mgr.where(
   8850                 other=other,
   8851                 cond=cond,

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/internals/managers.py in where(self, other, cond, align, errors, try_cast, axis)
    511             other = extract_array(other, extract_numpy=True)
    512 
--> 513         return self.apply(
    514             "where",
    515             align_keys=align_keys,

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, **kwargs)
    394                 applied = b.apply(f, **kwargs)
    395             else:
--> 396                 applied = getattr(b, f)(**kwargs)
    397             result_blocks = _extend_blocks(applied, result_blocks)
    398 

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/internals/blocks.py in where(self, other, cond, errors, try_cast, axis)
   1299         import pandas.core.computation.expressions as expressions
   1300 
-> 1301         cond = _extract_bool_array(cond)
   1302         assert not isinstance(other, (ABCIndexClass, ABCSeries, ABCDataFrame))
   1303 

~/miniconda3/envs/dts/lib/python3.8/site-packages/pandas/core/internals/blocks.py in _extract_bool_array(mask)
   2862 
   2863     assert isinstance(mask, np.ndarray), type(mask)
-> 2864     assert mask.dtype == bool, mask.dtype
   2865     return mask

AssertionError: object

Problem description

If the truthiness of boolean Series is defined, then it certainly should be for DataFrames.

Expected Output

Equivalent to doing this columnwise, i.e.:

expected = pd.concat([x.where(m) for (_, x), (_, m) in zip(x_df.items(), mask_df.items())], axis=1)
print(f"expected=\n{expected}")

expected=
        0
0 0.00000
1     nan
2     nan

Output of pd.show_versions()

INSTALLED VERSIONS

commit : d9fff27
python : 3.8.3.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-42-generic
Version : #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_HK.UTF-8
LOCALE : en_HK.UTF-8

pandas : 1.1.0
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 49.2.0.post20200714
Cython : None
pytest : 6.0.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.5 (dt dec pq3 ext lo64)
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.2.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.0
sqlalchemy : 1.3.18
tables : None
tabulate : 0.8.3
xarray : None
xlrd : None
xlwt : None
numba : None

[TomAugspurger]

Thanks for the report. Duplicate of #35429 I think, if you're interested in looking.