ENH: Restore the functionality of .fillna

[tomprimozic]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

The currently very useful behaviour of .fillna is being deprecated.

a = pd.Series([True, False, None])
a
# 0     True
# 1    False
# 2     None
# dtype: object

Using a.fillna raises a warning:

a.fillna(True)
# [/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/3077193745.py:1](https://file+.vscode-resource.vscode-cdn.net/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/3077193745.py:1): FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
#   a.fillna(True)

Full message of the warning is:

FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set pd.set_option('future.no_silent_downcasting', True)

The proposed solutions don't work:

a.fillna(True).infer_objects(copy=False)
# [/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/1224563968.py:1](https://file+.vscode-resource.vscode-cdn.net/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/1224563968.py:1): FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
#   a.fillna(True).infer_objects(copy=False)

maybe I misunderstood the Warning message?

a.infer_objects(copy=False).fillna(True)
# [/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/2319989247.py:1](https://file+.vscode-resource.vscode-cdn.net/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/2319989247.py:1): FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
#   a.infer_objects(copy=False).fillna(True)    # maybe I misunderstood the message?

Let's try to opt-in...

with pd.option_context("future.no_silent_downcasting", True):
  r = a.fillna(True)
r
# 0     True
# 1    False
# 2     True
# dtype: object

No, it's no longer a bool Series...

Some online resources suggest first casting to bool...

a.astype(bool)
# 0     True
# 1    False
# 2    False
# dtype: bool

Looks like this is a potential replacement for .fillna(False) but not for .fillna(True)...

Wait, there's a downcast parameter for .fillna!

a.fillna(True, downcast=True)
# [/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/4257582953.py:1](https://file+.vscode-resource.vscode-cdn.net/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/4257582953.py:1): FutureWarning: The 'downcast' keyword in fillna is deprecated and will be removed in a future version. Use res.infer_objects(copy=False) to infer non-object dtype, or pd.to_numeric with the 'downcast' keyword to downcast numeric results.
#   a.fillna(True, downcast=True)
# [/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/4257582953.py:1](https://file+.vscode-resource.vscode-cdn.net/var/folders/66/bjmfs8315pjdbztwxmyvd7x80000gn/T/ipykernel_85835/4257582953.py:1): FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
#   a.fillna(True, downcast=True)

Oh no, it's deprecated as well, not I got TWO warnings...

Feature Description

Restore the functionality of .fillna WITHOUT the Warning.

a = pd.Series([True, False, None])
a.fillna(True)
# 0     True
# 1    False
# 2     True
# dtype: bool

Alternative Solutions

Currently, the only "correct" option is to use nullable Boolean type.

a.astype('boolean').fillna(True).astype(bool)
# or, if we're happy keeping the type `boolean`...
a.astype('boolean').fillna(True)

This is overly verbose, but would be acceptable if boolean was inferred automatically for [True, False, None] (or [True, False, np.nan]), but currently it's not...

(The additional confusion is that integer nullable types are distinguished by uppercase (int64 -> Int64) but boolean nullable type isn't (bool -> boolean)... so it took me a very long time to even find this solution!)

Additional Context

No response

[rhshadrach]

Thanks for report! You mention that infer_objects didn't work, I'm not sure what you meant by this. Is it that you still get the warning? You need to use infer_objects with the future behavior like so:

ser = pd.Series([True, False, None])
with pd.option_context("future.no_silent_downcasting", True):
      print(ser.fillna(True).infer_objects(copy=False))