BUG/API: Series[object].fillna ignores downcast=None, "infer"

[jbrockmendel]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

ser = pd.Series([True, None, False, True, None], dtype=object)
>>> ser.fillna(True, downcast=None)
0     True
1     True
2    False
3     True
4     True
dtype: bool

>>> ser.fillna(True, downcast="infer")
0     True
1     True
2    False
3     True
4     True
dtype: object

Issue Description

Note that downcast=None is the default, so this also happens with just ser.fillna(True).

Disabling the special-casing for object dtype in Block._maybe_downcast breaks 9 tests. I need to check, think there may be some EA cases that aren't tested but where behavior would change if removed that special-casing.

Expected Behavior

I'd expect it to go the other way, with object dtype retained with downcast=None and bool inferred with downcast="infer"

Installed Versions

Replace this line with the output of pd.show_versions()