BUG: Reindex null timedelta series with pd.Timedelta fails

[ChiQiao]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

# Your code here
ts = pd.date_range("2020-01-01", "2020-01-03", freq="1d")
s = pd.Series(pd.Timedelta(None), index=ts)
s.reindex(pd.date_range("2020-01-01", "2020-01-04", freq="1d"), fill_value=pd.Timedelta(0))

Problem description

To begin with, s has the wrong type datetime64[ns] instead of timedelta64[ns]. This causes the reindex to fail.
I've also tested the code above on version 1.2.4, it does not produce error and the type is casted as object:

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

commit : c7f7443
python : 3.7.11.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : None.None

pandas : 1.3.1
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.2
setuptools : 52.0.0.post20210125
Cython : 0.29.24
pytest : 6.1.2
hypothesis : 6.14.1
sphinx : 4.0.2
blosc : None
feather : None
xlsxwriter : 1.4.4
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : 1.0.2
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.22.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 2021.07.0
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : 3.0.0
pyxlsb : None
s3fs : 0.4.2
scipy : 1.6.2
sqlalchemy : 1.4.22
tables : 3.6.1
tabulate : 0.8.9
xarray : 0.19.0
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.53.0

[simonjayhawkins]

Thanks @ChiQiao for the report

To begin with, s has the wrong type datetime64[ns] instead of timedelta64[ns].

pd.Timedelta(None) and pd.Timestamp(None) are both pd.NaT, instances of pandas._libs.tslibs.nattype.NaTType. There is no dtype associated with scalar pd.NaT.

The dtype for a Series constructed from a pd.NaT scalar being datetime64[ns] is expected behavior.

I've also tested the code above on version 1.2.4, it does not produce error and the type is casted as object

This now raises TypeError: value should be a 'Timestamp' or 'NaT'. Got 'Timedelta' instead. which seems reasonable. maybe value -> fill_value would be clearer.

this was changed in [a3bb751] REF: back DatetimeBlock, TimedeltaBlock by DTA/TDA (#40456)

There was no release note for this change and it does not appear that the change was intentional. However, I think the 1.2.5 behavior returning an object array and allowing a Timedelta as a fill value for a datetime Series was a bug.

cc @jbrockmendel

---

adding the correct dtype to the series explicitly gives the expected output

ts = pd.date_range("2020-01-01", "2020-01-03", freq="1d")
s = pd.Series(pd.Timedelta(None), dtype="m8[ns]", index=ts)
s.reindex(
    pd.date_range("2020-01-01", "2020-01-04", freq="1d"), fill_value=pd.Timedelta(0)
)

2020-01-01      NaT
2020-01-02      NaT
2020-01-03      NaT
2020-01-04   0 days
Freq: D, dtype: timedelta64[ns]

[ChiQiao]

Thanks @simonjayhawkins for triaging. If I understand your comment correctly:

1. pd.NaT can result from both pd.Timedelta and pd.Timestamp.
2. pd.Series constructed from pd.NaT has the type datetime64[ns].
3. We cannot fill a Series of the type datetime64[ns] with values of the type timedelta64[ns].

Individually they all make sense to me, but collectively they lead to a weird behavior:
A Series construct with "invalid" Timedelta cannot be filled with Timedelta but Timestamp only.

Furthermore, I found that only reindex causes this problem, and it works for fillna:

# Works, type converted
pd.Series(pd.Timedelta(None), index=[0]).fillna(pd.Timedelta(0))
# Fails
pd.Series(pd.Timedelta(None), index=[0]).reindex([0, 1], fill_value=pd.Timedelta(0))

[jbrockmendel]

Looks like in array_algos.take.take_nd the EA cases don't go through maybe_promote. Will also need to change ExtensionBlock.take_nd which doesn't go through array_algos.take.take_nd at all