Object dtype gives ValueError in _maybe_cast_with_dtype

[david33n]

import pandas as pd
import numpy as np

frame = pd.DataFrame(index=pd.Index([1, np.nan]))
fixed_index = frame.index.where(frame.index.notna(), 'a' + frame.reset_index().index.astype(str))

Traceback (most recent call last):
  File "<input>", line 4, in <module>
  File "C:\Users\Miniconda3\envs\testbed_dtype\lib\site-packages\pandas\core\indexes\base.py", line 3791, in where
    return self._shallow_copy_with_infer(values, dtype=dtype)
  File "C:\Users\Miniconda3\envs\testbed_dtype\lib\site-packages\pandas\core\indexes\base.py", line 567, in _shallow_copy_with_infer
    return Index(values, **attributes)
  File "C:\Users\Miniconda3\envs\testbed_dtype\lib\site-packages\pandas\core\indexes\base.py", line 385, in __new__
    data = _maybe_cast_with_dtype(data, dtype, copy)
  File "C:\Users\Miniconda3\envs\testbed_dtype\lib\site-packages\pandas\core\indexes\base.py", line 5457, in _maybe_cast_with_dtype
    data = data.astype(dtype)
ValueError: could not convert string to float: 'a1'

Problem description

Data is being cast to incorrect dtype based on inferred dtype and gives a ValueError.
Issue was not present in pandas==0.25.3

Expected Output

With pandas==0.25.3

fixed_index
Index([1.0, 'a1'], dtype='object')

Output of pd.show_versions()

python : 3.7.6.final.0
pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.2.0.post20200210
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : 1.3.0
xlsxwriter : None
numba : None

[jbrockmendel]

Looks like the issue is in Index.where, shouldnt pass dtype to shallow_copy_with_infer

[simonjayhawkins]

Issue was not present in pandas==0.25.3

regression in #27883 (i.e. 1.0.0)

e47362a is the first bad commit
commit e47362a
Author: jbrockmendel jbrockmendel@gmail.com
Date: Tue Aug 13 05:20:40 2019 -0700

CLN: Index.__new__ (#27883)

cc @jbrockmendel

[cgarciae]

I tested against master (895f0b4) and the issue is now gone, I think this can be closed now.

[jreback]

we would take a PR with a validation test

[cgarciae]

I see, I can add a test for this.

[cgarciae]

take

[simonjayhawkins]

I tested against master (895f0b4) and the issue is now gone, I think this can be closed now.

also seems fixed on 1.2.x (fixed in 1.2.0)

[rendner]

take