BUG: Column 'Int8Dtype' casted to 'object' type when melted

[etiennekintzler]

• [x ] I have checked that this issue has not already been reported.

• [ x] I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Pandas's dtype Int8Dtype is converted to object when melted.

Code Sample, a copy-pastable example

df = pd.DataFrame({"a": pd.Series([1, 2], dtype=pd.Int8Dtype())})
melted = df.melt()
print(melted.dtypes)

Problem description

When melted, the Int8 column is casted to object type.

Expected Output

melted.dtypes should return:

variable    object
value       Int8
dtype: object

Output of ``pd.show_versions()

INSTALLED VERSIONS

commit : b5958ee
python : 3.7.0.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-143-generic
Version : #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.5
numpy : 1.19.4
pytz : 2020.5
dateutil : 2.8.1
pip : 21.0.1
setuptools : 54.1.2
Cython : None
pytest : 6.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.19.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.3
numexpr : None
odfpy : None
openpyxl : 3.0.6
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : 1.0.8
s3fs : None
scipy : 1.5.4
sqlalchemy : 1.3.22
tables : None
tabulate : 0.8.7
xarray : None
xlrd : 2.0.1
xlwt : None
numba : 0.53.1

[mzeitlin11]

Thanks for the clear report @etiennekintzler! Issue persists on master, investigations to fix are welcome! (This looks like an issue with other ExtensionDType like StringDType as well)

[etiennekintzler]

Hi @mzeitlin11

After checking the function melt, it looks like the problem appears when the attribute DataFrame._values is accessed :

pandas/pandas/core/reshape/melt.py

Line 147 in 34b0bdc

mdata[value_name] = frame._values.ravel("F") # type: ignore[assignment]

For example

df_int8d = pd.DataFrame({"a": [1, 2]}, dtype=pd.Int8Dtype())
print(repr(df_int8d._values.dtype))

returns dtype('O') which doesn't seem right.

As expected pd.DataFrame({"a": [1, 2]})._values.dtype returns dtype('int64')

I don't know to what extent it is comparable with the attribute _values for Series but this issue doesn't appear for this class:

s = pd.Series(data=[1, 2], dtype=pd.Int8Dtype())
print(repr(s._values.dtype))

returns Int8Dtype() which is what we expect.

Issue persists on master, investigations to fix are welcome!

Would be glad to contribute however I am not very familiar with the pandas internals ; would you have any idea of which source file I could look into ? I've just found this https://github.com/pandas-dev/pandas/blob/v1.2.4/pandas/core/arrays/integer.py#L605-L608

[mzeitlin11]

Not sure about the best way to approach fixing this. The problem is that for melt we essentially want to take data and flatten it. We flatten it by converting to a numpy array and usingravel - the conversion to numpy is the problem since ExtensionDType data will become object.

There's also the further issue that for a case like

df = pd.DataFrame(
    {"a": pd.Series([1, 2], dtype=pd.Int8Dtype()),
     "b": pd.Series([1, 2], dtype=pd.Int16Dtype())}
)

keeping extension types would require invoking some find common type logic (maybe related to #22791)

So for fixing this issue it isn't limited to IntegerDType, so the file linked above probably won't be useful. It would be more about finding a different procedure for flattening the columns in a way which preserves types better (while still being performant). This sounds kind of like a concat, so I'd suggest looking into how something like pd.concat([df, df]) (using the example df from above) manages to maintain the type correctly.

Sorry I can't be of more help here - if you're looking to contribute you'd of course be welcome to look into this further (maybe someone with more experience with this part of the codebase has better suggestions for fixing this) but there are also tons of other issues which may be a more approachable starting point.