Extension dtypes are modified when performing groupby aggregate with agg dict

[griffindore]

Code Sample, a copy-pastable example if possible

x = np.array([0,1,2,3,4,5], dtype=np.uint8)    # some data 
y = np.array([0,1,2,3,4,5], dtype=np.int32)    # some other data 
g = ['foo', 'bar', 'baz', 'foo', 'bar', 'baz'] # grouping column 
df = pd.DataFrame({'x': x, 'y': y, 'g': g})    #  construct dataframe from np arrays preserving dtypes

# dtypes maintained through a groupby agg as expected
df.groupby('g').agg('first').dtypes
# Out[7]: 
# x    uint8
# y    int32
# dtype: object

# dtypes maintained through a groupby agg  as expected when agg_funcs is dict
df.groupby('g').agg({'x':'first', 'y':'first'}).dtypes
# Out[8]: 
# x    uint8
# y    int32
# dtype: object

# Use new extension dtypes
df = pd.DataFrame({'x': pd.array(x, copy=False), 'y': pd.array(y, copy=False), 'g': g})

# dtypes maintained through a groupby agg as expected when agg_funcs is a scalar
df.groupby('g').agg('first').dtypes
# Out[10]: 
# x    UInt8
# y    Int32
# dtype: object

# dtypes are not maintained through a groupby agg when agg_funcs is a dict and column dtypes are extension dtypes
df.groupby('g').agg({'x':'first', 'y':'first'}).dtypes
# Out[11]: 
# x    float64
# y    float64
# dtype: object

Problem description

When using Extension dtypes, StringDtype, BooleanDtype, Int64Dtype, Int32Dtype, etc., in a DataFrame, the dtypes are modified to float64 when performing a groupby.agg with a dict defining the reducing function per column. This change in dtypes doesn't occur with numpy dtypes, and it doesn't occur when using a single agg func as scalar instead of the dict.

Expected Output

df.groupby('g').agg({'x':'first', 'y':'first'}).dtypes
# Out[11]: 
# x    UInt8
# y    Int32
# dtype: object

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit : None
python : 3.6.7.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.1
numpy : 1.14.0
pytz : 2018.7
dateutil : 2.7.5
pip : 18.1
setuptools : 40.6.2
Cython : 0.29.1
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10
IPython : 7.3.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : 0.3.3
gcsfs : None
lxml.etree : None
matplotlib : 3.0.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : 0.1.3
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : 0.36.2

[griffindore]

This is also a regression from 0.25.3 and prior. I first noticed it with categoricals, but just now realized that you can set the new Int types in 0.25. dtypes are maintained in 0.25.3 when doing df.groupby.agg(agg_dict)

[simonjayhawkins]

Thanks @griffindore for the report. can confirm this is a regression

regression in #31359 (i.e. 1.0.0)

06ef193 is the first bad commit
commit 06ef193
Author: Kaiqi Dong kaiqi@kth.se
Date: Wed Jan 29 21:20:52 2020 +0100

BUG: regression when applying groupby aggregation on categorical columns (#31359)

@charlesdong1991

[simonjayhawkins]

When using Extension dtypes, StringDtype, BooleanDtype, Int64Dtype, Int32Dtype, etc., in a DataFrame, the dtypes are modified to float64 when performing a groupby.agg with a dict defining the reducing function per column. This change in dtypes doesn't occur with numpy dtypes, and it doesn't occur when using a single agg func as scalar instead of the dict.

StringDtype looks OK.

>>> import pandas as pd
>>>
>>> pd.__version__
'1.1.0.dev0+999.gc47e9ca8b'
>>>
>>> df = pd.DataFrame(
...     {
...         "a": pd.array(range(6), dtype="UInt8"),
...         "b": pd.array(range(6), dtype="Int32"),
...         "c": pd.array(list("abcdef"), dtype="string"),
...         "d": pd.array([0, 1] * 3, dtype="boolean"),
...         "g": ["foo", "bar", "baz"] * 2,
...     }
... )
>>>
>>> grp = df.groupby("g")
>>>
>>>
>>> grp.agg('first').dtypes
a      UInt8
b      Int32
c     string
d    boolean
dtype: object
>>>
>>>
>>> grp.agg({"a": "first", "b": "first", "c": "first", "d": "first"}).dtypes
a    float64
b    float64
c     string
d    float64
dtype: object