Append of categorical data where categories have different order

[gmeisl]

Code Sample, a copy-pastable example if possible

import pandas as pd

df = pd.DataFrame({"A":["a","b"]})
df2 = pd.DataFrame({"A":["a","b"]})

df["B"] = df["A"].astype('category')
df.B.cat.categories = ['c','d']

df2["B"] = df2["A"].astype('category')
df2.B.cat.categories = ['d','c']

dfa = df.append(df2)
print(dfa.B)

Problem description

When using df.append() on a dataframe with categorical columns where the categories are the same but in a different order ('c', 'd' vs 'd', 'c'), the wrong categories are associated to the appended dataframe.
In the example the result is
0 c
1 d
0 c
1 d

but should be
0 c
1 d
0 d
1 c

Expected Output

Correctly appended dataframe or error message.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 158 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.22.0
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[chris-b1]

thanks for the report, I'm guessing this snuck in due to the equality semantics with categoricals.

https://pandas.pydata.org/pandas-docs/stable/categorical.html#equality-semantics

[jschendel]

This looks to be a dupe of #20425, and appears to be fixed on master:

In [2]: pd.__version__
Out[2]: '0.23.0.dev0+759.g4a34497'

In [3]: df = pd.DataFrame({"A":["a","b"]})
   ...: df2 = pd.DataFrame({"A":["a","b"]})
   ...:
   ...: df["B"] = df["A"].astype('category')
   ...: df.B.cat.categories = ['c','d']
   ...:
   ...: df2["B"] = df2["A"].astype('category')
   ...: df2.B.cat.categories = ['d','c']
   ...:

In [4]: df.append(df2)
Out[4]:
   A  B
0  a  c
1  b  d
0  a  d
1  b  c

[chris-b1]

indeed, thanks @jschendel