BUG: attrs lost for Series in DataFrame

[RobertRosca]

• I have checked that this issue has not already been reported.

  • Has sort of come up in other issues related to metadata/attrs not propagating through properly, however I haven't seen any issues specifically about Series attrs being lost when put into a DataFrame.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

See this GitHub Gist Notebook.

Summary of the issue is:

import pandas as pd

#  Make some dummy dataframe
df = pd.DataFrame({'col1': [1, 2], 'col2': [3, 4]})

#  Make some dummy series
col1_w_attr = df["col1"].copy()
#  Add some attrs to it
col1_w_attr.attrs["test_attr"] = 1
df["col1_w_attr"] = col1_w_attr
#  Issue 1: The attrs are missing, call `__finalize__` manually
df["col1_w_attr"].__finalize__(col1_w_attr)

#  Repeat the above with another column:
col2_w_attr = df["col2"].copy()
col2_w_attr.attrs["test_attr"] = 2
df["col2_w_attr"] = col2_w_attr
df["col2_w_attr"].__finalize__(col2_w_attr)

#  Attrs are present as before
df["col2_w_attr"].attrs

#  Issue 2: col1_w_attrs is now missing its attrs:
df["col1_w_attr"].attrs

Problem description

First issue is that attrs are not propagated through when a Series is added to a DataFrame (likely related to #28283).

Second issue is that if you call __finalize__ to make sure that the attrs are present, once you add another column in or modify the dataframe, the attrs already present for a column are lost. My guess is that when the DataFrame is modified, finalize is not called per-column again, but I haven't looked into it much.

I'd be happy to help with this issue, however there are already many issues related to this topic and a few branches where others are working on attrs/metadata/finalize, so I'm a bit hesitant to start work as it'll probably conflict with what has already been done.

What would be the best place to discuss collaboration on this?

Expected Output

Expectation would be that you can attach some attrs to a Series, place it into a dataframe, modify/work with the dataframe, then access the attrs again.

Output of pd.show_versions()

import pandas as pd

pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.8.3.final.0
python-bits : 64
OS : Linux
OS-release : 5.7.0-3-MANJARO
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8

pandas : 1.0.5
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 19.2.3
setuptools : 41.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[TomAugspurger]

once you add another column in or modify the dataframe, the attrs already present for a column are lost. My guess is that when the DataFrame is modified, finalize is not called per-column again, but I haven't looked into it much.

Your right, __finalize__ is just called on the object self. We don't call __finalize__ on each column of a DataFrame independently, and I don't think we would want to (performance and potentially confusing behavior).

[RobertRosca]

Ah, so it's an intentional decision, yeah, I can see why that makes sense. Can I do a PR to clarify that in the documentation?

Also, is there a recommended way of dealing with the use case of wanting metadata to be attached to a column?

I'd like to use custom accessors to add in some convenience functions which need to store some metadata either in the attrs attribute or in a custom one and (ideally) some of these accessors can be called on a dataframe or a series.

Since finalize isn't called on all the columns, then one way I can think of doing this is to have the series accessor write the metadata into the DataFrame that the Series belongs do. Is there any way to access the 'parent' DataFrame from a Series, or some better way to do this?

[TomAugspurger]

Ah, so it's an intentional decision, yeah, I can see why that makes sense. Can I do a PR to clarify that in the documentation?

Sure, that would be great. I think we'll eventually want a page in the docs User Guide dedicated to attrs, but I don't think it's ready yet.

Also, is there a recommended way of dealing with the use case of wanting metadata to be attached to a column?

IMO, the metadata applies to the dataset as a whole, so I would have something like

df.attrs["descriptions"] = {"col1": "description 1", "col2": "description 2"]

And then we need to make sure that DataFrame.__getitem__ propagates the metadata by calling finalize.

[RobertRosca]

Sure, that would be great. I think we'll eventually want a page in the docs User Guide dedicated to attrs, but I don't think it's ready yet.

Yeah, there are still some PRs (like your one #33338) which should be closed before it's fully documented.

I added a small note to the documentation page for the series in PR #35456

IMO, the metadata applies to the dataset as a whole, so I would have something like

df.attrs["descriptions"] = {"col1": "description 1", "col2": "description 2"]

And then we need to make sure that DataFrame.__getitem__ propagates the metadata by calling finalize.

That's basically what I had in mind as well, but it'd be nice (if possible) to have it propagate the metadata through the first time when running __setitem__ or DataFrame.assign.

For example, I wanted to use this a bit like:

@pd.api.extensions.register_series_accessor('test')
class TestSeriesAccessor:
    def __init__(self, pandas_object: pd.Series):
        self._obj = pandas_object

    def method(self) -> pd.Series:
        res = self._obj

        # imagine some useful operations, that want to store metadata

        res.attrs['test'] = "some kind of metadata added by the method"

        return res

Then the behaviour would be like:

>>> df = pd.DataFrame({'col1': [1, 2, 3, 4, 5], 'col2':  [1, 2, 3, 4, 5]})
>>> col1_test = df.col1.test.method()
>>> col1_test.attrs
    {'test': 'some kind of metadata added by the method'}
>>> df["col1_test"] = col1_test
>>> df.attrs # My dream:
    {'col1_test': {'test': 'some kind of metadata added by the method'}}

But that would require calling __finalize__ on anything which could assign a series to a dataframe. And it adds in some confusion over how to deal with various scenarios like merging columns/dataframes, but I guess that as long as the approach is consistent and well documented it would be fine.

[jorisvandenbossche]

Another example where the current behaviour can lead to confusing behaviour, is when setting attrs directly on a column of a DataFrame, which are retained as long as the cache is not cleared .., see #38107

In [12]: df = pd.DataFrame({"a": [1, 2, 3]})
    ...: df["a"].attrs["key"] = "value"

In [13]: df["a"].attrs
Out[13]: {'key': 'value'}

In [14]: df._clear_item_cache()

In [15]: df["a"].attrs
Out[15]: {}