BUG/inconsistency: pd.Series(dtype=object).sum(skipna=True) does not respect skipna=True

[Code0x58]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np


class X:
    def __init__(self, n, generation=0):
        self.n = n
        self.generation = generation

    def __add__(self, other):
        return X(self.n + other.n, max(self.generation, other.generation) + 1)

    def __mul__(self, other):
        return X(self.n * other.n, max(self.generation, other.generation) + 1)

    def __repr__(self):
        return f'X({self.n}, {self.generation})'
    

s = pd.Series([X(1), X(2), np.nan, X(0)])

# these implicitly have skipna=True anyway
s.sum(skipna=True)

Issue Description

aggregation methods over objects does not respect skipna. Exception produced above

# sum, cumsum, prod, cumprod, ...
AttributeError: 'int' object has no attribute 'n'

Expected Behavior

NaN values are skipped.

Given that all-NaN values are documented to produce NaN

Installed Versions

INSTALLED VERSIONS
------------------
commit                : d9cdd2ee5a58015ef6f4d15c7226110c9aab8140
python                : 3.12.5.final.0
python-bits           : 64
OS                    : Linux
OS-release            : 4.15.0-213-generic
Version               : #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023
machine               : x86_64
processor             :
byteorder             : little
LC_ALL                : None
LANG                  : C.UTF-8
LOCALE                : C.UTF-8

pandas                : 2.2.2
numpy                 : 2.1.1
pytz                  : 2024.1
dateutil              : 2.9.0.post0
setuptools            : None
pip                   : 24.2
Cython                : None
pytest                : None
hypothesis            : None
sphinx                : None
blosc                 : None
feather               : None
xlsxwriter            : None
lxml.etree            : None
html5lib              : None
pymysql               : None
psycopg2              : None
jinja2                : None
IPython               : None
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : None
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : None
fsspec                : None
gcsfs                 : None
matplotlib            : None
numba                 : None
numexpr               : None
odfpy                 : None
openpyxl              : None
pandas_gbq            : None
pyarrow               : None
pyreadstat            : None
python-calamine       : None
pyxlsb                : None
s3fs                  : None
scipy                 : None
sqlalchemy            : None
tables                : None
tabulate              : None
xarray                : None
xlrd                  : None
zstandard             : None
tzdata                : 2024.1
qtpy                  : None
pyqt5                 : None

[rhshadrach]

Thanks for the report. For efficiency, pandas implements skipna=True by filling NA values with 0. I think to recreate a new array of smaller size would be prohibitively inefficient. If you'd like to be operable with pandas, I'd suggest implementing integer addition (at the very least, with the number 0).

[Code0x58]

That makes a fair bit of sense, and works out of the box for things like the built in complex, Fraction, and Decimal builtins which are probably more representative of normal use, as the relevant identity (i.e. 0 or 1) is use.

In the case of operations that produce scalars, like .sum(), .min(), etc. I'd have thought with the cost of operating with python objects that omitting invocations with nulls would actually ended up a little more efficient than dropping in 1 or 0 and throwing that at the next encountered value. I suppose given behaviour on empty series for .min(), .prod(), and sum() you'd want to have special handling everywhere NaN, 1, and 0 if everything else is unchanged, which seems fair. As a bit an extension, if you could provide the identity value and it supported __iadd__ or whatever else, you could likely end up with somewhat better efficiency as you can reasonably claim ownership over a provided identity parameter (or magic method?) and avoid creating intermediate objects.

I can see what you mean for efficiency of the expanding operations like .cumsum(), .cumin(), etc. as you'd need to either copy the object, or more precisely recreate an identity to accumulate to produce a new value. The puritan in me likes the idea of this over "you just get some default int", maybe with an interface like series.cumsum(identity_factory=X).

As much as I think there could be some nice performance gains, and maybe a little bit less surprise with documentation as it stands, this particularly niche case can be solved by handling NaN, 0, and 1 as you say. The "surprise" for the very odd user could be ironed out with documentation explaining what happens with the object dtype along with examples of handling those special cases.

That all said, as interesting as it I've found it to talk about, I don't have any real interest in promoting or working on any sort of "scalars might go faster and explicit identities for in-place ops may be faster still" or "explicit identities allow another way to do expanding". Even updating documentation feels like a bit of a stretch now as I guess this doesn't come up much, as even my case just came up from trying to hack up something into an existing project so have plenty of flexibility.