Skip to content

DataFrame.sum() inconsistent behavior treating full row/col of NANs #10785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
siyantan opened this issue Aug 10, 2015 · 1 comment
Closed

DataFrame.sum() inconsistent behavior treating full row/col of NANs #10785

siyantan opened this issue Aug 10, 2015 · 1 comment
Labels
Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Numeric Operations Arithmetic, Comparison, and Logical operations

Comments

@siyantan
Copy link

Seeing an odd issue. When doing a sum and the full column or row is NAN, I expect NAN as the result. Recently my system has been giving a sum of zero if the full row is NAN, unless skipna=FALSE is specified (but of course that screws up rows with only one NAN, for example).

I would have thought skipna=TRUE, which is default, should give NAN if the full row is NAN per the docs.

This seems to be dependant on some combination of package versions but I cannot figure out why. I reinstalled completely to a fresh Anaconda 2.3.0 amd64 and the behaviour is the same.

See screenshot. For In[6] I expect index [0] to be NAN, not zero.

image

My package list, for reference:

_license 1.1 py27_0
alabaster 0.7.3 py27_0
anaconda 2.3.0 np19py27_0
argcomplete 0.8.9 py27_0
astropy 1.0.3 np19py27_0
babel 1.3 py27_0
backports.ssl-match-hostname 3.4.0.2
bcolz 0.9.0 np19py27_0
bdateutil 0.1
beautiful-soup 4.3.2 py27_1
beautifulsoup4 4.3.2
binstar 0.11.0 py27_0
bitarray 0.8.1 py27_1
blaze 0.8.0
blaze-core 0.8.0 np19py27_0
blpapi 3.5.3
blz 0.6.2 np19py27_1
bokeh 0.9.0 np19py27_0
boto 2.38.0 py27_0
bottleneck 1.0.0 np19py27_0
cdecimal 2.3 py27_1
certifi 14.05.14 py27_0
cffi 1.1.0 py27_0
clyent 0.3.4 py27_0
colorama 0.3.3 py27_0
conda 3.15.1 py27_0
conda-build 1.14.1 py27_0
conda-env 2.3.0 py27_0
configobj 5.0.6 py27_0
cryptography 0.9.1 py27_0
cvxopt 1.1.7
cython 0.22.1 py27_0
cytoolz 0.7.3 py27_0
datashape 0.4.5 np19py27_0
dateutil 2.4.1 py27_0
decorator 3.4.2 py27_0
docutils 0.12 py27_1
dynd-python 0.6.5 np19py27_0
enum34 1.0.4 py27_0
fastcache 1.0.2 py27_0
flask 0.10.1 py27_1
funcsigs 0.4 py27_0
gevent 1.0.1 py27_0
gevent-websocket 0.9.3 py27_0
greenlet 0.4.7 py27_0
grin 1.2.1 py27_2
h5py 2.5.0 np19py27_1
hdf5 1.8.15.1 2
holidays 0.3.1
idna 2.0 py27_0
ipaddress 1.0.7 py27_0
ipython 3.2.0 py27_0
ipython-notebook 3.2.0 py27_0
ipython-qtconsole 3.2.0 py27_0
itsdangerous 0.24 py27_0
jdcal 1.0 py27_0
jedi 0.8.1 py27_0
jinja2 2.7.3 py27_2
jsonschema 2.4.0 py27_0
launcher 1.0.0 1
llvmlite 0.5.0 py27_0
lxml 3.4.4 py27_0
lz4 0.7.0
markupsafe 0.23 py27_0
matplotlib 1.4.3 np19py27_1
menuinst 1.0.4 py27_0
mistune 0.5.1 py27_1
mock 1.0.1 py27_0
multipledispatch 0.4.7 py27_0
networkx 1.9.1 py27_0
nltk 3.0.3 np19py27_0
node-webkit 0.10.1 0
nose 1.3.7 py27_0
numba 0.19.1 np19py27_0
numexpr 2.4.3 np19py27_0
numpy 1.9.2 py27_0
odo 0.3.2 np19py27_0
openpyxl 1.8.5 py27_0
pandas 0.14.1 np19py27_0
patsy 0.3.0 np19py27_0
pep8 1.6.2 py27_0
pillow 2.8.2 py27_0
pip 7.1.0 py27_0
ply 3.6 py27_0
psutil 2.2.1 py27_0
py 1.4.27 py27_0
pyasn1 0.1.7 py27_0
pycosat 0.6.1 py27_0
pycparser 2.14 py27_0
pycrypto 2.6.1 py27_3
pyflakes 0.9.2 py27_0
pygments 2.0.2 py27_0
pymongo 2.8 py27_0
pyodbc 3.0.7 py27_0
pyopenssl 0.15.1 py27_1
pyparsing 2.0.3 py27_0
pyqt 4.10.4 py27_1
pyreadline 2.0 py27_0
pytables 3.2.0 np19py27_0
pytest 2.7.1 py27_0
python 2.7.10 0
python-dateutil 2.4.2 py27_0
pytz 2015.4 py27_0
pywin32 219 py27_0
pyyaml 3.11 py27_1
pyzmq 14.7.0 py27_0
requests 2.7.0 py27_0
rope 0.9.4 py27_1
runipy 0.1.3 py27_0
scikit-image 0.11.3 np19py27_0
scikit-learn 0.16.1 np19py27_0
scipy 0.16.0 np19py27_0
setuptools 18.0.1 py27_0
six 1.9.0 py27_0
snowballstemmer 1.2.0 py27_0
sockjs-tornado 1.0.1 py27_0
sphinx 1.3.1 py27_0
sphinx-rtd-theme 0.1.7
sphinx_rtd_theme 0.1.7 py27_0
spyder 2.3.5.2 py27_0
spyder-app 2.3.5.2 py27_0
sqlalchemy 1.0.5 py27_0
ssl_match_hostname 3.4.0.2 py27_0
statsmodels 0.6.1 np19py27_0
sympy 0.7.6 py27_0
tables 3.2.0
toolz 0.7.2 py27_0
tornado 4.2 py27_0
ujson 1.33 py27_0
unicodecsv 0.9.4 py27_0
werkzeug 0.10.4 py27_0
xlrd 0.9.3 py27_0
xlsxwriter 0.7.3 py27_0
xlwings 0.3.5 py27_0
xlwt 1.0.0 py27_0
zlib 1.2.8 0

@jreback
Copy link
Contributor

jreback commented Aug 10, 2015

dupe of #9422

this was a change in bottleneck 1.0. see that issue for the soln, which is pretty straightforward.

@jreback jreback closed this as completed Aug 10, 2015
@jreback jreback added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Numeric Operations Arithmetic, Comparison, and Logical operations labels Aug 10, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Numeric Operations Arithmetic, Comparison, and Logical operations
Projects
None yet
Development

No branches or pull requests

2 participants