You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I seem to have encountered a bug while using DataFrame.apply(np.std) or DF.groupby.agg(np.std)
tested on:
debian 7.9 (wheezy), pandas version: 0.16.2.dev
ubuntu 15.10, pandas version: 0.15.0
more version details for the tested snippets below
test code:
importnumpyasnpimportpandasaspdprint("pandas version: ", pd.__version__)
# why is there a difference ?s=pd.Series([1.11] *10)
print(np.std(s))
#1.88486436615e-08print(np.std(s.values))
#2.22044604925e-16# why is it significantly != 0 ?# why is there a difference ?print(np.std(pd.Series([53426.7756333882,] *50)))
#0.0011048543456print(np.std(pd.Series([53426.7756333882,] *123)))
#0.000704429402084# doing that with data framesdf=pd.DataFrame([538512.198638,] *123)
print(df.apply(np.std))
#0 0.030867# dtype: float64print(np.std(df))
#0 0.030867# dtype: float64print(np.std(df.values))
#2.32830643654e-10
using the following data set even NaNs appear as result for std
funnily this happens if you add enough digits after the comma
importnumpyasnpimportpandasaspdprint("pandas version: ", pd.__version__)
df=pd.DataFrame([538512.1986379109,]*126)
df['uid']=15df.set_index('uid', inplace=True)
# why do I occasionally even get NaN for identical values?# NaN appears if there are enough decimal places after the comma print(df.groupby(level=0).agg([np.mean, np.std]))
# 0 # mean std#uid #15 538512.198638 NaNdf['bla'] =911.print(df.groupby(level=0).agg([np.mean, np.std]))
# 0 bla # mean std mean std#uid #15 538512.198638 NaN 911 0df['bla'] =911.12351171571542312243214print(df.groupby(level=0).agg([np.mean, np.std]))
# 0 bla # mean std mean std#uid #15 538512.198638 NaN 911 NaN
bug hypotheses:
this looks like a detail-problem regarding the way pandas applies functions
is the size of the mapped-on container calculated wrong? (e.g. not the single column series length is used, but the overall container's number of elements)
version details: (ubuntu 15.10)
INSTALLED VERSIONS
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-43-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_AT.UTF-8
I seem to have encountered a bug while using
DataFrame.apply(np.std) or DF.groupby.agg(np.std)
tested on:
test code:
using the following data set even NaNs appear as result for std
funnily this happens if you add enough digits after the comma
bug hypotheses:
this looks like a detail-problem regarding the way pandas applies functions
is the size of the mapped-on container calculated wrong? (e.g. not the single column series length is used, but the overall container's number of elements)
version details: (ubuntu 15.10)
INSTALLED VERSIONS
commit: None
python: 2.7.10.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-43-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_AT.UTF-8
pandas: 0.15.0
nose: 1.3.6
Cython: 0.23.3
numpy: 1.8.2
scipy: 0.14.1
statsmodels: 0.5.0
IPython: 2.3.0
sphinx: 1.2.3
patsy: 0.4.0
dateutil: 2.2
pytz: 2014.10
bottleneck: None
tables: 3.2.2
numexpr: 2.4.3
matplotlib: 1.4.2
openpyxl: 2.3.0-b1
xlrd: 0.9.4
xlwt: 0.7.5
xlsxwriter: None
lxml: 3.4.4
bs4: 4.3.2
html5lib: 0.999
httplib2: 0.9
apiclient: None
rpy2: 2.6.2
sqlalchemy: 1.0.8
pymysql: 0.6.2.None
psycopg2: 2.6.1 (dt dec mx pq3 ext lo64)
The text was updated successfully, but these errors were encountered: