Skip to content

difference in overflows handling while doing pow on large integers #15046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
yarikoptic opened this issue Jan 3, 2017 · 4 comments
Closed

difference in overflows handling while doing pow on large integers #15046

yarikoptic opened this issue Jan 3, 2017 · 4 comments
Labels
Compat pandas objects compatability with Numpy or Python functions Numeric Operations Arithmetic, Comparison, and Logical operations Testing pandas testing functions or related to the test suite

Comments

@yarikoptic
Copy link
Contributor

Output from running tests and troubleshooting

probably related to having numpy beta release or freshish numexpr...

$> nosetests -s -v --pdb pandas/tests/test_expressions.py
...
test_integer_arithmetic (pandas.tests.test_expressions.TestExpressions) ... > /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py(1018)raise_assert_detail()
-> raise AssertionError(msg)
*(Pdb) print msg
DataFrame.iloc[:, 0] are different

DataFrame.iloc[:, 0] values are different (85.16148 %)
[left]:  [2565992168703393792, 2110287360296998863, 0, 0, 2644539488379390509, -5278486589563110205, -4689350456247753643, 8450172506621111015, -2101438300051996672, 8028209804145959321, 302875106592253, -5278486589563110205, -4053175462519618527, -4071272324936764365, 0, 0, 0, 256, 8450172506621111015, 0, -145357957422043229, 3030756035345000135, 256, 0, -4071272324936764365, -5278486589563110205, -2863221430593058543, 0, 0, 0, 4951460147845608313, 0, -1595931050845505211, -5821812116355782473, -6284992600390200471, 2435553365352800019, -1595931050845505211, -4053175462519618527, 0, 3792227444683241861, -6776596920136667815, -5821812116355782473, 0, -8795647696940200745, 4593512421086651489, 256, 0, 0, 2644539488379390509, -5821812116355782473, -7924192819420782515, 1826852818786091687, 0, 2110287360296998863, -2559158286230232783, -4689350456247753643, -6646187150092009472, 8028209804145959321, 0, 437893890380859375, 0, -5278486589563110205, 0, 2467972595799031808, 2604998672350111773, -4689350456247753643, 7437354637604294721, 0, 285311670611, 5529354540715494413, 2435553365352800019, -1595931050845505211, 256, 823543, -2559158286230232783, 0, -3660930864499998817, 0, -4642015662142636065, -4071272324936764365, 4611686018427387904, 285311670611, 1, 0, 2565992168703393792, 8028209804145959321, -145357957422043229, 8450172506621111015, -5821812116355782473, 0, -4675932718989443137, 8037740809528595931, -4071272324936764365, 0, 6353754964178307979, 0, -1595931050845505211, -1595931050845505211, -3264474685660004015, 2823291043684304491, ...]
[right]: [-9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 302875106592253, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 256, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 256, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 256, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 437893890380859375, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 285311670611, -9223372036854775808, -9223372036854775808, -9223372036854775808, 256, 823543, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, 285311670611, 1, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, -9223372036854775808, ...]
(Pdb) up
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/src/testing.pyx(173)pandas._testing.assert_almost_equal (pandas/src/testing.c:3274)()
-> raise_assert_detail(obj, msg, lobj, robj)
(Pdb) 
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/src/testing.pyx(59)pandas._testing.assert_almost_equal (pandas/src/testing.c:4156)()
-> cpdef assert_almost_equal(a, b,
(Pdb) 
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py(1181)assert_series_equal()
-> obj='{0}'.format(obj))
(Pdb) 
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py(1313)assert_frame_equal()
-> obj='DataFrame.iloc[:, {0}]'.format(i))
(Pdb) 
> /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py(100)run_arithmetic_test()
-> assert_func(expected, result)
(Pdb) p df
        A   B   C   D
0      30  89  97  21
1      47  22  16  80
2      78  23  84  33
3      88  25  61  59
4      61   1  41  96
5      27  67  76  98
6      37  13  13  82
7      23  60  93   1
8      20   6  18  90
9      89  97  27  73
10     13  95  37  38
11     27  53  70  77
12     33  51  44  39
13     43  89  95   5
14     36  84  51  97
15     44   8  82  31
16     84  18  50  35
17      4  52  74  37
18     23  52  73  42
19     76  99  11  67
20     59  95  87  68
21     55  18  89  73
22      4  26  64  19
23     94  59  47  36
24     43  79  37  26
25     27  42  41  84
26     17  30   4   3
27     24  37  63  88
28     66  14  92  86
29     52  38  27  97
...    ..  ..  ..  ..
9971   25  13  53  68
9972    9  46  89  93
9973   70  53  47  33
9974   10  30  47   3
9975   98  15  98  43
9976   34   5  53  76
9977   21  29  65  95
9978   30  83  67  82
9979   52   1  58  17
9980    4  84  38  79
9981   83  96  57  12
9982   13  38  93  72
9983   58  64  24  13
9984   18  46   5  43
9985   70  84  88  45
9986   79  94  57  29
9987   18  31  63  57
9988   58  90  99   5
9989    8  88  77  55
9990   46  12  61  61
9991   69  70  96  45
9992   71  67  48  25
9993   77  68  46  16
9994   88  55  80  37
9995    6  93  76  73
9996   11  68  45  52
9997   57  79   3  69
9998   63  48  79   8
9999   11   5  70  16
10000  10  32  32  88

[10001 rows x 4 columns]
(Pdb) p op
<function pow at 0x7efedaaad758>

Problem description

Not sure what to blame here -- numexpr or numpy or ... What would you expect while trying to get e.g. 30**30? max for int of original pow base type?

Output of pd.show_versions()

# Paste the output here pd.show_versions() here $> ./ci/print_versions.py

INSTALLED VERSIONS

commit: ad13f08
python: 2.7.13.candidate.1
python-bits: 64
OS: Linux
OS-release: 4.8.0-1-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.19.0+git14-ga40e185
nose: 1.3.7
pip: 8.1.2
setuptools: 28.7.1
Cython: 0.25.2b0
numpy: 1.12.0b1
scipy: 0.18.1
statsmodels: 0.8.0rc1
xarray: None
IPython: 5.1.0
sphinx: 1.4.8
patsy: 0.4.1+dev
dateutil: 2.5.3
pytz: 2015.7
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.1
feather: None
matplotlib: 1.5.3
openpyxl: 2.3.0
xlrd: 1.0.0
xlwt: 0.7.5
xlsxwriter: None
lxml: 3.6.4
bs4: 4.5.1
html5lib: 0.999
httplib2: 0.9.2
apiclient: 1.5.2
sqlalchemy: 1.0.15
pymysql: None
psycopg2: 2.6.2 (dt dec mx pq3 ext lo64)
jinja2: 2.8
s3fs: None
pandas_datareader: None
@jreback
Copy link
Contributor

jreback commented Jan 3, 2017

#14498 you look like you have an older version here.

@yarikoptic
Copy link
Contributor Author

if you are referring to the version of pandas -- then nope -- all as fresh as it gets.

$> nosetests -s -v --pdb pandas/tests/test_expressions.py
test_binary_ops (pandas.tests.test_expressions.TestExpressions) ... > /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py(63)_evaluate_standard()
-> return op(a, b)
(Pdb) c
ERROR
test_bool_ops_raise_on_arithmetic (pandas.tests.test_expressions.TestExpressions) ... ok
test_bool_ops_warn_on_arithmetic (pandas.tests.test_expressions.TestExpressions) ... ok
test_boolean_ops (pandas.tests.test_expressions.TestExpressions) ... ok
test_float_arithemtic (pandas.tests.test_expressions.TestExpressions) ... ok
test_float_arithemtic_frame (pandas.tests.test_expressions.TestExpressions) ... ok
test_float_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ok
test_float_panel (pandas.tests.test_expressions.TestExpressions) ... ok
test_integer_arithmetic (pandas.tests.test_expressions.TestExpressions) ... > /home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py(1018)raise_assert_detail()
-> raise AssertionError(msg)
(Pdb) import pandas 
(Pdb) print pandas.__version__
0.19.0+303.gad13f08

just that nosetests adjusts PYTHONPATH while I ran ./ci/print_versions.py which picked up system wide one... sorry

@jreback
Copy link
Contributor

jreback commented Jan 3, 2017

might be numpy beta. we test with numpy master on travis, but not windows.

@jreback jreback added Compat pandas objects compatability with Numpy or Python functions Numeric Operations Arithmetic, Comparison, and Logical operations Testing pandas testing functions or related to the test suite labels Jan 3, 2017
@jreback jreback modified the milestones: Next Major Release, 0.20.0 Jan 3, 2017
raspbian-autopush pushed a commit to raspbian-packages/pandas that referenced this issue Mar 7, 2017
@yarikoptic
avoid overflows for now Due to a bug in current numpy beta (or numexp…
0d62bc8 
…r) tests would fail if operation on int leads to overflows (e.g. of pow operation).

  as a workaround for now -- avoid big ints

Origin:  Debian
Bug: pandas-dev/pandas#15046
Last-Update: 2017-01-04


Gbp-Pq: Name up_buggy_overflows
@jreback jreback modified the milestones: 0.20.0, Next Major Release Mar 23, 2017
raspbian-autopush pushed a commit to raspbian-packages/pandas that referenced this issue Apr 6, 2017
@yarikoptic @tillea
avoid overflows for now Due to a bug in current numpy beta (or numexp…
cca023d 
…r) tests would fail if operation on int leads to overflows (e.g. of pow operation).

  as a workaround for now -- avoid big ints

Origin:  Debian
Bug: pandas-dev/pandas#15046
Last-Update: 2017-01-04


Gbp-Pq: Name up_buggy_overflows
@mroeschke
Copy link
Member

This is working on master locally. Closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Compat pandas objects compatability with Numpy or Python functions Numeric Operations Arithmetic, Comparison, and Logical operations Testing pandas testing functions or related to the test suite
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yarikoptic @jreback @mroeschke