BUG: Determine cause of floating point exception with accelerated __mod__

[jtratner]

For some reason, using numexpr with % and zeros causes floating point exceptions. I.e., in this Travis CI test you can see the mysterious floating point exception occur:

#snippet from nosetests
........................................................S.................................................................
ci/script.sh: line 13:  4513 Floating point exceptionnosetests --exe -w /tmp -A "$NOSE_ARGS" pandas

The exception occurs when __mod__ is evaluated with str_rep of % (meaning a_values % b_values is passed to numexpr.evaluate), and it occurs in test_expressions.py under test_mixed_floats_arithmetic and test_integer_with_zeros in (for example) jtratner/pandas@e938da2 . Omitting str_rep from __mod__ fixes the issues, clearly showing it's occurring with numexpr. Going to investigate more in the future.

[jreback]

hmm wonder if there is a dtype that ne doesn't like or maybe nan with % not supported
?

[jtratner]

it's very simple to reproduce (and I just opened a numexpr issue). Modulus with zero isn't supported. Check it:

arr = np.arange(-5, 5)
# this will cause a floating point exception
ne.evaluate("arr % 0", locals())

Floats don't cause the problem

arr = arr.astype(float)
ne.evaluate("arr % 0", locals())

[jreback]

good to know.....this should still fallback to the numpy loop in evaluate in any event

[jtratner]

No, you can't catch it, it crashes the Python interpreter completely (see the end of this Travis build - https://travis-ci.org/jtratner/pandas/jobs/7917801 ). That's why in #3765 I've completely removed mod from being accelerated (for now). Floordiv just raises an numexpr error, so it's fine to still include.

[jreback]

oh....it DOESN't raise a FloatingPointException.....weird....ok...easy enough to not-accelarate (or alternativly you can cast to float64, sending in an additional parm, maybe cast=lambda x: x.astype(float64), sort of like com._fill_zeros does

of course easiest is simply to always cast mod and div to float, then you solve this problem and get truediv for free...

but IIRC we decided to push that off.....

[jtratner]

@jreback haha, yeah. Though would you cast it back to int afterwards? Actually, looks like you can just cast the other mod operand to float (so if it's 0 you convert it to 0.0 and it all works). I'll play around with it.

[jreback]

we dont' do any casting back on arithmetic ops that are handled thru _arith_ops...(e.g..-,+,/,**,%)
as we are using numpy facilities (these are all binary ops). You would in theory have to possibly split up by columns the resulting frame (e.g. say you were doing someting that resulted in some columns being representable as ints and some that were floats)....too much work, numpy will figure it out... :)

[jtratner]

so, basically, we just wouldn't pass it to numexpr at all for that case.
(easy to do, just pass str_rep as None)

On Mon, Jun 10, 2013 at 8:05 AM, jreback notifications@github.com wrote:

we dont' do any casting back on arithmetic ops that are handled thru
_arith_ops...(e.g..-,+,/,**,%)
as we are using numpy facilities (these are all binary ops). You would in
theory have to possibly split up by columns the resulting frame (e.g. say
you were doing someting that resulted in some columns being representable
as ints and some that were floats)....too much work, numpy will figure it
out... :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3819#issuecomment-19194592
.

[jtratner]

it occurred to me: except for the issue covered by _fill_zeros, does
numexpr always return the same types as numpy?

On Mon, Jun 10, 2013 at 8:07 AM, Jeffrey Tratner
jeffrey.tratner@gmail.comwrote:

so, basically, we just wouldn't pass it to numexpr at all for that case.
(easy to do, just pass str_rep as None)

On Mon, Jun 10, 2013 at 8:05 AM, jreback notifications@github.com wrote:

we dont' do any casting back on arithmetic ops that are handled thru
_arith_ops...(e.g..-,+,/,**,%)
as we are using numpy facilities (these are all binary ops). You would in
theory have to possibly split up by columns the resulting frame (e.g. say
you were doing someting that resulted in some columns being representable
as ints and some that were floats)....too much work, numpy will figure it
out... :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3819#issuecomment-19194592
.

[jreback]

they basically upcast if and convert to floats if needed AFAIK

[jtratner]

So, aside from the fill zeros, pandas doesn't change the type of numpy
arithmetic/binary operators?

On Mon, Jun 10, 2013 at 8:21 AM, jreback notifications@github.com wrote:

they basically upcast if and convert to floats if needed AFAIK

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3819#issuecomment-19195196
.

[jreback]

I think this i true for the binary arithmetic operators