-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect result from to_f64 on 0.000875 #118
Comments
Offending line is: I suppose I shouldn't be surprised that a floating point operation shouldn't be used here. To be explicit, this assertion passes as the second arg is equivalent to the closure above: assert_eq!(
BigDecimal::from_str("0.000875").unwrap().to_f64().unwrap(),
875.0 * 1e-6
); and this one fails, as you wrote above, as assert_eq!(
BigDecimal::from_str("0.000875").unwrap().to_f64().unwrap(),
875e-6
); I guess more bit-manipulation is required to fix this, as floating-ops should not be trusted. |
Starting with The goal is Break into floating point components:
To multiply by Take our risk on floating point math again....
and the mantissa part is one bit away from what we wanted ....
So that's a bummer |
Ah:
Which is the same number as the math above. Yeah, current implementation is off by one bit. |
Shoot, I was hoping the smaller number would allow more digits in the product, but we get the same mantissa. But I guess that can't happen if we "just" shift by 16 bits, as that will only change the exponent (by 16). |
If we have a 53 bit number we want to split into two smaller numbers, I think naturally we'd
or round up to
oh! or a power of 5?
Finally, some new mantissas. Actually, just adding those:
gives the number we've been looking for (with the 1 bit at the end). It's too late at night to figure out what to do with these, but I think I'm close. |
For completeness.... the And we have our So, with the original exponent of
and with that we have reached our goal of f64 with bits: |
Tracking fix in bugfix/to-f64 Works for normal values but needs more work to detect subnormal |
The first assert here passes, the second fails:
I would expect
to_f64
to give the same results asto_string
followed byparse
.The text was updated successfully, but these errors were encountered: