Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rounding to significant digits doesn't work as expected #430

Closed
hniksic opened this issue Oct 5, 2021 · 1 comment · Fixed by #434
Closed

Rounding to significant digits doesn't work as expected #430

hniksic opened this issue Oct 5, 2021 · 1 comment · Fixed by #434

Comments

@hniksic
Copy link

hniksic commented Oct 5, 2021

Let's say I want to round the number to three significant digits, rounding away from zero:

fn round3up(d: Decimal) -> Decimal {
    d.round_sf_with_strategy(3, RoundingStrategy::AwayFromZero).unwrap()
}

I would expect the following assertions to hold:

assert_eq!("12400", round3up("12301".parse().unwrap()).to_string());
assert_eq!("124", round3up("123.01".parse().unwrap()).to_string());
assert_eq!("1.24", round3up("1.2301".parse().unwrap()).to_string());
assert_eq!("0.124", round3up("0.12301".parse().unwrap()).to_string());

However, only the last assertion actually passes. round3up("12301") results in 12310, round3up("123.01") in 123.1, and round3up("1.2301") in 1.231. If I reduce the first argument of round_sf_with_strategy() to 2, then the first three assertions pass, but the last one doesn't, resulting in (correct) 0.13.

Am I misunderstanding the meaning of "significant digits" in the context of round_sf_with_strategy()? How would one implement consistent rounding to three significant digits?

@paupino
Copy link
Owner

paupino commented Oct 5, 2021

You're correct in thinking something doesn't look right. I'm in transit the next couple of days but will attempt to dig into this issue this weekend to see what's going on. Will keep you posted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants