You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
However, only the last assertion actually passes. round3up("12301") results in 12310, round3up("123.01") in 123.1, and round3up("1.2301") in 1.231. If I reduce the first argument of round_sf_with_strategy() to 2, then the first three assertions pass, but the last one doesn't, resulting in (correct) 0.13.
Am I misunderstanding the meaning of "significant digits" in the context of round_sf_with_strategy()? How would one implement consistent rounding to three significant digits?
The text was updated successfully, but these errors were encountered:
You're correct in thinking something doesn't look right. I'm in transit the next couple of days but will attempt to dig into this issue this weekend to see what's going on. Will keep you posted.
Let's say I want to round the number to three significant digits, rounding away from zero:
I would expect the following assertions to hold:
However, only the last assertion actually passes.
round3up("12301")
results in12310
,round3up("123.01")
in123.1
, andround3up("1.2301")
in1.231
. If I reduce the first argument ofround_sf_with_strategy()
to 2, then the first three assertions pass, but the last one doesn't, resulting in (correct)0.13
.Am I misunderstanding the meaning of "significant digits" in the context of
round_sf_with_strategy()
? How would one implement consistent rounding to three significant digits?The text was updated successfully, but these errors were encountered: