You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Testing my recent changes a bit more thoroughly, I believe there are two ways this can go.
(1) The user passes the following:
{
...
data: [ 0.1 ]
}
The value pulled from the JSON key data is now: 0.1000000000000000055511151231257827021181583404541015625. This value is indeed representable by fixed point, given enough fractional bits, so the error they receive is fractional width overflow, required fractional width: 55..
(2) The user passes in the data as strings, e.g.
{
...
data: [ "0.1" ]
}
This ensures that 0.1 indeed has the value 0.1, and thus the error they receive is not representable by fixed point numbers: 0.1.
To me, (2) seems less surprising to the user, and we can easily ensure that strings are actually passed for fixed point. Option (3) might be to explore other JSON libraries for Python that gives us more control over how numbers are interpreted.
The text was updated successfully, but these errors were encountered:
I'm going to spend some time boxing up Fixed Point number and Bitnum number representations in respective Python classes to make it a bit more clean and extensible. For simplicity, I'll just assume you pass in a string version of the number to the class.
Testing my recent changes a bit more thoroughly, I believe there are two ways this can go.
(1) The user passes the following:
The value pulled from the JSON key
data
is now:0.1000000000000000055511151231257827021181583404541015625
. This value is indeed representable by fixed point, given enough fractional bits, so the error they receive isfractional width overflow, required fractional width: 55.
.(2) The user passes in the data as strings, e.g.
This ensures that 0.1 indeed has the value 0.1, and thus the error they receive is
not representable by fixed point numbers: 0.1
.To me, (2) seems less surprising to the user, and we can easily ensure that strings are actually passed for fixed point. Option (3) might be to explore other JSON libraries for Python that gives us more control over how numbers are interpreted.
The text was updated successfully, but these errors were encountered: