Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Precision-correct normalization factor may exceed representation range #115

Closed
eyalroz opened this issue Feb 5, 2022 · 0 comments
Closed

Precision-correct normalization factor may exceed representation range #115

eyalroz opened this issue Feb 5, 2022 · 0 comments
Assignees
@eyalroz
Labels
bug Something isn't working resolved-on-develop A changeset fixing this issue has been commiutted to the development branch testing Only relevant to the testing programs rather than the main code itself

Comments

@eyalroz
Copy link
Owner

eyalroz commented Feb 5, 2022

We "normalize" our floating-point values to get their power-of-10 to 1, so that we are left with the base-10 mantissa as a number. This works fine. However, if we also require a precision value above 0, we multiply the normalization factor to also account for the precision, so that instead of, say, 1.234, for precision 2 we would get 123.4 , and can return the integral part of that value.

Now, if the original number is close to the edge of the representation range ( ~10^308) - the extra factor for the precision can bring it past the range, resulting in infinity, and messing up the rest of the calculation.

Unfortunately, there's no unit test coverage for this situation; so, we should add one and address it.
@eyalroz eyalroz added bug Something isn't working testing Only relevant to the testing programs rather than the main code itself labels Feb 5, 2022
@eyalroz eyalroz self-assigned this Feb 5, 2022
eyalroz added a commit that referenced this issue Feb 5, 2022
@eyalroz
Fixes #114, #115: Special-casing powers close to the extremal ones.
50cd7a8 
* Special-casing the power-of-10 function for 10^308 and 10^-308.
* Special-casing `get_normalized_components()` to not strive for better precision for near-extremal powers and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
eyalroz added a commit that referenced this issue Feb 5, 2022
@eyalroz
Fixes #114, #115: Special-casing powers close to the extremal ones.
8aee83a 
* Special-casing the power-of-10 function for 10^308 and 10^-308.
* Special-casing `get_normalized_components()` to not strive for better precision for near-extremal powers and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
eyalroz added a commit that referenced this issue Feb 6, 2022
@eyalroz
Fixes #114, #115: Special-casing powers close to the extremal ones.
10aca98 
* Special-casing the power-of-10 function for 10^308 and 10^-308.
* Special-casing `get_normalized_components()` to not strive for better precision for near-extremal powers and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
eyalroz added a commit that referenced this issue Feb 7, 2022
@eyalroz
Fixes #114, #115: Better handling of powers near the extrema of the r…
e91b7e5 
…epresentation range.

* Special-casing the power-of-10 function for 10^-308.
* Altered `get_normalized_components()` to not strive for better precision for near-extremal powers (at both ends) and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
eyalroz added a commit that referenced this issue Feb 7, 2022
@eyalroz
Fixes #114, #115: Better handling of powers near the extrema of the r…
e0ef6ca 
…epresentation range.

* Special-casing the power-of-10 function for 10^-308.
* Altered `get_normalized_components()` to not strive for better precision for near-extremal powers (at both ends) and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
eyalroz added a commit that referenced this issue Feb 7, 2022
@eyalroz
Fixes #114, #115: Better handling of powers near the extrema of the r…
6e16e71 
…epresentation range.

* Special-casing the power-of-10 function for 10^-308.
* Altered `get_normalized_components()` to not strive for better precision for near-extremal powers (at both ends) and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
eyalroz added a commit that referenced this issue Feb 10, 2022
@eyalroz
Fixes #114, #115: Better handling of powers near the extrema of the r…
52e9999 
…epresentation range.

* Special-casing the power-of-10 function for 10^-308.
* Altered `get_normalized_components()` to not strive for better precision for near-extremal powers (at both ends) and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
eyalroz added a commit that referenced this issue Feb 11, 2022
@eyalroz
Fixes #114, #115: Better handling of powers near the extrema of the r…
2b85ce1 
…epresentation range.

* Special-casing the power-of-10 function for 10^-308.
* Altered `get_normalized_components()` to not strive for better precision for near-extremal powers (at both ends) and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
@eyalroz eyalroz added the resolved-on-develop A changeset fixing this issue has been commiutted to the development branch label Feb 20, 2022
eyalroz added a commit that referenced this issue Feb 21, 2022
@eyalroz
Fixes #114, #115: Better handling of powers near the extrema of the r…
f83521b 
…epresentation range.

* Special-casing the power-of-10 function for 10^-308.
* Altered `get_normalized_components()` to not strive for better precision for near-extremal powers (at both ends) and settle for the "rougher" way to obtain the integral and decimal parts.
* Added a test-case of a near-extremal-power which fails due to the re-normalization even if the powering is correct.
@eyalroz eyalroz closed this as completed Mar 4, 2022
kuba2k2 pushed a commit to libretiny-eu/library-printf that referenced this issue Feb 25, 2023
@eyalroz
Fixes eyalroz#115: Unifying _ntoa_long and _ntoa_long_long into a…
b232802 
… single `ntoa` function.

* Defined an `NTOA_VALUE_TYPE` based on whether `long long` is supported.
* Using an `NTOA_ABS()` macro instead of repeating the one-liner code used earlier; and it eagerly casts to avoid problems with extremal negative values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eyalroz eyalroz

Labels
bug Something isn't working resolved-on-develop A changeset fixing this issue has been commiutted to the development branch testing Only relevant to the testing programs rather than the main code itself
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@eyalroz