Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auto scaling rounding wrong way on occasion. #334

Closed
scott-griffiths opened this issue May 17, 2024 · 0 comments
Closed

Auto scaling rounding wrong way on occasion. #334

scott-griffiths opened this issue May 17, 2024 · 0 comments
Assignees
@scott-griffiths
Labels
🐞 bug 🐞 Priority-Medium
Milestone
4.2.3

Comments

@scott-griffiths
Copy link
Owner

At the moment the autoscaling algorithm uses

log2 = int(math.log2(max_float_value))

The int() rounds towards zero, but we want to find the largest power of two less than this max value. Therefore we should round to -inf and use math.floor.

Need to write some failing tests (perhaps compare with gfloat) and fix it.
@scott-griffiths scott-griffiths added this to the 4.2.3 milestone May 17, 2024
@scott-griffiths scott-griffiths self-assigned this May 17, 2024
scott-griffiths added a commit that referenced this issue May 18, 2024
@scott-griffiths
Fix for Dtype auto scaling algorithm rounding the wrong way.
7611fc6 
For max values < 1 it was rounding the log towards zero instead of towards -infinity. Bug #334.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@scott-griffiths scott-griffiths

Labels
🐞 bug 🐞 Priority-Medium
Projects
None yet
Milestone
4.2.3
Development

No branches or pull requests
1 participant
@scott-griffiths