Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

logit function in backend/lite/utils.py is not good near 1/2. #694

Open
Schaechtle opened this issue Oct 29, 2018 · 0 comments
Open

logit function in backend/lite/utils.py is not good near 1/2. #694

Schaechtle opened this issue Oct 29, 2018 · 0 comments

Comments

@Schaechtle
Copy link
Collaborator

Found by Taylor, who writes:
"I realized the other day that the logit function I put in backend/lite/utils.py is not good near 1/2. For inputs between logistic(-1) = e^{-1}/(1 + e^{-1}) and logistic(+1) = 1/(1 + e^{-1}), it should be evaluated using the identity log(1/(1 - p)) = -log((1 - p)/p) = -log(1 + (1 - p)/p - 1) = -log(1 + (1 - p - p)/p) = -log(1 + (1 - 2p)/p) which can be computed safely using the log1p function by -log1p((1 - 2p)/p). The error of evaluating log(1/(1 - p)) directly can be quite large, while the error of evaluating -log1p((1 - 2p)/p) is bounded by 10eps. E.g., for p = .499999999, naive evaluation of log(1/(1 - p)) is wrong in about half the digits. I sketched some proofs and wrote a few more tests here:

(I chose to evaluate logistic(x) by e^x/(1 + e^x) for negative inputs there, but on reflection I see that it doesn't make a difference: for positive or negative inputs, the relative error of 1/(1 + e^{-x}) is bounded by 8eps + 4eps^2, since e^{-x}/(1 + e^{-x}) <= 2, which is exactly the same bound I computed for e^x/(1 + e^x). Maybe I'll just remove that case and reduce the amount of math to verify.)"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant