-
-
Notifications
You must be signed in to change notification settings - Fork 100
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sigmoid with residue #869
Sigmoid with residue #869
Conversation
PReLU forward and backward now working properly
Actually runs now
Changed some defaults to match Pytorch
Now gradients wont be set to 0 for large inputs
Now Sigmoidr layer can be used
right now it returns a minimum gradient of 0.0001, this maybe should be configurable. |
Since this can be accomplished by doing Thanks for the PR! |
This is not entirely right. While the forward pass is the same, the backwards pass is different. It makes sure that the gradients through the function are always at least some small epsilon. As in, if |
I don't know if this is something that should be in dfdx but it is useful for my use case where I need my model to return probabilities between 0 and 1, where each input is independent of all the others. If I use normal sigmoid I get vanishing gradients, so I made this.