-
-
Notifications
You must be signed in to change notification settings - Fork 46.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add perplexity loss algorithm #10718
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did the code passed the pre commit tests ??
This comment was marked as outdated.
This comment was marked as outdated.
>>> y_pred = np.array( \ | ||
[[[0.28, 0.19, 0.21 , 0.15, 0.15], \ | ||
[0.24, 0.19, 0.09, 0.18, 0.27]], \ | ||
[[0.03, 0.26, 0.21, 0.18, 0.30], \ | ||
[0.28, 0.10, 0.33, 0.15, 0.12]]]\ | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PEP8: Backslash line continuation should be avoided in Python.
>>> y_pred = np.array( \ | |
[[[0.28, 0.19, 0.21 , 0.15, 0.15], \ | |
[0.24, 0.19, 0.09, 0.18, 0.27]], \ | |
[[0.03, 0.26, 0.21, 0.18, 0.30], \ | |
[0.28, 0.10, 0.33, 0.15, 0.12]]]\ | |
) | |
>>> y_pred = np.array( | |
... [[[0.28, 0.19, 0.21 , 0.15, 0.15], | |
... [0.24, 0.19, 0.09, 0.18, 0.27]], | |
... [[0.03, 0.26, 0.21, 0.18, 0.30], | |
... [0.28, 0.10, 0.33, 0.15, 0.12]]], | |
... ) |
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All loss function files were consolidated into machine_learning/loss_functions.py
in #10737. Could you move your new code into that file?
machine_learning/loss_functions.py
Outdated
# Add small constant to avoid getting inf for log(0) | ||
epsilon = 1e-7 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please make epsilon
an optional function parameter so that users can change its value
machine_learning/loss_functions.py
Outdated
# Getting the matrix containing prediction for only true class | ||
true_class_pred = np.sum(y_pred * filter_matrix, axis=2) | ||
|
||
# Calculating perplexity for each sentence | ||
perp_losses = np.exp( | ||
np.negative(np.mean(np.log(true_class_pred + epsilon), axis=1)) | ||
) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# Getting the matrix containing prediction for only true class | |
true_class_pred = np.sum(y_pred * filter_matrix, axis=2) | |
# Calculating perplexity for each sentence | |
perp_losses = np.exp( | |
np.negative(np.mean(np.log(true_class_pred + epsilon), axis=1)) | |
) | |
# Getting the matrix containing prediction for only true class | |
# Clip values to avoid log(0) | |
true_class_pred = np.sum(y_pred * filter_matrix, axis=2).clip(epsilon, 1) | |
# Calculating perplexity for each sentence | |
perp_losses = np.exp(np.negative(np.mean(np.log(true_class_pred), axis=1))) |
You can use .clip()
to restrict the range of the array's values instead of adding epsilon
to every entry. This way, only the problematic values get changed while OK values remain the same. Note that this may change the value of your doctests.
Describe your change:
Checklist: