New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Does load-balanced loss help the loss converge？ #391

Open

mathfinder opened this issue Jun 13, 2023 · 0 comments

mathfinder commented Jun 13, 2023

mesh/mesh_tensorflow/transformer/moe.py

Line 935 in fbf7b1e

# LOAD BALANCING LOSS

I try load-balanced loss in my project and find load-balanced loss does not help loss converge.

Does it only balance the load, but does not help the loss convergence, or even slightly hurt the model?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment