New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add scaler.unscale_(optimizer) before clip_grad_norm_ #1196

Open

nickyi1990 opened this issue Oct 27, 2023 · 0 comments

nickyi1990 commented Oct 27, 2023 •

edited

Loading

examples/distributed/minGPT-ddp/mingpt/trainer.py

Line 103 in c67bbab

    
           torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.grad_norm_clip)

according to this link , we should add unscale before it when we need to modify or inspect gradients

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment