-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optionally logging reference response #847
Conversation
The documentation is not available anymore as the PR was closed or merged. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally like the idea of adding reference responses. Since this touches a lot of PEFT logic I'll let @younesbelkada make sure it doesn't break anything :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @vwxyzjn thanks a lot for iterating! I really like the optional_peft_ctx
context manager!
I left a suggestion to use self.accelerator.unwrap_model
once and make use of a local variable to pick up the correct ref model. What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very clean! Thanks very much Costa for your work on this! 🙏
* Optionally logging reference response * log ref rewards as welll * peft logic re-write * fix peft test case * refactor * push changes * test * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * quick fix * black --------- Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
This PR allows us to log the reference response in wandb as well, which can help us make a direct comparison of the data.
https://wandb.ai/costa-huang/trl/runs/0m8ylgjy
tested it to work with PEFT as well (needed to refactor the logic a bit) see https://wandb.ai/costa-huang/trl/runs/10i8zl27