Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optionally logging reference response #847

Merged
merged 11 commits into from
Oct 31, 2023

Conversation

vwxyzjn
Copy link
Contributor

@vwxyzjn vwxyzjn commented Oct 9, 2023

This PR allows us to log the reference response in wandb as well, which can help us make a direct comparison of the data.

https://wandb.ai/costa-huang/trl/runs/0m8ylgjy

image

tested it to work with PEFT as well (needed to refactor the logic a bit) see https://wandb.ai/costa-huang/trl/runs/10i8zl27

@vwxyzjn vwxyzjn marked this pull request as ready for review October 9, 2023 13:40
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Oct 9, 2023

The documentation is not available anymore as the PR was closed or merged.

Copy link
Member

@lvwerra lvwerra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally like the idea of adding reference responses. Since this touches a lot of PEFT logic I'll let @younesbelkada make sure it doesn't break anything :)

Copy link
Contributor

@younesbelkada younesbelkada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @vwxyzjn thanks a lot for iterating! I really like the optional_peft_ctx context manager!
I left a suggestion to use self.accelerator.unwrap_model once and make use of a local variable to pick up the correct ref model. What do you think?

trl/trainer/ppo_trainer.py Outdated Show resolved Hide resolved
trl/trainer/ppo_trainer.py Outdated Show resolved Hide resolved
trl/trainer/ppo_trainer.py Outdated Show resolved Hide resolved
trl/trainer/ppo_trainer.py Outdated Show resolved Hide resolved
trl/trainer/ppo_trainer.py Outdated Show resolved Hide resolved
trl/trainer/ppo_trainer.py Outdated Show resolved Hide resolved
trl/trainer/ppo_trainer.py Outdated Show resolved Hide resolved
vwxyzjn and others added 3 commits October 31, 2023 11:00
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
@vwxyzjn vwxyzjn requested a review from younesbelkada October 31, 2023 20:20
Copy link
Contributor

@younesbelkada younesbelkada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very clean! Thanks very much Costa for your work on this! 🙏

@vwxyzjn vwxyzjn merged commit 5b32372 into huggingface:main Oct 31, 2023
8 checks passed
lapp0 pushed a commit to lapp0/trl that referenced this pull request May 10, 2024
* Optionally logging reference response

* log ref rewards as welll

* peft logic re-write

* fix peft test case

* refactor

* push changes

* test

* Apply suggestions from code review

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* quick fix

* black

---------

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants