Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DPO] IPO Training loss #1022

Merged
merged 9 commits into from
Nov 24, 2023
Merged

[DPO] IPO Training loss #1022

merged 9 commits into from
Nov 24, 2023

Conversation

kashif
Copy link
Collaborator

@kashif kashif commented Nov 22, 2023

Implemented the IPO training loss from the paper:
A General Theoretical Paradigm to Understand Learning from Human Preferences
https://arxiv.org/pdf/2310.12036.pdf

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Nov 22, 2023 •
edited
Loading

The documentation is not available anymore as the PR was closed or merged.

younesbelkada
younesbelkada approved these changes Nov 23, 2023
View reviewed changes
Copy link
Contributor

@younesbelkada younesbelkada left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this great addition @kashif !

lewtun
lewtun approved these changes Nov 23, 2023
View reviewed changes
Copy link
Member

@lewtun lewtun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this so quickly @kashif, it's very elegant that you can now just swap out the losses for each method!

I left a nit about where to mention beta, but otherwise this LGTM

trl/trainer/dpo_trainer.py Outdated Show resolved Hide resolved
trl/trainer/dpo_trainer.py Show resolved Hide resolved
trl/trainer/dpo_trainer.py Outdated Show resolved Hide resolved
trl/trainer/dpo_trainer.py Show resolved Hide resolved
lvwerra
lvwerra reviewed Nov 23, 2023
View reviewed changes
Copy link
Member

@lvwerra lvwerra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good to me, left one comment

trl/trainer/dpo_trainer.py Outdated Show resolved Hide resolved
kashif and others added 3 commits November 23, 2023 12:30
lvwerra
lvwerra approved these changes Nov 24, 2023
View reviewed changes
Copy link
Member

@lvwerra lvwerra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@lvwerra lvwerra merged commit 55d7c95 into huggingface:main Nov 24, 2023
9 checks passed
@kashif kashif deleted the ipo branch November 24, 2023 14:53
lapp0 pushed a commit to lapp0/trl that referenced this pull request May 10, 2024
@kashif @lvwerra
[DPO] IPO Training loss (huggingface#1022)
e7b3503 
* initial IPO loss

* fix loss

* fixed comments

* added docs

* fix doc-strings

* add tests

* Update trl/trainer/dpo_trainer.py

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>

* fixes for review

* Added doc about beta in the Trainer's docstring

---------

Co-authored-by: Leandro von Werra <lvwerra@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@younesbelkada younesbelkada younesbelkada approved these changes

@lewtun lewtun lewtun approved these changes

@lvwerra lvwerra lvwerra approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@kashif @HuggingFaceDocBuilderDev @lvwerra @lewtun @younesbelkada