Clean up README and remove openrlbenchmark dependency #2085
Conversation
|
|
||
| **SFT:** | ||
|
|
||
| ```bash | ||
| trl sft --model_name_or_path facebook/opt-125m --dataset_name stanfordnlp/imdb --output_dir opt-sft-imdb | ||
| trl sft --model_name_or_path Qwen/Qwen2.5-0.5B --dataset_name trl-lib/Capybara --output_dir Qwen2.5-0.5B-SFT |
There was a problem hiding this comment.
The OPT models are really outdated and I think we should use the small Qwen models where possible in our examples since they train fast and are pretty good.
| # get dataset | ||
| dataset = load_dataset("stanfordnlp/imdb", split="train") | ||
| # load dataset | ||
| dataset = load_dataset("trl-lib/Capybara", split="train") |
There was a problem hiding this comment.
I think we should use small, but good quality datasets for our SFT / RM / preference optimisation examples. The Capybara dataset is nice since it's just 16k samples of high quality
| # load dataset and preprocess | ||
| dataset = load_dataset("trl-lib/Capybara-Preferences", split="train") | ||
|
|
||
| def preprocess_function(examples): |
There was a problem hiding this comment.
Note to self: we should align the reward trainer to be the same as the other ones where this preprocessing step is internalised
| ) | ||
|
|
||
| # train | ||
| trainer.train() | ||
| ``` | ||
|
|
||
| ### `PPOTrainer` | ||
| ### `RLOOTrainer` |
There was a problem hiding this comment.
I switched to RLOO because (a) it is better than PPO and (b) we're deprecating PPOTrainer and I didn't want to use PPOv2Trainer which looks a bit experimental in name
| @@ -209,20 +265,11 @@ cd trl/ | |||
| make dev | |||
| ``` | |||
|
|
|||
| ## References | |||
There was a problem hiding this comment.
These are now in the docs so can be removed IMO
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
|
||
| # get trainer | ||
| # configure trainer | ||
| training_args = SFTConfig(output_dir="Qwen/Qwen2.5-0.5B-SFT") |
There was a problem hiding this comment.
I'm using the same convention adopted in #2082
Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
lewtun
changed the title
|
OK to merge this @qgallouedec ? |
|
A few simplifications and we're good to go |
Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
What does this PR do?
This PR updates the README to display working examples and removes a bunch of other outdated bits. It also removes the
openrlbenchmarkdependency that was not being used in the core code and caused conflict withmake devBefore submitting
Pull Request section?
to it if that's the case.
documentation guidelines.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.