docs: Extend CLI basic usage examples to all supported CLIs #4425
+279
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Resolves huggingface#4378 - Add GRPO CLI examples with trl-lib/ultrafeedback-prompt dataset - Add RLOO CLI examples with AI-MO/NuminaMath-TIR dataset - Add KTO CLI examples with trl-lib/kto-mix-14k dataset - Add examples to all sections: Basic Usage, Config Files, Accelerate, accelerate_config, and dataset mixtures - Ensure parity in documentation coverage across all 6 training CLIs - Verified CLI commands exist in trl/cli.py (lines 47-51) - Verified datasets match official examples in examples/scripts/
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Member
sergiopaniego
left a comment
sergiopaniego
left a comment
There was a problem hiding this comment.
We could simplify the <hfoptions> by giving all of them the same id, so once you click on your preferred trainer, all of the options switch to it.
For the ... inline - ... w/ config file cases, we can expand these sections, concatenating both of them and only using the trainer name, so in the options we'd only use the trainer names.
behroozazarkhalili
added a commit
that referenced
this pull request
Nov 3, 2025
Addresses review comments on PR #4425: 1. Unified all <hfoptions> IDs to 'trainers' (5 sections) - Enables persistent trainer selection across all documentation sections - Improved user experience when navigating between examples 2. Consolidated inline/config file options (12 pairs → 12 single options) - Merged separate "inline" and "w/ config file" options - Used "Or with config file:" pattern for clarity - Applied to Sections 3 (Scaling) and 4 (Accelerate Config) Result: 30 balanced options (6 trainers × 5 sections) with improved navigation
qgallouedec
reviewed
Nov 4, 2025
Comment on lines
+60
to
+72
| trl grpo \ | ||
| --model_name_or_path Qwen/Qwen2.5-0.5B \ | ||
| --dataset_name trl-lib/ultrafeedback-prompt | ||
| ``` | ||
|
|
||
| </hfoption> | ||
| <hfoption id="RLOO"> | ||
|
|
||
| ```bash | ||
| trl rloo \ | ||
| --model_name_or_path Qwen/Qwen2.5-0.5B \ | ||
| --dataset_name AI-MO/NuminaMath-TIR | ||
| ``` |
Member
There was a problem hiding this comment.
they both miss a reward function or model
qgallouedec
reviewed
Nov 4, 2025
Comment on lines
+138
to
+157
| ```yaml | ||
| # grpo_config.yaml | ||
| model_name_or_path: Qwen/Qwen2.5-0.5B | ||
| dataset_name: trl-lib/ultrafeedback-prompt | ||
| ``` | ||
| Launch with: | ||
| ```bash | ||
| trl grpo --config grpo_config.yaml | ||
| ``` | ||
|
|
||
| </hfoption> | ||
| <hfoption id="RLOO"> | ||
|
|
||
| ```yaml | ||
| # rloo_config.yaml | ||
| model_name_or_path: Qwen/Qwen2.5-0.5B | ||
| dataset_name: AI-MO/NuminaMath-TIR | ||
| ``` |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #4378
Summary
Extends the CLI documentation to include basic usage examples for GRPO, RLOO, and KTO, achieving parity with the existing SFT, DPO, and Reward examples.
Changes
trl-lib/ultrafeedback-promptdatasetAI-MO/NuminaMath-TIRdatasettrl-lib/kto-mix-14kdatasetAll three CLIs now have examples in every documentation section:
--accelerate_configVerification
trl/cli.py(lines 47-51)examples/scripts/hfoptionsblocks properly balanced)Testing
Documentation structure validated using Python script to verify: