docs: Unify model examples to use trl-lib namespace

[behroozazarkhalili]

Summary

Unifies model namespace usage in documentation examples to use the common trl-lib namespace as requested in issue #4385.

Resolves #4385

Changes

Files Modified:

• docs/source/peft_integration.md - 1 model reference updated
• docs/source/use_model.md - 3 model references updated

Replacements:

• edbeeching/gpt-neo-125M-imdb → trl-lib/Qwen2-0.5B-XPO
• kashif/stack-llama-2 → trl-lib/Qwen2-0.5B-XPO

All personal developer namespace models in documentation examples now use the unified trl-lib namespace. Official organization models (meta-llama, microsoft, google, etc.) and research project references (cleanrl) are intentionally preserved as they serve specific purposes.

Verification

• ✅ All occurrences of personal developer namespaces replaced
• ✅ Replacement model (trl-lib/Qwen2-0.5B-XPO) is already widely used in TRL documentation
• ✅ No other personal developer namespace models found in docs
• ✅ Official and research project namespaces appropriately preserved

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

this one is missing:

$ python examples/scripts/evals/judge_tldr.py --model_name_or_path vwxyzjn/ppo_tldr --judge_model gpt-4o-mini --num_examples 1000

and all of these:

trl/examples/scripts/evals/judge_tldr.py

Lines 34 to 50 in 6f906d5

	python examples/scripts/evals/judge_tldr.py --model_name_or_path vwxyzjn/rloo_tldr --num_examples 1000
	Model win rate: 31.40%
	python examples/scripts/evals/judge_tldr.py --model_name_or_path vwxyzjn/rloo_tldr --judge_model gpt-3.5-turbo-0125 --num_examples 1000
	Model win rate: 51.60%
	python examples/scripts/evals/judge_tldr.py --model_name_or_path vwxyzjn/rloo_tldr --judge_model gpt-4o-mini --num_examples 1000
	Model win rate: 51.20%
	python examples/scripts/evals/judge_tldr.py --model_name_or_path vwxyzjn/ppo_tldr --num_examples 1000
	Model win rate: 46.30%
	python examples/scripts/evals/judge_tldr.py --model_name_or_path vwxyzjn/ppo_tldr --judge_model gpt-3.5-turbo-0125 --num_examples 1000
	Model win rate: 52.50%
	python examples/scripts/evals/judge_tldr.py --model_name_or_path vwxyzjn/ppo_tldr --judge_model gpt-4o-mini --num_examples 1000
	Model win rate: 63.00%

but they'll require train and push a model to the org trl-lib

[behroozazarkhalili]

@qgallouedec I've addressed the comments:

✅ Completed:

• Updated docs/source/use_model.md to use Qwen/Qwen3-0.6B (3 locations)
• Confirmed docs/source/peft_integration.md already reverted to edbeeching/gpt-neo-125M-imdb

⏳ Pending (blocked):

• examples/scripts/evals/judge_tldr.py updates: As you noted, this requires training and pushing PPO/RLOO TLDR models to the trl-lib org first. The current examples use vwxyzjn/rloo_tldr and vwxyzjn/ppo_tldr which don't have trl-lib equivalents yet.

Should we:

1. Keep the current examples until trl-lib models are available?
2. Track this as a follow-up issue once models are trained?
3. Use alternative trl-lib models (though they're different architectures)?