🪶 [GRPO] PPO Lite: Scale rewards by Std of Batch

[pramodith]

What does this PR do?

Updates the scale_rewards parameter to accept three different valid values none, group and batch to determine the level at which the std of rewards will be aggregated.

With this feature users can now train a model using the PPOLite technique outlined in https://arxiv.org/html/2508.08221v1, by setting scale_rewards = group and loss_type='bnpo' (for token level loss aggregration).

Fixes #3920

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[Copilot]

Pull Request Overview

This PR enhances the GRPO (Group Relative Policy Optimization) trainer by allowing flexible reward scaling strategies. It changes the scale_rewards parameter from a boolean to a string that accepts three values: "none", "group", and "batch", enabling different levels of standard deviation computation for reward normalization. This supports implementing the PPOLite technique from the referenced paper.

• Updates scale_rewards parameter to accept string values instead of boolean
• Implements batch-level reward scaling using global standard deviation across all processes
• Adds validation for the new parameter values and comprehensive test coverage

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
trl/trainer/grpo_trainer.py	Updates reward scaling logic to support group vs batch-level standard deviation computation
trl/trainer/grpo_config.py	Changes scale_rewards parameter type and adds validation for valid values
tests/test_grpo_trainer.py	Adds new test for batch-level reward scaling and updates existing test
docs/source/paper_index.md	Documents the PPOLite technique configuration example

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

very very nice!

[myron]

@pramodith thank you for adding!

in paper_index example, shouldn't the slale_rewards be 'batch' instead of 'group' ?

https://github.com/huggingface/trl/blob/main/docs/source/paper_index.md#part-i-tricks-or-traps-a-deep-dive-into-rl-for-llm-reasoning-lite-ppo

[pramodith]

@myron yes you're right! My bad, created a PR to fix that.