[algo] feat: add GSPO-token policy loss computation function

[0x404]

What does this PR do?

This PR implements the GSPO-token policy loss calculation proposed by paper https://arxiv.org/pdf/2507.18071

Test

Compared GRPO and GSPO under the same settings. GRPO uses the following script:

python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=$HOME/data/gsm8k/train.parquet \
    data.val_files=$HOME/data/gsm8k/test.parquet \
    data.train_batch_size=512 \
    data.max_prompt_length=512 \
    data.max_response_length=1024 \
    data.filter_overlong_prompts=True \
    data.truncation='error' \
    actor_rollout_ref.model.path=Qwen/Qwen2.5-3B-Instruct \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=128 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.entropy_coeff=0 \
    actor_rollout_ref.actor.policy_loss.loss_mode="vanilla" \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
    actor_rollout_ref.rollout.n=10 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.use_kl_in_reward=False \
    trainer.critic_warmup=0 \
    trainer.logger='["console","wandb"]' \
    trainer.project_name='verl_gspo_cmp' \
    trainer.experiment_name='qwen2.5-3B-GRPO' \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=1 \
    trainer.save_freq=20 \
    trainer.test_freq=5 \
    trainer.total_epochs=15 $@

GSPO uses the following script:

python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=$HOME/data/gsm8k/train.parquet \
    data.val_files=$HOME/data/gsm8k/test.parquet \
    data.train_batch_size=512 \
    data.max_prompt_length=512 \
    data.max_response_length=1024 \
    data.filter_overlong_prompts=True \
    data.truncation='error' \
    actor_rollout_ref.model.path=Qwen/Qwen2.5-3B-Instruct \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=128 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.entropy_coeff=0 \
    actor_rollout_ref.actor.policy_loss.loss_mode="gspo" \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
    actor_rollout_ref.rollout.n=10 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.use_kl_in_reward=False \
    trainer.critic_warmup=0 \
    trainer.logger='["console","wandb"]' \
    trainer.project_name='verl_gspo_cmp' \
    trainer.experiment_name='qwen2.5-3B-GRPO' \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=1 \
    trainer.save_freq=20 \
    trainer.test_freq=5 \
    trainer.total_epochs=15 $@

API and Usage Example

To use GSPO, users only need to set actor_rollout_ref.actor.policy_loss.loss_mode to gspo.

python3 -m verl.trainer.main_ppo \
  ... \
  actor_rollout_ref.actor.policy_loss.loss_mode="gspo" \
  ...

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
• Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

[gemini-code-assist]

Code Review

This pull request introduces the GSPO policy loss function. I've identified potential division-by-zero and floating-point overflows that could disrupt training. Addressing these will make the implementation more robust.

[vermouth1992]

Could you create a folder under recipe named gspo, and add two recipe script using QWen2.5 7b math and Qwen3 30b a3b following this?

• https://github.com/volcengine/verl/blob/main/recipe/dapo/test_dapo_7b_math.sh
• https://github.com/volcengine/verl/blob/main/recipe/dapo/test_dapo_qwen3_30b_math.sh

[0x404]

Could you create a folder under recipe named gspo, and add two recipe script using QWen2.5 7b math and Qwen3 30b a3b following this?

Of course, working on it.

[vermouth1992]

Here agg_loss should always be sentence-level?

[0x404]

Yes, "seq-mean-token-sum" looks most suitable. But from the paper's definition, maybe a "batch-mean-group-mean-token-sum" would be more accurate? Which sums the token loss of seq, then takes a mean of group seq loss at the group-level, then takes a mean/sum again at the batch level. there is a similar discussion in #2776

[mokeevdmitrii]

This is GSPO-token, so it should be exactly seq-mean-token-mean.

EDIT: hope i haven't hallucinated in my nightly calcs

The example in #2776 is about token-mean vs seq-mean).

Referencing the article:

1. First, we calc $s_i(\theta)$ and find token-level loss (inside the last sum, let's name it $pg_{i,t}$.

2. Second, we sum this loss over every sequence and divide it by sequence length $|y_i|$. This equals to:

seq_losses = torch.sum(loss_mat * loss_mask, dim=-1) / torch.sum(loss_mask, dim=-1)

in current verl implementation for seq-mean-token-mean. Let's name this $pg_{i}$.

3. Finally, we notice that there is a common denumerator, as the loss for a single group is $\dfrac{1}{G} \sum\limits_{i=1}^G pg_i$. So, "batch-mean-group-mean-token-mean" is not needed, as each "group-mean" operation has the same denumerator - this is equiv. to ""batch-group"-mean-token-mean" or simply "seq-mean-token-mean".

Notes:

1. seq-mean-token-sum would be wrong, as we are summing up $s_i(\theta)$ for each token in GSPO-token, so we must divide by seq-len here.

2. the fact that token-mean is a valid loss_agg_mode value here makes me a bit sad bcs the article is about sequence-level optimization)

[0x404]

@mokeevdmitrii You are right! we should use seq-mean-token-mean when using GSPO, thanks very much!

[CLAassistant]

All committers have signed the CLA.

[BounharAbdelaziz]

Just added the implementation of the sequence level optimization (GSPO vanilla) and improved the GSPO-token implementation by removing some redundant clamp for negative_approx_kl (at, I believe wrong late step).

I also added some details about the equations we are calculating at each step.

Will be sharing some xp results once ready.

[mokeevdmitrii]

This PR uses default cliprange settings (0.2 default as I am writing this comment, however the authors mention

that the clipping ranges in GSPO and
in previous algorithms (e.g., GRPO) typically differ in order of magnitude due to the distinct definitions
of importance ratios

I think this is worth testing if possible and (at least) mentioning this fact to the user of "loss_mode: gspo"

[BounharAbdelaziz]

This PR uses default cliprange settings (0.2 default as I am writing this comment, however the authors mention

that the clipping ranges in GSPO and
in previous algorithms (e.g., GRPO) typically differ in order of magnitude due to the distinct definitions
of importance ratios

I think this is worth testing if possible and (at least) mentioning this fact to the user of "loss_mode: gspo"

Indeed, it’s recommended (https://x.com/ChujieZheng/status/1948933507696525392) to be between 3e-4 and 4e-4.

[0x404]

Hi @BounharAbdelaziz, do you think we really need to keep both GSPO and GSPO-token?

For sentence-level GSPO, it shouldn't need seq_importance_ratio to do expand, but rather directly multiply with sentence-level advantage. However, our advantage here is token-level.

I think we might only need to implement GSPO-token, because GSPO-token and GSPO are equivalent (when the advantage has equal elements across the seq_length dimension). What do you think?

[chujiezheng]

I prefer GSPO-token, as it is theoretically equal to GSPO and offers higher flexibility for future potential features

[BounharAbdelaziz]

Hi @0x404, you are right. They fall into the same and you can technically make GSPO-token=GSPO by setting A_{i,t}=A_i. I'm pushing the update.

Thank you @chujiezheng for your input!

[BounharAbdelaziz]

@0x404 I see that you already did remove it, thanks!

[0x404]

I think this is worth testing if possible

results based on commit 4b247b2, GSPO using 4e-4 cliprange settings:

python3 -m verl.trainer.main_ppo \
    algorithm.adv_estimator=grpo \
    data.train_files=$HOME/data/gsm8k/train.parquet \
    data.val_files=$HOME/data/gsm8k/test.parquet \
    data.train_batch_size=512 \
    data.max_prompt_length=512 \
    data.max_response_length=1024 \
    data.filter_overlong_prompts=True \
    data.truncation='error' \
    actor_rollout_ref.model.path=Qwen/Qwen2.5-3B-Instruct \
    actor_rollout_ref.actor.optim.lr=1e-6 \
    actor_rollout_ref.model.use_remove_padding=True \
    actor_rollout_ref.actor.ppo_mini_batch_size=128 \
    actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.actor.use_kl_loss=True \
    actor_rollout_ref.actor.kl_loss_coef=0.001 \
    actor_rollout_ref.actor.kl_loss_type=low_var_kl \
    actor_rollout_ref.actor.entropy_coeff=0 \
    actor_rollout_ref.actor.policy_loss.loss_mode="gspo_token" \
    actor_rollout_ref.actor.clip_ratio_low=0.0004 \
    actor_rollout_ref.actor.clip_ratio_high=0.0004 \
    actor_rollout_ref.actor.clip_ratio=0.0004 \
    actor_rollout_ref.model.enable_gradient_checkpointing=True \
    actor_rollout_ref.actor.fsdp_config.param_offload=False \
    actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
    actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
    actor_rollout_ref.rollout.name=vllm \
    actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
    actor_rollout_ref.rollout.n=10 \
    actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=40 \
    actor_rollout_ref.ref.fsdp_config.param_offload=True \
    algorithm.use_kl_in_reward=False \
    trainer.critic_warmup=0 \
    trainer.logger='["console","wandb"]' \
    trainer.project_name='verl_gspo_cmp' \
    trainer.experiment_name='qwen2.5-3B-GSPO-clip1e-4' \
    trainer.n_gpus_per_node=8 \
    trainer.nnodes=1 \
    trainer.save_freq=20 \
    trainer.test_freq=5 \
    trainer.total_epochs=15 $@

[chujiezheng]

Thanks for your support of the GSPO algorithm!

We have updated the experimental details in the GSPO paper: https://arxiv.org/abs/2507.18071

I would suggest to try the following hyperparameters:

• actor_rollout_ref.actor.clip_ratio_low=3e-4
• actor_rollout_ref.actor.clip_ratio_high=4e-4

You may also try to turn off the KL loss by setting actor_rollout_ref.actor.kl_loss_coef=0

[0x404]

@chujiezheng Thanks, will try it out!

[chujiezheng]

Thanks for your test.

Based on the feedback from friends in other labs, the advantage of GSPO over GRPO (w/ Routing Replay) is mainly manifested on MoE models, while they perform similarly on dense models.

We did not do much test for GSPO on dense models internally. But our experience shows that dense models are much easier to RL train. So the results you obtain may be expected.

[chujiezheng]

Also, we did not do test in the zero setting, and we previously found that zero RL is also much easier than in the cold-start setting (perhaps because the response length in the zero setting is usually shorter).

[vermouth1992]

This clamp is unnecessary? Could you please confirm that @chujiezheng Thanks!

[chujiezheng]

Agreed it is unnecessary here

[chujiezheng]

You should change the calculation order to avoid precision error:

log_seq_importance_ratio = log_prob - log_prob.detach() + negative_approx_kl_seq.detach().unsqueeze(-1)

[0x404]

Thanks @chujiezheng ! all resolved.

[BounharAbdelaziz]

Also, we did not do test in the zero setting, and we previously found that zero RL is also much easier than in the cold-start setting (perhaps because the response length in the zero setting is usually shorter).

Great! Thanks a lot for sharing your insights!

[vermouth1992]

I guess this PR should be ready for merge?

[BounharAbdelaziz]

I did run an experiment using GSM8K on GSPO-token vs GRPO using the following "core" parameters:

GSPO-token:

clip_ratio_low=0.0003 # as recommended by the paper, see Sec. 5.1
clip_ratio_high=0.0004 # as recommended by the paper, see Sec. 5.1
train_batch_size=512
ppo_mini_batch_size=128 # maintain 4 mini-batches as recommended by the paper, see Sec. 5.1
ppo_micro_batch_size_per_gpu=8 # setup depending on your GPU memory
n_resp_per_prompt=16

use_kl_in_reward=false
kl_coef=0.0
use_kl_loss=false
kl_loss_coef=0.0

max_prompt_length=$((1024 * 2))
max_response_length=$((1024 * 8)) # yes this is a lot, and actually the response length mean converges towards 25x

GRPO shares the same setup with the main differences being:

clip_ratio_low=0.2
clip_ratio_high=0.28

Observations

Both algorithms tend to perform very similarly on the test set. Though, it seems like GRPO is learning a bit faster.

When examining the pg clipping factor, we can observe that it's huge sitting at 30% (double the one mentionned in the paper) and stabelizes around 10% while GRPO almost never clips, 0.03% on average. I believe that one need to tune the ϵ carrefully, perhaps @chujiezheng can share some insights on this?

With the above observations, I am currently running two other experiments for GSPO-token with clip_ratio_high=6e-4and clip_ratio_high=3.5e-4. The goal is to assess whether allowing more clipping might help, or if tighter clipping thresholds improve performance, though higher thresholds could increase the clipping factor even further.

Logs are attached for reference.

Sharing my experiments with the following two extra test for GSPO-token:

clip_ratio_low=0.0003
clip_ratio_high=0.00035

clip_ratio_low=0.0003
clip_ratio_high=0.0006

Results: no noticible difference. This joins the shared insights by @chujiezheng, i.e. GSPO is more suitable for MoEs.

[xihuai18]

Can someone explain why pg_clipfrac_lower is not used?

    # For compatibility, return zero for pg_clipfrac_lower (not used in standard GSPO)
    pg_clipfrac = verl_F.masked_mean(torch.gt(pg_losses2, pg_losses1).float(), response_mask)
    pg_clipfrac_lower = torch.tensor(0.0, device=pg_loss.device)

[TristanMeng]

should use Dual-clip ?