Skip to content

fix(rollout): normalize fan-out rewards by trajectory - #2302

Open
sxl1993 wants to merge 1 commit into
THUDM:mainfrom
sxl1993:fix/fanout-reward-normalization
Open

fix(rollout): normalize fan-out rewards by trajectory#2302
sxl1993 wants to merge 1 commit into
THUDM:mainfrom
sxl1993:fix/fanout-reward-normalization

Conversation

@sxl1993

@sxl1993 sxl1993 commented Aug 21, 2026

Copy link
Copy Markdown

Summary

  • count each unique rollout once in the custom fan-out GRPO baseline
  • broadcast the normalized trajectory reward to all emitted samples
  • reject inconsistent sibling rewards and handle singleton groups without NaNs

Motivation

For rewards [1, 1, 1, 0], where the first three samples belong to one rollout, the old hook used a sample-weighted baseline of 0.75. This change computes the trajectory-level baseline from [1, 0] as 0.5 and broadcasts [0.5, 0.5, 0.5, -0.5].

The default reward post-processing path is unchanged.

Test

  • added a CPU regression case to tests/test_sample.py

Count each rollout once in the custom GRPO baseline.

Broadcast the normalized reward to all samples it emitted.

Co-Authored-By: sxl1993 <121819792@qq.com>
@sxl1993
sxl1993 marked this pull request as ready for review August 21, 2026 08:55
@sxl1993

sxl1993 commented Aug 22, 2026

Copy link
Copy Markdown
Author

@zhuzilin Please review #2302. It fixes reward normalization for fan-out GRPO by normalizing per trajectory. The main thing to check is the rollout_id grouping logic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant