Skip to content

feat(v1): add Trace.reward_shaping for trainer reward-shaping terms - #2436

Draft
hallerite wants to merge 1 commit into
mainfrom
feat/trace-reward-shaping
Draft

feat(v1): add Trace.reward_shaping for trainer reward-shaping terms#2436
hallerite wants to merge 1 commit into
mainfrom
feat/trace-reward-shaping

Conversation

@hallerite

@hallerite hallerite commented Aug 24, 2026

Copy link
Copy Markdown
Member

Summary

Adds Trace.reward_shaping: dict[str, float] — additive training-reward terms a consumer's reward shapers record on a trace, keyed by shaper name (e.g. {"length_penalty": -0.1}).

This is a training annotation in the same sense as MessageNode.advantages: verifiers holds the field, the trainer fills it. Trace.reward stays the env's verdict; the trainer scores against reward + sum(reward_shaping.values()). Keeping the two apart means task-success metrics, curricula and evals keep reading the unshaped reward while credit assignment sees the shaped one.

Companion: PrimeIntellect-ai/prime-rl#3376 moves prime-rl's length penalty out of GRPO into a reward-shaping layer and writes its terms here.

🤖 Generated with Claude Code

Note

Add reward_shaping field to Trace model for trainer reward-shaping terms

Adds a reward_shaping: dict[str, float] field to the Trace pydantic model, defaulting to an empty dict. It stores additive training-reward terms keyed by shaper name. The field is included in JSON output from to_record() since it is not excluded.

Macroscope summarized e084c75. (Automatic summaries will resume when PR exits draft mode or review begins).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hallerite
hallerite force-pushed the feat/trace-reward-shaping branch from e084c75 to fd50b58 Compare August 24, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant