feat(rl): plumb GRPO flags to the agentic learner

[entrpn]

Description

Requires google/tunix#2061 to be merged first.

force_on_policy_ratio, use_rollout_logps, log_sampler_trainer_agreement
and loss_agg_mode were never passed into AgenticGrpoConfig, so setting
them on the command line had no effect and the learner silently used
dataclass defaults.

loss_agg_mode in particular was plumbed for the non-agentic branch only,
so any recent A/B of that flag through the agentic path was measuring the
default rather than the requested value.

Also logs the fully resolved GRPO config at startup, which is how the
above was found.

Tests

Measured on qwen3-4b (OpenMathInstruct-2, agentic rollout):

clipfrac 0.0006 -> 0.0000
log_ratio_abs 0.0157 -> 0.0000
step time 984s -> 742s (~24%)
Gradient magnitude is essentially unchanged; clipping was only firing on
~0.06% of tokens, all of it kernel noise rather than policy drift.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[gemini-code-assist]

Code Review

This pull request adds new configuration fields (use_rollout_logps, force_on_policy_ratio, and log_sampler_trainer_agreement) to the RL BaseModel and propagates them to the GRPO configuration in train_rl.py, along with logging the resolved GRPO configuration. The feedback suggests simplifying the multi-line formatting of the log_sampler_trainer_agreement argument in the GRPO configuration instantiation to a single line.

[gemini-code-assist]

The boolean argument log_sampler_trainer_agreement is unnecessarily wrapped in parentheses and split across multiple lines. It can be simplified to a single line to match the style of the other arguments in this call.

        log_sampler_trainer_agreement=trainer_config.rl.log_sampler_trainer_agreement,

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!