decouple force_on_policy_ratio from step0 reforward.

[entrpn]

Measured on qwen3-4b (OpenMathInstruct-2, agentic rollout):

• clipfrac 0.0006 -> 0.0000
• log_ratio_abs 0.0157 -> 0.0000
• step time 984s -> 742s (~24%)

Gradient magnitude is essentially unchanged; clipping was only firing on
~0.06% of tokens, all of it kernel noise rather than policy drift.

Also in this change:

• log_sampler_trainer_agreement: opt-in flag that spends one trainer
  forward pass to keep the sampler/trainer agreement metrics alive.
  Without it, enabling force_on_policy_ratio silently drops them.
• force_on_policy_ratio + off_policy_steps > 0 now warns instead of
  raising. The MLPerf reference pairs the two deliberately.
• Compare completion_tokens[-1] against EOS when counting clipped
  completions; the mask was being compared against a vocab id, so
  clip_ratio over-counted.
• Buffer metrics once per group instead of once per env/reward-time
  sub-key. Previously duplicated every metric, or dropped all of them
  when no timing keys were present.

Matches the MLPerf Qwen3.5-397B GRPO reference recipe.

Reference

Same implementation as nemo_rl for force_on_policy_ratio, found here: https://github.com/NVIDIA-NeMo/RL/blob/main/nemo_rl/algorithms/loss/loss_functions.py#L520C9-L525C36

Checklist

• I have added all the necessary unit tests for my change.
• I have verified that my change does not break existing code and all unit tests pass.
• I have added all appropriate doc-strings/documentation.
• My PR is based on the latest changes of the main branch (if unsure, rebase the code).
• I have signed the Contributor License Agreement.
• I have followed Contribution Guidelines.

[entrpn]

@lc5211 @tianshub resolved your comments, PTAL.

[tianshub]

@lc5211 are there any concerns if we disable this? probably no, but just to be aware this is another knob if we want to see the logged metrics

[lc5211]

It's used in combination with force_on_policy_ratio.

            or self.algo_config.log_sampler_trainer_agreement``` in this implementation so we are good. maybe let's add comment in line 486 just for the context purpose on when we enforce training logps recomputation.