fix: align evals with applied policies

[mikasenghaas]

Summary

• Route startup and later policy loads through one WeightWatcher update path. Advance policy.version and trigger evals only after inference applies the weights.
• Key fresh startup evals to policy v0. On resume, evaluate the loaded checkpoint only when retrigger_on_resume=true.
• Pause rollout scheduling across each weight apply so rollout provenance matches the policy served at dispatch.
• Remove the resume-step broadcast directory before the trainer rebuilds its startup handshake.
• Always broadcast and consume the final RL policy. Remove the conditional final-broadcast config and helper path.
• Remove the mixed-policy eval warning. Keep the minimum policy version in metrics and success logs.
• Remove the parenthesized step from eval success logs.

Verification

Before the fix, a 20-step reverse-text run with eval interval 1 reproduced the bug:

• Eval step 20 used policy v18.
• The final policy broadcast triggered eval step 21 with policy v20.
• The run wrote step-21 eval artifacts.

After the fix, this requested interval-10 run completed all 20 steps:

uv run rl @ examples/basic/reverse-text/rl.toml \
  --orchestrator.eval.interval 10 \
  --orchestrator.eval.num-examples 1 \
  --orchestrator.eval.group-size 1 \
  --orchestrator.eval.source '[{"name":"reverse-text-eval","env":{"taskset":{"id":"reverse-text"},"agent":{"harness":{"id":"null"},"runtime":{"type":"subprocess"}}}}]' \
  --no-monitors.wandb --no-dashboard

• Eval step 10 used policy v10.
• Final eval step 20 used policy v20.
• No step-21 eval artifacts were written.

A two-step run without evals applied final policy v2 and exited cleanly. This verifies unconditional final broadcast and consumption.

Fresh and resumed two-step smokes verified the startup matrix:

• A fresh interval-1 run evaluated policies v0, v1, and v2.
• Resume from v2 with retrigger_on_resume=false skipped v2, then evaluated v3 and v4.
• Resume from v2 with retrigger_on_resume=true evaluated v2, v3, and v4.
• Resume from v2 with interval 3 skipped v2 and evaluated v3 only.
• Every resumed startup rebuilt its v2 broadcast handshake and exited cleanly.

After the final review fixes, a fresh interval-1 run evaluated v0, v1, and v2. A resume from v2 with retrigger_on_resume=false skipped only v2, then evaluated v3 and v4.

uv run pytest \
  tests/unit/test_configs.py \
  tests/unit/orchestrator/test_batch.py \
  tests/unit/utils/test_pathing.py -q
# 156 passed

uv run ruff check \
  src/prime_rl/orchestrator/orchestrator.py \
  src/prime_rl/orchestrator/watcher.py \
  src/prime_rl/orchestrator/dispatcher.py \
  src/prime_rl/orchestrator/eval_source.py \
  src/prime_rl/utils/pathing.py \
  src/prime_rl/trainer/rl/train.py \
  src/prime_rl/utils/utils.py \
  packages/prime-rl-configs/src/prime_rl/configs/rl.py \
  packages/prime-rl-configs/src/prime_rl/configs/shared.py \
  tests/unit/utils/test_pathing.py \
  tests/utils.py
# All checks passed!

---

Note

High Risk
Changes core RL orchestration around weight updates, rollout scheduling barriers, and eval timing—incorrect ordering could cause wrong-policy evals or inference crashes during weight swaps.

Overview
Fixes evals running against stale or wrong policy versions by tying eval triggers and policy.version to inference actually applying weights, not merely publishing them.

Policy lifecycle: WeightWatcher now advances policy.version only after receive() completes, routes startup sync through sync_startup + shared on_update hooks, and registers orchestrator hooks for eval and dispatch-gate refresh. Batch shipping and shutdown waits refer to inference-applied versions.

Eval scheduling: Eval fires from trigger_eval after each applied policy (including startup v0 on fresh runs). Resume skips re-eval at the checkpoint step unless retrigger_on_resume=true; final eval uses force at max_steps without a spurious step-21 epoch. The dispatcher blocks new scheduling across weight swaps via policy_update_pending and a scheduling lock.

Broadcast config: Removes broadcast_final and final_broadcast_version; the trainer always broadcasts every step including the final policy, and resume cleanup deletes broadcast dirs from the resume step onward so the startup handshake is rebuilt.

^{Reviewed by Cursor Bugbot for commit 5fa6b01. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5fa6b01. Configure here.}