Skip to content

feat: report evaluation wall time - #2291

Open
skirdey wants to merge 1 commit into
THUDM:mainfrom
skirdey:codex/eval-wall-time
Open

feat: report evaluation wall time#2291
skirdey wants to merge 1 commit into
THUDM:mainfrom
skirdey:codex/eval-wall-time

Conversation

@skirdey

@skirdey skirdey commented Aug 20, 2026

Copy link
Copy Markdown

Summary

  • measure end-to-end evaluation wall time in RolloutManager.eval
  • report it as perf/eval_time through the existing eval metrics path
  • preserve custom evaluator metrics without mutating their dictionary

Motivation

Rollout generation already reports perf/rollout_time, but periodic evaluation has no corresponding duration. On long-running jobs this makes evaluation pauses indistinguishable from rollout, training, or weight-sync stalls.

Test plan

  • added a unit test for custom metric preservation and wall-time reporting
  • python3 -m py_compile slime/ray/rollout.py tests/test_rollout_metrics.py
  • isolated _eval_metrics contract execution passed
  • git diff --check

The full tests/test_rollout_metrics.py requires the project CUDA/SGLang test environment and is left to upstream CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant