prime-runs

[kcoopermiller]

wip!!!

Why

The eval CLI hand-rolled its own platform client. verifiers/v1/utils/platform.py was 317 lines of resolve env → create → post samples → finalize over raw httpx. PrimeIntellect-ai/prime#856 moves that lifecycle into prime-runs, a leaf SDK (httpx + prime-traces) where a run is an object held for the length of the run rather than three stateless calls at the end of it.

What

What is left in platform.py is wiring rather than transport:

run = open_run(config, push_state, num_examples=len(tasks))   # before the first rollout
if run.mode == "online":
    config.run.adopt_id(run.id)            # the platform's id is the run id

async def on_complete(episode):
    episode.record_run(EvalRunInfo(id=config.run.id, name=config.run.name))
    await append_episode(out, episode, write_lock)
    await asyncio.to_thread(run.log_episodes, [episode])   # streams as rollouts land

await asyncio.to_thread(finish_run, run, episodes, push_state)

Gone: push_traces, credentials, json_bytes, run_metrics, the batching loop, the payload ceiling, the in-memory buffer. trace_to_sample and build_samples moved to prime_runs.projection — that is knowledge about a platform wire format, not about an eval framework. build_samples stays here as a re-export because prime-rl imports it across the repo boundary (prime_rl/monitors/prime.py) against a released verifiers, so dropping the name would break its next upgrade; nothing imports trace_to_sample, so that one is gone.

file	what changed
utils/platform.py	open_run / finish_run / abort_run + PushState; run_config builds what is uploaded (see below); the SDK owns the rest
cli/eval/runner.py	both runners open the run before the first rollout, stream to it, and close it out on every exit path; a failing rollout cancels its siblings before the run is closed
cli/eval/main.py	drops the end-of-run push; records the @ file.toml the run was launched from
configs/cli/eval.py	RunConfig.adopt_id() and record_source(), both private attrs
cli/resolve.py	config_file_ref(argv) — the root-level @ <path>
cli/dashboard/eval.py	the footer reads the live run instead of an upload's outcome
tests/v1/conftest.py	the E2E fixture sets push=False

What the run carries

run_config() uploads, as the evaluation's metadata:

• the fields somebody actually set (model_dump(exclude_unset=True)), with the credential tables masked: every value under a client's headers table and under a seat's harness.env table. Those are the two free-form string tables in the config and the only places a credential can land — the API key itself never enters the config (api_key_var names an env var), and forward_env exists so harness secrets need not be written down either. Masked by location, whatever the key is called: no key-name heuristics to keep in sync with the dashboard;
• the v0 keys the dashboard's lists read unconditionally — model, num_examples (the runner passes the selected task count; -1 = "all" when unknown) and rollouts_per_example — which exclude_unset would otherwise drop, leaving the environment evals list showing "Unknown Model" / "0 examples × 1 rollouts";
• when the run was launched from one, the @ eval.toml itself byte for byte under metadata.config_source — but only if parsing it (TOML/JSON) finds neither table set. Otherwise, or when the file cannot be inspected, it is left out with a warning naming the table (never the values). A non-secret header in the file costs the verbatim copy, not the upload: the structured dump still goes up with the table masked.

This is the producer-side counterpart of the Config-tab work in platform#4740, which renders config_source verbatim (TOML highlighting) and falls back to the structured projection — now aware of the verifiers v1 keys — when there is no file.

Modes

--no-push keeps the run local. So does PRIME_RUNS_MODE=disabled — the SDK's own switch, honoured explicitly because open_run passes mode="online" and would otherwise override it. A run that stays local keeps its own uuid rather than adopting the SDK's disabled-… placeholder id. A run that cannot be opened (no API key, hub resolution failure) falls back to local with the reason in the dashboard footer; the eval itself never fails because of the platform.

Blocked on

prime-runs must be on PyPI (prime#872 → prime#856) before the [tool.uv.sources] git pin can go. Until then published verifiers wheels are uninstallable with pip, and the sandbox-side installs in the live E2E cannot resolve prime-runs>=0.1.0 — which is what the current red E2E job is.