prime-runs - #3459
Draft
kcoopermiller wants to merge 6 commits into
Draft
Conversation
Replace the hand-rolled TrainRun (RFT lifecycle, per-step metrics POSTs, every-10th-step Parquet presign->PUT->confirm, atexit failure marking) with prime_runs' pr.init(kind="train"): the SDK owns the uploads on a background thread with retries and backpressure, reports crashed on a process that exits without finalizing, and adds a Prime Traces sink for allowlisted accounts. episodes_to_parquet_bytes and SAMPLE_SCHEMA moved into prime_runs.projection; the kind/subset cohort filter stays here, the step cadence moves into the SDK's training samples sink. TEMPORARY: prime-runs is not on PyPI yet; pinned to the prime repo's feature/prime-runs-train-backend branch (#856 + #873) via uv source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AGeqWNdRB8ckrgch1tqVzJ
…tests - pyproject/uv.lock: the temporary git source points at the SDK branch head (aba26790) instead of a commit that only a merged PR ref still reaches; it picks up the per-upload sample id ranges, the training sink cooldown, replayable metrics and null handling for non-finite values. Dropped once 0.1.0 is on PyPI. - orchestrator: with an online platform run, episodes carry the platform's run id, so the run's traces can be queried by the id the dashboard shows. W&B keeps the launcher's PRL_RUN_ID either way. - monitor: log_metrics hops off the loop like log_episodes (a queue put can block under backpressure); the PRIME_RUNS_MODE value passes straight through; _base_url only strips /rft (the SDK strips /api/v1); a 60 s finish_timeout bounds the drain on finish and on the atexit crash path. - docs/training.md: the platform-monitoring section describes the SDK, the team requirement, PRIME_RUNS_MODE=disabled and RUN_ID attach. - tests/unit/monitors: first coverage of the monitor (init kwargs, attach, disabled switch, base URL). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces
monitors/prime.py'sTrainRunwithpr.init(kind="train", ...).What moved where
TrainRun.create/$RUN_IDattachpr.init(kind="train", model=, environments=, training=pr.TrainingSpec(...), config=, ...)/pr.init(kind="train", id=$RUN_ID)TrainRun.log_metrics(fire-and-forget task per POST)run.log_metrics(values, step=)— a queue put; the SDK's uploader owns retries,Retry-After, backpressureepisodes_to_parquet_bytes+SAMPLE_SCHEMA+ presign→PUT→confirmprime_runs.projection+ the SDK's training samples sink (same schema, same flow)step % 10cadence inlog_episodesstep_interval=10), keyed on each episode'srun.work.step— theTrainRunInfothe dispatcher stampsfinalize→POST /finalize, fallbackset_statusrun.finish()— drain, then an idempotent finalize; 409 on the failure path is treated as already-closed_mark_failedThe
kind == "train" and subset == "effective"cohort filter stays here. The monitor keepssanitize()so dropped non-finite metric paths are still named in the log (the SDK would drop them silently).$PRIME_API_BASEstill works — it historically points at…/api/v1/rft, so the monitor strips what the SDK re-adds.Decisions
run.work.step), not the batch step; the platform keeps one Parquet object per upload and the viewer unions every object under a step, so an off-policy episode landing in a later batch adds an object rather than replacing one, and each object numbers itssample_ids in its own range (SDK side,aba26790). "Step N" in the sample viewer therefore means dispatched at N, while the metrics at N describe the batch trained at N.PRL_RUN_ID(its id is fixed at spawn; nothing else reads that variable), and in attach mode the two already agree.Temporary
[tool.uv.sources]pinsprime-runsto the SDK branch head until 0.1.0 is on PyPI (PrimeIntellect-ai/prime#856); the line is dropped and the lock refreshed then.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fk5fBAAaTiPfJ5UpNyqxXR