Skip to content

AGENTS.md

Latest commit

 

History

History
463 lines (453 loc) · 70.6 KB

AGENTS.md

File metadata and controls

463 lines (453 loc) · 70.6 KB
Raw

Kev — small Jev-style decision models

Causal LM + LoRA run prefill-only with a block-causal mask (shared state prefix, isolated question branches) and a pointer readout over option boundary tokens, trained with log loss on public datasets plus generated policy and rule records. No text generation. The released family is Qwen3.5 (0.8B / 4B / 9B) plus Kev-27B on the post-trained Qwen3.8-27B; kev-0.5b (Qwen2.5-0.5B, the prototype this started as) and the Qwen3 generation are superseded but still published. See README.md (deep dive), PLAN.md (where the research stands, what we have learned, the standing rules and what is next; the full dated record of rounds 4-18 and everything before is frozen at git tag research-archive-2026-09-24), docs/autoresearch.md (the operating program for an unattended research session) and docs/model-cards/ (one card per checkpoint: recipe + metrics). README follows the Vercel Labs house style (tagline, for-the-badge badges, Highlights, Title Case sections, API tables, Authors + License); model cards are formal.

Commands

  • Env: uv sync (add --extra serve for FastAPI + the TypeSafe SDK; on Apple Silicon it also pulls mlx-lm for the MLX backend, --extra mlx alone for library use). kev is a real package (setuptools, installed editable by uv sync since #14), Python 3.12 or 3.13 (requires-python = ">=3.12,<3.14"; 3.14 has no wheels for the pinned torch, .python-version selects 3.13, uv sync --extra serve --python 3.13 overrides explicitly), transformers>=5.17,<6, peft>=0.21, torch>=2.6,<2.9. The dev group carries pytest, matplotlib and modal==1.5.5.
  • Train: uv run python -m kev.train --suite evals/v7/decision-v7 --base Qwen/Qwen3.5-4B-Base --base_revision <sha> --epochs 2 --lr 5e-5 --batch 4 --accum 2 --dtype bf16 --checkpointing 1 --p_none_pair 0.25 --device cuda --out runs/kev-4b (the released 4B/9B recipe; 0.8B uses --lr 1e-4 --batch 8). Records come from --suite (its training partition), --data (your own JSONL, optionally + --suite --replay N), or built on the fly from the public sources (--n_per_source, the smoke path). --help lists everything; the knobs that matter:
    • losses: --perm_kl/--perm_frac (permutation KL), --ord_w (ranked probability score for Score), --anchor/--anchor_w/--anchor_sources (KL toward the frozen base's zero-shot answers, from kev.anchors), --label_smoothing/--brier_w/--focal_gamma (the round-3 calibration screen; all default 0 and none is in a release).
    • augmentation / mix: --p_none, --p_none_distract, --p_distract, --p_none_pair, --synthetic_repeat, --public_frac, --train_sources, --holdout. --none_pair_max_state N (plan key none_pair_max_state, needs --p_none_pair > 0; absent = every record may pair, today's recipes byte for byte): only records whose state has at most N tokens (encode's count, <state> included) emit none pairs, drawn from each record's own stream (kev.train.none_pairs), and --length_sort counts their two siblings (the state twice more) in the record's cost, so a pass cannot outgrow the cost its run was cut to. A pair repeats the whole state, so on long states it tripled a pass past the GPU (round 21's projection). --pass_tokens_max N (plan key pass_tokens_max, needs --length_sort 1 and a hybrid base, whose passes are always rows or a shared prefix, what pass_tokens measures (an attention-only base runs the packed mask, refused once the model is built); not with --row_budget or --perm_kl; absent = today's plan byte for byte): the plan cuts each step on exact token shapes (kev.train.plan_shapes: every variant the epoch trains, siblings included, branches encoded without their state, the state counted once by state_token_counts) instead of characters, and a step whose costliest pass is over N padded tokens (pass_tokens) gets one more micro-batch per rank until none is (every rank the same count; a record over N alone is refused before the first step). Round 21 ran out of memory at its 62nd step on a characters plan: a token-dense PII state set the padding of 16 states at 5,877 tokens, a 98.7k-token pass costed in characters like its slot's other passes of 33-50k tokens. 27B, 8 H200: N = 40,960 peaks at 101.6-108.2 GiB per GPU on the worst passes the plan allows (scripts/sft_probe.py --passes, runs/sft-probe/r22-ceiling-27b-8xh200).
    • architecture / precision: --lora, --lora_targets all|dense|attn|qv (dense freezes the DeltaNet projections on hybrid bases), --head_dim, --option_isolation, --special_embeddings, --dtype (autocast) vs --weights_dtype (frozen backbone; bf16 is required by the fused MoE experts of 35B-A3B).
    • full-weight: --full_ft 1 --weights_dtype bf16 trains the whole text backbone (vision tower, LM head and MTP are never loaded) plus the head, with kev.full_ft.MasterAdamW (fp32 masters + moments; one GPU: in host memory, AutoJev's technique; under torchrun: FSDP2 shards, state on the GPUs, the head replicated). --shared_prefix (default on with --full_ft 1; hybrid backbones) runs each record's state once and its question branches from it (kev.shared_prefix: the DeltaNet recurrent and conv state and the attention keys/values of the state, functional, one checkpointed step per layer), exact to the row form (tests/test_unit.py, tests/test_model.py). --length_sort 1 deals each step's records into micro-batches of neighbours in length balanced by padded cost (--batch becomes the average), so no rank waits long on another. --row_budget N (one GPU): padded tokens per forward/backward pass, a record that does not fit is split by question (27B on one H200 needs 8192; not with --perm_kl or --anchor_w). Needs torch >= 2.8. --max_steps N stops early. Resume: --save_every_minutes M / --save_every_steps N write <out>/resume (fp32 masters + moments per rank, scheduler, RNG, data position; under torchrun from a host copy in a background thread, the interval stretched so blocking stays under 5 %), --resume 1 continues bit for bit (same arguments and world size), --stop_after N exits after a step. Snapshots: --snapshot_fractions 0.25,0.5,0.75 (of the optimizer steps) / --snapshot_every_steps N write loadable bf16 checkpoints into <snapshot_dir>/step-<N>/checkpoint (N zero-padded to 7 digits like resume points, step-0000389; --snapshot_dir, default <out>-snapshots; kev.full_ft.SnapshotWriter; at most kev.budget.MAX_SNAPSHOTS = 8 per run, the disk they may take next to two resume points, refused at parse time or before the first step): the final checkpoint's files (save_pretrained shards + head.pt + tokenizer; head.pt["snapshot"] = step, steps, epoch fraction, records seen) plus snapshot.json, written last, which marks it complete. Under torchrun the ranks gather into rank 0's host memory (the only blocking part) and rank 0 writes in a background thread; one GPU writes before going on. Snapshots are never deleted; a continued run skips complete ones and rewrites an incomplete one, and a resume point's latest.json waits for the snapshot being written, so no committed resume point passes a missing snapshot. training_metrics.json has per-epoch gradient norms before clipping (grad_norm: mean, max, clipped steps), LoRA runs too; full-weight runs also backbone_save_seconds (the final gather + write) and snapshots (blocking and write seconds per snapshot of that attempt; each snapshot.json has its own). Trials: see Studies. 27B, 8 H200, --batch 8 --accum 2 --length_sort 1, records shaped like the SFT corpus (scripts/sft_probe.py --mix all): 7.6 records/s, one epoch of the corpus (192k records) ≈ 7.1 h / $293, 71 GB peak per GPU (runs/sft-probe/sft2-*, PR #125); one H200 (row form, PR #122) 0.84 records/s on ~1,000-token records. Long states (scripts/sft_probe.py --state_tokens 16384,32768,49152,65536 --questions 3, --batch 1 --accum 1 --length_sort 1, 8 H200, runs/sft-probe/lc-27b-8xh200): 1.12 / 2.33 / 3.79 / 5.45 s per record (14.8k-12.0k record tokens/s), 78 / 96 / 115 / 134 GiB peak per GPU of 140. A micro-batch whose states are unpadded runs the state through SDPA's causal flash kernel with no mask (kev.shared_prefix); with the explicit 64k x 64k mask a 64k step took 150 s instead of 44 and filled the GPU (runs/sft-probe/lc-27b-8xh200-mask-ab).
    • Only one training process at a time: two on MPS slow each other ~10x.
  • Smoke: uv run python -m kev.train --n_per_source 40 --accum 4 --out runs/smoke (~1 min).
  • Benchmark (the eval path for everything current): uv run python -m kev.benchmark --run <run dir | Hub id[@rev]> --suite evals/v4/transfer-v4 --out runs/<name>; --remote <url> scores any System One endpoint (--remote-concurrency N keeps N requests in flight; rows are identical), --data x.jsonl your own labelled rows, --date_facts the opt-in preprocessing, --allow-test is the only way to read a locked test. Writes rows.json (per question, with logits) + report.json (accuracy, ECE/Brier/NLL, selective coverage and AURC, permutation, isolation; environment: the kernel set the logits depend on, from kev.predictors.kernel_environment: package versions, GPU, backbone dtype, attention implementation, the DeltaNet layer forward (kev.fused_qwen35 replaces it) and the conv and delta rule transformers bound; compare it before pairing two reads bit for bit). kev.calibrate --rows <rows.json> reports what one temperature fitted on those rows would do (raw / shipped / workload in-sample / workload group-disjoint OOF, paired bootstrap vs shipped; report only, writes calibration.json next to the rows); external-suite rows.json are committed for this. kev.evaluate is the legacy prototype eval (runs/kev, eval.json) and is not used for releases. kev.compare --candidate <dir> --reference <dir> pairs two result dirs (record-clustered bootstrap). kev.jev --suite ... --out ... scores Jev through Vercel AI Gateway (AI SDK experimental_evaluate, node worker in playground/scripts/jev-evaluate.mjs; needs AI_GATEWAY_API_KEY or --provision-scope; budget-capped).
  • Studies: kev.experiment --plan experiments/*.json --suite <suite> --out runs/<study> runs config-only trials (allowlist + ranges in experiment.py: DEFAULTS/CHOICES/validated_trial, provenance (provenance.json, copied into result.json; its environment is the kernel set the trial's reads were scored on, report.json's block from the same kev.predictors.kernel_environment, taken from the loaded predictor and replaced when a resume re-scores; trials before it have none), coverage/isolation gates, results.jsonl ledger, --transfer <suite> for an OOD read per trial, --aggregate to rank an existing directory, --resume for interrupted trials (evaluation; unfinished full-weight trials continue training from their resume point), --wait-pid to queue behind a training job). A full-weight trial runs under torchrun on every GPU of its container, writes a resume point every experiment.RESUME_MINUTES and is continued after a timeout by kev.rounds watch, not by Modal (kev.budget: FULL_FT_RETRIES continuations, up to 24 h per attempt, $1,000 per study; the bound counts every attempt): trials spawn with Modal's retries off, because Modal charged each killed timed-out attempt twice and gave round 22's trial two of its three attempts (scripts/modal_retry_probe.py); runs/<study>.spawn.json is the attempt ledger (calls = each trial's current call, attempts = all of them), and modal_app.py::resume --trial <label> (modal_app.continue_full_trial, what watch runs) spawns the next attempt only after a call ended by a timeout, within the ledger, with the study's GPU and timeout; --beyond-bound continues a trial without a ledger, uncounted. Each attempt is written to the ledger pending (a nonce) before its spawn; a pending entry blocks continuations until its call is adopted from the lease or it is kev.budget.LEASE_STALE old. The lease: every full-weight attempt holds <study>/<trial>/attempt.json on the kev-leases volume (modal_app.TrialLease: nonce, attempt, call id, heartbeat every LEASE_HEARTBEAT s from its own thread, marked ended on a clean exit); an attempt that finds another attempt's lease fresher than LEASE_STALE (900 s) returns {"refused": ...} without touching the trial, and continue_full_trial waits (bounded) for the old lease to end or go stale, so a continuation starts only after the old container is gone. It also keeps snapshots at experiment.SNAPSHOT_FRACTIONS (0.25, 0.5, 0.75 of its optimizer steps) in <trial>/snapshots/step-<N>/checkpoint (plan keys snapshot_fractions (a string, "none" for none) and snapshot_every_steps, which needs max_steps so validated_trial can check MAX_SNAPSHOTS; neither changes the config hash of a plan that omits it, nor the recipe). A 27B snapshot is ~51 GB of volume storage and never deleted: do not remove checkpoints or snapshots from the volume without asking. The volume copy is primary; a plan key snapshot_hub_repo (off by default) also mirrors each committed snapshot and the final checkpoint to that PRIVATE Hub model repo (kev.mirror, modal_app.run_mirror: a CPU container spawned after the commit, HF token from the Modal secret huggingface-secret or KEV_HF_SECRET; refuses a public repo, retries once, never fails training; the commit is recorded in snapshot.json["hub"] / checkpoint/hub.json, at <study>/<trial>/<step-N | final>/ in the repo). modal_app.py::mirror_snapshots --study X [--paths /runs/...checkpoint] [--repo jaredpalmer/kev-snapshots] [--dry-run] (re)uploads existing ones (a 27B checkpoint is ~51 GB; ask before mirroring those). The container commits the runs volume after every completed resume point and snapshot (a timeout skips the final commit); an attempt that fails with an error writes failed.json and returns {"failed": ...} instead of raising, so it is never continued (kev.rounds.poll_modal reports it as a failure; a timeout it reports as "timeout"). modal_app.py::pull (and kev.rounds watch) leaves full-weight shards (model*.safetensors directly in a checkpoint/ directory: final and snapshots) and resume points on the volume (--weights copies them); read a snapshot like any checkpoint: ::benchmarks --jobs "/runs/<study>/<trial>/snapshots/step-<N>/checkpoint@<suite>@<name>".
  • Rounds (every registered experiment since round 5): one spec per round, experiments/rounds/r<N>.json, committed before any training or read: studies (plan, GPU, timeout, budget), arms (<size>-<label>: trial + parent), parents (trial + where each of its reads lives), read tags -> suites (--allow-test for test panels, locked_test for the locked read), the rule (panels of reads, bootstrapped metrics, criteria on paired bounds, rank) and confirmation stages. kev/rounds.py is the one engine: uv run python -m kev.rounds {validate,launch,watch,launch-reads,readout,confirm} <spec>; watch polls the spawned trials (state in runs/<study>.watch.json, resumable; DNS/connection errors retried, a trial's own exception is a failure, a timed-out full-weight trial gets its next attempt within the spawn record's ledger), pulls each finished trial's study (one pull per study at a time, modal_app.pull_lock) and launches its reads once (per-arm lock; the launch intent is written first to runs/r<N>-reads-<arm>.json, and an arm launched within its reads' timeout is not relaunched; a finished call that maps to no arm is logged and makes watch exit non-zero), 60 s apart, then writes runs/r<N>-readout/round<N>.json; confirm <spec> --stage <s> writes runs/r<N>-verdict/<size>-<s>.json. Every side is served at the temperature fitted on its own development rows, unless the round registers a temperature pool for its arms (reads pooled, per-read sources allowlist, exclude_reads; refused inside any panel a temperature-dependent criterion reads; round 20). validate also refuses a pool that shares data with any arm's training (kev.rounds.pool_conflicts): (a) a pooled suite is an arm's training suite, a component its manifest names (sft-v1's inputs.components) or a plan's data suite; (b) it pools sources the arm trained on; (c) it reads the calibration or development partition of any training corpus (a suite with trainable sources). The arm's training comes from its study in the spec or its trial's provenance.json (manifest hash); a checkpoint arm is covered by the round's trial arms or names trained_on; training it cannot establish or list (no provenance, a manifest without sources, a data file outside evals/) is a problem for a new round, archived for a recorded one. Sources are compared by name, so pools use sources that are eval-only in Kev by construction; a sources allowlist must name sources its suite lists. Why: round 19 fitted its SFT arms' temperature on sft-v1 development rows, held-out items of the training sources, so in distribution (T 0.955, breadth-v1 ECE 0.059); round 20's pool of held-out datasets gave 0.0085 on the same checkpoint. Each arm's read-out records temperature_source (the pool with its reads and question count, or its trial's development rows and their suite; parents in parent_temperature_source: their trial's development rows plus shipped, head.pt's T when it is local or in the HF cache) and the printed table warns (!!!) when an arm is served at a training corpus's rows. From round 21 (POOL_REQUIRED_FROM) a rule or confirmation criterion the temperature moves (anything but accuracy) without a temperature pool is a problem, so validate and launch refuse the round; rounds <= 20 only get the warning, so their specs keep validating. validate/launch also warn (report only) when a parent's served T, fitted on a training corpus's development rows, is more than 0.05 from its shipped head.pt T. A panel with "by_length": true (or {edges, tokenizer}) adds acc / ECE / Brier / confident errors per state-token bucket (kev.metrics.calibration_by_length: under_8k, 8k_16k, 16k_32k, 32k_64k, 64k_plus and the tails 8k_plus/16k_plus/32k_plus; tokens = the encoded state segment, <state> included, as kev.model.encode builds it and kev.serve reports it, counted from the reads' suite records with kev.suite.ADMISSION_TOKENIZER unless given), and a criterion may read a bucket (long.ece_16k_plus.candidate <= 0.05). Panel filters (round 24, kev.rounds.panel_filter; the same rows leave both sides): source (a name or a list), exclude_sources, exclude_tasks, and exclude_file {path, sha256} (a JSON of ids / sources / tasks, checked against its hash; a private list, e.g. tasksource-heldout families, stays out of git and is restored with scripts/private_rows.py, so the public spec carries only the path and a salted file's sha256); "optional": true marks a report-only panel that no criterion or rank may read and whose absent reads never leave an arm incomplete; the temperature pool's ci: {level, samples, seed} adds each arm's bootstrap interval of its pooled T (temperature_ci). An arm may be a checkpoint without a trial ("checkpoint": "/runs/...", its "transfer" rows from a transfer_read), e.g. a WiSE-FT interpolation (scripts/interpolate_checkpoint.py, modal_app.py::interpolate: with the base, or --toward another checkpoint of the same base and revision, full weights or a LoRA merged in fp32, --blend_head to blend the pointer heads too; round 23); a LoRA checkpoint as a full-weight one, to start kev.train --full_ft 1 --init_from from it (scripts/merge_lora_checkpoint.py, modal_app.py::merge_adapter --lora <repo@rev> --out /runs/... [--like <full ckpt>]: the same fp32 W + delta rounded once as --toward at α 0, the LoRA's head and meta with lora 0, weights "full"; --weights_dtype bf16 writes an fp32-trained Kev as bf16, one rounding after the fp32 merge, which is how the MLX full-weight path is checked on small models; Kev-27B at /runs/r25-init/kev-27b-merged/checkpoint, weights f61fb0c5…, ~6 min on 8 CPUs, runs/r25-init/kev-27b-merged/merge.json); deltas are kev.rounds.paired (2,000 resamples, seed 0, micro). Rounds 5-18 are recorded specs ("archive": "research-archive-2026-09-24"): their plans, reads, data builders and per-round scripts live on that git tag, not on main; validate lists what this checkout lacks instead of failing, and launch/watch/launch-reads refuse a recorded round (a new round is a new spec, without archive, and must have its plans and parents' reads). A read of a removed suite (kev.suite.REMOVED_SUITES: scienthoon-v1, last read by round 22) is listed as archived for rounds up to its last_round (with or without archive; their read-outs come from the committed rows), refused for any later round, and never launched again (read_commands, load_split). tests/test_rounds.py reproduces the committed read-outs of rounds 5-18 and verdicts of 8/10/11/12/15 exactly; offline (CI) it runs round 5's read-out, round 15's locked verdict and round 20's six interpolated arms, whose rows are on main; round 19's read-out (and round 20's whole read-out, and rounds 22's and 24's) runs where the private dataset is readable (round 19's trials' sft-v1 development rows; rounds 22's and 24's tasksource-heldout reads, round 24's exclusion list, and ood / agents / guardrails rows: scripts/private_rows.py); every other case skips unless KEV_ROUNDS_ROOT points at a checkout with the archived rows and outputs (the research checkout, or a worktree of the tag plus its gitignored trial rows). kev.autoresearch session <specs> --spend-start X --spend-cap Y runs registered rounds to their read-outs under a spend cap and never confirms; it also keeps leaderboard, release-check, compare. An unattended session follows docs/autoresearch.md (start, spend rule, registration, run, what may not be touched, resilience, reporting, gotchas). Model-card numbers: scripts/release_numbers.py --release <name> reads experiments/releases/<name>.json (each arm a trial, served at its development-rows T, or a checkpoint with the round's registered temperature pool, as kev-27b-r23).
  • Frozen suites: evals/<version>/<suite>/{manifest.json, *.jsonl} with partitions train/calibration/development/test. Manifests pin dataset + base revisions and the sha256 of every partition. Current: evals/v7/decision-v7 (the release recipe), evals/v8/decision-v8, evals/v4/transfer-v4 and evals/v9/transfer-v9 (MMLU-Pro, buried states, unknowable) for OOD, evals/round3/{decision-r3,transfer-r3} (calibration audit; the 1,260-record final panel was read by the 2026-09-22 release confirmation and has been a short-state guard since round 11), evals/smoke-v1 for tests, plus evals/external/ (semif-v1, report only: sound labels but saturated, 130 of 144 right for every Kev-27B checkpoint; ekzhang-mmlupro-v1). Removed as unsound for a gate (kev.suite.REMOVED_SUITES refuses a read with the reason; the committed rows under runs/, e.g. runs/kev-*-{wanli,typesafe}-v1, stay as the record): external/scienthoon-v1 on 2026-09-27, and on 2026-09-30 external/wanli-v2 and wanli-v1 (a quarter of the pairs have two disagreeing annotators and a gold set to one of them) and external/typesafe-v1 (gold = the argmax of two closed frontier models' averaged answers; split-half r about 0 across checkpoints), with their builder scripts/freeze_semif_external.py and scripts/compare_typesafe.py. There is no pooled-externals guard: round 24's audited rule, which rounds 23-26 follow, reported SemIf, WANLI-v2 and TypeSafe without gating them (PLAN.md, standing rules)), evals/night2/ (delta training data, scripts/build_night2_data.py), evals/diagnostics/ (binding-v1), evals/hard-v1 (programmatically labelled skill records in seven families: long policy documents, trade-offs, probability, multi-hop, temporal/numeric, judging a proposed answer, missing-fact abstention; scripts/build_hard_v1.py + hard_v1_{common,policy,families,numeric}.py, labels from each family's solver over _meta.facts, templates 0-3 train / 4 development / 5 test; long_policy states reach ~5k tokens, so train with --max_state >= 5120; its 23 MB train partition is not in git; load_split fetches it from the kev-suites mirror (since 2026-09-30) and the builder regenerates it byte for byte, ~1 min; scripts/screen_overlap.py checks it against JevBench's public items, counts only, in overlap.json), evals/devtools-v1 (developer-tooling decisions from six licence-checked sources, human / heuristic / by-construction labels, no LLM labels; scripts/build_devtools_v1.py --reproduce-v1 rebuilds it byte for byte from cached downloads; When2Call and prompt injection are eval-only. Known defects, frozen: CodeReviewer ids came from the dataset's non-unique id field, so codereviewer/cls-test/13657 names two development records and codereviewer/cls-test/19245 two test records (paired comparisons drop both; 66 more ids repeat inside train or across train and an eval partition, so check train/eval overlap by text_sha256), and its CodeReviewer text_sha256 hashes the hunk without lines_before_hunk. Without --reproduce-v1 the builder makes line-based unique ids, keys the whole state and admits commitpackft records after their message question is added; see its docstring), and the real-document suites evals/documents-v1 (CFPB complaint narratives, product + issue Choice questions; its 23 MB train partition, Kev-4B's round-8 delta data, is not in git; load_split fetches it from the kev-suites mirror (since 2026-09-30)) and evals/documents-v2 (held-out test only, private mirror, manifest only): scripts/build_documents_v{1,2}.py -> label_documents_v1.py (AI Gateway teachers/judges, spend ledger) -> freeze_documents_v1.py (no flag: report + adjudication queue; --combine, --spot-check, --freeze ... --min-agreement 47); label provenance in runs/documents-v1-work/. evals/breadth-v1 is the eval-only generalisation panel mirroring the community Decision Index's five areas: 14 held-out datasets, MuSR, SATA-Bench, ChessBench, ContractNLI, HellaSwag (ActivityNet half), CLINC150, SGD/SGD-X, BRIGHT, BFCL, ToolRet, API-Bank, RouterBench, Humicroedit, cfcolor; GPQA skipped (gated). 150 records per dataset in development and in the locked test; >10-option tasks restricted to candidate subsets that contain the answer. Private mirror (manifest only in git; SATA-Bench NC, SGD share-alike, Humicroedit / cfcolor unlicensed text must not be public). scripts/build_breadth_v1.py rebuilds the partitions byte for byte; scripts/breadth_report.py --suite evals/breadth-v1 --result NAME=DIR ... scores rows per area with the Index's chance correction; baselines in runs/breadth-v1-*. These datasets must never enter a training corpus. evals/longdoc-v1 is a report-only long-document probe (eval-only, private mirror, development + locked test): state-length buckets 4k / 8k / 16k / 32k / 64k under the Qwen3.8-27B tokenizer, CUAD contracts (CC BY 4.0, expert clause labels, a target padded with other contracts) and generated agreement bundles (scripts/build_longdoc_v1.py + longdoc_v1_synthetic.py, byte for byte). Its manifest records kev.suite.SERVING_CONTEXT; kev.benchmark scores its 4k-16k buckets on the exact path and its 32k / 64k rows through the long-row rule (> ROW_PASS_TOKENS: state once, fused kernels, kernels: efficient). Read-out scripts/longdoc_report.py (--parity compares two reads row by row), overlap scripts/screen_longdoc_v1.py, serving cost scripts/longdoc_serving.py (modal_app.py::script); results in runs/longdoc-v1-report/README.md. Its synthetic part is at ceiling for every system read. evals/sft-v2 is the extended SFT corpus (private mirror jaredpalmer/kev-private-train, manifest only; built by kev-sft assemble-v2, assemble/build_v2.py): sft-v1 plus the private components tasksource-v1, longify (8k-64k states built from sft-v1 train), longdoc (code-assembled long documents), ood, tone (calm / frustrated / angry minimal pairs), injection, agents, guardrails-pii and guardrails-grounding, each hash-checked against its own manifest (recorded with path, commit and mirror revision under inputs.private_components). tasksource-v1's family list stays private: its records carry source tasksource. Every record is admitted at 64k states and screened (kev-sft assemble/screen_v2.py, the kev-sft screen rule) against every evaluation partition of every frozen Kev suite, the private evaluation mirror, JevBench public and the eval-only suites below; a non-clean _meta.variant moves to _meta.twin (kev.benchmark scores only clean rows). Its calibration/development partitions are held-out items of the training components: in distribution, in-trial screening only. evals/sft-v2-r21 is the same records with states of at most 32,768 tokens, round 21's training suite (PLAN "Round 21"; one epoch of the 64k corpus does not fit three 8-hour attempts on 8 H200s, so it also caps each of sft-v1's public sources at 1,200 train records and keeps only states up to 8k tokens in calibration/development, the in-distribution screening partitions no rule reads; none pairs gated at 8k, --none_pair_max_state). Eval-only suites set aside by its components (private mirror jaredpalmer/kev-private-evals, development only, report-only; source names shared with sft-v2's training records, so never a temperature pool): evals/ood-v2, evals/agents-ood-v1, evals/guardrails-ood-v1 (PII + grounding + injection). evals/tasksource-heldout-v1 (development + locked test; 24 whole dataset families held out of tasksource-v1: held-out datasets) publishes hashes and counts only; its family list is in the private kev-sft manifest, so its rows and per-source reports stay in the private dataset (scripts/private_rows.py), and with no sources listed it cannot be a temperature pool read. kev.jev --count-refusals counts HTTP 400/413/422, and a 5xx on a request past 1.5 x Jev's ~32k-token context (the gateway answers oversize with either), as refused. Partitions over ~10 MB are not in git; they are mirrored at the Hub dataset jaredpalmer/kev-suites (revision pinned in kev/suite.py: SUITES_REVISION) and load_split fetches + verifies them on first use. After freezing a new suite: hf upload jaredpalmer/kev-suites evals . --type dataset --include "*.jsonl" --include "*.json", bump SUITES_REVISION, gitignore the large partitions. Never modify a frozen file; new data = new version. Held-out suites whose text must never be public (a private test set) keep only manifest.json in git and name their own mirror in it, "mirror": {"dataset": "jaredpalmer/kev-private-evals", "revision": "<commit sha>"} (kev.suite.PRIVATE_DATASET, a private dataset repo): freeze locally, hf upload jaredpalmer/kev-private-evals evals . --type dataset --include "<suite path under evals/>/*.jsonl" (e.g. held/documents-v2/*.jsonl), write the returned commit into the manifest, gitignore the partitions. load_split fetches and hash-checks them for accounts with access (hf auth login / HF_TOKEN; Modal images already carry a locally fetched copy under evals/) and raises a PermissionError naming the repo for everyone else; tests/test_conventions.py fails if such a suite tracks a partition.
  • Modal (default for anything beyond smoke): modal_app.py; uv run modal run modal_app.py::{smoke,study,pull,resume, locked_test,evaluate,base_probe,benchmarks,smoke_base,anchors,sft_probe,gpu_tests,interpolate,release_copy,release_publish,release_verify}; uv run modal deploy modal_app.py once so studies survive a disconnect. Image = uv_sync of pyproject/uv.lock (Linux torch wheel is CUDA) + fla, triton>=3.7.1 and the causal-conv1d wheel (--no-deps, or it reinstalls torch's triton 3.4) + kev/ + evals/ + tests/; Volumes kev-hf-cache (HF_HOME) and kev-runs (trial outputs, pulled to runs/<study> then ranked by kev.experiment --aggregate). KEV_GPU picks the GPU type (H100 default; T4 for the free tier), KEV_APP_NAME isolates a research deployment, worker_environment propagates app/GPU/secret settings (a dependency list that differs inside the container fails with "Function has N dependencies but container got M"). Legacy checkpoints: Hub id, or modal volume put kev-runs runs/<run> /legacy/<run> then --existing /runs/legacy/<run>. Eval on CUDA turns TF32 and fused SDPA off in LocalPredictor (what "fp32-exact" means and does not, below); training keeps TF32 and may use --dtype bf16. Probes, external benches and new-base fit checks lived in modal_probe35.py until 2026-09-21 (folded in at 90990a5); the how-to is the kev-modal-study skill.
  • Figures: uv run python scripts/plot_family.py and uv run python scripts/plot_tweet.py regenerate docs/kev-family.png and docs/kev-benchmark{,-dark}.png from saved result files. Style lives in scripts/chartstyle.py (Geist type, Vercel color tokens, direct labels, no legends, one label/plot/value lane per bar set); new figures should import it rather than set their own rcParams. kev.plot (loss curves from train logs) is a debugging aid, not a README figure.
  • Current family (2026-09-21, all Qwen3.5 + the dates/unknowable delta): jaredpalmer/kev-9b (v2 since 2026-09-30: round 27's r18-9b/00-trial-0, the night2 checkpoint + one epoch on round 15's joint documents-v1 + hard-v1 + devtools-v1 data with replay 10000 (round 18, arm (a)), staged at /runs/release/kev-9b-r27/checkpoint, adapter 2b2a70cf…, T 2.19 from round 24's held-out pool; v1, night2-9b-du/00-trial-0 at T 2.30, at tag v1), jaredpalmer/kev-4b (r10-skills/00-trial-0 from the round-10 release: the round-8 checkpoint r8-small/00-trial-0 (the night2 checkpoint + one epoch on documents-v1 train) + one epoch on hard-v1 + devtools-v1 train; round-8 weights at tag r8-documents-release, night2 at night2-du-release; Qwen3 weights at tag qwen3), jaredpalmer/kev-0.8b (r15-08b/00-trial-0 from the round-15 release: night2-08b-du2/00-trial-0 + one epoch on documents-v1 + hard-v1 + devtools-v1 train together, replay 6000; previous at tag night2-du-release), jaredpalmer/kev-27b (v2 since 2026-09-30: round 23's 27b-k-w85, full bf16 weights d27af6ab… (0.85 round-22 full-weight SFT + 0.15 v1), staged at /runs/release/kev-27b-r23/checkpoint, T 1.32 from a held-out pool; v1, the LoRA r6-27b-v2/01-trial-1 from "B1 v2" in PLAN_27b.md at tag research-archive-2026-09-24, at tag v1-lora; base Qwen/Qwen3.8-27B rev 1d4bf0f2, post-trained, not -Base; bf16 only (51 GB of weights, ~66 GB resident when serving), so B200, H200 or H100 80 GB, no Mac path; the kev-deploy skill defaults it to B200, then H200, then H100; served fused with CUDA graphs like the others (v2: runs/serving-27b-r23; v1: LoadOptions.fused folds the adapter into the bf16 backbone, runs/fused-27b-*)). Pre-delta v7 checkpoints at tag v7-base (q35-9b/01-trial-1, q35-4b-s23/00-trial-0, q35-08b/02-trial-2). Calibration is built into each checkpoint: head.pt["temperature"] (fitted by scripts/calibrate_checkpoint.py on the trial's development rows; 27B 1.32 (on a held-out pool, round 23; v1 1.38), 9B 2.19 (on a held-out pool, round 27; v1 2.30), 4B 2.41 (2.96 after the round-8 delta, 2.14 before it), 0.8B 2.35 (2.41 before the round-15 delta)) is applied by PointerHead in eval mode; KEV_TEMPERATURE=1.0 overrides to raw. Those were fitted in distribution; the script now refuses rows that share data with the checkpoint's training (its own suite, sources it trained on, a training corpus's calibration/development partition; kev.rounds.pool_conflicts, training from head.pt or the trial's provenance, rows' suite from the kev.benchmark report.json beside them) unless --allow-in-distribution, which warns and is recorded in head.pt["temperature_fit"]["in_distribution"]; a manual --temperature needs --reason (recorded); temperature_fit["fit_rows"] records each rows file's suite, partition, sources and question count. Ship a temperature fitted on held-out datasets (round 20's pool). A trial's own calibration_fit (result.json, calibration/temperature.json) is labelled role: "in-trial screening; ... not a served or shipped temperature". The script also reports an out-of-fold grouped-CV ECE with bootstrap CIs alongside the in-sample fit (4B: 0.075 raw -> 0.020 OOF, separated) and stores it under head.pt["temperature_fit"]["cross_validation"]; re-run it after any new checkpoint before publishing. Opt-in: KEV_DATE_FACTS=1 (day counts). Delta data: evals/night2/ (scripts/build_night2_data.py). Previous generation, kept for Mac latency: kev-8b, kev-0.6b, kev-4b@qwen3 (cards *-qwen3.md). Qwen3.5 backbones are hybrid (Gated DeltaNet): DecisionModel.hybrid routes forward() (so kev.benchmark) through forward_rows_batch (one causal row per question, state repeated) and serving and probs() through _branch_rows_from_prefix (state once, #77); the packed block-causal mask is only valid on attention-only bases. Needs transformers>=5.17, peft>=0.21; CUDA wants flash-linear-attention + triton>=3.7.1 (in the Modal image). MPS has no fast DeltaNet kernels, so on Apple Silicon kev.serve runs these checkpoints through kev/mlx_model.py (mlx-lm's Metal kernels; M5, 5 questions on a ~270-token state: Kev-4B 721 ms new state / 136 ms cached state vs 3302 / 847 ms for torch bf16; parity with fp32 torch on the full decision-v7 development partition: 4B max |dp| 0.025, 1 flip in 1,264 questions; 0.8B max 0.054, 4 flips (runs/r4-mlx-parity-*)). Long states on the MLX server (scripts/mlx_long_states.py, M5 32 GB, 3 questions, one about a fact planted at 60% depth, runs/mlx-long-states/summary.md): the state runs into the prompt cache PREFILL_CHUNK = 1,024 tokens per pass (one pass kept every token's activations: Kev-4B at 8k took 4.6 GB above the weights instead of 1.2, same 6.5 s; exact up to float reassociation: fp32 on the CPU matches one pass to 1e-6, the 4B's first 8 layers included, while bf16 on Metal moves answers by up to ~0.01 (4B: 0.02) at any chunk size from 256 to 4,096 (a few tokens' keys/values in the 4B's deep layers move up to 0.7 of their norm, the same tokens wherever the passes start), since the kernels are picked by shape (a state of exactly k x 2,048 tokens happened to keep the one-pass bits at 2,048; real lengths do not), so the smallest chunk that costs no time wins; --prefill-ab, runs/mlx-long-states/prefill-ab-*.json; a state of up to 1,024 tokens is still one pass), and PrefixCache.make_room drops the states a batch will evict before it runs. Kev-0.8B at 8k / 16k / 32k / 65k: 1.4 / 3.2 / 8.0 / 21.2 s for a new state, 74 / 94 / 134 / 202 ms cached, 2.7 / 3.0 / 3.1 / 3.8 GB MLX peak (1.5 GB weights), planted fact right at every length, max |dp| 0.008 / 0.005 vs fp32 torch on the CPU at 8k / 16k. Kev-4B: 6.6 / 14.0 / 30.5 / 84.5 s new, 354 / 427 / 533 / 716 ms cached, 10.2 / 11.0 / 11.9 / 13.0 GB MLX peak (8.4 GB weights; 14.1 GB process footprint at 65k), planted fact right at every length (p 0.94-0.97, moves with the note when it names another office); its fp32 torch reference (18.6 GB) did not fit. Kev-9B was not run: its 19.3 GB of weights plus a working set did not fit the 19.8 GB that machine had free (other apps, 25 GB of swap in use); it has the 4B's attention layout (8 attention layers, 4 KV heads of 256: 33 KB of keys and values per state token), so a 65k state should add about what it adds to the 4B, ~4.5 GB. Plan and results: PLAN.md at tag research-archive-2026-09-24, History > "Qwen3.5 port". Full-weight checkpoints (Kev-27B v2) run on MLX too: kev.mlx_model.load_full builds mlx-lm's qwen3_5 model from the checkpoint's own config.json (model_type qwen3_5_text) and shards, language_model.model. prefixed onto the saved names, mlx-lm's sanitize for a transformers export (conv kernel axes, norms as 1 + w), LM head deleted before anything is materialized; strict (every name, shape and the head.pt dtype must match, else ValueError), nothing merged or cast. Load peak = the weights: Kev-4B written as bf16 full weights, 8.41 GB MLX peak for 8.41 GB of shards vs 15.87 GB for the LoRA path's merge; the same answers as the LoRA path bit for bit once the base's fp32 A_log / gated-norm tensors are rounded to bf16 as a full export holds them (otherwise max |dp| 0.018); vs fp32 torch on the same checkpoint max |dp| 0.015, 0 flips in 60 (runs/mlx-full-{4b,0.8b}, scripts/mlx_full_parity.py). Kev-27B v2: 51.2 GB of weights, so expected ≈ 51 GB plus working memory at load and after; untested on a Mac (PR #175 served v1 through the LoRA path on a 128 GB M5 Max: 52 GB steady, 97 GB peak at the merge).
  • Delta fine-tuning: kev.train --init_from <run dir | Hub id[@rev]> warm-starts LoRA + head (compatibility checked before load; source hashes in provenance; allowlisted in kev/experiment.py so studies can run cheap delta trials from a released checkpoint). Use lr <= 2e-5 for deltas. A full-weight checkpoint warm-starts a --full_ft 1 run (every backbone tensor copied over the base, coverage checked); LoRA and full do not mix.
  • Checkpoint layouts (kev.checkpoint, the loader rule): adapter_config.json -> LoRA on head.pt's base; no adapter and config.json + model*.safetensors (save_pretrained of the bf16 backbone, 5 GB shards + index) -> full weights, loaded from the checkpoint directory (meta.weights == "full", lora == 0, base/base_revision kept for the tokenizer). Full weights load in bf16 by default (KEV_DTYPE=fp32 upcasts); fused kernels and CUDA graphs apply as to a merged adapter; on Apple Silicon backend=auto runs them on MLX as saved (kev.mlx_model.load_full).
  • Publish: uv run python -m kev.publish --run runs/<run> --repo jaredpalmer/kev-<size> --card docs/model-cards/<name>.md (needs hf auth login; --private, --tag, --revision <branch> for candidates; --private creates a missing repo private and refuses a public one, kev.mirror.ensure_private). A full-weight checkpoint on the volume is staged and uploaded from CPU containers, never through the laptop: modal_app.py::release_copy --src /runs/.../checkpoint --dst /runs/release/<name>/checkpoint --expect <weights sha256> (scripts/release_checkpoint.py: refuses an existing destination, checks the copy's weights hash), then the release temperature into the copy's head.pt (scripts/calibrate_checkpoint.py), then modal_app.py::release_publish --run /runs/release/<name>/checkpoint --repo <private repo> --card <card> (kev.publish --private; shards linked, not copied). The approved public release: tag the current main first (kev-27b@v1-lora), then release_publish ... --public --confirm-public <repo> --replace (kev.publish --replace deletes every file the upload does not carry in the same commit; a full-weight upload into a repo still holding an adapter is refused without it, since the loader rule would pick the adapter), then modal_app.py::release_verify --jobs "<repo>@<suite>,..." --out runs/<name> (an H200 with no secret, no HF cache volume and no token) and scripts/compare_release_rows.py against the round's committed rows. Kev-27B v2 (PLAN.md "Released: Kev-27B v2", record runs/release/kev-27b-r23-published.json) is the worked example. Repos are named by base model size (Kev-0.5B = Qwen2.5-0.5B); versions within a size are Hub tags (hf repos tag create jaredpalmer/kev-0.5b vX.Y). Collection: huggingface.co/collections/jaredpalmer/kev-6aad9d0ea49f2589665e07cd. --run in serve/benchmark accepts a Hub id.
  • HF Space (public demo, ZeroGPU): huggingface.co/spaces/jaredpalmer/kev. Source in space/ (Gradio 6 app.py, presets.py mirrors the playground presets, README.md frontmatter models:/datasets: is what links the Space from the model and dataset pages). Publish with scripts/publish_space.sh [repo] [message]: it stages space/ + kev/{__init__,model,api,checkpoint}.py into one hf upload --type space commit, so the vendored modules never drift from the repo (a stale vendored model.py is how the hugging-apps Space broke on Qwen3.5). Any change to kev/model.py, kev/api.py or kev/checkpoint.py that affects serving should be republished. ZeroGPU rules: import spaces first, load on CPU in fp32 (PeftModel.from_pretrained(..., torch_device="cpu"), otherwise peft picks the faked cuda device and crashes), merge, then .to("cuda") once at module scope; a restart reloads both models (~3 min). Check with hf spaces logs jaredpalmer/kev and the gradio_client /decide endpoint; the Space is also in the Kev collection and needs PRO to exist.
  • Serve: uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b --port 8009 (--run defaults to runs/kev and falls back to runs/smoke; the playground proxies :8009)
    • TypeSafe-compatible: POST /v1/systemone, GET /v1/models (model cards for kev-latest and jev-latest, plus device, dtype, temperature and prefix-cache stats), an x-typesafe-request-id header on every response, and bearer auth when KEV_API_KEY is set (unset = open server).
    • SDK: TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8009", model="kev-latest")
    • Admission (kev.model.admit, the one check kev.serve and the Space call, the same on torch and MLX): a state over SERVE_MAX_STATE tokens (65,536, <state> included) gets a 422 naming its count, the limit and the fixes (the TypeSafe SDK raises TypeSafeUnprocessableEntityError with it, not retried), like a question row over SERVE_MAX_BRANCH; until this was fixed the server cut the state silently (only usage.input_tokens showed it; every Kev-27B deployment at kev-deploy's f2bb629 pin read 8,192 tokens). KEV_TRUNCATE_STATES=1 (Server.truncate_states) reads the first 65,536 tokens instead, and every response of such a server carries truncated plus usage.state_tokens / state_tokens_used (the SDK ignores the extra fields); a default server's body is unchanged. /v1/models reports max_state_tokens and truncate_states. No per-request switch: TypeSafe's own endpoint would ignore it. Benchmarks never truncate (LocalPredictor encodes strictly within the suite's context); --remote turns an endpoint's 400/413/422 into ContextOverflow at once (rejected under skip_overlong, else a clean stop), retries only 408/429/5xx and connection errors.
    • CUDA: bf16, fused kernels and CUDA graphs by default (LoadOptions.fused / LoadOptions.cuda_graphs, KEV_FUSED=0 / KEV_CUDA_GRAPHS=0 to decline; fused only when fla 0.5.2 is installed, checkpoint.fused_available, not in the serve extra). A server pass was kernel-launch bound (~60 ms on an H100 at any length). kev/fused_qwen35.py rewrites the merged Qwen3.5 layers with fla Triton kernels (it needs fla 0.5.2 exactly, pinned in the images, and refuses others: it patches fla's NB-keyed kernel launches; a pass continuing a cached DeltaNet state does not write it back). kev/cuda_graphs.py replays bucketed passes (state left-padded, rows right-padded, masked exactly; equal to eager up to bf16 reassociation) and owns admission (admits), batches (run), capture policy (capture_due: idle, or a bucket that keeps recurring) and stats(); a failed capture leaves its bucket eager. kev.serve.Server runs every pass on one model thread that batches whatever is queued (DecisionModel.probs_batch, CudaGraphs.run over typed Requests; probs_one is the per-request rule every backend shares; PrefixCache keeps only the states that survive a batch; length_groups groups passes by a token-cost model, PASS_TOKENS; long new states take their eager state pass EAGER_STATES at a time); /v1/systemone is async, Server.lock excludes the model thread, wait_idle() waits for answers and captures, every response carries server-timing. Measure with uv run modal run modal_app.py::serving --run <hub id> --gpu <GPU> --name <name> (scripts/serving_bench.py: latency, parity vs fp32, throughput at 1/8/32/64 in-process clients, --flags=--isolation for question isolation on the served path; reports in runs/serving-* (graphs only), runs/fused-* and, for the current checkpoints and code, runs/serve-* / runs/grouping-4b-h100 / runs/fused-27b-*, which the README serving table reads). tests/test_model.py::test_cuda_graphs_match_eager needs CUDA (run it on Modal). Over HTTP, Modal's asgi_app path caps a container at ~40-50 req/s in earlier measurements; the current direct Modal Server and rejected higher-limit web endpoint are paired by scripts/modal_http_bench.py (runs/modal-http-4b-h100/report.json: HTTP/process/model times, errors, kernel set). Loading merges the fp32 adapter straight into bf16 weights (same bits as the old fp32 merge + cast), so Kev-9B needs ~17 GB, not 36 GB.
  • Extra endpoints for the demo: POST /v1/systemone/permute (one Choice under n option orders), POST /v1/systemone/separate (each question alone; packed-vs-separate comparison). /v1/systemone also returns latency_ms.
  • Web demo: cd playground && npm run dev -- -p 3001 (:3000 is used by another project). Next 16 app router; /kev/* is rewritten to the FastAPI server (KEV_API, default http://127.0.0.1:8009); it uses only the /v1/* routes. Presets live in playground/src/lib/kev.ts.
    • /chess (src/components/chess-game.tsx, src/lib/chess.ts, chess.js): legal moves -> Choice options, board -> state, Score for eval; games in localStorage key kev.chess.v1.
    • React Compiler lint forbids sync setState in effects; schedule via setTimeout or move into handlers.
    • Next 16 dev only trusts localhost; other hostnames need allowedDevOrigins or the page SSRs but never hydrates (no console errors). 127.0.0.1 is allowed in next.config.ts. Verify hydration with agent-browser (CDP), not curl.
    • playground/AGENTS.md (regenerated by next dev) is the Next 16 rules file; read node_modules/next/dist/docs/ before writing app code.
  • Unit tests (no weights; the CI python job): uv run --extra serve python -m pytest tests/test_unit.py tests/test_research.py tests/test_generators.py tests/test_conventions.py tests/test_documents_tools.py tests/test_hard_v1.py tests/test_devtools_v1.py tests/test_breadth_v1.py tests/test_rounds.py tests/test_skill_scripts.py -q. tests/test_skill_scripts.py covers the kev-finetune stdlib scripts (stdlib only, no network). Weight-backed parity tests (runs/smoke-hl/00-trial-0/checkpoint + a Qwen2.5-0.5B / Qwen3.5-0.8B-Base download, ~2.5 min, local only): tests/test_model.py; tests/test_mlx.py (Apple Silicon, ~30 s) for the MLX backend. test_conventions.py is a table of "one canonical home" rules (head.pt via kev.checkpoint, KEV_* via LoadOptions.from_env, option keys via api.question_keys, context via model.fits/MAX_PACKED, device via kev.device, ...); add a row when a new helper becomes canonical. It also asserts every published README/model-card number in docs/claims.json traces to a committed report (scripts/verify_claims.py).
  • API tests (server must be up): KEV_BASE_URL=http://127.0.0.1:8009 uv run --extra serve python -m pytest tests/test_api.py -q
  • CI (.github/workflows/ci.yml) runs the unit job above and a playground job: npm ci && npm run lint && npx next typegen && npx tsc --noEmit -p ..

Skills

Repo skills (.agents/skills, tracked in git):

  • kev-verify: how to prove a change has no regression (unit suites, weight-backed parity, worktree parity harness against main) and ship it as a stacked, reviewed, squash-merged PR.
  • kev-pr-description: how to write the PR title and body (the acdlite / sebmarkbage essay style, with a weak/strong pair from a real Kev PR). Read it before opening any PR.
  • kev-modal-study: launching, monitoring and pulling Modal studies, base probes, remote benchmarks and new-base smoke checks.
  • thermonuclear-code-review: the strict structural review (standards adapted from cursor-team-kit's thermo-nuclear review, the approval bar, and the table of canonical homes for shared rules).
  • kev-knowledge: how to search the knowledge graph of every earlier Devin session on this repo (the private repo jaredpalmer/kev-knowledge: topic notes, session notes, timeline, PR back-links; refreshed nightly by a Devin Cloud automation from the Devin API) before re-deriving history. Read it when asked what was tried, decided or measured before, or why a rule exists.

Installed from other repos by npx skills add and pinned in skills-lock.json (deslop, unslop from cursor/plugins, grill-me from mattpocock/skills); .agents/skills/modal/ is gitignored and reinstalled with uv run modal skills install.

Published from this repo: skills/kev-deploy (npx skills add jaredpalmer/kev@kev-deploy), one self-contained Modal file (scripts/kev_serve.py, pinned KEV_REF, GPU list per released model from runs/serve-* / runs/fused-*: 0.8B L4, 4B L40S, 9B H100, 27B B200 (H200, H100); CUDA graphs captured at start (server.wait_idle()); 64 concurrent web inputs per container, autoscaling at 32; optional KEV_REGION; KEV_FLASH=1 = direct HTTP via app.server, target 64 per container, needs KEV_REGION, keeps one paid container up, tested with Modal 1.5.5; fla pinned to 0.5.2; bearer key via a deploy-time Modal secret) that serves a released Kev checkpoint as a System One endpoint in one modal deploy; bump its KEV_REF after a serving change it should pick up. And skills/kev-finetune (npx skills add jaredpalmer/kev@kev-finetune): the user-facing fine-tuning skill (SKILL.md for agents, README.md is the human cookbook, references/ the long-form docs). agentskills.io format (validate with uvx --from skills-ref agentskills validate skills/kev-finetune). Stdlib scripts: extract_workload (find Jev/TypeSafe call sites + labelled files, draft the spec), convert_data (CSV/JSONL -> records), generate_data (OpenAI-compatible endpoint), plan_size (paired McNemar sizing; --from-result post hoc), split_data (--holdout keeps real rows out of train). scripts/kev_modal.py is a self-contained Modal app (app kev-finetune, volumes kev-finetune-runs + kev-hf-cache) whose image clones this repo at KEV_REF and pip-installs it, so it needs no local clone; bump KEV_REF after merging a kev/ change the skill depends on. Launch-time settings go into the image env via SETTINGS (a Secret list that differs inside the container fails with "Function has N dependencies but container got M"). train derives every architecture flag from the init checkpoint's head.pt; results are the skill's own result.json shape (not a research trial's); publish is private by default; teardown removes runs / the endpoint / the volumes. Tests: tests/test_skill_scripts.py.

Layout

  • kev/data.py dataset -> typed records (Source, build, materialize, load_records), trainable/eval-only source policy, permutation / none-of-the-above / distractor augmentation
  • kev/composition.py generated rule structures (atoms, shapes, groups) behind the compositional training and eval records
  • kev/contrastive.py programmatic contrastive pairs with code labels: one sentence changed flips the label
  • kev/suite.py frozen suites: digest/manifest/read_manifest/load_split (Hub mirror), CONTEXT + admission, validate_training (trainable/eval-only policy), semantic_hash, UTF-8 JSON helpers, freeze CLI
  • kev/study_v3.py builds the v3+ decision/transfer suites (public pool + generated families, grouped splits)
  • kev/transfer_v9.py transfer-v9: transfer-v4 byte-for-byte plus MMLU-Pro, buried states and unknowable items
  • kev/model.py encode(), branch_mask(), fits/MAX_STATE/MAX_BRANCH/MAX_PACKED, PointerHead (carries the temperature), DecisionModel (packed mask, or hybrid row batches)
  • kev/full_ft.py full-weight training: MasterAdamW (fp32 masters, host or device), FSDP2 shard, rank_share, save_backbone, init_distributed
  • kev/train.py LoRA (or --full_ft) fine-tune: training_requests (suite / built / --data+--replay, context filter, policy checks, mix ablations), encode_batch + batch_loss (CE, anchor KL, permutation KL, RPS, smoothing/Brier/focal), main orchestration. Grad accumulation over small padded batches.
  • kev/anchors.py frozen-base zero-shot distributions per training question, the target for --anchor_w
  • kev/checkpoint.py Checkpoint (resolve run dir or Hub id, head.pt schema = Meta, the LoRA-vs-full loader rule full/shards/weights_sha256 (a full checkpoint loads in its head.pt weights_dtype, which must match config.json's dtype), load with LoadOptions -> torch DecisionModel or, with backend="mlx"/"auto", MLXDecisionModel; warm_start for deltas)
  • kev/mlx_model.py Apple Silicon backend: mlx-lm Qwen3.5 backbone, LoRA merged in fp32 on the CPU stream (merge_lora) or a full-weight checkpoint loaded as saved (load_full), Kev's encoder/rows and the torch PointerHead unchanged; state prefix = mlx-lm prompt cache filled PREFILL_CHUNK tokens per pass, branches on a replicated copy
  • kev/device.py default device and explicit selection (select, a child-process probe: CUDA/ROCm, then MPS, then CPU; KEV_DEVICE_PROBE_TIMEOUT, default 30s) / sync / empty_cache / allocated_bytes for cuda, mps, cpu
  • kev/metrics.py pure-numpy scoring of benchmark rows: ECE, Brier, NLL, selective prediction (tie-aware coverage@error, AURC), temperature fit, out-of-fold CV calibration report (cross_validated_temperature), paired bootstrap
  • kev/predictors.py LocalPredictor (checkpoint), RemotePredictor (System One endpoint), JevPredictor (AI SDK worker)
  • kev/benchmark.py rows from predictions, summarize(), evaluate_records(), CLI
  • kev/compare.py two saved result dirs -> paired bootstrap comparison + NLL sensitivity
  • kev/jev.py Jev through Vercel AI Gateway (key provisioning, budget cap) scored on a frozen suite
  • kev/experiment.py config-only study runner: trial allowlist, gates, temperature fit, ledger, aggregate/resume
  • kev/rounds.py registered rounds as specs: validate, launch, watch (pull + reads), read-out, confirmation; the served-vs-served paired read
  • kev/autoresearch.py spend-capped sessions over round specs (kev.rounds), plus the study leaderboard
  • kev/evaluate.py legacy prototype eval: acc/ECE, permutation stability, IIA shift, isolation probe, packed-vs-separate
  • kev/plot.py loss curve(s) from train logs + accuracy-vs-baselines bars from eval.json
  • kev/api.py TypeSafe request/response models; Noul/Choice/Score -> pointer options; question_keys; confidence formulas; with_date_facts
  • kev/serve.py FastAPI: /v1/systemone (+ permute, separate) and /v1/models; Server holds the checkpoint and the prefix cache
  • kev/publish.py run -> Hub repo: adapter, head.pt, tokenizer, trial result/provenance/log, the card as README.md; --tag, --revision, --private
  • modal_app.py every GPU entrypoint: trials, locked tests, probes, benches, anchors, smoke
  • scripts/ one-off builders and read-outs: calibrate_checkpoint.py, build_night2_data.py, build_binding_diagnostic.py, freeze_{semif,calibration_audit}.py, {build,label,freeze}_documents_v*.py, calibration_audit.py, review_calibration_screen.py, compare_{q35,night2}.py, temperature_groups.py, base_mmlu_probe.py, plot_*.py + chartstyle.py, publish_space.sh
  • tests/test_api.py conformance against the docs' example requests + official SDK

Notes

  • All JSON/JSONL is UTF-8 with LF endings regardless of platform locale: read/write through kev.suite.read_json/read_jsonl/write_json/write_jsonl (or pass encoding=), and .gitattributes pins *.json/*.jsonl to LF so sha256-checked partitions survive a Windows checkout (issue #12).
  • runs/ is gitignored except ledgers, reports, provenance and training logs (see .gitignore); never commit weights, prediction dumps or regenerated runs/leaderboard.*.
  • Delimiters reuse existing Qwen special tokens (<|fim_prefix|> etc.) to avoid resizing embeddings; peft trainable_token_indices leaked memory on MPS.
  • output_hidden_states=True on MPS blows memory; use the bare .model backbone's last_hidden_state.
  • PolyAI/banking77 uses a loading script (unsupported); use legacy-datasets/banking77.
  • User text is tokenized via model.user_tokens(), which rewrites <|name|> -> <¦name¦> so callers cannot forge option/branch delimiter tokens (the fast tokenizer ignores split_special_tokens).
  • Training data is built as TypeSafe-shaped requests and goes through api.to_record() (data.materialize), so train and serve text are identical.
  • Serving path (kev.checkpoint.Checkpoint.load + kev.serve; LoadOptions.from_env() reads the KEV_* variables at CLI entry points only): backend auto (MLX for hybrid checkpoints on MPS when mlx-lm is installed and fp32 was not asked for; KEV_BACKEND=torch|mlx to force; library callers get torch unless they ask; SCORING_INTERFACE in kev/model.py names what both implementations expose), LoRA merged in fp32 then cast (KEV_MERGE=0 to keep unmerged, torch only), bf16 by default on CUDA/MPS (KEV_DTYPE=fp32 for the exact path; L4 numbers in README "Serving Performance"), KEV_ATTN=sdpa default on MPS, KEV_SHAPE_BUCKET=64 on MPS, state-prefix LRU (KEV_PREFIX_CACHE=4 states and KEV_PREFIX_MAX_TOKENS=65536 state tokens in all, a longer state is not cached; the states a batch will evict are dropped before its passes, PrefixCache.make_room; KEV_PREFIX_MIN_TOKENS defaults to the model's prefix_min_tokens: 384 for attention-only torch models, 0 for hybrid and MLX ones because their miss path would recompute the state per question). Checkpoints trained with --weights_dtype bf16 always load bf16 with the adapter unmerged. Any change here must keep the parity tests in tests/test_model.py (merged vs unmerged, prefix vs full pass, bucket padding) and tests/test_mlx.py (MLX vs fp32 torch, prefix form vs row form, isolation; Apple Silicon only) passing; report numbers with the fp32 unmerged path. scripts/mlx_parity.py --run <ckpt> is the fuller read (60 records, latency of every path); scripts/mlx_full_parity.py --full <ckpt> the full-weight one (MLX full vs its LoRA source vs fp32 torch, each load's memory in its own process); MLX's fp32 GPU matmul is a reduced-precision fast path (~1e-3 relative on an M5), which is why the LoRA merge runs on mx.cpu.
  • Serving context is 65,536 tokens for the state and 73,728 for the state plus one question branch, so a question gets at least 8,192 (kev.model.SERVE_MAX_*; 8,192 / 8,192 until the long-context PR, kev.suite.SERVING_CONTEXT_8K, which the suites frozen before it record and their builders keep); kev.train --max_state goes up to the same 65,536 (MAX_TRAIN_STATE). The released checkpoints trained on 384 / 1,024, so longer inputs are untested for accuracy. No limit on questions per request: the row form runs rows_per_pass rows per forward pass (ROW_PASS_TOKENS, a 16,384-token budget counting the cached state per row), and an attention-only model switches from the packed mask to rows above it (DecisionModel.rows_form). Evaluation is fp32-exact (as the next item defines it) except long rows: on CUDA with the torch backend, a record whose longest row exceeds ROW_PASS_TOKENS runs under SDPA's flash / memory-efficient kernels (the exact math kernel's L x L scores do not fit; kev.predictors.long_row_kernels) and is labelled: "kernels": "efficient" on its prediction and rows, long_rows: {count, records, kernels, threshold} in report.json (both absent when no row is long, so other reads are unchanged; LocalPredictor). An fp32 backbone (the small family) stays fp32 there. transformers asks SDPA for grouped-query attention (enable_gqa) on an unmasked call, which is every long state, and only flash (no fp32) and math take it. So until the long-state-memory PR the fp32 state pass fell back to math, and Kev-0.8B / 4B / 9B ran out of memory on 32k-64k states (the 2026-09-30 family longdoc reads). Inside long_row_kernels an fp32 call repeats its keys and values instead and runs the memory-efficient kernel in fp32; bf16 calls (Kev-27B) are unchanged. On an H100, 16k / 32k / 64k states: Kev-4B 4.3 / 8.6 / 17.2 GiB over the weights and 3.0 / 6.8 / 17.2 s, Kev-9B 4.6 / 9.1 / 18.2 GiB and 5.1 / 10.9 / 25.5 s (math: 33.6-33.7 GiB at 16k, then out of memory, on an H200 too). Against math at 8k-16k: max |dp| 6e-4, 0 flips in 238 questions (Kev-4B and 9B, H100) (scripts/long_state_memory.py, runs/long-state-*). Kev-4B on longdoc-v1 development (H100, runs/longmem-kev-4b-longdoc-dev-r2) now reads all 1,200 records (its 480 32k / 64k records long, 6.3 / 16.8 s median), and its 720 records under the threshold have the logits of main bit for bit. A long row short enough to have fit before (just over ROW_PASS_TOKENS) moves by up to the parity above, so pair such rows only with reads from the same side of this change. This is CUDA-only: on CPU / MPS attention stays eager and exact and still materialises L x L. A long record on a hybrid torch backbone runs its state once through the shared prefix; the MLX backend's forward already does. Kev-4B on MLX, 64 questions on a 4.8k-token state: 4.3 s / 9.4 GB peak instead of 18.5 s / 24.7 GB in one pass. n_perm on /permute is 1..64.
  • What "fp32-exact" guarantees (runs/drift-v1/REPORT.md). A read is bit-reproducible for a fixed kernel set: the same kev code, image (torch, transformers, flash-linear-attention, Triton, causal-conv1d) and GPU type give bit-identical logits run to run and container to container, and a kev change that leaves the scoring path alone keeps them (the 2026-09-23 code and main agree bit for bit on Kev-27B v1). A kernel change moves them. LocalPredictor's switches reach only PyTorch. On CUDA, flash-linear-attention's Triton kernels run the Gated DeltaNet rule of every Qwen3.5 / Qwen3.8 backbone with TF32 dot products, so an fp32 Qwen3.5 read sits ≈ 0.003 in p from true fp32 (Kev-4B, semif-v1: 1 flip in 252; TRITON_F32_DEFAULT=ieee closes it to 4e-6). A bf16 backbone (Kev-27B) is bf16 in every kernel. In both cases an upstream bit change is amplified. #125 put causal-conv1d's CUDA kernel in the Modal image, replacing transformers' PyTorch convolution. That alone moved Kev-27B v1 by max |Δp| 0.03-0.06 (0-0.4 % flips, ECE +0.0008 on breadth-v1) and Kev-4B by 6e-4, so reads from before and after #125 differ by that much. CPU / MPS reads run no Triton or causal-conv1d and differ from CUDA reads by the same terms. Pair reads from the same kernel set. When a parent read is older than an image change, re-read the parent; a deployed app keeps the image of its last deploy. runs/drift-v1/probe_app.py swaps the convolution and the delta rule in place to measure one change.

Calibration Research

  • Metric version 2 accepts whole equal-confidence groups. paired_bootstrap(..., aggregation="micro") recomputes non-additive coverage/AURC for each paired record-group resample; default macro is equal task weight.
  • confident_error_rate divides confident errors by all questions. error_rate_at_0_9 divides by accepted questions. The empirical coverage-at-error envelope is not an unseen-data guarantee; freeze thresholds on a separate calibration set.
  • Local benchmarks save logits and effective inference temperature. Research studies explicitly score raw logits, fit on the calibration partition for both parent and candidate, and report raw/recalibrated metrics separately. Temperature can reorder confidence across multiclass questions, even at fixed option count.
  • Registered screen: experiments/calibration-audit-protocol.json; scripts/review_calibration_screen.py --study <name> --out runs/<new-review> verifies matched updates/tokens and gates against both unchanged parent and CE continuation. The screen ran and no candidate passed (round 3; PLAN.md Record, full text at tag research-archive-2026-09-24): the --label_smoothing/--brier_w/--focal_gamma losses exist but ship in nothing.
  • evals/round3/transfer-r3/test.jsonl was frozen as a fresh final panel; the loss screen found no qualifying candidate, so its first read was the 2026-09-22 release confirmation of the soft-target Kev-9B, and since round 11 it is half of the pooled short-state guard. It is spent as confirmation. Select and record a candidate before evaluating it; do not reuse previously inspected test partitions as untouched confirmation.
  • Isolate research deployments with KEV_APP_NAME=kev-calibration-audit. worker_environment propagates app/GPU/secret-name settings to prevent Modal dependency-count startup failures; secret values stay in Modal Secrets.
  • Verify live spend/rates with uv run modal billing summary --json and uv run modal billing rates --json. Training admission uses the actual configured CPU and maximum host memory, not the old 2-CPU/48-GiB assumptions. Initial cancelled startup calls and successful jobs are recorded separately.

Writing

  • Use simple technical English. For README tone, use Jared's older Formik, TSDX, Razzle, and Backpack READMEs as references: explain the developer's problem, address the reader directly, and show code early. Avoid slogans, canned contrasts, and repeated claims. Keep detailed experiment history in PLAN.md and the model cards rather than repeating it in the README.
  • README order is progressive: what Kev is, Highlights and the Models table (the first 30 seconds); Quick Start, Fine-Tune, Deploy and What to Expect (the first 5 minutes); then Playground, API, How It Works, Training, Benchmarks, Serving Performance, Limitations. The README describes the current family only. A release updates its Models row, the MODELS/LOCKED lists in scripts/plot_tweet.py and scripts/plot_family.py (then regenerate) and docs/claims.json; the dated release paragraph, the before/after numbers and the previous tag go in the model card and PLAN.md, not the README.
  • Pull requests teach. Write the body the way Andrew Clark (acdlite) and Sebastian Markbåge (sebmarkbage) wrote theirs on facebook/react before 2023: the problem as it exists today first, then the mechanism as an argument with one concrete artifact per idea, then what is uncertain and what was left out. Define the one term the change hinges on (pointer head, temperature, flip rate, AURC, prefix cache) the first time it appears, so an engineer who is not an ML researcher can follow. Numbers carry their provenance (checkpoint, suite + partition, n, report path); the decision rule is written before the result. A list of files changed is the diff again, not a description. Length follows novelty: one sentence for a one-line fix, headings for a new loss or serving path. The title becomes the squash commit subject. The kev-pr-description skill has the reference PRs and a weak/strong pair.