Snapshot: 2026-09-04. This is a single consolidated record of every number, every success, and every failure produced while building Kosh — the AI Finance Controller for Razorpay Buildathon 2026, Track 04. It is deliberately exhaustive: README.md gives the curated headline story and ARCHITECTURE.md gives the narrative build history; this file is the detailed ledger underneath both, kept as one file so numbers don't drift across three places. If you regenerate any benchmark, re-check the numbers here rather than trusting this file blindly — every figure below names its source file.
How this file is laid out. §§1–10 are the original seven-phase build (through commit 85350a3), left as originally written. §11 is the hardening sprint that followed — multi-seed validation, adversarial red-teaming, an all-LLM ablation, malformed-input handling, production hardening, and a final re-freeze. Code is frozen as of §11.8: no further engine commits, and anything found later goes into Known Limitations rather than a rushed fix against an already-reported number. §§12–15 are what came after the freeze — the FastAPI/dashboard stretch goals, the full-scale L3 run, validation past the frozen scales, and a robustness sweep — none of which touched the engine.
Test counts quoted inside a section are the count at that point in the build, not today's. Current state: 257 tests passing on Python 3.11 / 3.12 / 3.13 / 3.14.
Full pipeline (L0 → L1 → L2 → L4, no live LLM — L3's real numbers are in §4), against data/fixtures/run_{500,2000,10000} (seed 42), each a from-scratch synthetic dataset with 14 injected, labelled defect types and machine-readable ground truth.
| Records | Auto-match | Precision | Recall | False-match | F1 | Exceptions | ₹ at risk | Wall clock | Records/sec |
|---|---|---|---|---|---|---|---|---|---|
| 500 | 97.61% | 100.00% | 99.91% | 0.00% | 99.96% | 48 | ₹2,84,893.99 | 2.50ms | 184,480 |
| 2,000 | 97.74% | 100.00% | 99.95% | 0.00% | 99.98% | 139 | ₹16,72,013.03 | 7.72ms | 240,726 |
| 10,000 | 97.83% | 100.00% | 100.00% | 0.00% | 100.00% | 550 | ₹49,33,264.88 | 35.21ms | 264,630 |
Source: benchmarks/freeze_500.json, freeze_2000.json, freeze_10000.json.
False-match rate is the number that matters most. In finance a wrong match is worse than no match — an unmatched item sits in a queue for a human; a wrong match silently corrupts the books. It reads 0.00% at every scale, checked directly against ground truth, not asserted. It is not a vacuous zero: the eval harness is mutation-tested — inject 10 deliberately wrong links and it reports 0.24% (see tests/test_eval_baselines.py); drop half the true matches and recall correctly halves. The harness demonstrably fails when the engine is wrong.
| Baseline | Auto-match | Precision | Recall | False-match | Exceptions |
|---|---|---|---|---|---|
| Null (matches nothing) | 0.00% | — | 0.00% | 0.00% | 1,858 (100% of records) |
| Oracle (reads ground truth directly) | 97.85% | 100.00% | 100.00% | 0.00% | matches true unresolvable-defect count |
Source: benchmarks/run_null_run2000.json, run_oracle_run2000.json. These exist to prove the scorer is honest before any real matching logic existed — see tests/baselines/.
The brief gives a numeric target per phase. Below is target next to the real measured number, not rounded in its favor.
| Phase | Scope | Brief's target | Actual | Source |
|---|---|---|---|---|
| 1 | Data generator + ground truth | Byte-identical regen; every amount an integer; ~200 defects across all types; one full chain hand-verified | Verified; 14/14 defect types present (12 in the original brief, 2 added during hardening, §6.9) | data/fixtures/run_2000/manifest.json |
| 2 | Eval harness + baselines | Null → 0%/0%/0%/100%; oracle → ~100% precision/recall | Exact match, see table above | tests/baselines/ |
| 3 | L0 + L1 | Auto-match ≥88%; false-match 0.00%; <5s @ 2,000 records | 92.84%, 0.00%, 4.09ms | benchmarks/phase3.json |
| 4 | L2 subset-sum | Auto-match ≥94%; false-match 0.00%; p99 solve <250ms | 97.74%, 0.00%, 49.82ms p99 | benchmarks/phase4.json, phase4_solver_perf.json |
| 5 | L3 agent + L4 ledger | Auto-match ≥96%; false-match ≤0.1%; cost <$0.50/1000 records; every exception has a recommended action | 97.74%, 0.00%, $0.056/1000 records (real live run) | benchmarks/phase5.json, phase5_live_residual.json |
| 6 | Cash + dashboard + freeze | 3-scale freeze committed; results above the fold; fresh-clone make demo works |
Done, verified in isolated clone + clean venv | §1 above |
| 7 | Submission | README/ARCHITECTURE/demo script ready; public repo; 5-min video | Done. Public repo pushed, video recorded, submission form filed on 2026-09-05 | ✅ |
Every one of these targets is met. The one caveat: Phase 5's cost checkpoint had no real number behind it until a bug fix on 2026-08-31 (§6.13) — the number above is the first one that was ever actually computed rather than silently hardcoded to 0.
Share of correctly-matched links contributed by each deterministic layer, at all 3 scales:
| Layer | 500 | 2,000 | 10,000 | What only it can do |
|---|---|---|---|---|
| L0 exact-key | 97.76% | 99.33% | 99.82% | UTR/FK joins |
| L1 tolerance | 0.78% | 0.26% | 0.06% | UTR rekeyed with a transposed digit |
| L2 subset-sum | 1.47% | 0.41% | 0.12% | Consolidated payouts — one credit, 2–4 settlements, no per-settlement UTR |
| L3 agent (live) | — | 6 records seen, 0.024% of link-shares | — | Genuinely UNEXPLAINED variances no deterministic rule can decompose |
L3 saw 6 of 1,858 records (0.32%) on run_2000 — the other 99.68% cost zero LLM tokens. That is the deterministic-first thesis, quantified, not asserted.
run_2000's residual for L3 is 6 records (payments with a genuinely UNEXPLAINED fee/GST variance that L0–L2 and L4's deterministic classifier could not decompose). This section is the complete history of getting a clean, fully-verified result from a real model — including the three attempts that were not clean, and exactly what was wrong with each.
Model: nvidia/nemotron-3-ultra-550b-a55b via NVIDIA NIM. The brief specifies Anthropic's Claude; no Anthropic key was available, so this is a documented deviation — engine/llm/anthropic_client.py is spec-complete and unit-tested against a mocked client, but has never been run live.
- Result: 2 correct matches, 4
AGENT_INCOMPLETE. - What actually happened, verified by reading raw traces: all 4 incomplete cases showed genuinely thorough investigation — 3 separate
explain_variancecalls correctly probing fee, GST, and net variance independently (this is exactly thecompound_fee_tax_errordefect shape: two overlapping causes, neither hypothesis alone explains it) — then 2 more calls checking whether a settlement-batching explanation applied. One case was rejected bypropose_match's own validation on its literal last turn (it tried to assert a link to a bank_txn_id that wasn't the record under investigation's own direct relation) with zero turns left to retry a corrected call. - Diagnosis: the turn budget was genuinely one turn too tight for this defect class, not a capability gap.
- Along the way, this same live-run effort also caught and fixed two infrastructure bugs before reaching this result (§6.1, §6.2) and one earlier false-match bug during the original Phase 5 build (§6.3).
- Result: 3 matches, 3 exceptions. Zero
AGENT_INCOMPLETE— the budget fix worked. - But: independent verification (checking every claim against
ground_truth.jsonby hand, not trusting the pipeline's own score) surfaced two new bugs in the same run:- Two exceptions came back with invented category strings —
"FEE_CALCULATION_VARIANCE"and"unexplained_fee_variance"— neither a real category inRECOMMENDED_ACTIONS. Silently accepted, no error anywhere (§6.5). - One exception reported
amount_at_risk_paise: 58700(₹587, the payment's full gross) for a defect ground truth labels at 142 paise (₹1.42) — the model cited the wrong number entirely (§6.6).
- Two exceptions came back with invented category strings —
- Result: 0
AGENT_INCOMPLETE, 0 invented categories. But checkingcompute_defect_confusion's own scorecard revealed a third bug:FEE_VARIANCE/TAX_VARIANCEexceptions raised by L3 were invisible to per-defect-class scoring — counted as "missed" even though they had been correctly raised, becauseraise_exception'saffecteddict only ever carried a genericrecord_id/sourcepair, not thepayment_idfieldeval/metrics.py's scoring actually looks for (§6.7).
- Result (
benchmarks/phase5_live_residual.json, traces inbenchmarks/sample_traces_live/):
| Record | Outcome | Independently verified |
|---|---|---|
pay_3egKQ6BCralBAI |
TAX_VARIANCE, ₹0.90 at risk |
Ground truth: compound_fee_tax_error, ₹0.90 exactly. Judged immaterial, correctly not blocking anything else. |
pay_ymzQx3u8WEhd7G |
UNEXPLAINED_VARIANCE, ₹41.19 at risk |
Ground truth: ₹41.19 exactly. |
pay_OyvjU0Hc7g7Bi2 |
FEE_VARIANCE, ₹2,286.93 at risk |
Ground truth: ₹2,286.93 exactly. The largest of the 6 — real money, correctly flagged, not swept under the rug. |
pay_Yw6hEZsEyvZMNn |
UNEXPLAINED_VARIANCE, ₹1.42 at risk |
Ground truth: ₹1.42 exactly. |
pay_RMejvzSwrh9QXa |
Match → payment_settlement → setl_P65ExMApYLxX4s |
Confirmed against ground_truth.json's payment_to_settlement links directly — correct. |
pay_dGxUjmPIxeeXo4 |
Match → payment_settlement → setl_WCpMEdv6RwTwIE |
Confirmed against ground_truth.json directly — correct. |
- 5 of 6 exception amounts exact to the paisa against the generator's own labelled defect amount. Zero false matches. Zero
AGENT_INCOMPLETE. - Numbers: 51 LLM calls, 137,644 input tokens, 16,122 output tokens, 486.7s wall clock, $0.104 total, $0.056 per 1000 records (against the full 1,858-record dataset, matching
eval/metrics.py's own convention — not the 6 records L3 actually touched). - The one remaining imprecision, documented rather than hidden:
compute_defect_confusionshowscompound_fee_tax_error: {detected: 1, misclassified: 4, missed: 1}for this run. The 4 "misclassified" are cases where the model correctly raised an exception with the correct money and the correct record, but labelled itFEE_VARIANCEwhere the generator's own expected label isUNEXPLAINED_VARIANCE(a defensible call — the model named the anomalous leg it found rather than the fact that multiple legs are simultaneously unexplained). The 1 "missed" is the genuinely-immaterial ₹0.90 case, judged not worth a standalone exception alongside its correct settlement match. See thedefect_confusion_notefield in the committed JSON for the full reasoning.
A separate, hand-built 5-record set exercising every hard constraint at least once (an unexplained variance, an ambiguous refusal, a subset-sum batch, a high-value review flag, a turn-budget exhaustion), run against the same live model. Result: 2 matches, 5 exceptions (matches + exceptions overlap when a batch resolves as one match with a companion review exception), 39 LLM calls, $0.0733 total. Zero false matches. Its first-ever live run caught the false-match bug in §6.3 before this cleaned-up version was committed. Source: benchmarks/phase5_synthetic.json, benchmarks/sample_traces/.
| Correctly flagged as exceptions | Correctly resolved silently (must NOT become exceptions) | ||
|---|---|---|---|
missing_settlement |
20/20 | rounding_drift |
25/25 |
duplicate_payment |
20/20 | utr_mangled |
20/20 |
fee_mismatch_wrong_tier |
19/19 | settlement_split |
11/11 |
gst_variance |
15/15 | consolidated_payout |
8/8 |
period_cutoff |
14/14 | ||
orphan_chargeback |
14/14 | ||
unidentified_credit |
14/14 | ||
refund_misallocation |
9/9 | ||
fx_variance |
8/8 | ||
compound_fee_tax_error |
6/6 |
Zero misses, zero misclassifications, zero false exceptions on the deterministic-only run. The right-hand column is the harder half: four defect types exist specifically to test whether the engine raises a false alarm on something that should resolve silently — precision on not crying wolf is scored as hard as recall on real defects. Source: benchmarks/freeze_2000.json's defect_confusion field.
This project's own judging bar says an honest exception list beats a shorter, suppressed one. The same standard applies to the build process. Every one of the following was a real defect, caught by actually running something (a fresh clone, a live model, an actual click on the dashboard) rather than by inspection — in every case the mechanism that caught it is named, because "I checked" is not evidence and "I ran it and it failed this specific way" is.
Found: first live L3 batch run died with openai.InternalServerError: Error code: 504 and every result was lost — asyncio.gather propagates the first exception and discards every sibling result.
Fixed: added TransientBackendError (distinct from RateLimitedError — a 5xx means the request was never processed, so retrying is safe), retried with the same exponential backoff; each record in run_l3 now runs in isolation so one failure becomes its own honest AGENT_INCOMPLETE ledger entry instead of destroying five completed investigations.
Verified: two new tests (a transient error retried then succeeds; a permanently-failing record leaves its sibling's result intact). Commit bf9db67.
Later in the same live-run effort, the model returned NotFoundError: 404 (a genuine, temporary NIM-side outage, confirmed by a direct minimal call and client.models.list() still showing the model in the catalog). 404s are deliberately not retried (retrying a permanently-gone resource wastes budget) — the per-record isolation fix from §6.1 handled this correctly on the first try, proving itself against a harder failure mode than it was built for.
Found: during the original Phase 5 build, a live run showed the model doing genuinely excellent reasoning — correctly tracing payment → settlement → the settlement's own bank credit — then calling propose_match with all three IDs. The tool's _infer_link_type inferred a link type from table names alone: any (payment, bank) pair became "chargeback_payment", regardless of whether that bank row was actually a debit. The tool asserted "this payment has a chargeback" against what was actually its own settlement credit — a confirmed false link, exactly the class of error this whole project exists to hold at 0%.
Fixed: link-type inference now checks the bank row's actual debit_paise; a credit is rejected outright with a ToolError naming the real scope boundary (propose_match only asserts links to the record under investigation, not between two other bundled IDs).
Verified: new regression tests — the bad case is rejected outright (not partially applied), the genuine chargeback case still links correctly. Commit 78d339f. The run that surfaced this was NOT committed as the reference synthetic set, precisely because it contained a confirmed false match.
Found: reading raw trace files for all 4 AGENT_INCOMPLETE cases from the first clean live residual run (§4, Attempt 1) — every one showed genuinely thorough, correct investigation that simply ran out of budget by roughly one turn.
Fixed: DEFAULT_MAX_TURNS 8 → 12.
Side effect caught in the same pass: the trace cache key (_cache_key) didn't include max_turns — changing the budget without also bumping PROMPT_VERSION would have silently replayed a stale trace from the old budget forever. Fixed by adding max_turns to the key directly.
Verified: new test proving a changed max_turns misses the cache correctly. Commit 393b1fb.
Found: Attempt 2's live run (§4) returned "FEE_CALCULATION_VARIANCE" and "unexplained_fee_variance" — neither a real category — accepted with zero validation, silently breaking category-keyed scoring downstream.
Fixed: raise_exception now validates category against RECOMMENDED_ACTIONS's keys (also exposed as a JSON-schema enum on the tool spec, so the model sees the valid list up front), rejecting an invalid one with a ToolError that names the real options — giving the model a chance to self-correct within its remaining turns rather than polluting the ledger.
Verified: new test confirms an invented category is rejected and the loop doesn't silently close. Commit 393b1fb.
Found: the same Attempt 2 run reported amount_at_risk_paise: 58700 (₹587, the payment's full gross amount) for a defect whose real variance was 142 paise (₹1.42) — the model conflated "the transaction size" with "the amount actually at risk."
Fixed: the raise_exception tool schema now explicitly documents amount_at_risk_paise as "the specific unexplained amount... NOT the payment's or settlement's full gross/net amount," with a worked example.
Verified: Attempt 4's re-run shows 5 of 6 amounts exact to the paisa against ground truth — direct before/after evidence the fix worked. Commit 393b1fb.
Found: after fixing §6.5–6.6, compute_defect_confusion still showed compound_fee_tax_error: {detected: 2, missed: 4} — but 4 of those "missed" cases had, in fact, been correctly raised as exceptions. Root cause: raise_exception's affected dict only ever carried {"record_id": ..., "source": ...}; eval/metrics.py's CATEGORY_IDENTITY_FIELDS for FEE_VARIANCE/TAX_VARIANCE/etc. looks for "payment_id", a field that was never there. A correct exception, invisible to scoring, looks identical to a missed one.
Fixed: affected now also carries the source-specific ID field (payment_id/bank_txn_id/etc, via the same _ID_FIELD map get_record already uses), in both raise_exception and its HIGH_VALUE_MATCH_REVIEW companion, and in l3_agent.py's upstream-failure fallback path. Also added AGENT_INCOMPLETE to CATEGORY_IDENTITY_FIELDS so a turn-budget exhaustion now scores as "misclassified" (something was raised) rather than "missed" (nothing was raised).
Verified: new tests construct a real FEE_VARIANCE exception via raise_exception and confirm compute_defect_confusion now counts it correctly; a second test does the same for AGENT_INCOMPLETE. Commit 393b1fb.
Found: by actually running git clone && pytest — the judge's most natural command — rather than the sequence used during development (make demo first, which happens to generate the needed fixture as a side effect, masking the bug).
Fixed: tests/conftest.py gained a session-scoped autouse fixture that regenerates data/fixtures/run_2000 only if genuinely absent — regeneration is deterministic and takes under a second, so this is the honest fix rather than committing derived CSVs as source.
Verified: fresh clone + clean venv + git clone && pytest alone: 151/151 (was 144 passed / 7 failed); the auto-generated fixture reproduces committed benchmark numbers to 1e-12. Commit a0c6894.
Found: not a crash — an honestly-reported result that nonetheless left the architecture's core claim ("five layers, each earning its place") unproven on the actual demo data. Root cause: the original generator was too kind, not the engine too good.
Fixed: two new realistic defect injectors — consolidated_payout (one bank credit paying 2–4 same-day settlements, no per-settlement UTR, solvable only by subset-sum) and compound_fee_tax_error (fee tier AND tax rate both wrong at once, so no single-cause hypothesis decomposes it — genuine L3 work). Two choices deliberately cut against making the numbers look easy: L1's narration-similarity gate was made stricter so it correctly rejects consolidated payouts on amount rather than unfamiliar wording; a genuine ₹0-net-settlement ambiguity was fixed by excluding degenerate zero-value subset-sum terms rather than guessing, costing 2 links of recall (99.95%, not 100%) on purpose.
Verified: full re-run at all 3 scales, false-match rate held at 0.00% throughout; L2 and L3 now measurably non-zero (§3). Commit 7e43eb9.
Found: while checking Phase 5's own "<$0.50/1000 records" cost checkpoint against Attempt 4's live run — there was no real number to check. EngineMeta.cost_usd_micros existed in the contract and was correctly threaded through eval/metrics.py's cost_per_1000_records formula, but nothing anywhere ever set it. Every committed benchmark, including live runs with real tokens billed against a real account, reported exactly 0.
Fixed: engine/llm/pricing.py — a small per-model $/1M-token table (NIM's rate for nvidia/nemotron-3-ultra-550b-a55b sourced from its public OpenRouter listing, since NVIDIA doesn't publish per-token pricing for the hosted NIM preview endpoint itself; Anthropic's official rates included for the never-live AnthropicClient) feeding cost_usd_micros(model, input_tokens, output_tokens), wired through l3_agent.py's per-trace and aggregate cost, pipeline.py's meta construction, and a new "LLM cost" headline card in eval/report.py — shown only when llm_calls > 0, so a deterministic-only run never shows a misleading "$0.00".
Verified: recomputed retroactively from already-recorded token counts (no new API calls needed) for the two existing live-run artifacts; new unit tests for the pricing function (known model, zero tokens, unknown model reports 0 not a guess, result is always an int). Commit c4ae195.
Found: the exact same failure shape as §6.10, found by the same discipline — swept every field in engine/contract.py after finding the cost bug. aging_days is part of the brief's own exception-ledger shape and its own dashboard column; every row read "0d," for every exception, at every scale.
Fixed: infer_as_of_date(dataset) extracted to engine/io.py as a shared source of truth (deduped from cash/forecast.py's private, near-identical copy), plus aging_days(as_of, event_date_str). Every deterministic classifier now passes the record's own real underlying event date (a payment's captured_at, a settlement's settled_at, a bank line's value_date); L3's raise_exception and its companion do the same for the residual record under investigation.
Side bug caught in the same pass: infer_as_of_date crashed with max() arg is an empty sequence on a dataset with zero payments — true for two existing isolated-classifier unit tests. Fixed with a fallback chain (settlement/bank dates, then real wall-clock as a last resort for a fully-empty dataset) rather than coupling every isolated test to a full payments list.
Verified: every committed benchmark's accuracy numbers confirmed byte-identical before/after by diff (not assumed) — only aging_days and naturally-volatile timing fields changed. 5 new tests. Commit adc6f62.
Found: the fourth field in the same shape — never unset (always its dataclass default, "Reconciliation Ops"), but never overridden either. Every category, every exception, every scale showed the identical owner — a column that carries zero information is the same failure as one that's silently 0.
Fixed: SUGGESTED_OWNERS, a category-to-team map (Payments Ops, Finance Ops, Tax & Compliance, Disputes & Risk, Treasury, Finance Controller for high-value reviews, Reconciliation Ops kept as the genuine generalist catch-all), wired through both exception producers.
Verified: freeze_2000 now shows 6 distinct owners across 139 exceptions, not 1. New tests confirm three different categories get three genuinely different owners. Commit ebd040d.
Found: a pyflakes/coverage sweep, then manual verification — test_timeout_returns_timeout_status_not_a_hang's own instance (40 candidates, an exactly-reachable target) hit MAX_SOLUTIONS_TO_DETECT (2) within the first few root-to-leaf paths, empirically confirmed to resolve in under 0.1ms, 5/5 trials. The assertion accepted AMBIGUOUS/SOLVED/TIMEOUT as all "passing," so this had silently never exercised the branch it was named after.
Fixed: changed the target to one exactly unreachable (50 paise short of any achievable multiple, with zero tolerance) — zero solutions ever exist, so the search can never short-circuit and must genuinely explore the space. Confirmed empirically: exceeds a 10-second budget, and hits TIMEOUT deterministically (5/5 trials) at a realistic 50ms one. Assertion tightened to require exactly "TIMEOUT".
Verified: the fixed test passes and demonstrably exercises the real branch (unlike its predecessor). Commit fc71036.
Found: by actually clicking the live dashboard (via a local HTTP server, so JavaScript genuinely executes — a static file snapshot does not) rather than trusting a substring test. The existing test only checked that the string "function sortExceptions" appeared in the rendered HTML — it could never have caught a header wired to the wrong column. onclick="sortExceptions()" on the "Category" <th> called a function that only ever read dataset.amount. It looked correct only because the table's own default order is already amount-descending, so clicking "Category" appeared to do something (it just re-sorted by amount, invisibly).
Fixed: sortExceptions(column) is now parameterized ('amount' | 'category'), each row carries both data-amount and data-category, each header calls the function with its own real column name, with independent per-column ascending state. Kept the brief's own explicit "sortable by ₹ at risk" requirement on the amount column, while making "Category" genuinely sort alphabetically as its own label promises.
Verified: real .click() events dispatched on the actual header DOM elements on a live server — not just the function called directly — confirmed both columns sort correctly and independently. New tests check the header's onclick names its own column, not just that the function exists. Commit 85350a3.
Leftover unused imports/variables from earlier refactors (a Counter no longer needed after a rule change, an unused SettlementRow, unused dataclasses.field/asdict imports, six unused test imports, two unused test locals). All confirmed genuinely unreachable before removal; critically, regenerating the affected fixtures afterward reproduced byte-identical output (the removed code consumed no RNG state), so no committed benchmark was invalidated. Commits c0d796c, plus one more unused local found and fixed on 2026-09-01 in a test written that same day.
- 174/174 tests passing as of this snapshot (
pytest, ~4.9s). - Fresh-clone verified:
git clone→ isolated venv →pip install -e ".[dev]"→pytest→ 142 passed, 2 skipped (the 2 skips areanthropic/openai-dependent tests, gracefully skipped viapytest.importorskipwhen the optionalllmextra isn't installed — not a failure). pyflakesclean acrossengine/,eval/,cash/,data/,tests/.- Coverage (
coverage run -m pytest,engine/*,eval/*,cash/*): 86% overall.cash/forecast.py,engine/io.py,engine/baselines.py,eval/metrics.py,eval/io.pyall at 100%. The lowest-covered files (l2_subset.py70%,l3_agent.py70%) are dominated by--profileCLI entry points that are exercised by actually running them live (that's what producedphase4_solver_perf.jsonand every live L3 benchmark in §4), not by pytest unit tests — a deliberate choice, not a gap. One specific line inl2_subset.py's hot recursion path was manually traced and confirmed to be intentional defensive redundancy (the real enforcement happens one line earlier in the actual call structure), not a bug — left alone on purpose rather than "cleaned up" for a coverage number.
- Recall is 99.95% on
run_2000, not 100%: two settlements net to exactly ₹0, so a bank credit genuinely cannot evidence whether they rode along in a consolidated batch. Kosh refuses rather than guesses — this costs 2 links on purpose. PERIOD_CUTOFF's >4-day threshold is tuned to this fixture's observed distribution, not a law; one boundary case at exactly 3 days is genuinely indistinguishable from a slow weekend and is left as real residual rather than forced.propose_match's rationale-citation check (constraint 5) is a best-effort structural check — does the text contain a known record ID — not a semantic verification that the citation actually supports the claim. Documented as such, not silently assumed to be stronger than it is.- The live L3 residual on
run_2000is small (6 records). This was later addressed by running the agent againstrun_10000's 29-record residual — see §13 — which gave a real rate (25/29, 86%) and surfaced two findings six records could not: a turn-budget exhaustion, and the deterministic fallback outscoring the LLM on this one defect class. - No FastAPI layer, no live interactive dashboard — the static HTML report is the deliverable, per the brief's own explicit fallback preference over a broken from-scratch app.
- The brief specifies Anthropic's Claude; the real live run used NVIDIA NIM's Nemotron because that's the key available.
AnthropicClientis spec-complete and unit-tested against a mock, never run live. Documented, not hidden. auto_match_rateandhands_off_rateare currently defined identically (eval/metrics.py) — holds until a layer exists that can leave a record neither matched nor exceptioned.false_match_rateis computed against the engine's own asserted links, not total records — deliberate (can't be gamed by asserting fewer links), but must always be read next to auto-match rate, never alone.
Nothing. Demo video: recorded. Public repo push: done, github.com/bodapatisaikrishna/kosh, public, CI green on Python 3.11–3.14. Submission: filed on 2026-09-05.
Everything described above is complete, verified, and submitted.
git clone <this-repo> && cd kosh
pip install -e ".[dev]" # +.[llm] for the anthropic/openai-backed tests
pytest # 174 passed (142 passed, 2 skipped without the llm extra)
make demo # regenerates run_2000 + benchmarks/run_demo.html
python -m eval.report --fixtures data/fixtures/run_2000 --engine full --label freeze_2000
python -m engine.l2_subset --profile --trials 2000 --seed 42
export NIM_API_KEY=...
python -m engine.l3_agent --profile --backend nim --model nvidia/nemotron-3-ultra-550b-a55bFull command reference in README.md's "Everything else" section and the Makefile.
11. Hardening sprint (2026-09-01/02) — proof the results aren't an artifact, plus production hardening
A 7-task, gated sprint against the completed build above (§§1–10): prove the 0.00% false-match claim under multi-seed validation and adversarial red-teaming, measure an all-LLM alternative, harden malformed-input handling, and add the production properties (determinism, CI, a run manifest, pinned dependencies). Non-negotiable rules throughout: no benchmark regresses without being reported first, no new features, integer paise everywhere, every fix gets a regression test, enforce structurally not by prompting, never silently redefine a metric, report real numbers. Tasks 1–6 are done and committed; Task 7 (re-freeze + code freeze) has not run.
Goal: prove 0.00% false-match isn't a seed=42 artifact. scripts/multiseed.py generates a fresh 2,000-record fixture at each of 6 seeds to a temp directory, scores it with the full deterministic pipeline (run_full(client=None)), and deletes the fixture — only the summary is committed.
| Seed | Auto-match | False-match | Exceptions | L3 residual |
|---|---|---|---|---|
| 1 | 97.86% | 0.00% | 142 | 6 |
| 7 | 97.87% | 0.00% | 141 | 6 |
| 42 | 97.74% | 0.00% | 139 | 6 |
| 100 | 96.99% | 0.00% | 142 | 5 |
| 2026 | 97.85% | 0.00% | 140 | 5 |
| 31337 | 97.96% | 0.00% | 138 | 6 |
Mean auto-match: 97.71%, stddev: 0.36 percentage points. False-match: 0.00% at every single seed, no exceptions. All 6 fixtures independently show 14/14 defect types present. Source: benchmarks/multiseed/summary.json, tests/test_multiseed.py (regenerates live against the script's own function, not a frozen snapshot). Commit 04918c9.
Found (pre-existing, not new): L3 labelled a compound fee+GST error as whatever single category the model happened to check first (FEE_VARIANCE), citing one leg's delta even when two legs partially offset — 744 paise reported vs. the true net 183 paise.
Fixed structurally, in the tool layer, not the prompt (engine/l3_tools.py): explain_variance_tool now records a VarianceObservation per call; raise_exception coerces category to UNEXPLAINED_VARIANCE and recomputes amount_at_risk_paise from the bottom-line (largest abs(expected_paise), not largest abs(delta_paise) — that was the original bug) whenever 2+ genuinely unexplained legs are on record, deduped so a retried identical comparison can't inflate the count. A single unexplained leg below the threshold deliberately does not coerce — the false-positive guard that keeps single-cause FEE_VARIANCE exceptions correctly labelled. 7 new tests.
Deterministic layers confirmed byte-identical before touching anything live (L0/L1/L2 untouched by this fix). PROMPT_VERSION bumped 1→2 to bust the trace cache, then the real 6-record run_2000 residual re-run live against nvidia/nemotron-3-ultra-550b-a55b:
| Run | compound_fee_tax_error |
Note |
|---|---|---|
| Before the fix | 1 detected / 4 misclassified / 1 missed | Original bug |
| After (run A) | 6/6 correctly UNEXPLAINED_VARIANCE |
Model checked both legs |
| After (run B, minutes later) | 4/6 UNEXPLAINED_VARIANCE, 1 correctly TAX_VARIANCE (single-cause, correctly not coerced), amounts exact in all 6 either way |
Model checked only one leg that run — genuine LLM run-to-run variance, not a bug (see the trace: unexplained_leg_count: 1) |
Both runs: zero AGENT_INCOMPLETE, zero false matches, zero wrong amounts, all 6 amounts exact to the paisa. Reported as both real numbers, not the better one cherry-picked. Source: benchmarks/phase5_live_residual.json, benchmarks/sample_traces_live/. Commit f553bd7.
Post-freeze addendum (2 more real runs, done specifically because 2 data points isn't enough to claim a variance range): a 3rd run scored 5/6 detected, all 6 amounts still exact — benchmarks/phase5_live_residual_run3.json. A 4th scored 3/6 detected and surfaced a genuinely new finding: for pay_ymzQx3u8WEhd7G, the model investigated only the fee leg (expected 29,195 paise, observed 30,732 paise, delta 1,537 paise — arithmetic exactly correct) and never called explain_variance on the GST leg at all, so the coercion's unexplained_leg_count: 1 correctly did not fire — but the record's true ground-truth amount is 4,119 paise, not 1,537. The "all amounts exact to the paisa" claim above holds for those first two runs specifically, not as a universal guarantee — across all 4 real runs, detected counts are 6/4/5/3 out of 6 (not the narrower 4-6 the first two suggested), and a single-leg investigation's reported amount is only exact when that one leg happens to dominate the record's true compound total, which was true in every earlier case by coincidence and false in this one. Not a computation bug — the leg's own math is correct — a live-model investigation-completeness limitation, disclosed here rather than smoothed into a cleaner-looking range. Source: benchmarks/phase5_live_residual_run4.json, benchmarks/sample_traces_live_more_runs/run4_pay_ymzQx3u8WEhd7G_amount_understated.json.
Seven inputs hand-engineered to induce a false match at a specific layer, run through run_full(client=None) — no LLM, no cost. Every outcome must be REFUSED (no wrong link) or CORRECT (the one true link); FALSE_MATCH is a bug to report, not a test to adjust.
| # | Attack | Layer targeted | Outcome |
|---|---|---|---|
| a | Two settlements, UTRs differing by a transposed digit, truncated narration | L0/L1 | REFUSED |
| b | Bank credit's amount coincidentally sums two unrelated settlements 60 days out | L2's date window | REFUSED |
| c | Duplicate payment, one order, only one genuinely settled | L0/L4 | CORRECT (flagged DUPLICATE_PAYMENT) |
| d | Refund reduces one settlement's net to match a second, unrelated settlement | L1 | REFUSED |
| e | Fee-adjusted net coincidentally equals a different zero-fee settlement's gross | L1 | REFUSED |
| f | Identical UTR text on two separate bank rows, same settlement | L0 double-claim | REFUSED (after a real bug was found and fixed — see below) |
| g | Two different subsets of three settlements both sum to one bank credit | L2's ambiguity guard | REFUSED |
Attack f found a real bug: l0_deterministic.py matched the same settlement to two different bank rows sharing one UTR — a genuine double-claim, silently double-counting cash. The first fix attempt ("a settlement claimed once can't be claimed again") broke a real feature — settlement_split legitimately reuses one UTR across two genuine partial payouts — inflating unidentified_credit_paise on run_2000 by ~3.65x, caught by the existing reconciliation-identity test and reverted before being committed. The correct fix, engine/pipeline.py::_reconcile_settlement_credit_sums, checks the sum of a settlement's linked credits against its own net_paise (±₹1 tolerance) across L0+L1+L2's combined output, not a claim count — refuses on a genuine over-claim, still allows a legitimate split. Verified against both scenarios directly, then all 8 committed benchmarks regenerated and diffed byte-for-byte (only volatile timing fields moved). 2 new tests.
0 of 7 attacks produced a FALSE_MATCH in the committed state (1 of 7 did during development — caught and fixed before ever being committed). Source: benchmarks/adversarial.json. Commit 3da7998.
Goal: measure, not assert, that deterministic-first is the right architecture. run_llm_only routes every payment/settlement/bank record (348 on sample_200 — 202+71+75) straight to L3, bypassing L0–L2 and L4 entirely.
| Engine mode | Fixture | Auto-match | False-match | Precision / Recall | Cost |
|---|---|---|---|---|---|
| Null | run_2000 | 0.00% | 0.00% | 0.00% / 0.00% | $0 |
| L0+L1 | run_2000 | 92.84% | 0.00% | 100.00% / 99.54% | $0 |
| L0+L1+L2 | run_2000 | 97.74% | 0.00% | 100.00% / 99.95% | $0 |
| Full (client=None) | run_2000 | 97.74% | 0.00% | 100.00% / 99.95% | $0 |
| All-LLM | sample_200 (348 records) | 81.52% | 0.00% | 100.00% / 77.36% | $6.58 total, $35.76/1000 records |
348/348 records, zero crashes, 1,207 LLM calls (nvidia/nemotron-3-ultra-550b-a55b). False-match holds at 0.00% even with every deterministic layer disabled — the same "refuse rather than guess" discipline (confidence < 0.85 → raise_exception) still holds under load. Recall drops because more genuinely ambiguous records get correctly refused rather than confidently matched, at 637x the cost ($35.76/1000 here against the $0.056/1000 the real live L3 run in §4 actually spent) of the deterministic layers resolving the same 99.68% of records for a rounding error. Source: benchmarks/ablation_llm_only.json.
Two crash fixes, both found because this is the first mode that ever routes a never-captured ("failed") payment to L3 at all: _residual_aging_days/_aging_days_hint/find_candidates all raised ValueError on an empty captured_at — including inside the per-record failure-isolation's own fallback path, meaning the crash-recovery mechanism itself wasn't crash-safe. Fixed to degrade to aging_days=0 gracefully; 4 new regression tests.
A third, more serious gap, found live: the first full-batch attempt hung 10+ hours with zero progress and no error, surviving an overnight laptop sleep. lsof showed all 8 DEFAULT_CONCURRENCY connections ESTABLISHED, 0% CPU — every worker thread's client.complete() call was blocked on a TCP connection gone half-open across the sleep; the openai SDK's own 600s read timeout never fired because nothing ever arrived to time out against, and the existing per-record isolation only helps once something actually raises. Fixed with an outer PER_RECORD_TIMEOUT_SECONDS=900 ceiling via asyncio.wait_for, proven with a test that a hung record times out without blocking its sibling. Disclosed, not hidden: this bounds the batch's logical progress, but asyncio.run()'s own cleanup can still be slow to return if the underlying OS thread never returns at all (Python can't force-kill a thread) — a materially better failure mode than before, not an absolute guarantee.
A mistake made investigating it, disclosed rather than glossed over: the stuck process was killed on a several-minutes-stale snapshot (139 records, unchanged since the sleep) without re-verifying freshness immediately before acting. By the time the kill happened, the run had already recovered on its own and reached 159 records at its fastest pace of the entire run (~2.5/min). No data was lost — every trace and cache entry persists as it completes, so the relaunch resumed instantly from 159/348 — and the timeout fix stands on its own merits regardless, but the kill itself was premature. Full account in ARCHITECTURE.md. Commit fc03983.
Found (pre-existing): engine/io.py::load_dataset did zero validation — a missing column raised a bare KeyError, a non-integer amount a bare ValueError, both with no file/row/field context; csv.DictReader silently clobbers a duplicate header column and silently drops row-overflow fields under a None key.
Fixed, wrapped not replaced — load_dataset's signature is unchanged, but each of the 4 CSVs now goes through header validation, a data-rows check, and per-row checks (field overflow, integer parsing, non-negativity where the domain requires it, date format, duplicate primary keys), raising DatasetValidationError(filename, reason, row, field) — never a silent skip or coercion. Fails at the file boundary: a whole broken file is scanned fully before stopping, rather than aborting on the very first bad row anywhere.
Sample error, verbatim: orders.csv: missing required column(s): ['invoice_no']
Two real domain findings, checked empirically before hard-coding a rule: a payment.captured_at of "" is the correct value for a status="failed" payment (never captured, genuinely no timestamp) — not validated as malformed. bank.credit_paise/debit_paise are deliberately not validated non-negative — data/generator/defects.py's settlement-adjustment injector can legitimately drive one below zero (confirmed: seed 100 in Task 1's sweep produced a real credit_paise: -34800); non-negativity is enforced on gross_paise instead, checked clean across all 4 fixtures.
12 tests (tests/test_malformed_input.py, later 13 — see §11.10) cover all 10 required cases — missing column, duplicate header, non-integer amount, negative-where-only-positive-valid, unexpected date format, empty file, headers-only file, non-UTF8 bytes, duplicate primary key, row with more fields than header — plus a clean-fixture sanity check and the failed-payment-empty-date non-false-positive check. All 8 committed benchmarks regenerated and diffed byte-for-byte afterward — zero drift. Commit 0e3a68d.
6.1 Determinism — tests/test_determinism.py: two run_full runs of run_2000 are byte-identical (timing stripped) and never produce a duplicate match or exception, within or across runs.
6.2 CI — .github/workflows/ci.yml, matrix Python 3.11 + 3.12, pip install -e ".[dev,llm]", pytest. A real gap found while building it: openai was a genuine runtime dependency (used by NimClient) never declared in pyproject.toml's llm extra — test_nim_client.py was silently skipping via pytest.importorskip. Fixed; verified in a clean Python 3.12 venv: 215 passed, 0 skipped (previously 2 would have skipped).
6.3 Run manifest — eval/manifest.py::build_run_manifest: git commit SHA + dirty-tree flag, record counts, engine/seed/model, tracked package versions, per-input-file SHA256 hash. Wired into eval.report.run_eval's output (additive "manifest" key) and the HTML report's footer (flags a dirty-tree run visibly). 7 new tests.
6.4 Dependency pinning — dropped pydantic, fastapi, uvicorn from pyproject.toml (confirmed zero imports anywhere in the codebase — dead weight from the original brief's suggested stack). Exact-pinned pandas==2.3.3, pytest==9.1.1, anthropic==0.82.0, openai==2.53.0. requirements-lock.txt via plain pip freeze from a clean venv (no new lock tooling). Verified end-to-end: a fresh venv installed from the lockfile alone passes the full suite (222 passed) and make demo runs clean (97.74% auto-match, 0.00% false-match, unchanged).
Correction (§11.9): this sweep was incomplete. It checked only the three packages named in its own plan and missed
pandas— which was not an extra but the sole entry in[project.dependencies], and is likewise imported nowhere. Removed later; see §11.9.
Commit 218db04.
- 223/223 tests passing as of commit
fc03983(up from 174 at the start of the sprint — 49 new tests across Tasks 1–6, zero deletions). pyflakesclean acrossengine/,eval/,cash/,data/,tests/,scripts/after every task.- Zero unexpected benchmark drift across all 6 committed tasks — every diff was byte-for-byte confirmed after every task that touched matching logic (Tasks 2, 3), and the tasks that didn't (1, 4, 5, 6) never invoke the affected code paths from any frozen benchmark's CLI mode.
make freeze regenerated all 6 headline benchmarks (freeze_500, freeze_2000, freeze_10000, phase3, phase4, phase5) from scratch — fresh fixture generation at each scale, not a re-score of the existing CSVs — against the final code from Tasks 1–6. Every regenerated file diffed programmatically against its previously-committed version, stripping only the fields that are expected to move on any regeneration (generated_at_unix, wall_clock_seconds, records_per_second, and the new manifest block Task 6.3 added):
Result: all 6 files identical. auto_match_rate, false_match_rate, precision, recall, exceptions_detail, defect_confusion, and the cash-reconciliation numbers are byte-for-byte unchanged at every scale — exactly as expected, since nothing in Tasks 1, 4, 5, or 6 touches any code path these deterministic-CLI benchmarks exercise, and Tasks 2/3's matching-logic changes were already regenerated and diffed at the time they landed (§11.2, §11.3). The only genuine content change in all 6 files is the new, additive manifest block (git SHA, dirty-tree flag, record counts, package versions, per-input-file SHA256) — by design, not drift.
One honest note on the manifest's own git_dirty flag: it reads true on this regeneration, because the regeneration itself (this Makefile target, these very benchmark files) is what's being committed — there's no way to freeze the committed state of a change before making it. Read git_dirty: true on these 6 files as "generated as part of landing this exact commit," not as evidence of an unrelated uncommitted change.
Docs reconciled: README gained the Task 1 multi-seed table (directly beneath the 3-scale results table) and its Makefile command-reference line now lists multiseed/adversarial/verify-deterministic/freeze. The compound_fee_tax_error table cell and Limitations bullet were already reconciled to the real 4–6/6 number during Task 2 (§11.2) — re-checked here, no further change needed. ARCHITECTURE.md's Task 2–5 narrative sections were likewise already accurate as of their own tasks — re-checked, no further change needed. Makefile gains the freeze target itself (previously three manual one-off commands, now one reusable command producing the correctly-named freeze_*/phase* files, not the raw run_<label> names eval.report's CLI writes by default).
Final numbers, all 3 scales, vs. previously committed:
| Records | Auto-match | False-match | Precision / Recall | Exceptions | Change |
|---|---|---|---|---|---|
| 500 | 97.61% | 0.00% | 100.00% / 99.91% | 48 | none |
| 2,000 | 97.74% | 0.00% | 100.00% / 99.95% | 139 | none |
| 10,000 | 97.83% | 0.00% | 100.00% / 100.00% | 550 | none |
223/223 tests passing, pyflakes clean, at the moment of freeze.
Code freeze declared as of this commit. No further engine commits follow. Any non-critical bug found after this point is recorded in §8 (Known Limitations), not patched against an already-reported and re-verified number.
A full read of the code rather than the docs, done from a judge's point of view, found three defects invisible in the write-up but visible to anyone who clicks or runs anything. All three are docs/tests/presentation-wiring — no engine/*.py logic was touched, so the freeze holds.
1. The demo's climax silently did nothing. The demo walkthrough's "click one exception → full agent reasoning trace" beat was dead. In the default make demo dashboard, 0 of 139 exceptions carried a trace link. The 6 UNEXPLAINED_VARIANCE rows are exactly the 6 records with real committed live traces, but eval/report.py's SAMPLE_TRACES_DIR looked only in benchmarks/sample_traces/ (the hand-built synthetic set — pay_unexplained, btxn_batch), while the real traces live in benchmarks/sample_traces_live/. All 6 rendered "no agent trace (deterministically classified)."
The obvious one-line fix would have introduced a worse bug, caught before writing it: record ids are seed-derived, so run_2000 and run_10000 (both seed=42) share 2001 payment ids while holding genuinely different records — verified directly against both fixtures. Adding the directory to the lookup would have made freeze_10000's dashboard attach pay_OyvjU0Hc7g7Bi2's real run_2000 agent trace to an unrelated FEE_VARIANCE payment of the same id — an LLM reasoning in detail about the wrong record, strictly worse than showing nothing. The trap never fired before only because the synthetic set's hand-written ids can't collide with generated ones. Fixed as a fixture-scoped lookup (SAMPLE_TRACE_SOURCES, mapping each trace directory to the fixture it's valid for), with the collision itself as a regression test.
2. The most load-bearing credibility claim cited a test that did not exist. README and §1 above both state the harness is mutation-tested — "inject 10 deliberately wrong links and it reports 0.24%; drop half the true matches and recall halves" — and §1 cited tests/test_eval_baselines.py by name. That file had 4 tests, none of them a mutation test. The claim is what turns 0.00% false-match from suspicious into verified, and a dead citation is disproportionately damaging precisely because every other citation in these docs resolves.
The claim itself was true, just never committed: injecting 10 wrong links yields 10/4172 = 0.2397% → 0.24%, and halving the matches yields recall exactly 0.5. Both are now real tests in the file that was already being cited, asserting the exact rate, the documented "0.24%" string, and that dropping matches does not register as a false match.
3. pandas was a fourth dead dependency. Task 6.4 (§11.6) removed pydantic/fastapi/uvicorn but checked only the three named in its own plan — missing pandas==2.3.3, the sole entry in [project.dependencies], imported nowhere. data/generator/emit.py's own docstring says it uses stdlib csv instead of pandas; eval/manifest.py was tracking the version of a package nothing imports. Removed, along with its 5 transitive deps (numpy, python-dateutil, pytz, six, tzdata) — lockfile down from 29 packages to 23. Kosh has zero runtime dependencies, which is a better story than the one previously told and is now stated as such in the README.
Verification: 227/227 tests passing (223 + 2 mutation + 2 trace-scoping), pyflakes clean, all 6 frozen benchmarks re-diffed with only the intended trace_file field moving and every accuracy number byte-identical, and both a .[dev,llm] and a lockfile-only clean-venv install re-verified end to end.
Task 5 (§11.5) flagged, but deliberately did not fix, a real generator-side quirk: a large-enough negative net-adjustment delta could drive an injected bank credit below zero (a real credit_paise: -34800 row, surfaced by the Task 1 multi-seed sweep at seed 100). It was spun off as a separate tracked task rather than folded into Task 5's own scope. That task ran independently in its own git worktree and produced one commit, reviewed and merged here after the fact - the only engine/*.py change since code freeze, and disclosed as such rather than folded in silently.
Fix: data/generator/defects.py::_adjust_bank_credit_for_settlement now floors the adjusted credit at max(0, ...) - a negative bank credit isn't a real value; money moving the other way is a debit. engine/io.py re-enables non_negative=True on bank.credit_paise/debit_paise, closing the validation gap Task 5 had deliberately left open for this exact quirk. 2 new tests (a generator-level regression forcing the original failure shape, and a malformed-input test that a negative bank credit is now rejected) plus a new scripts/verify_no_negative_bank_credit.py regenerating all 6 multiseed seeds and asserting none produce a negative credit or debit.
Independently re-verified before merging, not taken on the commit message's word: full suite (229/229, up from 227), pyflakes clean, all 6 multiseed seeds confirmed clean by direct re-run, all 4 reference fixtures (run_500/run_2000/run_10000/sample_200) regenerated and diffed byte-for-byte identical to their committed CSVs (none of them ever hit this path, so nothing about them changes), and all 6 frozen benchmark JSONs re-diffed with zero change to any accuracy number.
Everything above (§§1–11) is the submission: code-frozen, benchmarked, judge-fix-verified, publicly pushed. This section covers work done after that point, at the user's explicit request for "the best project with all included" — specifically the two optional items the original brief allowed but the shipped submission didn't build (README.md's own Limitations: "No FastAPI layer and no live/interactive dashboard — the static HTML report is the deliverable, per the brief's own explicit fallback preference"). Recorded separately from §11 because it's a deliberate scope extension, not a continuation of the hardening sprint or a freeze violation — no engine/*.py file was touched to build it.
What was built: api/ (FastAPI: a live-triggerable run endpoint, historical-benchmark endpoints, and static mounts for the trace files the dashboard links to) and dashboard/ (Next.js + Recharts, matching the brief's own named stack), both local-only by explicit decision — no hosting, no public exposure of a compute-triggering endpoint.
Design discipline carried over from the rest of this project: the API is a thin wrapper around code that already exists and is already trusted — data.generator.generate.run and eval.report.run_eval — not a second implementation. tests/test_api.py asserts this directly: a live run at seed=42/records=2000/engine=full produces output byte-for-byte identical (modulo timing/manifest) to the committed freeze_2000.json.
A real bug caught before shipping, in the same spirit as everything else in this log: the first version generated live runs into a randomly-named temp directory. eval/report.py's trace-drilldown lookup (SAMPLE_TRACE_SOURCES, from the earlier judge-fix round) matches on the fixture directory's name being "run_2000", not its content — so a live seed=42 run would reproduce run_2000's data exactly, yet its exception drill-down links would all silently come back empty, because the scratch directory had the wrong name. Fixed by naming the scratch directory run_<records> — the same convention a real fixture at that scale would use. Verified against the actual running API, not just pytest: a live seed=42 run's UNEXPLAINED_VARIANCE exceptions link to the real, already-committed agent traces in benchmarks/sample_traces_live/, and a freeze_10000 view still shows zero trace links through the dashboard — the same fixture-scoping collision guard proven in the earlier judge-fix commit, now proven again through a second consumer.
Safety: api/runs.py::ENGINE_ALLOWLIST = (null, oracle, l0l1, l0l1l2, full) — the costed llm-only engine is asserted absent from it at import time, and no request field can supply an LLM client, model, or key. Tested directly (test_the_costed_engine_is_never_reachable_from_the_api), not just implied by the allowlist's existence. GET /api/benchmarks/{name} uses an explicit filename allowlist, never a path built from request input — tested against literal traversal attempts, which 404 like any other unknown name.
Dependencies: fastapi/uvicorn re-added to pyproject.toml as an api extra (Task 6.4 removed them for zero usage — that reasoning doesn't apply once they're genuinely imported by api/main.py). Exact-pinned to the versions actually resolved (fastapi==0.141.1, uvicorn==0.52.4 — a version bump from the brief's >=0.110 floor was required: fastapi==0.110's bundled starlette TestClient predates a breaking change in the httpx version already pulled in transitively by anthropic/openai). Lockfile regenerated to include the api extra's closure.
Verification: 244/244 tests passing (229 + 15 new), pyflakes clean including api/. Verified live, not just in pytest: the API server and dashboard dev server both started and driven through an actual browser — the run form submitted, all four panels rendered with real numbers, an exception row expanded and its trace link resolved with a real 200 (not a 404), the layer-waterfall chart confirmed to render real proportioned bars (not an empty grid), and the historical-benchmark picker confirmed against freeze_10000's real record count (9,317). No engine/*.py file was modified.
Every earlier live L3 number came from run_2000's 6-record residual: too few to state a rate. The agent was therefore run against run_10000's 29-record residual — same code, same model (nvidia/nemotron-3-ultra-550b-a55b), fresh cache so nothing replayed.
Run: 29 records, 245 LLM calls, 1,214s wall clock, 616,815 input / 70,701 output tokens, $0.464 at NIM's published rate. Source: benchmarks/phase5_live_residual_10k.json.
Headline metrics unchanged, and this is the point: auto-match 97.83%, false-match 0.00%, precision 100%, recall 100%, 550 exceptions — identical to the deterministic freeze_10000 run. L3 sees 29 of 9,317 records; it cannot move the top line, and didn't.
Per-class result: compound_fee_tax_error 25/29 detected (86%). Every one of the 4 misses was checked by hand against ground_truth.json:
| Record | What happened | Amount |
|---|---|---|
pay_4cXgekcH1NC0sO |
TAX_VARIANCE instead of UNEXPLAINED_VARIANCE |
130 paise — exact |
pay_l23vdlUo60FAXP |
FEE_VARIANCE |
20,579 paise — exact |
pay_z8mbHMXa8wIMih |
TAX_VARIANCE |
33 paise — exact |
pay_wkjO7N4t4iTfU1 |
AGENT_INCOMPLETE, 12 turns exhausted |
fallback record-level hint (1,475,545 paise) vs. true 182 paise |
Three of four are label-only misses with the money exactly right: the model found the true variance, then named a single cause because the multi-leg coercion correctly refused to fire on one unexplained leg.
Two things six records could never have shown:
13.1 — The 12-turn budget is not always enough. Raised from 8 after the original 4-of-6 exhaustion (§4), and every run_2000 run since showed zero AGENT_INCOMPLETE — which made 12 look settled. At 29 records, one exhausted (~3%). The isolation path did exactly its job: the record is on the ledger with the reason attached rather than crashing the batch. But its amount_at_risk_paise is _amount_hint's record-level figure, not a measured variance — 1,475,545 paise against a true defect of 182. Honest as "unknown, needs review," misleading if read as ₹ at risk. Trace committed: sample_traces_live_10k/agent_incomplete_turn_budget.json.
13.2 — On this class, the deterministic fallback beats the LLM: 29/29 vs 25/29. With client=None every residual record is blanket-labelled UNEXPLAINED_VARIANCE, which for a defect defined as undecomposable is correct 100% of the time. The LLM attempts a specific cause and is wrong 4 times in 29. This is not an argument against L3 — it is the sharpest available statement of where L3 does and does not earn its cost: on residuals that can genuinely be decomposed, not on a class whose correct answer is "no single cause explains this."
Both are recorded because the run produced them, not because they help.
Four claims were each proven at exactly one scale. All four were re-tested at the largest scale available, deterministic-only, free to run.
14.1 — Cash identity ties at 10,000. The brief's own test: "reconciled cash vs book cash, with the delta fully explained — if those don't tie, you have a bug." Previously verified on run_2000 and sample_200 only. At run_10000: book ₹6,94,27,826.66 vs reconciled ₹6,52,77,273.37, a gap of ₹41,50,553.29 — named to the paisa, remainder exactly 0.
14.2 — Determinism holds at 10,000. tests/test_determinism.py runs on run_2000. Re-run at 10k: two passes produced identical output across 19,610 links and 550 exceptions, no duplicates.
14.3 — Seeds × scale, tested together. The committed sweep is 6 seeds at 2,000; the 10,000 result was one seed. Those two dimensions had never been crossed, so "0.00% at scale" rested on a single seed. Five seeds at 10,000: auto-match 97.83–97.84%, 100% recall on every seed, 0.00% false-match on every seed — a tighter spread than at 2,000 (96.99–97.96%), which is what a larger sample should do. Source: benchmarks/multiseed_10k.json.
14.4 — Throughput out to 93,030 records (10× the freeze). "Throughput" is in the track's own bar and was proven only to 9,317.
| Scored | Wall clock | rec/s | Auto-match | False-match |
|---|---|---|---|---|
| 9,317 | 40ms | 231,191 | 97.83% | 0.00% |
| 23,228 | 111ms | 209,943 | 97.83% | 0.00% |
| 46,506 | 318ms | 146,364 | 97.82% | 0.00% |
| 93,030 | 638ms | 145,773 | 97.82% | 0.00% |
Scaling exponents between consecutive points: 1.106, 1.520, 1.006. The mid-range 1.52 is a single step, not a trend — the final doubling is linear to three decimal places and the rate plateaus at ~146k/sec. L2's subset-sum is the only superlinear component and stays bounded by construction (≤40 candidates, 250ms deadline), which is why the curve flattens rather than blowing up.
One correction this forced: the README previously headlined 219,659 rec/s, a run_2000 figure. It does not survive scale — the sustained rate is ~146k/sec, and that is now the number quoted. The higher small-N rates are amortisation, not a faster code path. Source: benchmarks/scaling_100k.json.
15.1 — Degenerate and minimum-size inputs. Nothing below 200 records had ever been run. The brief's own floor is 50.
| Orders | Scored | Auto-match | False-match | Cash ties |
|---|---|---|---|---|
| 1 | 1 | 100.00% | 0.00% | yes |
| 2 | 3 | 33.33% | 0.00% | yes |
| 5 | 6 | 83.33% | 0.00% | yes |
| 10 | 11 | 90.91% | 0.00% | yes |
| 50 (brief's minimum) | 43 | 95.35% | 0.00% | yes |
| 200 | 185 | 97.84% | 0.00% | yes |
No crashes, no divide-by-zero on an empty layer, and the book-vs-reconciled identity ties to the paisa at every size including a single-record dataset. Low auto-match at n=2 is not a defect: the generator's max(1, ...) floor guarantees every defect type appears at least once, so a 3-record dataset is almost entirely defective by construction.
15.2 — L2's published p99 is a stress number; on real data the solver is ~2,500× faster. benchmarks/phase4_solver_perf.json reports p99 = 49.8ms against a 250ms budget, measured on 2,000 synthetic worst-case instances (40 candidates each). Re-run to confirm it still holds: p99 44.9ms, max 99.2ms, status counts identical (SOLVED 919 / AMBIGUOUS 1004 / NONE 77 — deterministic).
Instrumenting a real run tells a different and equally honest story:
| Fixture | L2 invocations | p50 | max | Statuses seen |
|---|---|---|---|---|
run_2000 |
22 | 0.005ms | 0.018ms | SOLVED 8, NONE 14 |
run_10000 |
24 | 0.004ms | 0.015ms | SOLVED 8, NONE 16 |
On production-shaped data the solver is invoked ~24 times and never exceeds 0.018ms — four orders of magnitude inside its budget — and AMBIGUOUS never occurs naturally; it only fires under the deliberately-constructed adversarial attack g. The synthetic profile measures the safety margin, not the cost.
15.3 — Memory at 100,000 orders: 319 MB peak RSS, producing 192,741 links and 5,135 exceptions. Never previously measured.
15.4 — Python version range. pyproject.toml declares requires-python = ">=3.11" — unbounded — while CI tested only 3.11 and 3.12. Verified 3.14 locally with the full [dev,llm,api] extras first, then widened the matrix. All four now green on GitHub runners: 257 passed on 3.11, 3.12, 3.13 and 3.14 (run 33867354652).