Three sources must tie out to the paisa:
| Source | What it is | Key fields |
|---|---|---|
| A. Orders | Merchant's ERP/invoice export | order_id, gross amount, date |
| B. PG ledger | Payment gateway transaction + settlement reports | payment_id, order_id, gross, MDR fee, GST, net, settlement_id, utr |
| C. Bank statement | Bank credits with free-text narration | value_date, credit_paise, narration (UTR buried in it) |
The hard part: one bank credit is a batch. It equals the sum of many payments' net amounts, minus refunds, minus chargebacks, plus/minus adjustments. When the UTR in the narration is mangled or missing, the system must solve for which set of payments explains that credit.
An LLM doing arithmetic across thousands of rows is slow, expensive, and wrong. Kosh is a layer cake, and each layer only ever sees what the layer above it couldn't resolve:
┌─────────────────────────────────────────────────────────────┐
│ L0 Deterministic joins (exact keys) ~85% target │
├─────────────────────────────────────────────────────────────┤
│ L1 Tolerance matching (±amount, ±date) ~8% target │
├─────────────────────────────────────────────────────────────┤
│ L2 Combinatorial solver (subset-sum) ~4% target │
├─────────────────────────────────────────────────────────────┤
│ L3 Claude agent (residual only) ~3% target │
├─────────────────────────────────────────────────────────────┤
│ L4 Exception ledger (honest remainder) whatever's left│
└─────────────────────────────────────────────────────────────┘
The agent never sees the ~97% that deterministic code already handled. That's the whole competitive story, and it's also just correct engineering: cheap, fast, auditable layers absorb everything they can before anything nondeterministic runs.
Every phase's own numeric checkpoint against the brief's stated target, and the full submission ledger, live in RESULTS.md rather than here, so a number never has two homes to drift apart in. What follows is the narrative: why each phase is built the way it is, and every bug that shaped it.
data/generator/ builds one internally-consistent "clean world" (orders → payments → refunds → settlements → bank statement, all tying out exactly), then mutates it with 12 configurable defect injectors (defects.py). Ground truth is recorded from the pre-mutation clean world plus a labelled record of every mutation, so eval/ (Phase 2) can score an engine's output against a graph that is true by construction, not by inference.
Key modules:
fees.py— the single MDR + GST model, integer-only, imported unchanged byengine/in later phases so a fee bug shows up as a failing test, not a silently-accepted false match.world.py— builds the clean, defect-free dataset.defects.py— the 12 injectors, detailed below.emit.py— serializes to the four CSV schemas +ground_truth.json+manifest.json.trace.py— hand-verification CLI: bank credit → settlement → payments → orders, with aTIES OUT/OFF BYverdict.
| Metric | Value |
|---|---|
| Orders | 2,000 |
| PG payments (incl. injected duplicates) | 2,020 |
| Settlements | 305 |
| Bank statement rows | 351 |
| GMV | ₹14,779,996.39 |
| Injected defects | 188 (~9.4% of orders), all 12 types present |
A small companion fixture, data/fixtures/sample_200 (seed 7, 1 month), is committed to the repo so real output is inspectable without running the generator.
Realistic Indian PG economics, so failures concentrate where they should:
| Method | MDR |
|---|---|
| UPI | 0.00% (zero-MDR P2M — real regulatory position) |
| RuPay debit | 0.00% |
| Card (domestic) | 2.00% |
| Netbanking | 1.90% |
| Wallet | 2.00% |
| Card (international) | 3.00% |
Zero-MDR UPI/RuPay means most of that volume reconciles trivially (net == gross), which is intentional, not a shortcut: the interesting defects (fee-tier errors, GST variance, FX drift) concentrate in card and international volume, exactly like a real merchant's exception queue. GST is 18% of the fee, and rounding is integer half-up applied at each step (fee first, then GST on the rounded fee), not once at the end — that ordering produces genuine ±1 paisa drift, one of the 12 defect types and also just how real gateways compute it. All money is integer paise, enforced by an AST-level lint (tests/test_no_floats.py) that fails the build on a stray /, round(), or float literal in the fee/settlement arithmetic.
| # | Defect | Expected exception category |
|---|---|---|
| 1 | Payment captured but never settled | MISSING_SETTLEMENT |
| 2 | Duplicate payment row | DUPLICATE_PAYMENT |
| 3 | Paisa rounding drift (±1–3p) | resolvable — not an exception |
| 4 | Wrong MDR tier applied | FEE_VARIANCE |
| 5 | GST ≠ 18% of fee | TAX_VARIANCE |
| 6 | Refund booked against wrong order | REFUND_MISALLOCATION |
| 7 | Chargeback debit with no linked payment | ORPHAN_CHARGEBACK |
| 8 | Settlement lands after month-end cutoff | PERIOD_CUTOFF |
| 9 | UTR truncated/mangled in narration | resolvable via subset-sum — not an exception |
| 10 | International payment with FX rate drift | FX_VARIANCE |
| 11 | Non-PG bank credit (direct customer NEFT) | UNIDENTIFIED_CREDIT |
| 12 | One settlement split across two bank credits | resolvable via subset-sum |
Three of the twelve are marked resolvable: they exist specifically to test that a reconciliation engine is smart enough not to raise a false exception over ordinary rounding drift, a truncated bank narration, or a split settlement batch.
engine/contract.py defines the interface every layer built in Phases 3–5 must speak: an EngineOutput of Matches (link_type, left_id/right_id, layer, confidence, evidence) and ReconExceptions (category, severity, amount_at_risk_paise, affected, recommended_action). Nothing downstream — eval/, the future dashboard, Phase 5's agent tools — touches raw CSV rows directly; everything speaks this contract.
eval/metrics.py scores an EngineOutput against ground_truth.json: throughput (records/sec, LLM cost in integer micro-dollars), accuracy (auto-match rate, per-layer contribution, false-match rate, precision/recall, per-defect-class confusion matrix), and an honest exception summary (count, ₹ at risk, category/severity breakdown). eval/report.py emits a timestamped JSON + a self-contained report.html per run.
Two baseline "engines" (engine/baselines.py) exist purely to validate the harness before any real matching logic is written:
| Baseline | What it does | auto-match | precision / recall | false-match | exceptions |
|---|---|---|---|---|---|
null |
asserts nothing | 0.00% | 0.00% / 0.00% | 0.00% | 100% of records |
oracle |
reads ground_truth.json directly |
97.85%* | 100.00% / 100.00% | 0.00% | 139 (exactly the unresolvable defects) |
* Not 100%: payments hit by missing_settlement or duplicate_payment have no true settlement link to match — they correctly land on the exception ledger instead.
null_baselineasserts nothing and raises one exception per captured payment → must score 0% auto-match, 0% recall, 0.00% false-match, 100% of records exceptioned.oracle_baselinereadsground_truth.jsondirectly and asserts exactly the true link graph → must score ~100% precision/recall, 0.00% false-match, and an exception ledger that exactly matches the unresolvable defects (no misses, no misclassifications, and — critically — no false exceptions raised over the 3 defect types that should resolve silently).
Both are frozen in tests/baselines/*.json as a regression guard: if the scoring logic in eval/metrics.py drifts, or a baseline's own behavior drifts, pytest fails immediately. false_match_rate is the metric this whole harness is built to protect — it's computed against the engine's own assertions (wrong ÷ total asserted), not against total records, specifically so it can't be gamed by asserting fewer links.
Key design point an engine must respect from Phase 3 onward: engine/io.py never reads ground_truth.json. Only eval/io.py and the oracle baseline do — the oracle exists to test the scorer, not to reconcile anything, and a real engine that read ground truth would invalidate every accuracy number this project exists to produce.
engine/l0_deterministic.py runs the exact-key cascade in three steps: orders.order_id ↔ pg_payments.order_id and pg_payments.settlement_id ↔ pg_settlements.settlement_id are direct, unambiguous FK joins (confidence 1.0, always). The third step, pg_settlements.utr ↔ a UTR pulled out of bank_statement.narration, is where real ambiguity can arise: engine/normalize.py extracts a UTR-shaped token ([A-Z]{4}[NR]\d{1,11}) from free text and classifies it as a full 16-char exact match, a ≥12-char truncated prefix, or nothing usable. A full match does an O(1) dict lookup against known settlement UTRs; a prefix match requires the prefix to identify exactly one settlement — two or more candidates means it's refused, not guessed, and falls through to L1.
engine/l1_tolerance.py picks up whatever L0 couldn't place — in practice, a bank credit whose UTR was rekeyed with a transposed digit (still 16 chars, still syntactically valid, but matching no real settlement). It requires all three: amount within ±300 paise, date within ±3 days, and a narration-similarity score ≥0.80 against a settlement-vocabulary bag (engine/normalize.py:settlement_narration_similarity) that distinguishes a genuine (even truncated) settlement narration from a customer-name or chargeback one. The same rule applies: more than one settlement satisfying all three tolerances means the candidate is ambiguous and is refused, never picked by "closest".
A subtlety worth naming: a bank row is excluded from settlement-matching only when it's a genuine debit (debit_paise > 0), never merely because credit_paise == 0 — a settlement can legitimately net to exactly ₹0 (e.g. after a period_cutoff shift on a small settlement), and that's still a real credit leg, not a debit. An earlier version of l0_deterministic.py filtered on credit_paise <= 0 and silently dropped these; caught by the reference-fixture recall dropping to 99.94% instead of 100%, which is exactly why every checkpoint is measured against run_2000, not eyeballed.
engine/fees.py re-exports the generator's compute_expected_fee unchanged and adds explain_variance(observed, expected, ...), which decomposes a paise delta into MATCH / ROUNDING / FEE_TIER / GST_RATE / REFUND, or honestly returns UNEXPLAINED when none of those hypotheses fit. This function does no matching itself — Phase 3's pipeline (engine/pipeline.py) doesn't call it yet — but it's the piece Phase 5's exception ledger and Phase 5's agent tools both build on, so it's built and tested now.
Result on data/fixtures/run_2000: 92.84% auto-match, 100.00%/99.54% precision/recall, 0.00% false-match, ~4ms wall clock — see benchmarks/phase3.json. (The 97.85%-and-100%-recall figure once here described an earlier, 12-defect-type fixture; phase3.json has since been regenerated against the current 14-type suite, and this line is now the number that file actually holds.) Phase 3's pipeline doesn't raise any exceptions yet (L2/L3/L4 don't exist), so whatever L0/L1 can't place is simply absent from the match list for now, not silently claimed as reconciled.
engine/l2_subset.py handles the "one bank credit is a batch" problem literally: given a credit L0/L1 couldn't place, and a pool of settlements within a same-day/T+1 date window (excluding any settlement already confidently matched elsewhere), find the subset whose net amounts sum to it within tolerance.
Both algorithms the spec suggests have a scaling trap specific to this domain. Meet-in-the-middle costs O(2^(n/2)) in the candidate count — dangerous at the spec's own max_terms=40. A bitset DP (one Python bigint, one bit per achievable paise value) is the classic alternative, and it's genuinely cheap as a one-off feasibility check — but reusing it per DFS node during subset recovery costs O(bit-length) at every node, and bit-length here tracks money magnitude (paise), not candidate count. This was caught by profiling, not by inspection: an early version comfortably passed correctness tests, then blew the 250ms budget (p99 over 1.2s) on nothing more exotic than 40 candidates around ₹50,000 each, because every prune check was building or shifting a multi-megabit integer.
The shipped solver drops the bitset entirely and prunes with a plain O(1) sum-bound at every DFS node (the most any remaining branch could still add) — weaker pruning in theory, but its cost depends only on n, so a node-count-based time check gives a real, predictable worst case regardless of how large the amounts are. Result: p50 ~2ms, p99 ~50ms, max ~101ms over 2,000 synthetic instances at n up to 40 (benchmarks/phase4_solver_perf.json) — comfortably inside the 250ms cap with margin, not hugging it.
Those are worst-case numbers, and worth separating from the real ones. The profile above runs 2,000 synthetic instances deliberately built at the max_terms=40 ceiling. Instrumenting the solver during an actual pipeline run tells a very different story: on run_2000 it is invoked 22 times, max 0.018ms; on run_10000, 24 times, max 0.015ms — four orders of magnitude inside the 250ms budget, and AMBIGUOUS never occurs naturally (only under the deliberately-constructed adversarial attack g). That gap is the design working as intended rather than a discrepancy: bounding the search by max_terms and a node-count deadline is what makes the worst case survivable, and real settlement batches are nowhere near that worst case. Both numbers are reported — the synthetic one is the safety margin, the real one is the cost.
The ambiguity guard mirrors L0/L1's: more than one distinct subset satisfying the target returns AMBIGUOUS with the alternatives found (capped at 2 — "more than one exists" is all that matters), never a pick. A pool over max_terms, or a search that exhausts its time budget, both escalate the same way as a genuine NONE — this layer simply declines to match, leaving the credit for L3/L4.
On data/fixtures/run_2000, L2 contributes zero matches. L0+L1 already achieve 100% recall on every tie-outable settlement_bank_txn link — a split settlement resolves via both credits sharing one UTR (L0), and a mangled UTR resolves via amount+date+narration tolerance (L1) — so nothing genuinely batched-and-unrecoverable reaches this layer in the current defect suite. Rather than manufacture a residual to make L2 look busy, this is reported as-is: L2's correctness (a genuine multi-settlement batch, the ambiguity guard, the already-matched exclusion, the date-window prune) and its timing are verified directly, by tests/test_l2_subset.py and tests/test_l2_subset_perf.py respectively, independent of whether this dataset happens to invoke it.
Designing L4's classifier surfaced that bank_statement.csv's chargeback narration carried no reference back to its originating payment at all — legitimate and orphan chargebacks used the same template pool and a meaningless random reference, so they were textually identical. No engine, deterministic or LLM, could have told them apart from the public data sources as they stood; exposing the true link in ground_truth.json alone would only have fixed scoring, not given anything a real detection capability.
Fixed generator-side: data/generator/ids.py::derive_dispute_ref(payment_id) is a pure, deterministic transform embeddable in narration — the same "the engine can independently recompute this" pattern a settlement's UTR already uses, just applied to a different link. A legitimate chargeback's narration now embeds derive_dispute_ref of its real payment_id; an orphan chargeback (defects.py) gets a same-shaped but collision-checked random token, so format alone still can't distinguish them — only the lookup can. engine/l0_deterministic.py::match_chargeback_payment is the 4th L0 exact-key join this enables, and ground_truth.json gained a 4th link type (chargeback_to_payment) for scoring it. Verified purely additive against run_2000: auto-match and false-match rates unchanged; the new link type alone scores 100% recall, 0% false-match on its 8 legitimate chargebacks.
engine/exceptions.py::classify_deterministic is where "deterministic first" earns its keep for real: every category with an honest, generalizable rule is classified here, using only the four public CSVs plus explain_variance (built in Phase 3, unused until now) — FEE_VARIANCE/TAX_VARIANCE from explain_variance's FEE_TIER/GST_RATE causes, REFUND_MISALLOCATION/FX_VARIANCE from a settled payment's gross disagreeing with its order's (refund_id present vs. international with none, respectively — the two injectors never touch the same field combination, so the categories never collide), MISSING_SETTLEMENT/DUPLICATE_PAYMENT from grouping captured payments by order and checking settlement_id presence, and ORPHAN_CHARGEBACK/UNIDENTIFIED_CREDIT from whatever L0-L2's residual still couldn't place.
PERIOD_CUTOFF is the one genuinely calibrated rule: a settlement's gap from its own payments' latest capture to its settled_at is empirically 1-4 days when normal and 3-31 days when genuinely cut off, with exactly one boundary case (3 days) indistinguishable from ~20 normal same-gap settlements using this signal alone. The threshold is set at >4 days: 13/14 caught, zero false positives — reported as one honest miss rather than 14/14 bought with ~20 false alarms.
A real scoring bug was caught verifying this against the eval harness: eval/metrics.py::compute_defect_confusion matched an exception to a defect by checking whether any value in the exception's affected dict overlapped the defect's identity, which let an unrelated TAX_VARIANCE exception's settlement_id context field get blamed for a different PERIOD_CUTOFF defect on the same settlement — reported as "misclassified" instead of the true "missed". Fixed with CATEGORY_IDENTITY_FIELDS, the symmetric counterpart to DEFECT_IDENTITY_FIELDS: matching now compares an exception's own primary identity field to a defect's identity, never an incidental context value.
Result on run_2000: 0 LLM calls, 132 exceptions, every non-resolvable defect category covered (8 of 9 at 100% recall, PERIOD_CUTOFF at 13/14), every exception carrying a real recommended_action and non-empty evidence_chain — see benchmarks/phase5.json.
The brief specs the Anthropic SDK (claude-sonnet-5). No Anthropic key was available when this was built; an NVIDIA NIM key was, and NIM serves open models (Llama, Nemotron, ...) via an OpenAI-compatible API, not Claude. Rather than couple the agent loop to one SDK's shape, engine/llm/base.py defines a minimal LLMClient protocol (ToolSpec, ToolCall, Message, AssistantTurn, one complete() method). engine/llm/anthropic_client.py implements it against the real Anthropic SDK and is unit-tested against a mocked client — spec-complete, ready the moment a key exists, but never run live. engine/llm/nim_client.py implements the same interface against NIM's OpenAI-compatible endpoint and is what actually ran for real, against nvidia/nemotron-3-ultra-550b-a55b. This is a documented deviation from the literal brief for the reference numbers, not a silent substitution.
engine/l3_tools.py implements the brief's 7 tools (get_record, find_candidates, compute_expected_fee, explain_variance, solve_subset, propose_match, raise_exception) with every hard constraint enforced structurally, not just prompted: a ToolContext.known_ids set rejects any id a tool didn't hand out (constraint 1); compute_expected_fee/explain_variance are the only arithmetic path (constraint 2); confidence < 0.85 rejects propose_match outright (constraint 3); severity is recomputed from the amount via severity_for_amount, never trusted from the model, and a match over ₹50,000 auto-emits a HIGH_VALUE_MATCH_REVIEW companion exception regardless of confidence (constraint 4); propose_match's rationale must cite a known id (constraint 5, best-effort); and engine/l3_agent.py's loop itself raises an AGENT_INCOMPLETE exception on a record's behalf if it exhausts its turn budget without closing the loop (constraint 6, enforced at the infrastructure level — never a silent drop).
run_2000's residual for L3 is empty (L0-L2 and L4's deterministic classifier already resolve everything), so L3's real capability was proven on a small hand-built synthetic exercise set (engine/l3_agent.py --profile) instead — 5 records engineered to need UNEXPLAINED_VARIANCE reasoning, a clean high-value match, weak/uncertain evidence, a genuine 2-settlement batch requiring find_candidates → solve_subset chaining, and an unresolvable ambiguity. Run for real:
- A genuine false-match bug was found and fixed before the reference run. The model correctly traced a payment → its settlement → the settlement's own bank credit (excellent, fully-evidenced reasoning) and bundled all three ids into one
propose_matchcall._infer_link_typeinferred a link type from table names alone, so any(payment, bank)pair becamechargeback_paymentregardless of whether that bank row was actually a debit — here it wasn't, so the tool asserted a chargeback that never happened. Fixed to check the bank row's actualdebit_paise; a credit is now rejected with an error explaining the real scope boundary (propose_matchonly asserts links to the record under investigation). Regression-tested intests/test_l3_tools.pybefore re-running live. - The clean re-run: zero false matches.
pay_unexplainedandpay_ambiguouscorrectly classified/refused;btxn_batchcorrectly chainedfind_candidates→solve_subset→propose_matchacross both settlements, with theHIGH_VALUE_MATCH_REVIEWcompanion firing automatically;pay_high_valueandpay_weak_evidencehitAGENT_INCOMPLETEafter thorough, correct-but-unfinished investigation exhausted the 8-turn budget - constraint 6's fallback working exactly as designed, not a false positive, but a real signal the turn budget is tight for a genuinely uncertain case.
Committed as benchmarks/phase5_synthetic.json, full traces in benchmarks/sample_traces/.
Settlement SLA reuses data.generator.calendar.settlement_date unchanged — the same one-source-of-truth pattern as engine/fees.py re-exporting compute_expected_fee. compute_inflow_curve buckets unsettled captured payments' net by their own method's expected settlement date over the next 14 days; compute_stuck flags anything past its own SLA + 1 day grace, still unsettled — distinct from a payment still legitimately in flight.
The hard part is compute_cash_reconciliation: book cash (accrual, gross on every captured payment) vs. reconciled cash (cash basis, actual net bank movement) must tie to the paisa, per the brief. Verified the same way every identity in this project has been — computed against run_2000, and iterated until the residual hit exactly zero, not assumed correct from the derivation alone:
- First pass was off by ~₹3.1 lakh. Cause:
unsettled_gross_paisewas summed onnet_paiseinstead ofgross_paise— an unsettled payment hasn't been through the fee-deduction pipeline at all, so its full gross belongs in the gap, not a net figure that doesn't apply yet. - Second pass was off by exactly -18 paise. Cause: the fee/GST term was reconstructed as
fee_paise + gst_paise, which silently dropsrounding_drift(applied directly tonet_paise, never touching the fee/GST fields). Replaced withgross_paise - net_paisedirectly — the true source of truth for what a settlement actually paid out, regardless of which defect caused the gap. - Also fixed: sign errors on
adjustments_paiseandunidentified_credit_paise(both increase actual bank cash, so they should shrink the book-reconciled gap, not widen it), and a missinglegitimate_chargeback_paiseterm entirely, found via L0'schargeback_paymentmatches.
Residual is now exactly 0 on both run_2000 and sample_200. Separately, _infer_as_of_date originally took the max date across payments+settlements+bank — since settlements/bank trail captures by the T+N lag, that pushed "today" past every unsettled payment's own expected date and made the 14-day curve trivially all-zero. Fixed to use only capture dates.
Per the brief's own fallback rule ("a working CLI + report.html beats a broken dashboard, judges grade accuracy not CSS"), the 4 panels live in eval/report.py's existing static HTML report rather than a new Next.js subsystem:
- Headline strip — gained "₹ reconciled" (previously missing).
- Layer waterfall — unchanged from Phase 2.
- Exception queue — now sortable (click the Category header; vanilla JS, no framework) and click-to-expand per row, showing the full
evidence_chain,affectedids,recommended_action, and a link to the agent trace when one exists (checked againstbenchmarks/sample_traces/— the 5 live L3 records have one, everyrun_2000exception is deterministically classified and correctly shows none). - Cash position — a 14-day inflow bar chart plus the full book-vs-reconciled table, rendered as the named components from Part 1, which sum exactly.
Verified interactively in the browser, not just by reading the HTML source: bar heights checked against the underlying data via DOM inspection, click-to-expand confirmed to actually toggle the detail row's computed display style, and the sort function confirmed to actually reorder rows by amount.
Benchmark freeze: data/fixtures/run_{500,2000,10000} (seed 42) through the complete pipeline, committed as benchmarks/freeze_{500,2000,10000}.{json,html} — consistent across a 20x size range (see README's results table), wall clock scaling linearly.
Fresh-clone checkpoint verified for real: cloned into an isolated temp directory, pip install -e . into a brand-new venv, make demo ran clean. That same run caught a real gap — pytest crashed at collection (not just failed) because test_anthropic_client.py/test_nim_client.py import anthropic/openai unconditionally, and those packages live in the optional llm extra, not the base install. Fixed with pytest.importorskip: a bare install now gets 2 graceful skips (140 passed, 2 skipped) instead of an interrupted run; the dev environment with the llm extra is unaffected (151/151).
The layer cake's whole claim is that each layer absorbs what the one above it can't. On the original fixture that claim was unproven for half the stack: L2 and L3 both contributed exactly zero. That was reported honestly, but honesty about an unproven claim is not the same as proving it — and the fault was a generator that was too kind, not an engine that was too good. Two injectors were added, both modelled on what real gateways actually do:
13. consolidated_payout — one bank credit paying 2–4 same-day settlements at once, carrying only a batch reference (PYT…, deliberately not UTR-shaped) and no per-settlement UTR. Real daily payout runs look exactly like this. L0 structurally cannot join it (no UTR to key on); L1 cannot either (no single settlement's amount matches a 2–4 way sum). Only subset-sum can explain it.
14. compound_fee_tax_error — a payment where the fee tier and the tax rate are both wrong at once. explain_variance's fee-tier hypothesis assumes correct GST and its GST-rate hypothesis assumes the correct fee, so neither can decompose a delta caused by both. The injected rates (1150/1375/2550 bps) are deliberately disjoint from KNOWN_WRONG_GST_BPS, so the GST branch cannot accidentally catch it. The result is a genuinely UNEXPLAINED variance — real work for L3, and verified to be exactly that: the residual set equals the injected set, no false residual and none missed.
Two things worth naming about how this was done, because both cut against making the numbers look good:
- The L1 test was made stricter, not weaker. A consolidated credit initially scored 0.71 narration similarity, so L1 rejected it — but for the wrong reason (unfamiliar wording).
CONSOLIDATED/CONSOL/PAYOUTwere added to the settlement vocabulary (genuine domain wording any ops team reads as a settlement credit), raising it to 0.86 so it now passes L1's gate. L1 must therefore reject it on amount alone, which is the real reason L2 is required. - A degenerate ambiguity was fixed properly rather than papered over. Two consolidated payouts returned
AMBIGUOUSbecause one member settlement nets to exactly ₹0 — including or excluding a zero-value term gives an identical total, so k such terms multiply the solution count by 2^k while carrying no information. The guard was refusing correctly but uselessly.l2_subsetnow excludes zero-amount candidates from the pool, so the meaningful members resolve unambiguously and the ₹0 settlement is simply never claimed. That costs 2 links of recall (99.95%, not 100%) and is the honest answer: a bank credit genuinely cannot evidence whether a ₹0 settlement rode along inside it.
This also exposed a real test bug I had previously mis-diagnosed. test_run_l3_aggregates_matches_and_exceptions_across_records had failed intermittently once, and I attributed it to my own edit script rewriting a file mid-import. That was wrong. run_l3 executes records concurrently, and the test scripted a single shared FakeClient queue — so which record received which scripted turn was a genuine race. It only surfaced now because the propose_match false-match fix made the unfavourable interleaving fail loudly instead of silently asserting a bad link. FakeClient gained per-record scripting and a lock; the test is now deterministic.
Hardening gave L3 a genuine, non-empty run_2000 residual for the first time (6 records). Running it for real against nvidia/nemotron-3-ultra-550b-a55b surfaced a real production-shape gap before it produced a real result:
A single 504 killed the whole batch. The first live attempt died with openai.InternalServerError: Error code: 504 from inside asyncio.gather — one upstream 5xx, and every sibling record's result was discarded with it, not just the one that failed. _call_with_backoff only retried RateLimitedError (429); a 5xx/timeout/connection error wasn't a case it had been built for at all. Fixed in two parts, deliberately kept separate: (1) TransientBackendError, a new exception distinct from RateLimitedError — translated from InternalServerError/APITimeoutError/APIConnectionError in both NimClient and AnthropicClient — retried with the same backoff, because a 5xx means the request was never meaningfully processed and retrying it is safe; (2) each record's run_one() task wrapped in its own try/except so any failure, retried-out or not, becomes an honest AGENT_INCOMPLETE ledger entry rather than destroying the other five records' results. Two new tests cover both paths directly: a client that fails once then succeeds (proves the retry), and a client that always fails for one specific record while succeeding for another (proves isolation).
The isolation fix then proved itself against a harder failure than the one it was built for. The model started returning NotFoundError: 404 mid-session — not a 5xx, a "this model doesn't exist right now" from NIM's side, confirmed by a direct minimal call outside the harness while models.list() still listed it in the catalog: a transient provider-side outage, not a config error. 404 is deliberately not retried (retrying a genuinely-gone resource wastes budget for nothing), so all 6 records correctly fell through to AGENT_INCOMPLETE with the real reason recorded — the batch didn't crash, it produced 6 honest ledger entries. Re-run after the outage cleared, the same model returned real results.
The first clean result, independently checked against ground_truth.json directly rather than trusted from the pipeline's own summary: 2 of 6 records correctly matched — both cross-checked by hand against ground_truth.json's order_to_payment links — and 4 of 6 ended AGENT_INCOMPLETE, exhausting the 8-turn budget. Zero false matches. Checking why 4 of 6 ran out of budget, rather than accepting the number, found a real cause and three more real bugs, none of which the summary numbers alone would have surfaced.
The turn budget was actually 1 turn short, not just "tight." Reading all four AGENT_INCOMPLETE traces turn-by-turn: three showed the same shape — get_record, compute_expected_fee, three separate explain_variance calls (correctly probing fee, GST, and net variance independently, because this is compound_fee_tax_error, where no single hypothesis decomposes the delta), two more get_record calls, then find_candidates on the very last turn to rule out a batching explanation - a completely legitimate investigation that simply needed one more turn to call raise_exception. The fourth had already reasoned correctly and tried to close the loop, but propose_match rejected its argument (it tried to assert a bank-credit id that only relates to the settlement, not the payment under investigation) on the last available turn, with no room left to retry with a corrected argument. DEFAULT_MAX_TURNS raised from 8 to 12. While making this change, a second latent bug surfaced: _cache_key() never included max_turns, only PROMPT_VERSION - changing the budget without remembering to bump that constant would have silently replayed the old, incomplete cached trace forever. Fixed by adding max_turns to the key directly, so this class of bug can't recur regardless of what a future change remembers to bump.
Re-running with the new budget surfaced two more real bugs, both about ledger fidelity, not link correctness. raise_exception accepted whatever category string the model supplied with zero validation; the live model invented FEE_CALCULATION_VARIANCE and unexplained_fee_variance - neither a real category - which would have silently broken every category-keyed downstream consumer (the dashboard's by-category breakdown, compute_defect_confusion's exact-match scoring) with no error anywhere. Fixed the same way every other hard constraint in this file is enforced: structurally, not just prompted - category is now an enum in the tool's own JSON schema, and raise_exception rejects anything outside RECOMMENDED_ACTIONS's keys with a ToolError, giving the model a real chance to retry within its remaining turns instead of polluting the ledger. Separately, one exception reported amount_at_risk_paise: 58700 - the payment's full gross - when its own rationale text correctly identified the actual variance as 142 paise; the tool schema's description of that field said nothing about what it should mean, so it was clarified explicitly ("the specific unexplained amount... NOT the payment's full gross/net amount").
Re-running again to exercise both fixes found a fourth, deeper bug, this time in the scoring code itself, not the agent. raise_exception had always stamped affected as {"record_id": ..., "source": ...} only - a different convention from engine/exceptions.py's own, which always uses the source-specific field (payment_id, bank_txn_id, ...). eval/metrics.py's CATEGORY_IDENTITY_FIELDS looks up exceptions by that source-specific field for every category except UNEXPLAINED_VARIANCE (which happens to check record_id too). The result: every FEE_VARIANCE/TAX_VARIANCE/etc. exception L3 ever raised was invisible to compute_defect_confusion - not misclassified, not detected, just silently absent, indistinguishable from "nothing was raised at all." This had been true since Phase 5 and nothing had ever surfaced it, because no one had checked the live run's own defect_confusion breakdown against its exception list by hand until now. Fixed by having raise_exception stamp the source-specific field too (reusing get_record's own _ID_FIELD mapping, the same fix applied to the failed-record fallback path in l3_agent.py), and by giving AGENT_INCOMPLETE its own CATEGORY_IDENTITY_FIELDS entry so a turn-budget exhaustion now reads as "misclassified" (something was raised, wrong label) rather than "missed" (nothing was raised) - a materially more honest signal for a reader of the per-defect-class table. Three tests cover this directly, exercising the real raise_exception function rather than hand-built exception objects.
The final, fully-verified result (benchmarks/phase5_live_residual.json, traces in benchmarks/sample_traces_live/), independently checked field-by-field against ground_truth.json rather than trusted from any summary number: 0 of 6 AGENT_INCOMPLETE, 0 invented categories, 0 false matches. 5 of 6 exception amounts matched the generator's own labelled defect amount exactly to the paisa; the sixth cited the largest single unexplained leg's delta (₹7.44) rather than the smaller net delta (₹1.83) when two legs partially offset - a real but minor imprecision, documented rather than hidden (see defect_confusion_note in the committed JSON and README's Limitations). compute_defect_confusion now honestly shows 1 detected / 4 misclassified / 1 missed for compound_fee_tax_error, replacing a {detected: 2, missed: 4} reading that had been silently wrong the whole time. This took three live re-runs and four real bugs, on top of the two already found in the section above - six in total from one 6-record residual, none of which a false-match-rate or auto-match-rate summary alone would ever have caught.
Cost, checked against Phase 5's own "<$0.50/1000 records" checkpoint, and found to not exist at all. EngineMeta.cost_usd_micros existed in the contract and was threaded through eval/metrics.py's cost_per_1000_records calculation, but nothing ever set it - every committed benchmark, including live runs with real tokens billed against a real account, reported cost_usd_micros: 0. Token counts were tracked correctly throughout; only the USD conversion was missing, silently, since Phase 5. Fixed with engine/llm/pricing.py: a small per-model $/1M-token table (NIM's rate for nvidia/nemotron-3-ultra-550b-a55b sourced from its public OpenRouter listing, since NVIDIA doesn't publish per-token pricing for the hosted NIM preview endpoint itself; Anthropic's official published rates included for AnthropicClient, never run live) feeding a cost_usd_micros(model, input_tokens, output_tokens) function, wired through l3_agent.py's per-trace and aggregate cost, pipeline.py's meta construction, and a new headline card in eval/report.py (shown only when llm_calls > 0, so a deterministic-only run never shows a misleading "$0.00"). The final run's real number: $0.104 total, $0.056 per 1000 records - comfortably inside the brief's target even at the higher 12-turn budget and the extra tokens six re-runs cost, but the point was never that it passed; it's that it couldn't have been checked at all before this fix.
Same bug shape as cost_usd_micros, found by the same discipline: don't trust a field just because it's in the contract - check whether anything actually sets it. ReconException.aging_days is part of the brief's own exception-ledger shape (category · severity · amount_at_risk · aging_days · evidence_chain · recommended_action · suggested_owner) and is rendered as its own "Aging" column in the dashboard's exception queue - every row read 0d, for every exception, at every scale, the whole time. Nothing in engine/exceptions.py or engine/l3_tools.py had ever computed it; every ReconException construction site either omitted the field (defaulting to 0) or, in one case, took a d["aging_days", 0] fallback that only ever had 0 to fall back to.
Fixed by giving both producers a real, shared notion of "now" and "when did this happen": engine/io.py gained infer_as_of_date(dataset) (moved from cash/forecast.py's private, near-identical copy - one source of truth for what "today" means for a given fixture, the same principle as compute_expected_fee) and aging_days(as_of, event_date_str). Every deterministic classifier in engine/exceptions.py now passes the record's own underlying event date - a payment's captured_at for MISSING_SETTLEMENT/FEE_VARIANCE/REFUND_MISALLOCATION/etc, a settlement's settled_at for PERIOD_CUTOFF, a bank line's value_date for ORPHAN_CHARGEBACK/UNIDENTIFIED_CREDIT. engine/l3_tools.py's raise_exception and its HIGH_VALUE_MATCH_REVIEW companion do the same via _residual_aging_days, looking up the record under investigation's own date field. aging_days is clamped at 0 (never negative - a negative value would only mean a data-quality issue in the fixture, not something a ledger should ever display).
This surfaced a second, smaller bug while fixing the first: infer_as_of_date crashed with max() arg is an empty sequence on a dataset with zero payments - true for two existing unit tests that construct a bank-only Dataset to isolate ORPHAN_CHARGEBACK/UNIDENTIFIED_CREDIT classification. Rather than coupling every isolated classifier test to a full payments list, infer_as_of_date was made to fall back to the latest settlement/bank date, and finally to the real wall-clock date only if the dataset carries no dates at all - a degenerate case that only a synthetic test fixture, never a real one, would hit.
Purely additive: every committed benchmark's auto_match_rate/false_match_rate/precision/recall/exception count is byte-identical before and after (verified by diff, not assumed) - only aging_days (and the naturally-varying wall_clock_seconds/generated_at_unix) changed. benchmarks/freeze_{500,2000,10000}.json, phase3.json, phase4.json, phase5.json, and both baselines regenerated and re-verified.
Prompted by finding three "field exists, nothing sets it" bugs above, a deliberate sweep of every field in engine/contract.py turned up a fourth, quieter version of the same shape: ReconException.suggested_owner was never unset (it always had its dataclass default, "Reconciliation Ops"), but nothing ever overrode it either - every category, at every scale, showed the identical owner. Not wrong, exactly, but a column that carries zero information is the same failure as one that's silently 0: a real ops org doesn't route an FX rate dispute and a chargeback dispute to the same desk, and a judge scrolling the exception queue would see one unchanging value in a column that exists specifically to answer "who handles this."
Fixed with SUGGESTED_OWNERS, a category-to-team map in engine/contract.py next to RECOMMENDED_ACTIONS (same pattern, different question - "who" instead of "what"): MISSING_SETTLEMENT/DUPLICATE_PAYMENT/REFUND_MISALLOCATION to Payments Ops, FEE_VARIANCE/PERIOD_CUTOFF to Finance Ops, TAX_VARIANCE to Tax & Compliance, ORPHAN_CHARGEBACK to Disputes & Risk, FX_VARIANCE to Treasury, HIGH_VALUE_MATCH_REVIEW to Finance Controller (the project's own namesake role), and UNIDENTIFIED_CREDIT/UNRECONCILED/AGENT_INCOMPLETE/UNEXPLAINED_VARIANCE staying with Reconciliation Ops as the genuine generalist catch-all for open-ended investigation. Wired through both producers - engine/exceptions.py's _exc() and engine/l3_tools.py's raise_exception plus its HIGH_VALUE_MATCH_REVIEW companion. freeze_2000 now shows 6 distinct owners across its 139 exceptions, not 1. Baselines (engine/baselines.py) deliberately left untouched - they exist to validate the harness, not to model realism, same exemption as their aging_days.
L3 could correctly find that a payment's fee tier AND tax rate were both wrong at once (compound_fee_tax_error - two overlapping causes, so no single-cause hypothesis decomposes it), but had no structural way to say so: it labelled the record with whichever single category it asked for (FEE_VARIANCE), citing whichever leg's own delta happened to be largest - even when that leg's delta wasn't the actual net amount at risk (fee and GST deltas partially offsetting means the real bottom line is smaller than either leg alone). Per-class scoring on the live residual: 1 detected / 4 misclassified / 1 missed.
Fixed structurally, in the tool layer, not the prompt - the same "recompute, don't trust" pattern as severity_for_amount. engine/l3_tools.py gained: a VarianceObservation recorded on every explain_variance call (observed/expected/delta/cause); _unexplained_legs(ctx), deduped on (observed, expected) so a model re-checking itself with an identical call never inflates the count; _bottom_line_leg(legs), the leg with the largest abs(expected_paise) - the net-level comparison, not the largest delta. In raise_exception, if 2+ distinct undecomposable legs are on record, the category is coerced to UNEXPLAINED_VARIANCE and the amount to the true bottom-line delta, with the coercion and every leg logged into the evidence chain - never silent. PROMPT_VERSION bumped 1→2 first, so the cache-busting this requires actually happens (the exact trap max_turns hit once before).
7 new tests, the false-positive guard written first: one unexplained leg alone must never coerce (over-firing would turn every genuine single-cause FEE_VARIANCE into a vague UNEXPLAINED_VARIANCE and cost accuracy on a class that scores perfectly today). Deterministic suite re-verified byte-identical against every committed benchmark before spending anything on a live re-run - L0/L1/L2 are untouched by this change.
Two live runs, both real, both fully verified against ground truth, genuinely different results - and that's the honest finding. The first (36 minutes, a badly degraded NIM endpoint that day - see below) scored compound_fee_tax_error 6/6 detected: the model investigated thoroughly enough on every record to surface 2+ legs each time. A second run minutes later scored 4 detected / 1 misclassified / 1 missed: reading the misclassified record's own trace, unexplained_leg_count: 1 - the model checked only the GST leg this time and never separately checked the fee leg, so the coercion correctly did not fire on one data point. This is not a flaw in the fix; it's the false-positive guard working exactly as designed, against genuine run-to-run variance in how thoroughly a live model chooses to investigate. All 6 amounts were exact to the paisa against the generator's own labelled defect in both runs, and zero false matches in either. benchmarks/phase5_live_residual.json (the second, fully-clean run) is the committed reference; its compound_fee_tax_error_note field records the first run's 6/6 result for comparison.
Two data points isn't enough evidence for a variance claim, so two more real runs were done later specifically to check it (post-code-freeze, gathering more evidence rather than changing the engine - see benchmarks/phase5_live_residual_run3.json/run4.json). A 3rd run scored 5 detected / 1 misclassified, all 6 amounts still exact. A 4th scored 3 detected / 3 misclassified and surfaced something the first 3 runs hadn't: for pay_ymzQx3u8WEhd7G, the model called explain_variance on the fee leg (expected 29,195 paise, observed 30,732 paise, delta 1,537 paise - arithmetic exactly correct) and never called it on the GST leg at all, so unexplained_leg_count: 1 and the coercion correctly did not fire - but this record's true ground-truth amount is 4,119 paise (₹41.19), not 1,537 (₹15.37). In the earlier runs' single-leg cases, the one investigated leg happened to be close to the full compound total by coincidence of that record's own composition; this record's composition made the gap material instead of negligible. Corrected finding, across all 4 runs: detected counts for this class are 6, 4, 5, 3 out of 6 (not the narrower 4-6 range the first two runs alone suggested), and the reported amount is only guaranteed exact when the model's investigation actually covers the record's full compound variance - which it does most of the time, but not always. The tool layer itself is not at fault here (it correctly reports exactly what it was given evidence for, and refuses to guess at the rest); this is a live-model investigation-completeness limitation, disclosed rather than smoothed over. Trace: benchmarks/sample_traces_live_more_runs/run4_pay_ymzQx3u8WEhd7G_amount_understated.json.
Two more real bugs, found while chasing an unrelated anomaly during Task 3, both resolved before either live run above was trusted:
- A first attempt at Task 3's "attack f" (see below) broke
settlement_split, caught immediately, reverted. Not part of Task 2's own scope, but discovered and fixed in the same window - see the next section. - The first live run of this task's own fix showed a
settlement_splitanomaly (false_exception_raised: 11instead ofcorrectly_resolved: 11) that survived extensive isolation testing:git stashback to the true pre-sprint commit plusclient=Nonegave 11/11 correct; a scriptedFakeClientclosing the loop immediately gave 11/11 correct; three fresh-process repeats gave 11/11 correct every time. The anomaly did not reproduce in the second live run (11/11 correct there). Documented rather than chased further: observed once, in one 36-minute run against a degraded endpoint with unusually high retry/backoff activity, never reproduced in three separate controlled tests or a second live run. If it recurs, the isolation tests above are the starting point, not a fresh investigation.
Adversarial attack (Task 3, attack f): the identical UTR text appears in two separate bank rows. engine/l0_deterministic.py::match_settlement_bank_txn iterated every bank row independently with no check at all for whether a settlement had already been claimed by an earlier row in the same pass - it happily matched both rows to the same settlement, which would double-count that settlement's cash in any real reconciliation.
The first fix was wrong, and broke a real feature - caught immediately, not shipped. The obvious guard ("a settlement claimed once can't be claimed again") was implemented in both L0 and L1 (which had the identical gap one layer downstream, never having been told which settlements L0 already claimed). Running the full suite immediately after: 4 failures, all in the cash-reconciliation identity test. Root cause: settlement_split (11/11 correctly resolved, a real and already-correct feature) deliberately puts the same UTR on two bank rows on purpose - data/generator/defects.py::_inject_settlement_split splits one settlement's payout into two genuine partial deposits (part_a/part_b summing exactly to net_paise), both carrying the settlement's own UTR, because that's what a real split payout looks like on a bank statement. The one-claim-only guard couldn't tell that apart from a true collision and refused the second, legitimate half every time - unidentified_credit_paise on run_2000 jumped from ₹1,20,449.00 to ₹4,39,856.37, a ~3.65x inflation, caught by test_reconciliation_ties_exactly_on_run_2000 failing outright. Reverted via git checkout within the same session, before it was ever committed.
The correct fix is an accounting check, not a claim-count check, in engine/pipeline.py::_reconcile_settlement_credit_sums - a post-pass over every settlement_bank_txn match together, run once after L0+L1+L2 have all contributed (none of the three layers can see the others' matches while running, so this is the first point anything can check one settlement's full picture at once). A settlement's linked credits summing to at most its own net_paise (+₹1 tolerance, for paisa-level rounding elsewhere in the chain) is accepted regardless of how many bank rows that is - correctly allowing settlement_split. A sum exceeding net_paise means something is genuinely wrong, and every link for that settlement is refused rather than guessing which one is real - same "never guess" principle as every ambiguity guard inside a single layer, just applied across all three layers' combined output.
Verified properly this time, having just been burned by shipping a plausible-looking guard without full verification: the fix was checked against both scenarios directly (a real collision, refused correctly; a legitimate split, both parts still match) before touching the test suite, then all 8 committed benchmark/baseline reports were regenerated and diffed byte-for-byte against their committed values (only the expected volatile timing fields changed) - not just the one test that had caught the regression the first time.
The other six attacks in tests/adversarial/attacks.py (REFUSED/CORRECT/FALSE_MATCH, run via run_full(client=None) - no LLM involved) all came back clean on the first attempt, no fixes needed:
| # | Attack | Layer | Outcome |
|---|---|---|---|
| a | Two settlements, identical amount/date, UTRs differing by the last two digits (transposed); narration truncated to a shared 14-char prefix | L0 | REFUSED |
| b | A bank credit's amount coincidentally equals the sum of two unrelated settlements dated 60 days earlier | L2's date window | REFUSED |
| c | Duplicate payment on one order, only one genuinely settled | L0/L4 | CORRECT (flagged DUPLICATE_PAYMENT, no false settlement link) |
| d | A refund reduces one settlement's net to coincidentally match a second, unrelated settlement's net | L1 | REFUSED |
| e | A card settlement's fee-adjusted net coincidentally equals a different, zero-fee settlement's gross; a third stray credit carries that amount | L1 | REFUSED |
| g | Three settlements where two different subsets both sum to one bank credit | L2's ambiguity guard | REFUSED |
Every attack shares one fixture-builder module and one run_all_attacks() function with the pytest suite (tests/adversarial/test_adversarial.py) and the regenerable artifact (scripts/run_adversarial.py → benchmarks/adversarial.json), so the committed JSON and what the tests actually checked can never drift apart - the same AttackResult list backs both. make adversarial regenerates it. 0 of 7 attacks produced a FALSE_MATCH in the final, committed state (1 of 7 - attack f - did during development, caught and fixed before being committed at all; see above).
The point of Task 4 was to measure, not assert, that deterministic-first is the right architecture: route every payment/settlement/bank record straight to L3 (engine/pipeline.py::run_llm_only), bypassing L0–L2 and L4 entirely, and see what an all-LLM system actually costs and scores against the same fixture. On data/fixtures/sample_200 that's 348 investigations (202 payments + 71 settlements + 75 bank rows), not the ~6-record residual the real pipeline ever sends to L3.
Two crashes, found by being the first mode that ever routes a "failed" payment (empty captured_at) to L3 at all - run_llm_only is a materially different caller than the real pipeline, which never sends a never-captured payment to L3 in the first place (L0/L4 handle it first). _residual_aging_days (l3_tools.py) and its twin _aging_days_hint (l3_agent.py) both called aging_days(as_of, event_date) on an empty string and raised ValueError - and critically, _aging_days_hint is called from inside run_one's own except-block, the fallback path that exists specifically to catch and isolate a failing record without taking down the batch. A safety net that itself throws on the exact input class it's supposed to handle isn't a safety net. Fixed by making both functions return 0 gracefully on an empty event_date; find_candidates got the same treatment (raises a proper ToolError instead of crashing when the record under investigation itself has no captured_at, plus a truthy-guard in its bank-debit candidate loop). Four regression tests added across tests/test_l3_tools.py and tests/test_l3_agent.py.
A third, more serious gap, found live: the first full-batch attempt hung for 10+ hours with zero progress and no error after the laptop it ran on went to sleep overnight. lsof on the stuck process showed all 8 DEFAULT_CONCURRENCY connections ESTABLISHED and 0% CPU - every worker thread's client.complete() call was blocked on a TCP connection that had gone half-open across the sleep, and the openai SDK's own 600s read timeout never fired because nothing ever arrived to time out against. run_one's existing per-record isolation (engine/l3_agent.py) only protects the batch once an upstream call actually raises - it does nothing for a call that never returns at all, and with every one of the 8 concurrency slots wedged the same way, the whole asyncio.gather simply stopped moving. Fixed with an outer PER_RECORD_TIMEOUT_SECONDS = 900.0 ceiling via asyncio.wait_for around each record's investigation, proven with a regression test that a fake client hanging on one record times out into an honest AGENT_INCOMPLETE without blocking its sibling record's completion (test_a_hung_client_call_times_out_instead_of_stalling_the_whole_batch). Disclosed, not hidden: asyncio.wait_for cancels the await, not the underlying OS thread (Python can't force-kill a thread), so asyncio.run()'s own cleanup can still be slow to return if a thread is blocked forever rather than just past the ceiling - a materially better failure mode than before (every healthy record still completes and is persisted to disk on time) but not an absolute guarantee against every possible hang.
A mistake made investigating it, corrected rather than glossed over: after finding the above and writing the fix, the process was killed on the strength of the last-known snapshot (139 records, unchanged since the sleep) without re-checking freshness immediately before acting. By the time the kill actually happened, the run had already recovered on its own and reached 159 records at its fastest pace of the whole run (~2.5 records/min) - a healthy, fast-progressing process was killed on stale evidence. No data was lost (every trace and its cache entry is written to disk as it completes, so the 159 already-done records were simply reused on the next run), and the timeout fix above is independently correct regardless of whether this specific run turned out to have actually been stuck - but the kill itself was premature, and the fix for that is procedural, not code: verify freshness (a fresh timestamp on the most recent file, not just a raised count) immediately before any destructive action, not from a read that's already several minutes old by the time the decision is acted on.
The result, on the second (resumed, cache-hit) run: 348/348 records, zero crashes, wall clock ~53 minutes, 1,207 LLM calls. 81.52% auto-match, 0.00% false-match, 100% precision, 77.36% recall, at $6.58 total ($35.76/1000 records) - see the README's ablation table for the full comparison against L0+L1, L0+L1+L2, and full. False-match rate holding at 0.00% even with every deterministic layer disabled is the same "refuse rather than guess" discipline (constraint 3: confidence < 0.85 → raise_exception) doing its job under load; the recall drop is that discipline correctly declining to guess on records the deterministic layers would have resolved with certainty. Source: benchmarks/ablation_llm_only.json.
engine/io.py::load_dataset did zero validation before this: every field was a raw int(r["..."]) or r["..."] dict lookup, so a missing column raised an uncaught KeyError, a non-integer amount an uncaught ValueError, both with Python's default traceback and no file/row/field context - and csv.DictReader doesn't raise on a duplicate header (a later column silently clobbers an earlier one) or on a row with more fields than the header (the overflow lands under a None key, silently, unless something checks for it). A real system ingests files produced by other systems - a bank export, an ERP dump - and those break in ways a synthetic generator never will.
Wrapped, not replaced: load_dataset's public signature and return type are unchanged, so every existing call site keeps working. Each of the four CSVs now goes through a small per-file validation pass before constructing its rows: header validated first (missing columns, duplicate columns, or a genuinely empty file with no header at all), then a data-rows check (a header with zero rows is itself rejected - a merchant's payments file being unexpectedly empty is a real ingestion red flag worth failing loudly on, not silently reconciling "nothing" and reporting a misleading 100% match rate), then per-row checks: field-overflow (csv.DictReader's None-key convention), integer parsing, non-negativity where the field's real-world semantics require it, date/datetime format, and duplicate primary keys. Every problem raises DatasetValidationError(filename, reason, row, field) - never a silent skip, never a coerced value. Fails at the file boundary, not the first bad row anywhere: a whole file is cheap to fully scan, so if row 47 and row 48 are both broken it's more useful to see both than to stop at 47 - but the first file with any problem stops the whole load, rather than reading all four and reporting a confusing partial picture.
Two real, if minor, findings surfaced while deciding exactly which fields "only positive is valid" actually applies to - checked empirically against the generator and every committed fixture, not assumed:
- A
payment.captured_atof empty string is not a malformed date - it's the correct value for astatus="failed"payment, which never captured and so genuinely has no capture timestamp. The date-format check only fires on a non-empty, unparseable value. bank.credit_paise/debit_paiseare validated non-negative - but this was briefly relaxed. A defect injector indata/generator/defects.pyapplied a net-adjustment delta directly to a settlement's linked bank credit (linked[0].credit_paise += delta_paise), and a large-enough negative delta on an already-small credit drove it below zero - confirmed empirically: none of the four committed reference fixtures (run_500/run_2000/run_10000/sample_200) hit it, but the Task 1 multi-seed sweep produced a realcredit_paise: -34800(seed 100 originally; seed 1 after later hardening commits shifted the RNG). While that was possible, enforcing non-negativity here would have made a real generator output fail to load, so the check was off and the quirk flagged as a generator-side finding. That finding is now fixed:_adjust_bank_credit_for_settlementfloors the adjusted credit at 0 (a negative credit doesn't correspond to anything a real bank statement would show - money moving the other way is a debit). All six multiseed seeds and the four reference fixtures were re-verified to carry no negative credit/debit (scripts/verify_no_negative_bank_credit.py; fixtures also byte-identical before/after the floor), so the io.py check is back on.settlement.adjustment_paiseis still deliberately never checked - it's a genuinely signed figure by design (a gateway recovery or credit can be negative), same principle asaging_days's own "know the domain before hard-coding a rule" discipline elsewhere in this codebase.
13 tests in tests/test_malformed_input.py cover all 10 required cases (missing column, duplicate header, non-integer amount, negative-where-only-positive-valid, unexpected date format, empty file, headers-only file, non-UTF8 bytes, duplicate primary key, row with more fields than header) plus one confirming a clean fixture set still loads unchanged, one confirming the failed-payment/empty-captured_at case specifically is not a false positive, and one confirming a negative bank.credit_paise is now rejected (see the generator floor above). All 8 committed benchmark/baseline reports regenerated and diffed byte-for-byte afterward - purely additive, zero drift.
No new engine code, per the schedule's own rule — never cut the generator's ground truth, the eval harness, or the exception ledger; everything else is presentation.
README.mdrestructured to the brief's own mandated order: one-line description, results above the fold, reproduce in 3 commands (git clone && pip install -e . && make demo), an architecture summary + link here, then limitations. The per-phase deep-dive narrative that used to live in README (fee model, defect list, reference-run stats, baseline table) moved into this file's Phase 1/2 sections instead, so there's exactly one place each fact lives rather than two copies drifting apart — fixed one stale cross-reference (ARCHITECTURE.mdlinking to a README anchor that no longer existed) while doing it.
The brief's own two optional "if time allows" items - a FastAPI API layer and an interactive Next.js dashboard - built after the code freeze and the judge-fix round, as a deliberate, dated scope extension (see RESULTS.md, "Post-submission stretch goals"), not a reopening of the frozen engine.
Design constraint that shaped everything else: the API is a thin wrapper around code that already exists and is already trusted (data.generator.generate.run, eval.report.run_eval), not a second implementation. api/runs.py::run_live generates to a scratch directory and calls run_eval directly - the exact same function the CLI and scripts/multiseed.py already call - then deletes the scratch directory. This means the API's live-run endpoint and the frozen freeze_2000.json benchmark are, byte for byte, the same code path; tests/test_api.py asserts this directly rather than trusting it by inspection.
A real bug caught before it shipped: the first version generated to a randomly-named tempfile.mkdtemp() directory. eval/report.py's trace-drilldown lookup (SAMPLE_TRACE_SOURCES, added in the earlier judge-fix round) matches on the fixture directory's own name being "run_2000", not on its content - so a live run at seed=42/records=2000 would reproduce run_2000's data exactly, yet its trace links would all silently come back None, because the scratch directory wasn't named run_2000. Fixed by giving the scratch directory the same conventional name (run_<records>) a real fixture at that scale would have - run_live never touches data/fixtures/, so there's no collision risk, just a name match where one is needed. Verified live (not just in pytest): a seed=42 run through the actual running API links to the real committed traces in benchmarks/sample_traces_live/, and a freeze_10000 view through the dashboard still shows zero trace links, proving the fixture-scoping still holds.
Safety boundary, enforced at the orchestration layer, not just documented: api/runs.py::ENGINE_ALLOWLIST is (null, oracle, l0l1, l0l1l2, full) - llm-only (the costed, all-LLM ablation engine) is asserted absent from it at import time, and no request field can supply an LLM client, model name, or API key. This is the same client=None invariant eval/report.py's own ENGINES dict has always enforced for the CLI, now extended to a form a browser can submit. CORS is restricted to the dashboard's own dev-server origin (http://localhost:3000) - this was built and verified local-only, per an explicit decision, not deployed anywhere.
Frontend: Next.js (App Router) + Recharts, matching the brief's own named stack literally. Four panels mirror eval/report.py:render_html's structure exactly (headline cards, a layer-waterfall bar chart, a sortable click-to-expand exception queue with the same trace-drilldown behavior, cash position), plus one genuinely new capability neither the static report nor a CLI can offer: a live run form (engine/records/seed/months → "Run live") that re-renders the same four panels from a fresh result. A second mode browses the already-frozen historical benchmarks without regenerating anything.
Three additive metrics, none touching the matching engine (engine/l0_deterministic.py through pipeline.py untouched throughout).
Fee leakage (eval/metrics.py::compute_fee_leakage): total money the merchant was overcharged in gateway fees, tax on those fees, and FX, aggregated only from FEE_VARIANCE/TAX_VARIANCE/FX_VARIANCE exceptions - deliberately excluding MISSING_SETTLEMENT/PERIOD_CUTOFF/DUPLICATE_PAYMENT/UNIDENTIFIED_CREDIT/REFUND_MISALLOCATION/ORPHAN_CHARGEBACK, which are timing, duplication, or attribution problems, not overcharges. On run_2000: ₹10,475.40 across 42 records. Reported as a lower bound, always: a compound error coerced to UNEXPLAINED_VARIANCE (see the Task 2 hardening-sprint section above) contains real fee leakage this number can't count, because its cause was never isolated to a single fee leg.
Cash forecast viewpoint (cash/forecast.py, eval/report.py --as-of): compute_forecast already accepted an as_of_date parameter; it was simply never threaded through from the CLI. The default behavior (infer_as_of_date - the dataset's own latest capture date) is correct, but it's the least informative viewpoint on a historical fixture: almost nothing is still in flight by the dataset's own last day, so the 14-day inflow curve renders as 2 nonzero days totalling ₹4,990 against ₹4.9 lakh stuck. An earlier, operator-chosen --as-of shows the same underlying data from a viewpoint where more payments haven't settled yet - 4 nonzero days (₹84,621.75) seven days earlier, 8 nonzero days (₹2,08,228.03) thirty days earlier. This is a presentation choice about which day to view the same correct data from, not a correctness fix - the report always labels which viewpoint it's showing (ForecastResult.as_of_source: "supplied" vs "inferred"), so an operator-chosen viewpoint can never be silently mistaken for the system's own inference. make demo-cash runs the demo at 30 days before the fixed dataset end, for exactly this reason.
Exception aging (eval/metrics.py::compute_exception_summary's new "aging" key): median/max/bucketed aging plus a count breaching the industry-standard 48-hour SLA for open reconciliation breaks. On run_2000: median 35 days, max 89, 129 of 139 past 48 hours. This number is not alarming and must never be presented as if it were: run_2000 is a fixed, historical 3-month synthetic fixture, scored against its own end date, not a live queue that's actually been sitting untouched for months - large aging here is the expected consequence of scoring a static historical snapshot, not a finding about operational neglect. Suppressing this framing would be dishonest in one direction; reporting the raw number without it would be dishonest in the other. Both the HTML report and this note state it plainly, every time the number appears.
All three verified to change nothing about matching: auto_match_rate, false_match_rate, precision, recall, and exception count are byte-identical to the pre-existing frozen benchmarks throughout - confirmed by diff after each change, not assumed.