v3.12 lot 2 — make the router prompt a pure append

[Kurtisone]

Branch: v3.12-pure-append (base: main, after PR #17 / lot 1)
5 patches · 588 tests green (baseline 577) · ruff check and ruff format --check clean on every commit individually

Measured on the target box (Steam Deck → NiPoGi, llama.cpp server, 13 tools loaded).

Why

Lot 1's instrumentation, plus the LLAMA_CPP_CACHE_PROMPT fix, took a routing
run from ~52 s to ~11.5 s. The remaining ~8.5 s was prompt processing: the
whole prompt was being re-evaluated on every call even though only ~25 tokens
of it changed since the previous one.

Bisecting the cache behaviour with reuse_partiel.py produced a cliff rather
than a curve:

prefix relationship	ms/token
identical prompt	0.09
appended at the end	0.28
inserted at 90%	2.08
inserted at 75%	12.40
completely different	12.27

The model that fits: a pure append continues from the live slot state and
costs almost nothing. Any insertion further up requires a rewind — restoring
the recurrent state from its last checkpoint and replaying forward — and past a
certain depth no checkpoint remains, so the whole prompt is recomputed.

The server-side lever is exhausted. -ctxcp 16 -cms 128 were confirmed applied
and had no effect: in llama-server's help --ctx-checkpoints is an alias of
--swa-checkpoints, so it governs sliding window attention, and this model has
no SWA layers. -c 16384 -np 1 moved nothing either (~5%, inside the noise),
though it is worth keeping for the window headroom.

That leaves the application-side lever, which is this branch.

What was actually in the way

The prompt was laid out as:

[static template][history][step_context][closing instructions][User: <input>]

Two things broke the append property, and only one of them was the closing
instructions.

Closing instructions after a growing block. A fixed ~50-token block sat
after a section that grows every turn, so each turn inserted text into the
middle of the prompt rather than extending its end.

The live turn and its own history rendering were different bytes. Turn N's
message appeared at the bottom of prompt N as \nUser: <message>\n and came
back in prompt N+1, through memory.json, as - they said: <message>. Moving
the closing instructions alone would not have helped: the common prefix would
still have ended where the history block ended.

Three smaller ones behind those:

• the vague-file-reference instruction was the tail of _format_history(),
  so every new turn was inserted in front of it — the same defect one level
  down, invisible from the template;
• the history header was emitted only when history was non-empty, so it
  first appeared on turn 2 — an insertion in front of turn 1, i.e. one
  guaranteed cache miss per conversation;
• _MAX_HISTORY_ENTRY truncated user entries at 120 chars while the live
  line is never truncated, so any multi-line question diverged from its own
  history rendering at character 120.

What changed

The layout is now:

[static template + closing instructions + history header][history]
[step_context][User: <input>]

• chore(router) — drop the dead /no_think token. Qwen3's soft switch was
  removed in Qwen3.5, and the GBNF grammar forces the first generated token to
  be { anyway, so no reasoning block was ever reachable. It sat in front of
  the entire cacheable prefix.
• refactor(router) — hoist the closing instructions, the vague-file-reference
  instruction and the history header into the static block. The first two are
  reworded to address the last User: line, since the whole conversation now
  sits between them and the message they refer to. The file instruction is
  gated on the tool set rather than on history: a per-turn gate is what
  breaks the prefix, and the presence of files/review is what actually
  makes it meaningful. The tool set is fixed for the lifetime of the process,
  so the block stays static, and a tool not opted into via ENABLED_TOOLS is
  still never named in the prompt.
• fix(router) — both renderings of a user message go through a single
  render_user_turn(), newlines included, so they cannot drift apart in a
  future edit. No change was needed on the persistence side:
  memory.add_exchange(user_input, output) already stores the message
  verbatim, which was the one precondition that could have made this
  impossible.
• fix(router) — _MAX_USER_HISTORY_ENTRY (4000) for user entries; assistant
  entries keep 120.
• test(router) — eleven tests pinning the invariant.

Results

Prefix (string-level, deterministic)

Divergent tail between consecutive prompts, 12-turn conversation:

	pure append	tail (median)
before	0 / 11	573 chars
after	11 / 11	0

The tail was a constant ~573 chars regardless of conversation length.

Prompt processing

ms_per_token on a growing 12-turn conversation, from llama-server's own
timings:

turn	before	after
0 (cold)	11.04	11.00
1	2.99	0.50
2	3.12	0.55
3	3.35	0.57
4	3.24	0.63
5	3.35	0.64
6	3.17	0.81
7	3.76	0.86
8	2.65	1.30
9	2.46	1.11
10	2.66	1.08
11	2.57	1.42
median (warm)	3.12	0.81

3.9× faster. In absolute terms on a ~2700-token prompt: ~8.4 s → ~2.2 s of
prompt processing per routing call, which accounts for most of the ~8.5 s
residual lot 1 left behind.

Two caveats worth recording rather than burying.

main measured 3.12, not the ~1.80 the bisect table predicted for an insertion
this close to the end. The real-world cost of an insertion is higher than the
synthetic curve suggests; that table understates it.

And ms_per_token has become a misleading metric on this branch. prompt_n
counts all ~2700 tokens even though only ~25 are new, so it mixes two
quantities. After this change, absolute prompt_ms is the number to watch.
That also explains why repeated runs of the same arm disagree (0.62, 0.81, 0.80
medians across three runs): the residual absolute cost is not yet pinned down,
which is lot 3's problem, not this branch's.

Routing accuracy

The measurement that can regress silently. The GBNF grammar guarantees the
output shape, so "did it emit valid JSON" comes back green whatever happens
and is not evidence of anything.

29 fixtures, run through the real build_router_prompt → call_llm →
parse_router_output path with production ENABLED_TOOLS:

	pass	fail	manual
before	22	5	2
after	25	2	2

Agreement 25/29 (86%). Four decisions changed, no regressions:

fixture	before	after
a03 list files in a directory	shell (fail)	files (pass)
b02 script request after a files turn	chat (fail)	code (pass)
c02 "analyse le contenu", path only in history	chat (fail)	review (pass)
c05 control, no real path anywhere	files (manual)	review (manual)

c02 is the result that matters. It is the vague-file-reference fixture: the
path exists only in an earlier turn, and the instruction that resolves it moved
from directly adjacent to the message to roughly 3000 tokens above it. It not
only still works, it now works where it previously failed — plausibly because
the rewording targets the last User: line explicitly.

b02 is a "answer the last message" case, which is what the reordering
targets, so the mechanism is clear there too. a03 I cannot explain beyond
"the history block reads more clearly"; it is credited to luck until reproduced.

Order-independence check. The fixture set was re-run in reverse order
against the same arm: 29/29 agreement, bench median 0.80 vs 0.81. The four
changes above are attributable to the prompt, not to slot state or fixture
ordering.

Two pre-existing failures, not regressions

Both fail identically on main and on this branch. Separate issues, not
blockers here.

• c03 — with two file paths in history, "relis ce fichier" resolves to
  the older one instead of the most recently mentioned. The instruction says
  "most recently mentioned"; the model does not honour the ordering.
• e02 — with a [web_search] result already in step_context, the
  router calls web_search again instead of answering from it. A loop risk.

Two findings from the unscored fixtures

c05 and c05's sibling f01 have no correct automatic answer and are
recorded for reading by hand. Both surface the same gap:

• c05 (no real path anywhere in the conversation) → review with
  {"file_path": "src/forge/main.py"}. The path is invented. The
  instruction forbids inventing file content — "Do NOT invent file content
  from memory" — but says nothing about inventing a path when none exists.
• f01 → review with {"file_path": "<same path as above>"}, a literal
  placeholder emitted as if it were a path.

Neither is caused by this branch (c05 also routed to a file tool on main),
and f01 did not follow the injected instruction in its step_context —
it stayed on review rather than shell, which is the one point on which the
step_context positioning argument is positively confirmed. But the
instruction has a hole worth closing in a follow-up: "if no real path is
present, ask rather than guess".

Deliberate asymmetry

Only the user half of a turn is rendered symmetrically. Assistant turns
render as (you answered: ...), not Assistant: ....

The bullet-summary format existed for a real reason, recorded in the code: a
full User: ... / Assistant: ... dialogue completes a pattern that contradicts
what the examples above teach (User: X → JSON), and a 9B follows the nearest
surface pattern. An assistant turn only ever appears in history, never as a
live line, so nothing constrains its shape — and spending that freedom here
means the only thing that ever follows a bare User: line in this prompt is
JSON. The GBNF grammar is the structural backstop underneath.

Known limits, accepted rather than fixed

• today_line() is in the static header, so the entire prefix is
  invalidated once, at midnight. Moving it to the tail would trade one daily
  full recompute for a rewind on every turn. Worth knowing before reading a
  benchmark that straddles midnight as a regression.
• step_context is this run's own tool output and is never persisted, so
  the first call of the turn after a multi-step run cannot extend the last call
  of that run. Since step_context sits just before the live user line, the
  rewind covers both. Moving it after the user line would buy those tokens back
  and is deliberately not done: it would put untrusted tool output in the last
  position before generation, the strongest position in the prompt and exactly
  what the E-2 provenance markers defend against. A test asserts the rewind
  stays a tail and leaves the persisted history intact.
• A user message over 4000 chars diverges at the cap. Bounded and one-shot:
  the truncated form is stable from then on, so the next turn is a pure append
  again. Truncating at ingestion would remove it entirely and is not done —
  orchestrator.py:353 records that capping what gets persisted silently
  corrupted the web UI, which renders memory.json through GET /history.
• Compaction rewrites the head of the history block and invalidates
  everything. compaction.py already documents this. The consequence for this
  branch: eviction must happen in large batches, never one turn at a time, or
  every turn past the threshold is a full recompute and the whole gain
  evaporates.

Testing

test_router_pure_append.py — eleven tests: turn 2 extending turn 1; a
ten-turn conversation where each turn extends the previous; the header present
from turn 1; live and history renderings being the same bytes; the assistant
asymmetry; nothing emitted after the live user line; the static prefix being a
function of the tool set alone. Two boundaries are pinned as bounded damage
rather than as invariants, since neither can be made to hold: the over-cap
message and the step_context rewind.

A regression here has no functional symptom — every prompt stays correct, every
answer stays correct, and the only evidence is that runs get slower. That is
exactly how the old layout survived unnoticed, so each test was checked against
the defect it is meant to catch, by reintroducing the defect and confirming the
failure:

reintroduced defect	caught
fixed block after the live user line	yes
old divergent bullet rendering	yes
history header conditional on non-empty history	yes
user cap back at 120	initially not

The bounded-rewind test is written in terms of _MAX_USER_HISTORY_ENTRY, so it
passes at any value including 120. Only a test anchored to a concrete message
of ordinary length — a pasted traceback, a multi-line question — actually holds
the cap in place, which is why
test_a_realistically_long_user_message_is_still_a_pure_append exists.

Three existing tests were updated rather than deleted, since they asserted
invariants this branch deliberately inverts: the history-block marker string,
the file instruction being absent without history (now gated on tools), and
test_history_is_passed_as_context_not_dialogue, replaced by its inverse and
exercised through the real persistence path rather than a hand-built history
list.

Two fixtures in the A/B harness were wrong on the first pass and were fixed
before the numbers above were taken. d01-d03 asserted that the answer
content contained the remembered subject, which can only pass if the model
picks chat — routing to recall puts a query in the content and the answer
arrives from the tool afterwards, so a correct route was being scored as a
failure. b04 scored research as a failure on a question about quantization,
which is a defensible reading of that question rather than evidence the model
answered an earlier turn; the fixture now ends on a message that operates on
the conversation itself, so only chat is defensible.