feat/rag-intake — stop storing a compacted block as one averaged vector

[Kurtisone]

Base aa39191 (main, after PR #27). Twelve patches, 1103 tests green (baseline 1045),
ruff check . and ruff format --check . clean on the whole tree, series re-verified
with git am on a fresh clone. Author and committer are Kurtisone on all twelve.

Every number below was measured on the Deck against the real store, and the
before/after run was reproduced twice on two independently migrated copies.

The problem this closes

The recurring diagnosis on the vector store has been "it holds compaction pointers and
almost no facts". Reading it directly for the first time on 2026-08-22 turned that from
an inference into a count — 16 entries, 11 of them whole compacted blocks — and produced
a second number that nobody had asked for:

"Quels outils as-tu accès ?"                 0.9386   vs a history_summary that
                                                      contains the question almost
                                                      word for word
"Le serveur de test tourne sur quel port ?"  0.671    vs a short fact

About 0.27 of distance lost to burying a sentence in a block, on a store where the
whole hit/miss gap that day was 0.0422. That is not a model limitation and not a
threshold problem — a cutoff adds honesty, it does not create proximity. It is a bug in
what Forge writes.

The mechanism is in rag._embed: text past EMBEDDING_MAX_CHARS is split into chunks,
each chunk is embedded, and the chunk vectors are averaged into one. The mean of a
dozen unrelated subjects is close to no question in particular. compaction's
rag_pointer strategy fed it the entire evicted window as a single string, so every
compaction pass produced exactly that.

What changes

One entry per exchange. forge/transcript.py holds a single definition of how a
conversation is written down and where one exchange ends — render, blocks, split —
and the test that earns the module is the round trip split(render(msgs)) == blocks(msgs).
The unit is a user turn plus whatever answered it: a message on its own retrieves either
a question with no answer or an "oui, 8080" that has lost its subject.

rag.remember_many writes N units as N rows in one transaction. It costs no extra
embedding calls — _embed already made one request per chunk. The change is that the
chunks stay apart instead of collapsing into their mean. Two deliberate asymmetries with
remember(): a degenerate item is skipped rather than raised on (the caller is archiving
a block it did not write, and one one-word message must not fail the whole compaction),
and an EmbeddingError propagates before the commit, so a block is never half-indexed —
a pointer claiming a range that does not hold what it says is worse than no pointer.

Two things stop at the store boundary, both found by reading the real store:

• an earlier compaction pointer is not indexed again. It is a reference to another entry;
  stored, it becomes a memory whose entire content is "N messages were compacted, see
  security: audit lot 1 — XSS, unauth API, shell allowlist, git tool (+ example compose) #12" — it answers no question and sits at middling distance from all of them.
• raw router JSON is unwrapped. Entry v3.9 — Compaction, drawer, and a more reliable router for file operations #9 of that store was a {"tool": "code", ...}
  object swallowed from an assistant turn by a version that did not unwrap tool output.
  The envelope is the noise; the answer inside it is real.

_POINTER_RE and _pointer() are two statements of the same string, so a test pins them
together. A pointer that stops matching its own detector gets indexed as conversation and
nothing anywhere fails.

deploy/rag_resplit.py re-slices what is already stored. Without it the fix applies
only to future compactions and the store stays mostly made of the shape the change exists
to remove — which also means the next measurement against it measures the old problem.
Dry run is the default, --backup is offered, and pieces are inserted before the block is
deleted: an interrupted run leaves a visible duplicate rather than a missing entry.

bench/rag_dilution.py measures the claim instead of asserting it. The same sentence
is planted three ways — alone, buried in a transcript stored as one entry, and the same
transcript stored per exchange — and the same question is asked of each, so only one thing
moves. It prints the mechanism it depends on (how many chunks the buried form is averaged
from) and states plainly what a null result would mean: the dilution comes from somewhere
other than the averaging, and the change must not be defended with this number. The figure
is a lower bound — a real evicted block is an order of magnitude longer than the planted one.

What this does not do

It does not create facts. The store is 69% compaction pointers because nothing writes
fact entries unless a human types !remember, and that half of the intake problem is
untouched here. It needs an LLM pass under a grammar plus a grounding check, and it cannot
be evaluated while the store is still made of blocks — the granularity has to land, and be
measured, first.

Measured, not argued

All three checks were run on the Deck against the real store.

The premise, isolated on a planted sentence by bench/rag_dilution.py — the same
sentence, the same question, only the storage shape moving:

alone      0.5109   rank 1/1
buried     0.9559   rank 1/1     (23 exchanges, 3987 chars, 3 chunks averaged)
split      0.5109   rank 1/23

cost of burial: +0.4450

Splitting does not recover most of the gap between a block and the bare sentence — it
recovers all of it, to four decimal places.

The migration, run twice from the same backup — once before patch 0007 and once
after — on a store that had meanwhile taken a live compaction pass:

first run    16 entries -> 283      10 blocks -> 277 units
second run   33 entries -> 293      10 blocks -> 270 units

The seven-unit difference is 0007 doing its job: seven of the ten blocks opened on a
compaction pointer, so each had been producing one unit whose entire content was
"[59 messages précédents compactés — voir mémoire vectorielle #12]". After 0007 the
store holds zero bare pointers and zero raw router-JSON entries.

The consequence, six questions before and after:

before   hits 0.8934  0.671  0.9386  |  misses 0.9891  0.9356  1.098
         worst hit 0.9386 > best miss 0.9356   ->   NO GAP
after    hits 0.4498  0.671  0.7695  |  misses 1.0619  0.8337  1.0369
         worst hit 0.7695 < best miss 0.8337   ->   gap 0.0642

The short fact at 0.671 did not move by a single digit — only the entries that had been
buried in a block did. That is the control, and it is what turns RECALL_MAX_DISTANCE
from a guess into a choice. Every distance recorded before this branch (the five rows of
2026-08-19 at 0.90–1.0015) measured blocks that no longer exist and is now stale.

The six figures came back identical to four decimals on both migrated copies, which
is worth stating plainly rather than claiming as a second win: removing the seven
pointer-only entries changed nothing measurable here, because those entries were never
the nearest for any of these six questions. 0007 is hygiene and a guarantee that the two
intake paths cannot diverge — it is not, on this evidence, a distance improvement.

Confirmed in real use, with the cutoff active. "Quel est le modèle exact de ma voiture ?"
drops all five candidates (1.0755–1.1620), reaches the error node and makes zero LLM
calls — 4 s instead of 17. "Comment s'appelle mon chat ?" is dropped too rather than
answered from user: Bonjour Forge, comment je m'appelle ?, which was the near-duplicate
case this was expected to get wrong.

The first migration found two bugs in this branch

Patch 0007 exists because of the first. Patches 0001-0006 shared the cutting between
compaction and the migration and left the filtering behind, so seven of the ten
re-sliced entries began with a compaction pointer and each produced a unit whose entire
content is "[59 messages précédents compactés — voir mémoire vectorielle #12]". About a
dozen noise entries out of 278 — precisely the drift the shared module was written to
prevent, one level down.

forge/transcript.py now owns the whole sequence, and both callers are the same three
steps: units(messages) = blocks(indexable(messages)) and split(text) = units(parse(text)),
with a test asserting that equality on one input. The pointer builder and its regex moved
there too, and the pointer check no longer looks at the role — a re-parsed block can hand
a pointer back under whatever role preceded it, so the shape is the evidence, not the
speaker.

Known limit, named

A single message longer than EMBEDDING_MAX_CHARS is still averaged: the migration
logged rag.embed_chunked chunks=5 chars=6082 on six of the 270 units. Cutting at
exchange boundaries cannot help there, and cutting inside a message would separate a
statement from its subject — the failure this branch exists to avoid, one level down.
Those six units are the shape the rest of the store no longer has.

RECALL_MAX_DISTANCE=0.95, and why not the midpoint

The harness suggests 0.81, halfway-plus between 0.7695 and 0.8337. It is right about the
arithmetic and wrong for this store, because the three misses are not one family. Two are
"nothing here is relevant" (1.0619, 1.0369), which is what the setting is for. The third —
"Comment s'appelle mon chat ?" at 0.8337 — is close only because the store now literally
contains user: Bonjour Forge, comment je m'appelle ?: lexically near, semantically not
an answer. No threshold separates that cleanly, and one set at 0.81 leaves 0.04 of
headroom above the worst genuine hit.

The two errors are not symmetric. Too high lets a bad answer through, and a bad answer
gets argued with. Too low answers "je n'ai rien en mémoire" while the answer sits in the
store, and that gets believed. 0.95 cuts both real misses with ~0.18 of headroom and does
not pretend to solve a near-duplicate by picking a number. recall.dropped logs every cut
with its id and distance, so a value that bites in the wrong place is visible.

It is the one active line in .env.example. The code default stays unset — a deployment
that has not measured its own store is not handed someone else's number — and two tests
pin the value against the reasoning in config.py and pin that the default is still off.

Patch 0008: a log line that could not answer its own question

compaction.indexed reported messages=35 indexed=17 entries=17, and that was read as
proof the filter had fired. It was not. Seventeen units out of thirty-five messages is
what exchange grouping does on its own, two messages at a time, and the line carried no
number that could tell a dropped pointer from an ordinary pair. It now reports
messages / kept / dropped / units / entries, and dropped > 0 is the filter and nothing
else. transcript.indexable also takes a source label, because a resplit run was printing
compaction: model wrapped a substantive answer… while no compaction was happening.

Patches 0009-0012: found by using it

Everything below was discovered by running the migrated store, not by review.

0009 — the same exchange must not occupy three of five recall slots. A live recall
returned five entries and three were the same text at 0.8306 (#205, #227, #233).
Compaction windows overlap, so an exchange sitting in the tail of one and the head of the
next was indexed twice, and re-slicing made it three rows. A sweep of the real store found
72 duplicates out of 293 entries — a quarter of it. remember_many now skips exact
duplicates, within the batch and against the store, scoped to the project. Exact match
only: a near-duplicate is a judgement with a threshold to tune, an identical string is a
fact. Not applied to remember() — a human asserting the same fact twice is saying they
think it was forgotten.

0010 — say what was stored. Remembered (#305). is a receipt for a transaction nobody
can check; the entry that provoked this went in as pocresseur 5500U and the typo
surfaced days later through a debugging tool. The confirmation now echoes the text, its
kind, and how many entries of that kind exist — the same lesson files:write learned in
v3.11, when a created file answered with a byte count and had to be opened by hand.

It also flags a word that appears nowhere else in the store but sits within 0.85 of one
that does. The dictionary is the store itself, and that is the design: a French
spellchecker on this corpus is a machine for breaking identifiers — NiPoGi,
sqlite-vec, busctl, aardvark-dns, GBNF are precisely the tokens carrying the
information. Vocabulary drawn from what has already been written knows them because they
were used, and sharpens with every entry instead of needing an allow-list. It only ever
suggests: the entry is stored unmodified whatever the check thinks, because silently
rewriting a memory entry is the one place in Forge where being approximately right is
worse than being wrong — nobody re-reads an entry, so it returns weeks later as a fact
with no trace of the edit.

0011 — two bugs in that check, both found on its first real run. J'utilise aardvark-dns pour la résolution DNS was flagged on utilise and résolution, two words
written a dozen times in that store: the check subtracted the new text from the vocabulary
before searching, which deleted the exact hit proving each word was fine. Fixing that
exposed the second — rag.remember commits the row before the check runs, so the entry
was its own dictionary. Both directions are pinned, since either bug alone makes the
feature silently do nothing or silently cry wolf.

0012 — !forget where the message that recommends it is displayed. The confirmation
ends with "!forget 312 puis réécris-la", and that command existed only in the REPL, so
the web UI answered "commande inconnue". DELETE /memory/{id} was already there; only the
binding was missing. It is also the first UI command taking an argument, so the dispatcher
now passes one.

Two findings this branch does not fix

A stored refusal outranks the answer. Asked "Tu peux me lister mon matériel ?", the
nearest entry at 0.4519 is user: Tu peux lister mon matériel ?\nassistant: Je ne peux pas lister ton matériel… — an exchange in which Forge failed. The real answer, #138, is
second at 0.7891, and the model echoed the refusal. Indexing exchanges puts the question
text inside the entry, so an entry that is mostly the question and nothing else is the
closest possible neighbour of that question. These refusals were always in the store; the
granularity made them competitive.

The embedding matches on text overlap, not meaning. A fact reading Matériel : NiPoGi AM06PRO, processeur Ryzen 5500U, 32 Go de RAM ranks 109th at 1.0496 for "Tu peux me
lister mon matériel ?" — with the word matériel in it — while #138, which contains the
whole question verbatim, sits at 0.7891. The same fact is rank 1 for "Quel processeur a
mon NiPoGi ?" and rank 2 for "Combien de RAM a le NiPoGi ?". Recall works when the question
is phrased in the words of the fact and fails otherwise. That is a property of the
embedding model on short French, not an intake defect, and it kills the rule this branch
was about to adopt for automatic extraction ("a fact must name its category" — measured,
falsified, dropped).

Both belong to the next branch, along with automatic fact extraction, which cannot be
evaluated until they are settled.