Skip to content

Commit 1050707

Browse files
Two things the kit told me by being used, not read
The Tavily collection printed `0.92 best of 13 topic terms, 8 capture(s) >= 0.50` on a corpus where one capture in eight was about Tavily. The signal is not broken: the topic was "data retention training zero enterprise opt out", seven common words, so the search matched the words and the signal then measured the same words back. A high score means nothing unless the topic carried distinctive terms, and `topicMatch` now says so where someone reading the number will find it. It still decides nothing, which is why it did no harm. The same summary ran two lines together - `spent 8 (...)topic 0.92 ...` - for a missing newline. The third is a real bug, and I found it by obeying the tool. `partial-render` told me to record my review "in a `renderReview:` front-matter line". Doing exactly that on a live corpus turned a 1-warning PASS into a blocking `provenance/body-unmodified` failure and left the warning standing, because `verifyLedger` hashes the capture whole. The check's own comment, ten lines above the message, explains why the note must NOT go there - the message contradicted it. It now names the EVIDENCE row and the `[render-reviewed: ...]` syntax the parser actually reads, and says not to edit the capture. The existing test pinned that front-matter does not SATISFY the review. Nothing pinned that the message does not SEND you there. Now something does; confirmed red first. ARCHITECTURE said twelve checks and CHECK_NAMES holds thirteen - corroboration went in without it. Fixed while here, along with the README's test count. 874 passed, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 7d3b3f9 commit 1050707

7 files changed

Lines changed: 72 additions & 5 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ else, and it never touches the corpus:
145145
| `decompose.mjs` | Phase 0 coordination: recipe parsing, docs-host discovery, material gathering (scrape-budget accounting via the corpus's `cacheDecision` — a cache hit is not an attempt), draft MAP writing. Contains **no judgment** — it seeds the checklist and gathers material; it never writes a status. | — |
146146
| `dimensions.mjs` | The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (`coverageOfUniversals`). Recipes add on top of it; nothing replaces it. | — |
147147
| `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). | ADR-0003 (corpus-adjacent), ADR-0020 | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not.
148-
| `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape — twelve, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. | ADR-0004, ADR-0026 |
148+
| `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. | ADR-0004, ADR-0026 |
149149
| `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 |
150150
| `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 |
151151
| `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **One deliberate exception to "no CLI parsing"**: the commit gate's staged-path list arrives as *data on a pipe* (`stagedPathsFromStdin` / `splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"** — conflating them would hand the verdict an empty staged set, which it allows, i.e. a gate that silently stops gating. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). | ADR-0002, ADR-0007, ADR-0008, ADR-0020 | **An unreadable index BLOCKS (fixed 2026-09-21).** The commit gate judges staged bytes (ADR-0024); when `git checkout-index` fails it used to judge the working tree and note that it had. Noting is not enough — the two differ exactly when it matters, so an environmental git failure silently reinstated the defect ADR-0024 exists to prevent. It now blocks with `judged: 'nothing'` and names `--no-verify` as the recorded override. A project that is not a git repository still judges the tree: there is no index to fail to read. Index materialisation is also **chunked** (250 paths per call) rather than spreading every tracked path into one argv, which on Windows hit the ~32k command-line cap for a large corpus and produced exactly that git failure.

‎research-kit/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -395,7 +395,7 @@ node research-kit/bin/selftest.mjs # all of it
395395
node research-kit/bin/selftest.mjs gate hook # just these files
396396
```
397397

398-
872 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
398+
874 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
399399
the assertions settled (ADR-0021), and each test is raced against a watchdog
400400
(`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).
401401

‎research-kit/bin/research.mjs‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,8 @@ depth ${run.depth} (budget ${run.budget} scrapes)
117117
collected ${run.collected}
118118
cached ${run.cached}
119119
failed ${run.failed}
120-
spent ${run.spent} (budget consumed: collected + failed)`);
120+
spent ${run.spent} (budget consumed: collected + failed)
121+
`);
121122

122123
// The topic signal, printed at the one moment it helps: the pages are on disk and nobody
123124
// has read them yet. It decides nothing - see `topicMatch` for the two thresholds that were

‎research-kit/lib/checks.mjs‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -373,11 +373,19 @@ function captureCompleteness(corpus) {
373373
{ file: capture.file }));
374374
continue;
375375
}
376+
// The instruction names the EVIDENCE row on purpose. It used to say "a `renderReview:`
377+
// front-matter line", which is the one place the note must NOT go - the paragraph above
378+
// explains why, and the message contradicted it. A reviewer who followed it literally
379+
// got a BLOCKING `provenance/body-unmodified` failure and kept this warning, because
380+
// editing a capture breaks the hash recorded at fetch. Measured, not reasoned: doing
381+
// exactly what the old text said turned a 1-warning PASS into a 1-blocking FAIL.
376382
out.push(finding('warn', 'capture-completeness', 'partial-render',
377383
`${capture.file} contains ${hits} render-failure notice(s) from the page itself - `
378384
+ 'it arrived whole, so `completeness` cannot see this. Confirm the text cited from it is '
379-
+ `present, then record what you checked in a \`renderReview:\` front-matter line of at least ${MIN_RENDER_REVIEW} characters`
380-
+ (reviewed ? ` (the current one gives only "${reviewed}")` : ''),
385+
+ `present, then write \`[render-reviewed: what you checked]\` (at least ${MIN_RENDER_REVIEW} `
386+
+ 'characters) into the Finding cell of an EVIDENCE row that cites this capture. Do not edit '
387+
+ 'the capture: the ledger hashes it whole, front-matter included'
388+
+ (reviewed ? ` (the current note gives only "${reviewed}")` : ''),
381389
{ file: capture.file }));
382390
}
383391
if (!out.length) out.push(finding('pass', 'capture-completeness', 'completeness', 'no closed unknown rests on a partial capture alone'));

‎research-kit/lib/research-run.mjs‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -470,6 +470,20 @@ export function searchUsage(root, { now = new Date(), provider = '' } = {}) {
470470
* moment it helps - right after collecting, before anyone has read the pages - and decides
471471
* nothing. A reviewer seeing 0.25 on a fresh collection knows to check the URLs; on the EUDR
472472
* run that is precisely the step that caught it, performed by hand.
473+
*
474+
* A HIGH SCORE MEANS NOTHING WHEN THE TOPIC IS MADE OF COMMON WORDS, and that is not
475+
* hypothetical either. The first real use after shipping was a collection on Tavily's terms,
476+
* topic "...data retention training on inputs outputs zero retention enterprise tier opt
477+
* out". It reported **0.92, eight of eight above threshold** - on a corpus where SEVEN OF
478+
* EIGHT captures were generic "zero data retention" marketing from unrelated vendors and
479+
* only `tavily.com/terms` was on topic. Every one of those pages genuinely contains "data",
480+
* "retention", "training", "zero" and "enterprise", so the arithmetic was right and the
481+
* signal was useless.
482+
*
483+
* So the number is informative in ONE direction only: a low score is worth investigating,
484+
* and a high score is worth nothing unless the topic carried distinctive terms. `deforestation`
485+
* and `2023/1115` are distinctive; `data` and `service` are not, and this function cannot
486+
* tell the difference without a corpus-wide notion of rarity it does not have.
473487
*/
474488
const TOPIC_STOPWORDS = new Set(['the', 'a', 'an', 'of', 'for', 'and', 'or', 'to', 'in', 'on',
475489
'at', 'by', 'after', 'before', 'current', 'date', 'with', 'from', 'is', 'are', 'as', 'eu',

‎research-kit/test/checks.test.mjs‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -567,6 +567,26 @@ test('capture-completeness: the render review is read from the ROW, never the ca
567567
'a note in the capture front-matter must NOT satisfy the review - the ledger hashes it');
568568
});
569569

570+
test('partial-render tells the reviewer where the note goes, and it is not front-matter', () => {
571+
// The test above pins that front-matter does not satisfy the review. This pins that the
572+
// MESSAGE does not send the reviewer there - which it did, for as long as the check
573+
// existed. Found by following it on a real corpus: adding the `renderReview:` front-matter
574+
// line it named turned a 1-warning PASS into a blocking `provenance/body-unmodified`
575+
// failure, and left this warning standing. An instruction that breaks the gate when
576+
// obeyed is worse than no instruction, so the wording is load-bearing and pinned here.
577+
const dir = makePassingProject();
578+
const capture = readCorpus(dir).captures.entries[0];
579+
corrupt(dir, capture.file, (text) => `${text}\n\nThere was an error while loading.\n`);
580+
581+
const hit = runCheck('capture-completeness', snapshot(dir)).find((f) => f.rule === 'partial-render');
582+
assert.equal(hit.severity, 'warn');
583+
assert.match(hit.detail, /\[render-reviewed: /, 'must name the syntax the check actually parses');
584+
assert.match(hit.detail, /EVIDENCE row/, 'must name where the note goes');
585+
assert.match(hit.detail, /Do not edit the capture/, 'must say why it cannot go in the capture');
586+
assert.ok(!/front-matter line/.test(hit.detail),
587+
'must not direct the reviewer to edit front-matter - the ledger hashes the capture whole');
588+
});
589+
570590
test('a row named INSIDE a single-witness note is not thereby cited', () => {
571591
// The footgun this removes. `citedIds` returns every E-## mention in the cell, and the
572592
// reason a reviewer writes lives in that same cell - so explaining why no second witness

‎research-kit/test/collect.test.mjs‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -385,3 +385,27 @@ test('topicMatch REPORTS and never decides - the thresholds that were rejected',
385385
assert.equal(m.pass, undefined, 'a pass/fail field would make this a gate');
386386
assert.equal(m.severity, undefined);
387387
});
388+
389+
test('topicMatch scores high on a generic topic even when the corpus is off-topic', () => {
390+
// The limitation, pinned so the number is never read as a quality score. The first real
391+
// use after shipping reported 0.92 with eight of eight above threshold, on a corpus where
392+
// seven of eight captures were generic "zero data retention" marketing from unrelated
393+
// vendors and only one was the document being researched.
394+
//
395+
// The arithmetic was right. Every one of those pages genuinely contains "data",
396+
// "retention", "training" and "zero" - so a topic built from common words matches
397+
// anything, and this function cannot tell a distinctive term from a common one.
398+
const genericTopic = 'terms of service data retention training on inputs outputs zero retention enterprise';
399+
const unrelated = [
400+
'Zero data retention explained: how enterprise AI vendors handle training data and service inputs and outputs.',
401+
'What is zero data retention? A glossary entry about training, retention and enterprise service terms.',
402+
];
403+
const m = topicMatch(genericTopic, unrelated);
404+
assert.ok(m.best >= 0.5, `a generic topic matches unrelated pages: got ${m.best}`);
405+
assert.ok(m.strong > 0, 'and they clear the threshold, which is exactly the problem');
406+
407+
// The contrast: distinctive terms behave the way the signal is useful for.
408+
const distinctive = 'EU Deforestation Regulation 2023/1115 operators SMEs';
409+
assert.equal(topicMatch(distinctive, unrelated).strong, 0,
410+
'a distinctive topic does NOT match unrelated pages');
411+
});

0 commit comments

Comments
 (0)