Merge pull request #41 from StepenkoAnatoli/feat/mirror-detection

corroboration counts documents, because counting hosts inverted on a mirror

Author: StepenkoAnatoli

docs/ARCHITECTURE.md

@@ -146,6 +146,7 @@ else, and it never touches the corpus:
 | `dimensions.mjs` | The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (`coverageOfUniversals`). Recipes add on top of it; nothing replaces it. | — |
 | `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). | ADR-0003 (corpus-adjacent), ADR-0020 | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not.
 | `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape — twelve, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. | ADR-0004, ADR-0026 |
+| `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 |
 | `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 |
 | `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **One deliberate exception to "no CLI parsing"**: the commit gate's staged-path list arrives as *data on a pipe* (`stagedPathsFromStdin` / `splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"** — conflating them would hand the verdict an empty staged set, which it allows, i.e. a gate that silently stops gating. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). | ADR-0002, ADR-0007, ADR-0008, ADR-0020 | **An unreadable index BLOCKS (fixed 2026-09-21).** The commit gate judges staged bytes (ADR-0024); when `git checkout-index` fails it used to judge the working tree and note that it had. Noting is not enough — the two differ exactly when it matters, so an environmental git failure silently reinstated the defect ADR-0024 exists to prevent. It now blocks with `judged: 'nothing'` and names `--no-verify` as the recorded override. A project that is not a git repository still judges the tree: there is no index to fail to read. Index materialisation is also **chunked** (250 paths per call) rather than spreading every tracked path into one argv, which on Windows hit the ~32k command-line cap for a large corpus and produced exactly that git failure.
 | `doctor.mjs` | Read-only diagnostics: machine + project + gate + chain health in one report. Consumes the verdict context (`verdictContext` / `readGateState`) for the chain-verified corpus and the gate-state triple; adds one machine read of its own (the git-resolved `hooksPathEffective`) for report. **Role-aware** (ADR-0010): the Firecrawl findings are informational on a `builder`, and the handoff findings (ADR-0011) are blockers there — the collection entrypoints' refusal is what makes that split honest. The handoff findings come from the owner, not from a second composition: doctor passes the corpus it already read to `verifyHandoff(root, { corpus })`, so the arrival question has one implementation and one set of pins. One deliberate write: the repository-local hooksPath override log (the third override is silent by nature — the bypassed hook cannot report it). `gateHealth` and `runDoctor` accept explicit Git/config/settings paths and a child environment so diagnostics can run against disposable machine fixtures without consulting host state. | ADR-0010, ADR-0011 |
@@ -433,7 +434,7 @@ Counted against the filesystem, not estimated:
 
 | | |
 |---|---|
-| `lib/` modules | **48** (44, plus the 4 extracted into `lib/release/`) — every one has a row above |
+| `lib/` modules | **49** (45, plus the 4 extracted into `lib/release/`) — every one has a row above |
 | `bin/` entrypoints | **29**, of which **4 are Python**: 3 conformance runners and the module they share |
 | `schemas/` | **12** |
 | `conformance/` vector packets | **3** |

docs/adr/0036-the-gate-can-see-single-sourcing.md

@@ -160,6 +160,9 @@ be one witness; ADR-0036 already said that. It is that two hosts can be one witn
 staleness the freshness check cannot see*, because `freshness` grades when the page was
 **fetched** and this one was fetched today.
 
+> **Superseded the same day — the "no code changes" conclusion below was wrong, and is
+> now implemented as `lib/similarity.mjs`. See the amendment that follows.**
+
 **No code changes.** Detecting mirrors means comparing document content or parsing version
 strings out of vendor pages, and a check that guessed at either would be the reasoning-grader
 ADR-0013 refused. What changes is what a reviewer is told to do: **`independent` is a prompt
@@ -170,6 +173,72 @@ Recorded live rather than reasoned about: the corpus that found it
 `single-source` warnings instead — the correct trade, and the one the check's grades
 currently push against.
 
+## Amendment, 2026-09-21 — the mirror IS detectable, and "grading meaning" was a dodge
+
+The amendment above concluded that a mirror could only be caught by "comparing document
+content … which would be the reasoning-grader ADR-0013 refused". **The first half named the
+solution and the second half talked itself out of it.**
+
+Comparing document content is not grading meaning. It asks whether two byte streams are
+substantially the same text, which is arithmetic. It judges nothing, cannot tell a correct
+page from a wrong one, and does not try — `lib/similarity.mjs` says so at the top. ADR-0013
+refuses checks that grade *how an agent reasoned*; it does not refuse checks that measure.
+Reaching for it here was an argument for doing nothing.
+
+### What was built
+
+`corroboration` no longer counts hosts. It groups an unknown's rows into distinct
+**documents** first, then counts hosts among those:
+
+| rule | when | severity |
+|---|---|---|
+| `single-source` | one row | `warn` |
+| `one-voice` | several rows, one host after grouping | `warn` |
+| `mirror` | several hosts, but all one document | `warn` |
+| `independent` | several distinct documents, several hosts | `pass` |
+
+A three-row unknown where two rows mirror each other still counts the third, and the
+`independent` detail now states how many rows were folded — so three rows buying two
+witnesses is visible rather than absorbed.
+
+### The threshold is measured, not chosen
+
+Every capture pair in this repository was compared — 47 captures, 1081 pairs:
+
+| | Jaccard |
+|---|---|
+| the known stale mirror (`nodejs.org` × `lira.epac.to`, six majors apart) | **0.3879** |
+| highest-scoring genuinely *different* cross-host pair | **0.0275** |
+| median cross-host pair | 0.0000 |
+
+A fourteen-fold gap. `MIRROR_THRESHOLD = 0.25` sits ~9× above the strongest false-positive
+candidate and below the hardest true positive — and the *stale* mirror is the hard case,
+because a current one scores ~0.99. The 128-hash sketch tracks exact Jaccard to within
+0.048 across all 1081 pairs, verified rather than assumed.
+
+### Two properties that matter more than the threshold
+
+**`null` never means "different".** A document below `MIN_SHINGLES` is unjudgeable, and two
+unjudgeable rows stay two documents rather than merging into an invented mirror. The
+conservative direction for a check that can only ever *reduce* apparent corroboration.
+
+**Hashing is seeded FNV-1a, not `Math.random`.** A fingerprint that decides a gate finding
+has to grade the same corpus identically on every machine, or a corpus passes on one
+developer's laptop and fails in CI for reasons nobody can reproduce.
+
+### What it changed in this repository
+
+Nothing, and that is the finding. All five corpora report identical verdicts — no existing
+citation was secretly a mirror. The check was built for a hole found by collection, not for
+a defect already in the corpus, and it now stands in front of the corroboration sweep that
+will add ~30 second sources, where a mirror slipping in as `independent` is a live risk
+rather than a hypothetical one.
+
+One existing assertion moved: `corroboration: two hosts is independent support` matched
+`2 rows across 2 hosts` and now matches `2 distinct documents across 2 hosts`. Its two rows
+name a capture the fixture does not have, so neither has a sketch — which makes it the
+regression test for the `null` rule as well.
+
 ## What this ADR does not claim
 
 That two hosts make a claim true. Two vendors can repeat one another, and a specification

research-kit/README.md

@@ -395,7 +395,7 @@ node research-kit/bin/selftest.mjs            # all of it
 node research-kit/bin/selftest.mjs gate hook  # just these files
 ```
 
-829 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
+832 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
 the assertions settled (ADR-0021), and each test is raced against a watchdog
 (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).
 

research-kit/lib/checks.mjs

@@ -9,6 +9,7 @@
 
 import { hostOf, PATHS, resolve, exists, ageInDays } from './core.mjs';
 import { captureOf, traceOf, citedIds } from './corpus.mjs';
+import { documentGroups } from './similarity.mjs';
 import { coverageOfUniversals } from './dimensions.mjs';
 import { verifyLedger } from './provenance.mjs';
 
@@ -534,8 +535,31 @@ function corroboration(corpus) {
       continue;
     }
 
+    // Several hosts is where the host heuristic used to stop and say `independent`. A
+    // mirror passes that test while carrying LESS than either genuine page, so the rows
+    // are grouped into distinct DOCUMENTS before the hosts are counted again.
+    const groups = documentGroups(rows.map((row) => corpus.captures.sketches?.get(captureOf(corpus, row)?.file) ?? null));
+    if (groups.length === 1) {
+      out.push(finding('warn', 'corroboration', 'mirror',
+        `${unknown.id} cites ${rows.length} rows across ${hosts.size} hosts but they are the same document - a republished copy is one witness, and the copy may be the stale one`,
+        { row: unknown.id, line: unknown.line }));
+      continue;
+    }
+
+    // Hosts that survive as DISTINCT documents. A three-row unknown where two rows mirror
+    // each other still counts the third, so this reports what is actually independent.
+    const distinctHosts = new Set(groups.map((g) => hostOf(rows[g[0]].url)).filter(Boolean));
+    if (distinctHosts.size === 1) {
+      out.push(finding('warn', 'corroboration', 'one-voice',
+        `${unknown.id} cites ${rows.length} rows across ${hosts.size} hosts, but after grouping republished copies only ${[...distinctHosts][0]} remains - one voice`,
+        { row: unknown.id, line: unknown.line }));
+      continue;
+    }
+
+    const mirrored = rows.length - groups.length;
     out.push(finding('pass', 'corroboration', 'independent',
-      `${unknown.id} rests on ${rows.length} rows across ${hosts.size} hosts`,
+      `${unknown.id} rests on ${groups.length} distinct documents across ${distinctHosts.size} hosts`
+        + (mirrored ? ` (${mirrored} of ${rows.length} rows are republished copies and were not counted twice)` : ''),
       { row: unknown.id, line: unknown.line }));
   }
 

research-kit/lib/corpus.mjs

@@ -13,6 +13,7 @@ import {
   PATHS, HEADERS, resolve, relative, exists, isDirectory, readText, readJson,
   writeText, appendLine, listFiles, sha256File, ageInDays, hostOf,
 } from './core.mjs';
+import { sketch } from './similarity.mjs';
 
 // ---------------------------------------------------------------- table machinery
 
@@ -161,6 +162,7 @@ export function readCaptures(root) {
   const dir = resolve(root, PATHS.raw);
   const entries = [];
   const problems = [];
+  const sketches = new Map();
   for (const name of listFiles(dir).sort()) {
     if (name.startsWith('.')) continue;
     const abs = path.join(dir, name);
@@ -172,6 +174,7 @@ export function readCaptures(root) {
     if (!front.url) {
       problems.push({ kind: 'capture-no-url', file: rel, detail: 'capture has no url in its front-matter' });
     }
+    sketches.set(rel, sketch(body));
     entries.push(captureEntry({
       file: rel,
       url: front.url,
@@ -193,7 +196,10 @@ export function readCaptures(root) {
     const held = byUrl.get(entry.url);
     if (!held || String(entry.retrieved) >= String(held.retrieved)) byUrl.set(entry.url, entry);
   }
-  return { entries, byUrl, byFile, problems };
+  // Sketches ride alongside the index rather than inside `captureEntry`, because an entry
+  // is serialised into artifact manifests and a 128-value fingerprint per capture would
+  // bloat every package to answer a question only one check asks (ADR-0036 amendment).
+  return { entries, byUrl, byFile, sketches, problems };
 }
 
 /** The one writer of the in-memory index, for a URL collected mid-run. */