Skip to content

Commit 96aa972

Browse files
Merge pull request #57 from StepenkoAnatoli/feat/report-closest-pair
The obvious repair for the mirror blind spot makes it worse, measured
2 parents c5d7107 + 9ccb65b commit 96aa972

6 files changed

Lines changed: 126 additions & 9 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -422,6 +422,7 @@ each line — is in the ADR named beside it and in
422422
| **No dispatch input reaches a shell**: `${{ inputs.x }}` inside a `run:` block is substitution *before* the shell parses, so inputs arrive through `env:` and then an argv array. Enforced across **every** workflow, not only the newest | [ADR-0033](adr/0033-the-collector-refuses-rather-than-degrades.md), ADR-0020 |
423423
| The collector's spend gate is a **protected environment whose existence is proven**, because GitHub silently creates an *unprotected* one when a workflow names a missing environment; a missing credential **refuses** rather than falling back to a keyless route | ADR-0033 |
424424
| A claim's **support is weighed, not just checked**: `corroboration` reports whether a closed unknown rests on one reading, several readings of one source, a **republished copy of one document across hosts** (`mirror`, via `lib/similarity.mjs`), or genuinely distinct documents — it never fails a corpus on its own, because single-sourcing is often the right answer. Rows are **deduped by id first**: `citedIds` returns every `E-##` mention, so prose naming one row twice used to produce it twice, and once grouping folded the duplicate it reported a "republished copy" about a row that was merely mentioned twice | [ADR-0036](adr/0036-the-gate-can-see-single-sourcing.md) |
425+
| `corroboration` **reports the closest pair it measured** on an `independent` verdict, and does not act on it. The threshold has two measured blind spots, and containment - the obvious repair for the second - ranks a genuinely different pair (0.506) ABOVE a real mirror (0.380) over 2404 pairs, so trying harder would report second sources as copies. The number informs a reviewer; the grade does not move | ADR-0036 |
425426
| A finding a reviewer has **considered and overruled** is recorded, not repeated forever: `[single-witness: …]` in an unknown's Evidence cell and `[render-reviewed: …]` in an evidence row's Finding cell turn a true-but-permanent warning into a pass **that prints the reason**. A reason under 30 characters is named (`single-witness-unreasoned`) rather than obeyed, and a note left on an unknown that has since been corroborated is reported (`single-witness-stale`) so the acknowledgement cannot rot. The render note lives on the ROW because `verifyLedger` hashes the whole capture file — a capture is evidence and stays byte-immutable, a judgement about it is corpus prose | [ADR-0038](adr/0038-a-judgement-the-gate-can-read.md) |
426427
| The collector lock is exclusive **on a local filesystem, and not promised on a network one**: Node states "the exclusive flag might not work with network file systems", a condition ADR-0020 never carried. Measured 2026-09-22 - 24 racers, 1 winner, 23 EEXIST on win32 - so the guarantee holds where it was tested and nothing detects where it is standing. Evidence: `docs/decisions/2026-09-22-wx-lock/` | ADR-0020 |
427428
| An entrypoint **refuses a flag it does not know** rather than ignoring it: `refuseUnknownFlags` in `lib/core.mjs`, called immediately after `parseFlags` in all eleven bins. `parseFlags` knows no flag names by design, so an entrypoint that skips the check gets silence — the flag is parsed, never read, and the command proceeds to its default. On 2026-09-22 that default was **a real collection**: `research.mjs --totally-made-up-flag` spent 26 credits against this repository's corpus, and `audit.mjs` wrote twelve files. The same survey found a dead `--max-scrapes` in two workflows, where the page bound people believed it set was really `plan.maxScrapes` — real in `collect.yml`, and **absent** in `live-collection.yml`, whose `max_pages` was validated and then ignored | ADR-0020 |

‎docs/adr/0036-the-gate-can-see-single-sourcing.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,53 @@ dilution pushes such pairs down. The threshold was chosen from a 14× gap; this
262262
the observed gap to about 1.6×. That is not a failure and it is not a reason to move the line
263263
on one data point — it is the note that the line is now known to have traffic near it.
264264

265+
## A second blind spot, and why the obvious repair was rejected, 2026-09-22
266+
267+
Using the kit on an unrelated question surfaced a mirror the check does not catch.
268+
`node.readthedocs.io` is an unofficial copy of the Node.js documentation and scored **0.054**
269+
against `nodejs.org` — nowhere near the 0.25 line. Both pages carry the same caveat in drifted
270+
wording ("might not" against "may or may not work with network file systems"), so they are one
271+
document at two vintages, exactly the case this check exists for.
272+
273+
**The cause is the metric, not the number.** Jaccard is `|A n B| / |A u B|`, which is symmetric:
274+
a 29 KB snapshot inside a 465 KB current page scores low however completely it is contained,
275+
because the union is dominated by everything the live page has since grown.
276+
277+
### Containment was measured and is worse
278+
279+
The textbook repair is containment, `|A n B| / min(|A|,|B|)` — "what fraction of the smaller
280+
document is in the larger". Measured over **2404 pairs** in this repository, cross-host only,
281+
since the mirror rule never fires within a host:
282+
283+
| pair | Jaccard | containment |
284+
|---|---|---|
285+
| `node.readthedocs.io` × `nodejs.org` — a real mirror | 0.054 | **0.380** |
286+
| `modelcontextprotocol.io` × its own `GOVERNANCE.md` — genuinely different | 0.136 | **0.506** |
287+
288+
**The genuine pair outranks the mirror.** Any threshold catching the mirror at 0.380 also flags a
289+
legitimate second source at 0.506 — and the remedy a reviewer reaches for on a `mirror` finding
290+
is to drop one of the rows. A check that tried harder here would delete real corroboration.
291+
292+
So the repair is refused, and refused on measurement rather than on taste.
293+
294+
### What ships instead: the number, not a verdict
295+
296+
`independent` now reports the closest pair among the rows it counted —
297+
`rests on 3 distinct documents across 3 hosts; closest pair 0.16`. The grade is unchanged.
298+
299+
Across this repository the reported values run 0.00 to 0.16, and the single 0.16 is the
300+
`modelcontextprotocol.io`/`GOVERNANCE.md` pair that had to be checked by hand. A reviewer
301+
seeing 0.16 is pointed at the one pair worth a look; a reviewer seeing 0.00 is not. That is
302+
the honest limit of a mechanical check: it can say how close two documents came, and it cannot
303+
say whether that closeness means anything.
304+
305+
### An accounting error found in passing
306+
307+
`similarity.test.mjs` used `node:test` while every other file uses the shared harness, so its
308+
**ten tests ran and were never counted** — the suite reported 845 while executing 855. The file
309+
is converted to the harness. The count guard exists because "a stale count is the first claim a
310+
reader checks"; a count that silently omits a whole file is the same failure from the inside.
311+
265312
## What this ADR does not claim
266313

267314
That two hosts make a claim true. Two vendors can repeat one another, and a specification

‎research-kit/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -395,7 +395,7 @@ node research-kit/bin/selftest.mjs # all of it
395395
node research-kit/bin/selftest.mjs gate hook # just these files
396396
```
397397

398-
845 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
398+
855 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
399399
the assertions settled (ADR-0021), and each test is raced against a watchdog
400400
(`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).
401401

‎research-kit/lib/checks.mjs‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99

1010
import { hostOf, PATHS, resolve, exists, ageInDays } from './core.mjs';
1111
import { captureOf, traceOf, citedIds } from './corpus.mjs';
12-
import { documentGroups } from './similarity.mjs';
12+
import { documentGroups, closestPair } from './similarity.mjs';
1313
import { coverageOfUniversals } from './dimensions.mjs';
1414
import { verifyLedger } from './provenance.mjs';
1515

@@ -635,7 +635,8 @@ function corroboration(corpus) {
635635
// Several hosts is where the host heuristic used to stop and say `independent`. A
636636
// mirror passes that test while carrying LESS than either genuine page, so the rows
637637
// are grouped into distinct DOCUMENTS before the hosts are counted again.
638-
const groups = documentGroups(rows.map((row) => corpus.captures.sketches?.get(captureOf(corpus, row)?.file) ?? null));
638+
const sketches = rows.map((row) => corpus.captures.sketches?.get(captureOf(corpus, row)?.file) ?? null);
639+
const groups = documentGroups(sketches);
639640
if (groups.length === 1) {
640641
shape = { rule: 'mirror', detail: `cites ${rows.length} rows across ${hosts.size} hosts but they are the same document - a republished copy is one witness, and the copy may be the stale one` };
641642
} else {
@@ -646,11 +647,18 @@ function corroboration(corpus) {
646647
shape = { rule: 'one-voice', detail: `cites ${rows.length} rows across ${hosts.size} hosts, but after grouping republished copies only ${[...distinctHosts][0]} remains - one voice` };
647648
} else {
648649
const mirrored = rows.length - groups.length;
650+
const closest = closestPair(sketches);
649651
shape = {
650652
corroborated: true,
651653
rule: 'independent',
654+
// The closest pair is REPORTED, never acted on. The threshold has two measured
655+
// blind spots and containment - the obvious repair - ranks a genuinely
656+
// different pair above a real mirror, so trying harder would report second
657+
// sources as copies. A number a reviewer can act on beats a verdict that
658+
// pretends to more certainty than the measurement supports.
652659
detail: `rests on ${groups.length} distinct documents across ${distinctHosts.size} hosts`
653-
+ (mirrored ? ` (${mirrored} of ${rows.length} rows are republished copies and were not counted twice)` : ''),
660+
+ (mirrored ? ` (${mirrored} of ${rows.length} rows are republished copies and were not counted twice)` : '')
661+
+ (closest === null ? '' : `; closest pair ${closest.toFixed(2)}`),
654662
};
655663
}
656664
}

‎research-kit/lib/similarity.mjs‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,36 @@ export function isMirror(a, b) {
101101
return score === null ? false : score >= MIRROR_THRESHOLD;
102102
}
103103

104+
/**
105+
* The closest any two of these sketches came, or `null` if fewer than two are judgeable.
106+
*
107+
* WHY THIS IS REPORTED RATHER THAN ACTED ON. The threshold has two measured blind spots and
108+
* neither is closable with a better number:
109+
*
110+
* - a stale mirror of a page that has since GROWN scores low, because the shared text is
111+
* diluted by everything the current page gained (`node.readthedocs.io` vs `nodejs.org`,
112+
* 0.054 at a 6.4x size ratio);
113+
* - and CONTAINMENT, the obvious repair for that, ranks a genuinely different pair ABOVE
114+
* that mirror (0.506 for `modelcontextprotocol.io` vs its own `GOVERNANCE.md`, against
115+
* 0.380 for the mirror). Measured over 2404 pairs on 2026-09-22.
116+
*
117+
* So a check that tried harder here would start reporting real second sources as copies,
118+
* and the remedy a reviewer would reach for - dropping one - is the opposite of what the
119+
* corpus needs. The number is surfaced instead: `independent (closest pair 0.38)` invites a
120+
* look, `closest pair 0.00` does not, and neither changes the verdict.
121+
*/
122+
export function closestPair(sketches) {
123+
let best = null;
124+
for (let i = 0; i < sketches.length; i += 1) {
125+
for (let j = i + 1; j < sketches.length; j += 1) {
126+
const score = similarity(sketches[i], sketches[j]);
127+
if (score === null) continue;
128+
if (best === null || score > best) best = score;
129+
}
130+
}
131+
return best;
132+
}
133+
104134
/**
105135
* Group rows into distinct DOCUMENTS. Each group is a set of indices whose sketches are
106136
* near-duplicates of one another.

‎research-kit/test/similarity.test.mjs‎

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,18 +10,17 @@
1010
// used to grade `independent` and now grade `mirror` - is proven at the check level, in
1111
// checks.test.mjs, which is where the old behaviour actually existed.
1212

13-
import test from 'node:test';
14-
import assert from 'node:assert/strict';
15-
import fs from 'node:fs';
16-
import path from 'node:path';
1713
import { fileURLToPath } from 'node:url';
14+
import { test, describe, assert, fs, path } from './harness.mjs';
1815

1916
import { parseCapture } from '../lib/corpus.mjs';
2017
import {
21-
sketch, similarity, shingles, documentGroups, isMirror,
18+
sketch, similarity, shingles, documentGroups, isMirror, closestPair,
2219
MIRROR_THRESHOLD, MIN_SHINGLES,
2320
} from '../lib/similarity.mjs';
2421

22+
describe('similarity');
23+
2524
const here = path.dirname(fileURLToPath(import.meta.url));
2625
const repo = path.resolve(here, '..', '..');
2726
const seaRaw = path.join(repo, 'docs', 'decisions', '2026-09-21-sea-assets', 'research', 'raw');
@@ -106,3 +105,35 @@ test('unjudgeable rows each stand alone rather than merging', () => {
106105
// Two nulls must not become one group: "I cannot tell" is not "they are the same".
107106
assert.equal(documentGroups([null, null]).length, 2);
108107
});
108+
109+
// ---------------------------------------------------------------- closestPair
110+
111+
test('closestPair reports the tightest pair, and null when nothing is judgeable', () => {
112+
const canonical = sketch(body(seaRaw, '-nodejs-'));
113+
const mirrored = sketch(body(seaRaw, 'epac'));
114+
const other = sketch(body(seaRaw, 'bundling'));
115+
116+
// The mirror is the tightest pair present, so it is what gets reported.
117+
const three = closestPair([canonical, mirrored, other]);
118+
assert.ok(three > 0.35 && three <= 1, `expected the mirror to dominate, got ${three}`);
119+
assert.equal(closestPair([canonical, mirrored]), similarity(canonical, mirrored));
120+
121+
assert.equal(closestPair([]), null, 'nothing to compare is null, not 0');
122+
assert.equal(closestPair([canonical]), null, 'one document has no pair');
123+
assert.equal(closestPair([null, null]), null, 'unjudgeable pairs report null, never 0');
124+
});
125+
126+
test('closestPair is REPORTED, not acted on - the verdict does not move with it', () => {
127+
// The measured reason, recorded so nobody "fixes" this into a blocking rule later:
128+
// containment - the obvious repair for a size-mismatched mirror - ranks a genuinely
129+
// different pair (0.506) ABOVE a real mirror (0.380). A check that tried harder would
130+
// report real second sources as copies, and the remedy a reviewer reaches for is to drop
131+
// one. So the number informs and the grade is unchanged.
132+
const canonical = sketch(body(seaRaw, '-nodejs-'));
133+
const other = sketch(body(seaRaw, 'bundling'));
134+
const score = closestPair([canonical, other]);
135+
assert.ok(score !== null);
136+
assert.ok(score < MIRROR_THRESHOLD, 'this pair must remain below the mirror line');
137+
// Two distinct documents stay two distinct documents whatever the number says.
138+
assert.equal(documentGroups([canonical, other]).length, 2);
139+
});

0 commit comments

Comments
 (0)