You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 90f22c2
Browse filesBrowse the repository at this point in the historyBrowse files
corroboration counts documents, because counting hosts inverted on a mirror
ADR-0036 recorded that a republished page scores `independent` - the
registry's best grade - for one witness plus a staleness `freshness` cannot
see, then concluded no code change was possible because detecting it would
mean "grading meaning" and ADR-0013 refuses that.
The first half named the solution and the second half talked itself out of
it. Comparing document content is not grading meaning. It asks whether two
byte streams are substantially the same text, which is arithmetic. ADR-0013
refuses checks that grade how an agent REASONED; it does not refuse checks
that measure. Reaching for it was an argument for doing nothing.
`corroboration` now groups an unknown's rows into distinct DOCUMENTS before
counting hosts:
single-source one row warn
one-voice several rows, one host after grouping warn
mirror several hosts, all one document warn
independent several distinct documents, several hosts pass
A three-row unknown where two rows mirror still counts the third, and the
`independent` detail states how many rows were folded, so three rows buying
two witnesses is visible rather than absorbed.
The threshold is measured, not chosen. All 47 captures in this repository,
1081 pairs:
known stale mirror (nodejs.org x lira.epac.to, six majors apart) 0.3879
highest genuinely different cross-host pair 0.0275
median cross-host pair 0.0000
A fourteen-fold gap, so 0.25 is not delicate - ~9x above the strongest
false-positive candidate, below the hardest true positive. The stale mirror
IS the hard case; a current one scores ~0.99. The 128-hash sketch tracks
exact Jaccard to within 0.048 across all 1081 pairs, verified not assumed.
Two properties that matter more than the threshold:
- `null` never means "different". Below MIN_SHINGLES a document is
unjudgeable, and two unjudgeable rows stay two documents rather than
merging into an invented mirror - the conservative direction for a check
that can only reduce apparent corroboration.
- Hashing is seeded FNV-1a, not Math.random. A fingerprint that decides a
gate finding must grade the same corpus identically everywhere, or a
corpus passes on a laptop and fails in CI for reasons nobody reproduces.
It changed no verdict in this repository, and that is the finding: all five
corpora report identically, so no existing citation was secretly a mirror.
It was built for a hole found by collection, and it now stands in front of
the corroboration sweep that will add ~30 second sources, where a mirror
entering as `independent` is a live risk rather than a hypothetical one.
Regression proven at the check level against the real two captures rather
than synthetic near-duplicates: the same pair that graded `independent`
grades `mirror`. The unit tests in similarity.test.mjs make no claim to have
run against the old code - the module is new - and say so.
832 passed, 0 failed. All five corpora PASS, every chain verifies.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/ARCHITECTURE.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -146,6 +146,7 @@ else, and it never touches the corpus:
146
146
|`dimensions.mjs`| The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (`coverageOfUniversals`). Recipes add on top of it; nothing replaces it. | — |
147
147
| `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). | ADR-0003 (corpus-adjacent), ADR-0020 | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not.
148
148
|`checks.mjs`| The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape — twelve, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. | ADR-0004, ADR-0026 |
149
+
| `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 |
149
150
|`preflight.mjs`|**The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 |
150
151
| `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **One deliberate exception to "no CLI parsing"**: the commit gate's staged-path list arrives as *data on a pipe* (`stagedPathsFromStdin` / `splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"** — conflating them would hand the verdict an empty staged set, which it allows, i.e. a gate that silently stops gating. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). | ADR-0002, ADR-0007, ADR-0008, ADR-0020 | **An unreadable index BLOCKS (fixed 2026-09-21).** The commit gate judges staged bytes (ADR-0024); when `git checkout-index` fails it used to judge the working tree and note that it had. Noting is not enough — the two differ exactly when it matters, so an environmental git failure silently reinstated the defect ADR-0024 exists to prevent. It now blocks with `judged: 'nothing'` and names `--no-verify` as the recorded override. A project that is not a git repository still judges the tree: there is no index to fail to read. Index materialisation is also **chunked** (250 paths per call) rather than spreading every tracked path into one argv, which on Windows hit the ~32k command-line cap for a large corpus and produced exactly that git failure.
151
152
| `doctor.mjs` | Read-only diagnostics: machine + project + gate + chain health in one report. Consumes the verdict context (`verdictContext` / `readGateState`) for the chain-verified corpus and the gate-state triple; adds one machine read of its own (the git-resolved `hooksPathEffective`) for report. **Role-aware** (ADR-0010): the Firecrawl findings are informational on a `builder`, and the handoff findings (ADR-0011) are blockers there — the collection entrypoints' refusal is what makes that split honest. The handoff findings come from the owner, not from a second composition: doctor passes the corpus it already read to `verifyHandoff(root, { corpus })`, so the arrival question has one implementation and one set of pins. One deliberate write: the repository-local hooksPath override log (the third override is silent by nature — the bypassed hook cannot report it). `gateHealth` and `runDoctor` accept explicit Git/config/settings paths and a child environment so diagnostics can run against disposable machine fixtures without consulting host state. | ADR-0010, ADR-0011 |
@@ -433,7 +434,7 @@ Counted against the filesystem, not estimated:
433
434
434
435
|||
435
436
|---|---|
436
-
|`lib/` modules |**48** (44, plus the 4 extracted into `lib/release/`) — every one has a row above |
437
+
|`lib/` modules |**49** (45, plus the 4 extracted into `lib/release/`) — every one has a row above |
437
438
|`bin/` entrypoints |**29**, of which **4 are Python**: 3 conformance runners and the module they share |
`${unknown.id} cites ${rows.length} rows across ${hosts.size} hosts but they are the same document - a republished copy is one witness, and the copy may be the stale one`,
545
+
{row: unknown.id,line: unknown.line}));
546
+
continue;
547
+
}
548
+
549
+
// Hosts that survive as DISTINCT documents. A three-row unknown where two rows mirror
550
+
// each other still counts the third, so this reports what is actually independent.
`${unknown.id} cites ${rows.length} rows across ${hosts.size} hosts, but after grouping republished copies only ${[...distinctHosts][0]} remains - one voice`,
0 commit comments