Publish the gap-audit branch; fix #319 (unreadable quote markers) and #333 (scripts without word spaces)

CHANGELOG.md

@@ -5,6 +5,46 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in
 
 ## Unreleased
 
+- The MCP `fetch_corpus` link is built with the platform's own encoder
+  (`pathToFileURL(path.resolve(file))`) instead of pasting `file:///` onto the path. A `#` in a
+  package name became a fragment and a `?` a query, so the link addressed a file that does not
+  exist; a space was left unencoded; and a relative path was anchored at the filesystem root
+  instead of the working directory (rank 13, `docs/review-2026-10-04-gap-audit.md`).
+- Near-duplicate detection reads a **word in any script**: the Latin-ASCII `[^a-z0-9]` class
+  stripped every non-Latin letter, so a page in Russian, Arabic, Hebrew, Greek or Devanagari
+  shingled to nothing, `sketch` returned null for it, and a republished copy of it was reported
+  as an independent witness - the one finding this module exists to make, inverted. Accented
+  Latin no longer collapses to its ASCII skeleton, and canonically equivalent text is one word
+  (rank 17, `docs/review-2026-10-04-gap-audit.md`; rank 14 of `docs/review-2026-10-03-gap-audit.md`).
+  ASCII input is unchanged, so `MIRROR_THRESHOLD` still rests on the corpus it was measured
+  against (ADR-0163).
+- Near-duplicate detection also reads Chinese, Japanese, Thai, Lao, Khmer and Burmese, the
+  scripts that put no space between their words. A page in them was one "word" per clause, or
+  one per page, so it shingled to too little, `sketch` returned null for it, and a republished
+  copy of it was an independent witness again. In those scripts a character is a token and a
+  shingle is fifteen of them: the length at which real Chinese, Japanese and Thai text scores
+  on the scale `MIRROR_THRESHOLD` was measured on, where the plain character 5-gram put two
+  different Thai texts at 0.18. `MIN_SHINGLES` asks the same amount of text of a page in every
+  script. Every other page shingles as before (#333, ADR-0176).
+- The collector stamps a fetch's ledger `at` with the moment it happened instead of midnight
+  of its day; a failed fetch already did, so the two sorted against each other wrongly, and the
+  day the capture was retrieved is unchanged in `retrieved` (rank 9,
+  `docs/review-2026-10-04-gap-audit.md`).
+- The WARC export's metadata record names the resource block's own hash (`body-sha256`)
+  beside the capture file's, a ledger `scrape` entry with no date is left out instead of
+  writing a record with an empty mandatory `WARC-Date`, a partial export exits 2
+  (`INCOMPLETE`), and the file it wrote is re-read (rank 7,
+  `docs/review-2026-10-03-gap-audit.md`; rank 15, `docs/review-2026-10-04-gap-audit.md`).
+- The `citations` check reads a `[quote: ...]` marker however deeply its brackets nest (a
+  Markdown link whose own text held brackets was not a marker at all, so the quote went
+  unchecked), holds every fragment within one passage instead of anchoring words scattered
+  over a page, and measures the short-quote floor over each fragment rather than the joined
+  text (rank 4, `docs/review-2026-10-03-gap-audit.md`).
+- A `[quote` the gate cannot read - no closing bracket, a space before the colon, no colon - is
+  `citations/quote-unreadable`, blocking: it was not a marker at all, so an invented sentence
+  in that shape passed where the same sentence written `[quote: ...]` failed, and `measure`
+  counted its row as unanchored. `measure` now counts it as a quote not found. A `[quote` in a
+  code span, a URL or Markdown link text is not an attempt (#319, ADR-0175).
 - `collect-remote` refuses empty repository, topic, client-ref, prior and prefer
   values by name before creating an output directory or reading a token (#287).
 - Invalid `--snapshot-max-age-ms` values are usage errors with exit 2. Library callers

docs/adr/0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md

@@ -0,0 +1,74 @@
+# ADR-0161 — The WARC export names the block's own hash, and leaves out an entry it cannot date
+
+- **Date:** 2026-10-17
+- **Status:** accepted
+- **Area:** `research-kit/lib/warc.mjs`, `research-kit/bin/export-warc.mjs` (ADR-0090). A fix
+  to the export's own integrity, not a new capability: it adds no command, flag, transport,
+  provider, configuration key or check.
+- **Freeze:** lifts the feature freeze (ADR-0117) for the one `metadata` field
+  `body-sha256`, by the precedent of ADR-0140, which lifted it for the `<capture>.source.html`
+  file and two ledger fields. Nothing else is new: the exit code and the re-read are fixes to
+  an existing command's result, not a new command.
+- **Refines:** ADR-0090 (the corpus as WARC 1.1). Its decision stands.
+
+## Context
+
+The 2026-10-03 gap audit (`docs/review-2026-10-03-gap-audit.md`, rank 7, VERIFIED) found
+these defects in the export ADR-0090 introduced; the 2026-10-04 gap audit
+(`docs/review-2026-10-04-gap-audit.md`, rank 15, which re-verified rank 7) restated them and
+added one:
+
+- The `metadata` record's `capture-sha256` is the hash of the **whole capture file** — the
+  value the ledger holds — but the `resource` record's block is the body **after** the
+  front matter, so a consumer cannot check a block against the field that names its file.
+- A ledger `scrape` entry with no `at` writes an empty `WARC-Date`. WARC 1.1 makes
+  `WARC-Date` mandatory; the clause is captured in the extraction audit's copy of the
+  specification (`docs/decisions/2026-10-04-gap-audit-extraction/research/raw/`), which the
+  audit cites as C:E-08.
+- A capture left out is named on stdout, but the command exits 0, so a caller cannot tell a
+  partial export from a whole one.
+- The file written is never read back: the bytes split in memory are the ones in hand, not
+  the bytes on disk, so a write that landed corrupted is reported as an archive.
+
+## Decision
+
+The `metadata` record gains `body-sha256`, the hash of the block the `resource` record
+carries, beside `capture-sha256`, the whole file the ledger hashed. A ledger `scrape` entry
+the export cannot date is left out and named like any other entry it cannot use, so no
+record carries an empty mandatory `WARC-Date`. The command exits 2 (`INCOMPLETE`) when
+anything was left out and 0 otherwise, and re-reads the file it wrote and splits it again
+before it reports success.
+
+The exit code is 2, not 1: `README.md` maps `PASS` to 0, `FAIL`/`REOPEN` to 1 and
+`INCOMPLETE` to 2, and a partial export is INCOMPLETE — the command did what it could and
+says so with its own code, distinct from both a whole export and a failed one. The first
+implementation of this ADR exited 1 (the 2026-10-03 audit's wording); the re-verification
+says 2, and the kit's own mapping agrees, so 2 stands.
+
+## Rejected alternatives
+
+- **Write `capture-sha256` over the block it accompanies.** Then the field no longer equals
+  the hash the ledger holds, so it stops tying the archive to the evidence — the one thing it
+  is for. Two hashes, each naming its own object, keep both checks.
+- **Keep writing the empty `WARC-Date` and warn.** `WARC-Date` is mandatory, so the record is
+  not conformant whatever is printed beside it; a capture that cannot be dated is a corpus
+  defect the caller should be able to act on, not a warning next to a written file.
+- **Abort the whole export on an undated entry.** The export already names and continues past
+  a capture it cannot use (missing, edited, outside the project). An undated entry is the same
+  kind of per-entry defect; a partial archive is more useful than none, with exit 2 saying it
+  is partial.
+- **Exit 1 for a partial export.** `README.md` reserves 1 for a `FAIL`/`REOPEN` verdict and 2
+  for `INCOMPLETE`; the export produced something, incompletely, which is INCOMPLETE. Exit 1
+  was the 2026-10-03 audit's first wording and was superseded by the re-verification.
+- **Trust the in-memory bytes and skip the re-read.** `writeBytes` writes a scratch file and
+  renames it, so the window is small, but the on-disk file is the product; splitting the bytes
+  in hand says nothing about the bytes that landed.
+
+## Consequences
+
+`body-sha256` is an extension field in the `metadata` record's `application/warc-fields`
+block. WARC 1.1 says a reader ignores a field name it does not recognize, so an archive tool
+that does not know it is unaffected. A consumer can now tie a block to the ledger entry
+(`body-sha256` beside `ledger-seq` and `capture-sha256`) and check it against the resource
+record's own `WARC-Block-Digest`. The freeze lift is for `body-sha256` alone: it is not
+permission to add further fields to the export without their own ADR.

docs/adr/0162-a-quote-is-one-passage-read-at-any-bracket-depth.md

@@ -0,0 +1,76 @@
+# ADR-0162 - A quote is one passage, read at any bracket depth, its floor measured per fragment
+
+- **Date:** 2026-10-17
+- **Status:** accepted
+- **Area:** `research-kit/lib/quotes.mjs`, `research-kit/lib/checks.mjs` (ADR-0087). A fix to
+  the quote check's own reading, not a new capability: it adds no command, flag, transport,
+  provider, configuration key or check.
+- **Refines:** ADR-0087 (a `[quote: ...]` anchor must occur in the capture) and ADR-0126 (the
+  short-quote floor is words AND characters). Both decisions stand.
+
+## Context
+
+The 2026-10-03 gap audit (`docs/review-2026-10-03-gap-audit.md`, rank 4, VERIFIED) found three
+defects in the `[quote: ...]` check; the 2026-10-04 gap audit
+(`docs/review-2026-10-04-gap-audit.md`, rank 4, which re-verified it) restated them:
+
+- The marker regex read **one** level of brackets. A passage two levels deep - a Markdown link
+  whose own text held brackets - was not read as a marker at all: `matchAll` found nothing, so
+  the quote was not checked, and nothing said so.
+- The fragments of one quote could be **arbitrarily far apart**. `anchorFound` searched each
+  fragment from the end of the previous match with no upper bound, so
+  `[quote: free ... requests ... credits]` anchored three words scattered over a pricing page
+  and passed as verified.
+- The short-quote floor was measured over the **joined** fragments. Three one-word fragments
+  read as three "words" and cleared a floor none of them reaches alone.
+
+The second audit named a fourth item in the same rank - "a quote that normalises to nothing is
+counted as found". That was closed on 2026-10-08 (an anchor that normalises entirely to empty
+cannot count as found; formatting-only fragments beside real text are still ignored). It is not
+re-done here, and its behaviour is kept.
+
+## Decision
+
+The marker reader counts brackets instead of matching a fixed depth: a marker opens at
+`[quote:` and closes at the `]` that balances it, so a passage may hold a code subscript, a
+Markdown link, or a link whose own text holds brackets, with no depth limit.
+
+A quote is one passage. `anchorFound` requires each fragment to start within `MAX_QUOTE_GAP`
+(300 characters of the compared, normalised text, about a paragraph) of the end of the one
+before it. Each fragment's placement is tried at every occurrence, not only the first: with a
+bound, a first occurrence out of context would otherwise reject the passage the quote meant.
+
+The short-quote floor is measured **per fragment**, by `quoteTooShort`: the quote is short only
+when EVERY fragment is under both floors. A single long or multi-word fragment is enough for the
+quote to be specific.
+
+## Rejected alternatives
+
+- **Keep the one-level regex and name an unreadable marker instead** (`quote-unparsed`). The
+  audit allows either. Refused: a Finding may legitimately discuss the marker syntax - this
+  repository's own decision projects write `[quote: ...]` as an example - and a warning on an
+  empty or unterminated marker would fire on that prose. Counting brackets reads the real
+  passage and does not fire there.
+- **Bound the gap by "the same paragraph".** `normalizeForMatch` collapses runs of whitespace,
+  so paragraph breaks are already gone from the compared text; checking them would need a second
+  normalised form and a second matcher for one rule. The character bound is simpler and testable.
+- **Bound the gap relative to the quote's own length.** A short quote would then get the
+  tightest bound, where the scattered-word case lives, and it would read as more precise while
+  being less predictable. A single documented constant is clearer.
+- **Fuzzy-match within the bound.** ADR-0087 refused fuzzy matching (a 0.92 similarity accepts
+  an invented word inside a real sentence). Nothing here reopens it.
+- **Warn rather than fail when fragments are far apart.** Each fragment still occurs, so the
+  outcome is not the same as a refuted quote. Kept as the existing `quote-not-found` fail: the
+  quote AS A PASSAGE does not occur, which is what the check claims.
+
+## Consequences
+
+`MAX_QUOTE_GAP` and `quoteTooShort` are exported; `anchorFound` takes `{ maxGap }` so a caller
+can tighten the bound. `measure.mjs` counts quotes through `anchorFound`, so its count follows
+the same rule - a scattered quote is no longer counted as found there either. `quoteChars`
+remains for measurement and orientation, but the check no longer uses it. `quoteWords`, whose
+last caller was the check, is removed: an export nothing references fails `repo-hygiene` since
+#297 (found when this branch was merged with it, 2026-10-19).
+
+A quote whose fragments genuinely lie in one passage is unaffected: the repository's own
+decision corpora hold four ellipsis quotes, and all three projects still pass their gates.

docs/adr/0163-a-shingle-is-a-word-in-any-script.md

@@ -0,0 +1,69 @@
+# ADR-0163 - A shingle is a word in any script, and text is compared as it was written
+
+- **Date:** 2026-10-17
+- **Status:** superseded in part by ADR-0176 (in a script without word spaces a character is a token, and a shingle is fifteen of them; the word rule for every other script stands)
+- **Area:** `research-kit/lib/similarity.mjs`
+- **Supersedes nothing.** The word basis, the sketch size and `MIRROR_THRESHOLD` are kept as
+  they were decided and measured; what changes is which characters count as a word.
+
+## Context
+
+`shingles` folded a page to words with `[^a-z0-9\s]` -> space. That class is Latin ASCII. So:
+
+- A page in **any other script** - Russian, Arabic, Hebrew, Greek, Devanagari - lost every
+  letter to a space. The page shingled to **nothing**, `sketch` returned `null` for it, and
+  `documentGroups` reads `null` as "cannot tell" rather than "different", so a republished
+  copy of such a page counted as an **independent witness**. That is the exact inversion this
+  module exists to prevent (rank 17, `docs/review-2026-10-04-gap-audit.md`; rank 14 of
+  `docs/review-2026-10-03-gap-audit.md`), and it applied to every corpus not written in the
+  Latin alphabet.
+- **Accented Latin** collapsed to its ASCII skeleton, so a page's words were hashed as
+  something the page did not say - the accent was not noise, it was dropped.
+- **Canonically equivalent text** was two words spelled two ways: `é` (U+00E9) and `e` + U+0301
+  are the same character and shingled differently.
+
+## Decision
+
+A word is a maximal run of `\p{L}` (letters), `\p{N}` (numbers) and `\p{M}` (combining marks -
+they carry the vowels of Devanagari and Arabic), in any script; everything else is a separator,
+as punctuation always was. Text is composed to **NFC** before that, so canonically equivalent
+input is one word rather than two.
+
+ASCII input is untouched, and that is load-bearing rather than lucky: after `toLowerCase`, the
+old class kept exactly `a-z`, `0-9` and whitespace, which is what these keep too. No ASCII
+page's shingles move, so `MIRROR_THRESHOLD = 0.25` and the pinned exact Jaccard (0.3879 for the
+known stale mirror against `nodejs.org`, 0.0275 for the highest genuinely different cross-host
+pair, both measured 2026-09-21) still describe the corpus they were measured on.
+`test/similarity.test.mjs` pins that equivalence against the old rule.
+
+## Rejected alternatives
+
+- **Shingle over characters (character n-grams) instead of words.** This is the robust,
+  language-agnostic basis and it would also fingerprint a script that does not separate words
+  with spaces at all - Chinese, Japanese, Thai - where a whole page is one token and still
+  shingles to nothing. Refused **for now**: the threshold and the pinned Jaccard were measured
+  on the word basis, so changing the basis changes what those numbers mean and it needs its own
+  measurement over real non-Latin pairs, not a ride-along in a bug fix.
+- **Segment scripts without word spaces** (split CJK ideographs, dictionary-segment Thai). A
+  per-script rule this module cannot measure from the corpus it has; and it would silently give
+  those pages a sensitivity nobody has checked, which is the failure mode this module's
+  measured constants exist to prevent.
+- **NFKC normalisation.** It folds **distinct** characters together (`①` into `1`), inventing an
+  equivalence the page did not have. NFC composes only canonically equivalent sequences, so it
+  cannot create a false match - it can only recover a true one.
+- **Transliterate to ASCII, or write the limitation down and keep it.** That is what the code
+  did. It loses the text: a page in another script was not merely unjudgeable, it was reported
+  as a second independent witness, which is the best grade in the registry for one document.
+
+## Consequences
+
+A page in a space-separated script is now fingerprinted, and two copies of one are one document.
+A page written in a script with **no word separators and no punctuation** still shingles to
+nothing and stays unjudgeable. That is recorded, not hidden, as a KNOWN-UNKNOWN: it needs a
+measured basis of its own, and the trigger to take it is a corpus in such a script that needs
+duplicate detection.
+
+A page whose text is accented Latin may move slightly, because its words are now themselves
+rather than their ASCII skeleton; `mirror` is a finding a reviewer reads, never a gate failure
+(ADR-0108). The root corpus and every project under `docs/decisions/` keep their grades,
+because their captures are ASCII.