From 8ee1be40648a659fe6d28f7859393cb5a759a59a Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli Date: Sat, 17 Oct 2026 03:27:36 +0300 Subject: [PATCH 01/12] warc: name the block's own hash, exit INCOMPLETE on a partial export, and re-read what was written What changed: `research-kit/lib/warc.mjs` names the resource block's own hash (`body-sha256`) beside the whole capture file's `capture-sha256`, and leaves out a ledger `scrape` entry it cannot date instead of writing a record with an empty mandatory `WARC-Date`; `research-kit/bin/export-warc.mjs` exits 2 (`INCOMPLETE`) when anything was left out and re-reads the file it wrote, splitting it again before it reports success. Why: rank 7 of the 2026-10-03 gap audit (`docs/review-2026-10-03-gap-audit.md`) and rank 15 of the 2026-10-04 gap audit (`docs/review-2026-10-04-gap-audit.md`), which re-verified rank 7 and added a part the first omitted. A consumer could not verify a block against the field that names its file; a scrape entry with no `at` produced a non-conformant record; a partial export exited 0; and the file written was never read back. What it touched: `research-kit/lib/warc.mjs`, `research-kit/bin/export-warc.mjs`, `research-kit/test/warc.test.mjs`, `docs/ARCHITECTURE.md` (two rows), the new `docs/adr/0161-...md` and its index row in `docs/adr/README.md`, `CHANGELOG.md`. ADR-0161 records the decision and lifts the ADR-0117 freeze for the one field `body-sha256`, by the ADR-0140 precedent (a source file and two ledger fields). What you verified: - verified: the warc group is green on this tree - 11 passed, 0 failed (`selftest.mjs warc`). - verified: red-first - with the CLI stashed at its pre-fix exit 1, the re-stated test is red (10 passed, 1 failed); restored, green. - verified: WARC 1.1 makes `WARC-Date` mandatory and a reader ignores an unrecognized field name, against the captured specification (`docs/decisions/2026-10-04-gap-audit-extraction/research/raw/2026-10-04-the-warc-format-github-626d7a34.md`, lines 356, 385, 407). - verified: the test count is unchanged at 1864 (a test was re-stated, not added). - expected: the commit gate's full offline suite is green; it runs for this commit because the commit stages `research-kit/`, so a red suite would block it rather than land it. - untested: the re-read's failure branch (a write that landed corrupted on disk). `writeBytes` writes a scratch file and renames it, so the branch is hard to trigger on purpose; the happy path runs on every export test. What you got wrong and fixed: - mistake: the export exited 1 on a partial result, and ADR-0161 did not lift the freeze for the field it added. - where: `research-kit/bin/export-warc.mjs` (the final `process.exit`), `docs/adr/0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md`, `docs/adr/README.md`, `docs/ARCHITECTURE.md`. - impact: a caller reading the exit code could not tell a partial export (`INCOMPLETE`) from a failed one, and the freeze lift for `body-sha256` was missing from the record. Nothing had been pushed, so nothing delivered was affected. - cause: ADR-0161 was written from the 2026-10-03 audit alone; the 2026-10-04 audit re-verifies rank 7 and says 2, and `research-kit/README.md` maps `INCOMPLETE` to 2, which would have settled it. - fix: exit 2 (`INCOMPLETE`), recorded in ADR-0161, its index row, `ARCHITECTURE.md` and the changelog; the fourth part (re-read the written file) was added in the same change. - verified: `selftest.mjs warc` green after the fix; red-first shown by stashing the CLI. --- CHANGELOG.md | 5 ++ docs/ARCHITECTURE.md | 4 +- ...fiable-beside-the-capture-that-names-it.md | 74 +++++++++++++++++++ docs/adr/README.md | 1 + research-kit/README.md | 2 +- research-kit/bin/export-warc.mjs | 22 +++++- research-kit/lib/warc.mjs | 18 ++++- research-kit/test/warc.test.mjs | 45 +++++++++++ 8 files changed, 163 insertions(+), 8 deletions(-) create mode 100644 docs/adr/0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 8f64a765..30239cba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,11 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in ## Unreleased +- The WARC export's metadata record names the resource block's own hash (`body-sha256`) + beside the capture file's, a ledger `scrape` entry with no date is left out instead of + writing a record with an empty mandatory `WARC-Date`, a partial export exits 2 + (`INCOMPLETE`), and the file it wrote is re-read (rank 7, + `docs/review-2026-10-03-gap-audit.md`; rank 15, `docs/review-2026-10-04-gap-audit.md`). - `collect-remote` refuses empty repository, topic, client-ref, prior and prefer values by name before creating an output directory or reading a token (#287). - Invalid `--snapshot-max-age-ms` values are usage errors with exit 2. Library callers diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 23221272..d4dddb59 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -208,7 +208,7 @@ is an open decision (2026-09-30): | `firecrawl.mjs` | **All** vendor knowledge: the CLI transport (`exec` seam, which carries an **argv array** and spawns with `shell: false` — no URL or query is ever concatenated into a shell string), the Windows `.cmd`-shim resolution (`resolveProgramPath`/`resolveInvocation`: full path off PATH+PATHEXT, a real `.exe` spawned directly, a `.cmd`/`.bat` run through cmd.exe as argv and only after every argument passes `cmdSafeArg` — a **denylist** of cmd.exe metacharacters (`CMD_UNSAFE_CHARS`), refused and never re-quoted — a **denylist**, so an ordinary query with a space passes while an interpreter character does not. `unsafeCmdArgs(argv)` is the plural reporting form and the **single owner** of "which arguments would be refused": `resolveInvocation` calls it rather than filtering inline, so the refusal and the message naming the offenders cannot drift apart. Measured against the allowlist the ADR-0029 source tree still uses, the two agree on every character that can break out of cmd quoting — `"` `%` `&` `|` `<` `>` `^` `(` `)` `` ` `` tab newline — and differ on `'` `{` `}` `\` and **any non-ASCII character**, which an allowlist of ASCII punctuation refuses by construction. That is why `é` is a permitted argument here and a hostile one there, and why that tree's hostile-argv corpus could not be ported), payload normalisation (`normalizeSearch`/`normalizeScrape`/`normalizeMap` — `normalizeSearch` reads the CLI's real `{success, data: {web: […]}, creditsUsed}` object, pinned at `test/fixtures/firecrawl-search-1.23.3.json`), status parsing (`--status`, the flag; `firecrawl status` is not a command), `map`/`search`/`scrape` adapters. `command(args)` survives as a **display-only** rendering (dry-run output, the ledger's `cmd` annotation) that nothing executes. Stamps `transport: 'firecrawl-cli'` — narrowed to `firecrawl-cli-anonymous` by `transport.mjs` when the CLI has no credential — and **grades** completeness at the 1500-character bar rather than asserting `full`. **Tested CLI: 1.25.3** (from 1.25.2, 2026-10-03: the release is one fix - `--objective` optional for alexandria feedback, firecrawl/cli#301 - with nothing on scrape, search or `--status`, and its output, captured for three credits, reads unchanged, `test/fixtures/*-1.25.3.*`; from 1.24.6 on 2026-10-02, `docs/decisions/2026-10-02-firecrawl-cli-1-25`: its `--status`, search and scrape output captured for three credits and read unchanged by the parsers, `test/fixtures/*-1.25.2.*` - the scrape receipt goes to stderr, `data.tools` is populated and reaches no result; before that from 1.23.3, 2026-09-26): 1.23.3's bundled axios 1.15.2 fails behind an HTTPS proxy that requires CONNECT tunnels (405 on every call); 1.24.6 bundles axios 1.18.0, and its status, search and scrape output were read by these parsers before the move - search's new fields are additive `search` reports `searchesUsed: 1` and a `creditsEstimate` of 2 per 10 results, rounded up, at least 2 (2026-09-27, ADR-0065, E-03/E-14): Firecrawl searches had cost credits the kit never recorded. The CLI runs with `maxBuffer: CHILD_OUTPUT_LIMIT` and names an overflow (2026-09-28, ADR-0082): under the 1 MiB default a large answer was lost to ENOBUFS. **`creditsExhausted(text)`** (2026-09-28, ADR-0086) recognises the API's 402 text, "Insufficient credits ...", which is all the CLI passes on - the status never reaches the kit. `normalizeSearch` and `normalizeMap` drop a row that is not an object (2026-09-29, Arena break test): one null row in the CLI's stdout threw on `row.url` and took every valid result with it - the shape `serpapi.normalizeSearch` was hardened against on 2026-09-28. A field of the wrong type - url, title, description, body, status - reads as empty in normalizeScrape and normalizeSearch, as in serpapi and searxng; one hostile shape is fed to all three parsers (2026-09-30). `rateLimitWaitMs` caps a vendor-stated wait at `MAX_RATE_LIMIT_WAIT_MS` (60 s): the number is the vendor's text, and it is slept with the corpus lock held (2026-10-01). The payload is the **first complete JSON value** in stdout (`parsePayload`: each bracket tried as a start, the value ending where its brackets balance outside strings, at most 64 starts), and `scrape` refuses an exit of 0 that printed nothing, no readable JSON, or a page with no text - each a named failure, never a capture (2026-10-02, break-test pass 3: a banner with a bracket, a receipt line after the JSON, a cut payload and an empty stdout had each become a "collected" capture of 0 characters). **A scrape is a live fetch** (2026-10-03, ADR-0139, gap audit rank 2): the argv carries `--max-age 0`, because the vendor serves a cached copy up to two days old by default and a capture stamped with today's date could be yesterday's page - reproduced live, and the kit's own 1.25.3 fixture is a `cacheState: hit`; `normalizeScrape` keeps `metadata.cacheState` and `cachedAt`, which `writeRaw` writes into the front matter and `appendFetch` into the ledger entry when the vendor sends them, so a cached answer, if one still arrives, says so. | ADR-0005, ADR-0020, ADR-0086 | The argv also asks `--format markdown,rawHtml` (ADR-0140, 2026-10-03): the page's HTML returns beside the Markdown at no extra credit (a live scrape billed one; the pricing page names only JSON, Question and Highlight as formats that add credits), and `normalizeScrape` hands it back as `source`. | `bin/mcp-server.mjs` | The stdio entrypoint an MCP client launches as a subprocess. No flags: a stdio server's configuration is its environment. Calls `honourEnvProxy()` before touching either stream, because where only the startup flag exists the server starts again on the same pipes (ADR-0047). **stdout carries messages and nothing else** - a stray `console.log` corrupts the stream and surfaces as a malformed response to whatever request was in flight - so every diagnostic goes to stderr, including the one line reporting whether a credential was found. It calls `tolerateClosedStdout()` before its first write (2026-09-30): a client that disconnects mid-response made the kernel's EPIPE an unhandled `error` event, a raw stack and exit 1 that read as a server defect. A real pipe test (`| head -c 100`) and a source-order pin cover it. **Startup refusal does not echo argv values** (2026-10-09): it reports the count and environment-variable remedy on stderr, before runtime/proxy/token work. The direct check includes positional arguments; `--help` is handled first, and a bare invocation still starts normally. | ADR-0034 | | `bin/measure.mjs` | The judge-free citation report for the current project (ADR-0089); `--json` for machines. A report, never a gate; exit 2 outside a research project. | ADR-0089 | -| `bin/export-warc.mjs` | The corpus as one WARC 1.1 `.warc.gz` for archive tools (ADR-0090); `--out` (default `research-corpus.warc.gz`), written whole with `writeBytes` after `readWarc` re-reads it; a refused write is named by `writeFailure`, exit 2 (2026-09-29). A copy, never a gate; exit 2 outside a research project. | ADR-0090 | +| `bin/export-warc.mjs` | The corpus as one WARC 1.1 `.warc.gz` for archive tools (ADR-0090); `--out` (default `research-corpus.warc.gz`), written whole with `writeBytes` after `readWarc` splits the bytes in hand, then re-read from disk and split again before it is reported; a refused write is named by `writeFailure`, exit 2 (2026-09-29). A copy, never a gate; exit 2 outside a research project, and exit 2 (`INCOMPLETE`) when a capture was left out (named on stdout), so a partial export is not read as a whole one (rank 7, 2026-10-03 gap audit; rank 15, 2026-10-04 gap audit, ADR-0161). | ADR-0090 | | `bin/disclosure.mjs` | Probes a run and reports what a stranger can read. Calls `honourEnvProxy()` before its first request (ADR-0047). Exit codes ARE the finding, so it can gate something later: `0` shape only, `1` content readable, `2` subject leaked, `3` could not measure - which includes any probe that never reached GitHub, unless an exposure was measured anyway (`exitCode` in the lib). An unknown flag matching token/auth/key/secret is refused with the reason - authenticating this probe would answer a different question. | ADR-0035 | | `bin/collect-remote.mjs` | The one command an agent or a desktop app needs: dispatch the collector on GitHub, learn which run it started, wait, download, unwrap, and validate with the same code a human would run. Calls `honourEnvProxy()` before its first request, so a proxy-only network can dispatch at all (ADR-0047). Prints the run id **before** it waits, so a caller that dies mid-wait can still find its run - and a caller that never sees that line knows the dispatch itself did not land. **No `--token` flag**, and an unknown flag matching token/auth/key/secret/pat is refused with the reason rather than the spelling. Exit codes separate the four things a caller does differently about: `0` valid (still not authorized to build), `1` invalid package, `2` run failed or incomplete, `3` could not start, `4` dispatched and still running with the run id on stdout. An `--out` it cannot write is refused (exit 3, `OUT_DIR`) before the token is read or a run dispatched (2026-09-27); `--no-wait` downloads nothing and skips the check. Each failed poll the watch survives is printed with its number (2026-10-01). `--run-id ` picks up a run already dispatched - after `--no-wait`, a timeout (exit 4) or a lost connection - waits, downloads and judges it, and refuses `--topic` beside it (2026-10-01). The remedies and exit 4 had pointed at that flag while it did not exist, so the only way back to a run was to pay for another. A failure that names no way out now says how to pick the run up. A bare `--out`, `--workflow` or `--ref` is refused as `-- needs a value` (exit 3) beside the five checked values (2026-10-11, #270): the parser reads a bare flag as `true`, so `--out` had become the folder `true`, created before the token was asked for, and `--workflow` or `--ref` would have dispatched against the literal name "true". The shared query/URL validator refuses CR/LF inside an individual item before output-folder checks, token lookup or dispatch (#376, 2026-10-12). Rejected URL diagnostics pass through `redact` before stderr (#378, 2026-10-12), including credentials supplied by the caller rather than this process. Caller references use checkClientRef before token lookup, output-folder creation or dispatch (#375, 2026-10-13); CR/LF are refused by the shared checker, and diagnostics are redacted. | ADR-0033 | | `bin/researcher-release.mjs` | The release CLI: `validate`, `conform`, `fi-validate`. **A usage error returns 2 from every subcommand**, and a status maps to its own code — `PASS` 0, `FAIL`/`REOPEN` 1, `INCOMPLETE` 2, `BLOCKED` 3. `4` is the unrecognised-status fallback and never a usage error, so a caller can distinguish "did not run" from "ran and produced something unknown". `--help` prints the usage to stdout and exits 0 (2026-09-27), as do `path-authority`, `ledger-conformance`, `fi-sidecar-conformance` and `artifact`. All five had exited 2. `--report` is written through `writeText` (2026-09-28, ADR-0079), whole or not at all. `fi-validate --report` names a report path the environment refuses through `writeFailure` and exits 2 (2026-09-30); it had crashed with a stack and exit 1, its code for FAIL. `test/hardening.test.mjs` now requires every `bin/` command that imports a writer from `lib/` to call `writeFailure`, or to name the handler that makes it unnecessary. | — | @@ -225,7 +225,7 @@ is an open decision (2026-09-30): | `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' `verifyLedger` reports a ledger-named capture it cannot read as `raw-unreadable` instead of throwing out of preflight (ADR-0077). It refuses a capture that is not a REGULAR FILE before it opens it, for the reason `readText` does (2026-09-28, Arena break test 12): a fifo named by the ledger hung every command that verifies a corpus, and a ledger symlinked to `dev/zero` ended them with `std::bad_alloc` rather than a verdict. A lock write that fails (a full disk) closes its descriptor and removes the empty lock it just created before the error reaches the caller; it leaked both (2026-09-28). | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). A source entry without `sourceSha256` is a `body-unmodified/unhashed` problem, never skipped (2026-10-04, break-test pass 4 F4): the no-hash skip was written for the first ledgers' captures, and a source never had that history, so a rechained entry naming an unverifiable sibling is refused. | ADR-0003 (corpus-adjacent), ADR-0020 `verifyLedger` does not read a capture that lies outside the project (2026-09-30, ADR-0076); it reports `raw-outside`, which preflight fails. A chain naming `../elsewhere` or a link leading out of `research/raw` had verified a file the corpus does not hold. `verifyLedger` reads the path through `projectFile` (2026-09-30, ADR-0109). `acquire` throws `LOCK_STUCK` when a lock it judged stale cannot be removed for any reason but ENOENT (already gone is the race the loop retries): the failed removal fell through to `continue`, skipping the deadline and the sleep, so a read-only research/raw holding a dead pid's lock, or a lock path that is a directory, spun a whole core forever and printed nothing - unreachable by `--report-on-signal`, since a synchronous loop never returns to the event loop (break-test 2026-10-01). The wait-out refusal is coded `LOCK_HELD`; `withLock(root, fn, { waitMs })` is the test seam for its 120 s wait, in the spirit of `materializeIndex`'s injectable git. Evicting on age stays rejected (ADR-0025). **`assertAppendable(root)`** (ADR-0122, 2026-10-02) is the one function that establishes a ledger can be appended to: an unparsed line (`LEDGER_DAMAGED`), a torn tail (`LEDGER_TORN_TAIL`), and a chain whose `seq`/`prev`/`entrySha256` no longer link (`LEDGER_CHAIN_BROKEN`, named by line and rule, remedy `git checkout -- research/raw/.fetches.jsonl` - the kit never rewrites a hash). `chainProblems(entries)` is the link check alone, shared with `verifyLedger`; `appendFetch` asks `assertAppendable` under the lock, and `runResearch` asks it before the first fetch. Until then a parsed-but-broken chain was appended to, and the paid-for fetch was refused by handoff with the rest. `isLineEndingRewrite` keeps its own CRLF-only fold, deliberately not core's `foldLineEndings` (2026-10-03): it asks whether autocrlf explains a mismatch, and autocrlf never makes a lone CR. Tail repair recognizes the leading BOM on a one-entry ledger as the reader does, so a valid first entry is never dropped as a torn tail; repairing a later append preserves the prefix (#296). | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not. The two sanctioned rewrites, `repairLedgerTail` and `rebuildLedger`, write through `writeText` (2026-09-28, ADR-0079): they emptied the ledger before writing it, so one cut short left a truncated ledger. Both look for the ledger **before** taking the lock (2026-09-28): the lock lives in `research/raw/`, a gate marker, so `doctor --fix-arity` in a folder with no ledger made it a gated project missing its scaffold. A fetch entry names its source sibling (`source`, `sourceSha256`, ADR-0140, 2026-10-03), and `verifyLedger` checks every file an entry names the same way - the capture and the source - so an edited source is `body-unmodified` and a missing one `raw-missing`, and the commit gate refuses them as it refuses an edited capture. `sourceOmitted` rides on an entry whose source was given and not kept (2026-10-04). | `quotes.mjs` | **Quote anchors** (2026-09-28, ADR-0087): `quoteAnchors(text)` reads the `[quote: ...]` markers of a Finding cell, split into fragments on an ellipsis - a marker may hold balanced brackets one level deep, a code subscript or a Markdown link (2026-09-29: it ended at the first `]`, and the stub it kept was what got checked); `normalizeForMatch` is the comparison-only form (NFKC, Markdown link text, quotes, dashes, emphasis, escapes, whitespace, case); `anchorFound(fragments, body)` requires at least one nonempty normalized fragment and matches every nonempty fragment in order, exact after normalization, with no fuzzy fallback; `MIN_QUOTE_WORDS` is 3, and `MIN_QUOTE_CHARS` is 40 (2026-10-02, ADR-0126): a quote is short only under both, measured by `quoteWords` and `quoteChars` across the fragments - two rows of the kit's own decision project quoted a 130-character dependency map, one whitespace-separated "word", and were warned as anchoring almost nothing. Pure; `checks.mjs` reads the capture and reports. `quoteAnchors` takes one pair of quotation marks around the whole passage off before matching (`unwrap`): `[quote: "the sentence"]` was quote-not-found for a sentence the capture held word for word (2026-10-01). `normalizeForMatch` also drops Unicode's format characters (`\p{Cf}`: zero-width space, soft hyphen, word joiner, the joiners and directional marks) after NFKC, on both sides: `## [\u200B](...#bundled-skills) Bundled skills` in this repository's own corpus made `[quote: ## Bundled skills]` unfound, and 32 of its 188 captures hold such characters (break-test 2026-10-01). An anchor that normalizes entirely to empty cannot count as found: raw formatting characters or three formatting-only words used to clear the short-quote floor while every search needle was skipped, falsely passing citations and inflating the measurement count (2026-10-08). Formatting-only fragments alongside real text remain ignored; nonempty fragments still require exact ordered matches. | ADR-0087 | | `measure.mjs` | **How a corpus's citations hold up, without a judge** (2026-09-28, ADR-0089): `measureCorpus(root)` - link works (capture on disk, in the ledger, unmodified, not an error page), quote-anchored rows and quotes found (`quotes.mjs`), full captures, closed unknowns on a `P` row and on two or more hosts, known unknowns, ledger verifies; `factCheck` is named and never computed. `renderMeasure` for people; `bin/measure.mjs` prints either, `--json` for machines, refuses outside a project, and never gates. | ADR-0089 | -| `warc.mjs` | **The corpus as WARC 1.1** (2026-09-28, ADR-0090): `exportWarc(root)` - a `warcinfo` record, then per ledger `scrape` entry whose capture is on disk and unchanged a `resource` record (the text after the front matter, `text/markdown`) and a `metadata` record (`WARC-Concurrent-To`; transport, completeness, the ledger's hash and seq); `WARC-Block-Digest: sha256:`, record IDs derived from the ledger entry (`uuidFrom`, RFC 9562 v8), each record its own gzip member; an edited capture is left out and named in `skipped`. `readWarc` splits a stream back into records and throws on bad framing. Validated with warcio on every corpus. `exportWarc` reads a ledger-named capture only if it is inside the project (`isInside` and `realInside`, ADR-0076) and a regular file (2026-09-30); anything else is `skipped` with its reason. A ledger naming `../` or a link out of `research/raw` had packed that outside file into the archive, and a FIFO would have hung the export. `exportWarc` reads the path through `projectFile` (2026-09-30, ADR-0109). The record is parsed from the bytes whose hash passed, never a second read of the file (2026-09-30). | ADR-0090 | +| `warc.mjs` | **The corpus as WARC 1.1** (2026-09-28, ADR-0090): `exportWarc(root)` - a `warcinfo` record, then per ledger `scrape` entry whose capture is on disk and unchanged a `resource` record (the text after the front matter, `text/markdown`) and a `metadata` record (`WARC-Concurrent-To`; transport, completeness, the capture file's hash, the block's own `body-sha256` and the ledger's seq); `WARC-Block-Digest: sha256:`, record IDs derived from the ledger entry (`uuidFrom`, RFC 9562 v8), each record its own gzip member; an edited capture is left out and named in `skipped`, as is a ledger entry the export cannot date (WARC-Date is mandatory; rank 7, 2026-10-03 gap audit, ADR-0161). `readWarc` splits a stream back into records and throws on bad framing. Validated with warcio on every corpus. `exportWarc` reads a ledger-named capture only if it is inside the project (`isInside` and `realInside`, ADR-0076) and a regular file (2026-09-30); anything else is `skipped` with its reason. A ledger naming `../` or a link out of `research/raw` had packed that outside file into the archive, and a FIFO would have hung the export. `exportWarc` reads the path through `projectFile` (2026-09-30, ADR-0109). The record is parsed from the bytes whose hash passed, never a second read of the file (2026-09-30). | ADR-0090 | | `prior.mjs` | **The registered prior: what was expected BEFORE the evidence, and the proof that it was.** `registerPrior` writes `research/PRIOR.md` and chains an `op: 'prior'` entry carrying its sha256, so two things stop being editable at once - the text, through `body-unmodified`, and its POSITION before the first scrape, through the chain itself. Three dishonest paths, each measured rather than argued: revising the prediction once the answer is known fails the chain; registering late is refused by the tool; and chaining one late BY HAND with every hash correct is caught by `provenance/prior-precedes-collection`, which exists because `appendFetch` is a public export and the gate does not get to trust that the CLI was used. **Nothing grades the prediction** (ADR-0013) - not its specificity, not its reasonableness, and above all not whether it turned out right, since a prior that was wrong has done its job and a gate that preferred correct ones would teach the opposite habit. The single content rule is a length floor, stated as a floor. Absence is SILENT at the gate: a prior is optional, and a warning on every project that skipped one is a warning nobody reads. `bin/prior.mjs` registers and reads it back; `bin/research.mjs` prints a one-line nudge before a spending run, because after the first page lands there is no second chance. **It also reaches the REMOTE path, and that half is what makes the rest useful**: the collector workflow scaffolds its own project on the runner with an empty ledger, so a prior registered on the operator machine cannot reach the corpus that comes back - and every corpus collected since `collect-remote` shipped came from that route. The prediction therefore travels as a dispatch input and is registered on the runner between the map and the collection, through `env` and an argv array like the topic (ADR-0020), with the topic's measured disclosure profile (ADR-0035). A test pins that the registration step precedes `collect`, because if it ever drifts below it, `bin/prior.mjs` refuses and the feature stops working silently on a workflow CI never runs. | ADR-0013, ADR-0020, ADR-0039 | | `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. `partial-render` applies only to a capture behind an EVIDENCE row some unknown cites (ADR-0099): on a context-only row its instruction had no cited text to confirm. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. The same duty covers what a PASS claims: `gate-integrity` used to pass with "no override in effect" when what it measured was that no override was RECORDED - and `git commit --no-verify`, the escape hatch this kit documents most prominently, records nothing. It now says what it counted and names the bypass it cannot see. Found by sweeping all thirteen pass messages for claims wider than their measurement; it was the only one. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. `collection-attempts` counts an attempt through the rows a KNOWN-UNKNOWN cites, not only a URL typed into it (2026-09-26): an E-row is a capture of its URL, so citing one is the stronger proof of having reached for it. Until then a KNOWN-UNKNOWN citing the page it tried was warned as never attempted - found by the `2026-09-26-actions-sept-changes` project's own gate, and no test exercised its verdict (only the registry-order test named it). `citations/url-mismatch` (2026-09-27) fails a row whose URL is not, by `urlKey`, the page its capture was fetched from: a row could name another page and pass, and the URL cell is what a reader is shown as the source. `provenance/ledger-unparsed` names its repair (2026-09-27): a torn last line gets `doctor --fix-arity`, which drops exactly that; a broken line inside the chain is restored from git. It gave only the JSON error. A row is never compared with itself (2026-09-27): `supersededRows` skips a same-ID pair and `hygiene/duplicate-url` leaves the same ID to `duplicate-id`; a row pasted twice failed with "E-01 has been superseded by E-01". `hygiene/future-date` fails a Retrieved date more than a day ahead (2026-09-27): 2030-01-01 had passed, hiding the row's age from every freshness check. `hygiene/duplicate-id` fails rather than warns (2026-09-27, ADR-0054): two rows under one ID leave every citation, COVERED cell and brief reference to it ambiguous, because lookups keep the last row. A contract with two U-1 rows had passed. `corroboration` weighs each unknown ID once, so it no longer prints the same single-source warning twice. `hygiene/brief-stale` warns when a stamped brief's inputs hash no longer matches the corpus (2026-09-27, ADR-0055) and names the command that redrafts it; `hygiene/brief-unstamped` warns on a drafted brief with no stamp (2026-10-03, ADR-0138) - its currency cannot be checked and it keeps its approval, so the gate says so and names the redraft that stamps it; a template or a hand-written legacy brief gets no such warning. `hygiene/duplicate-url` treats two same-day rows for one URL as a refresh when the ledger records a fetch for each (2026-09-27, ADR-0060); a pasted row, with one fetch behind two rows, is still a duplicate. `discovery-contract/build-intent` tells a missing heading from an empty one, from the corpus's `intentHeading` (2026-09-27): a renamed heading read as "is empty". **`citations` checks quote anchors** (2026-09-28, ADR-0087): each `[quote: ...]` in a Finding must occur in the row's capture (`quote-not-found`, blocking) and carry at least three words or 40 characters (`quote-too-short`, warn; the character floor is ADR-0126, 2026-10-02); the matching lives in `quotes.mjs`. `citations/raw-error-status` warns when a row cites an HTTP 4xx/5xx capture from before that refusal, unless the row names the status itself (this repository's E-15 keeps a 404 as the record of a failed guess). **`unknown-closure/secondary-only`** (2026-09-28) warns when a CLOSED unknown cites no `P` row and at least one `S` row: the registry promised "a primary row" and only `L` was checked, so three corpora closed everything on `S` rows with no warning (found by the first measurement). Retrieval dates compare by `compareText` (2026-10-01). `runChecks` appends each check's findings one at a time (2026-10-01, break-test PR #188): `push(...check.run())` passed every finding as an argument, and past the engine's limit - 60,000 to 130,000 findings from one check, by Node line - the verdict died with a raw RangeError; preflight, doctor, brief and audit crashed, and the commit gate answered internal error. Reachable from about 66,000 evidence rows. The other spread sites (`priorOrder`, doctor's `gateHealth`) are bounded by the contract's unknown count and left as they are. **`hygiene` fails a malformed ID** (2026-10-03, audit G2): each `malformed-id` problem the corpus reader records becomes a `fail` beside `duplicate-id`, for the same reason - the ID is how every check finds a row; `corpus-shape` skips the kind so one typo is not reported twice. A `table-split` problem fails the same way (2026-10-03, gap audit rank 1): the row no check reads is named with its file and line, and `corpus-shape` does not repeat it. **Freshness is judged by the capture's own date** (2026-10-03, audit G8): `unknown-closure` ages a cited row by `freshnessDate` - the capture's front-matter `retrieved`, written by the collector at the fetch and under the ledger's hash, with the table's cell as the fallback for a row whose capture carries no parseable date - and names the judging date when the two differ; `hygiene` warns `date-mismatch` when a row's cell and its capture name different calendar days (`retrievedDay`), with the remedy of writing the capture's date in the row. Editing E-19's cell alone had made its stale warning disappear while the capture on disk still said 2026-09-20. The capture judged is the one the row NAMES (`ownCapture`, review of G8 the same day): `captureOf` answers a blank Raw cell with the URL's latest capture, and judged by that a row with no Raw cell borrowed a later fetch's date, lost its stale warning and drew a date-mismatch telling the operator to write the later date in; a row naming no capture is judged by its cell and has nothing to reconcile against. A future date in the capture fails `future-date` as one in the cell does, naming the capture, and the stale hint's "fresher capture of the same URL" is found by capture date (`freshnessDay`), not by the cells. **Same-day supersession follows the ledger** (2026-10-03, audit G4): `supersededRows` orders two rows of one URL by the day freshness judges them by (`freshnessDay` - the capture's date; ordered by the table cell, one edit to an older row's date had failed the gate on the genuinely newest row and told the operator to cite the older one; review of G4, the same day) and then by the ledger position of each row's cited capture (`ledgerRank`, the rule `newer` applies for the collector); a capture no fetch recorded ranks below every fetched one, and only among those does the table's order still break the tie. With A fetched, then B, then A again, the collector had held A current while the check said B superseded it, and swapping two evidence rows swapped the verdict. `uncited-capture` says "was collected" only of a file a ledger scrape entry names; a file under research/raw no fetch produced is named as such (2026-10-04, break-test pass 4 F5a). `discovery-contract/contract-missing` states that the contract is missing or could not be read (2026-10-08): the reader uses the same null fallback for absence and refused reads, so that finding cannot establish absence. Both states keep the existing failing rule; no filesystem-state API or new check is introduced. **Evidence URL comparisons use the adopted page identity** (2026-10-09, audit rank 14): supersession groups, stale-citation fresher-row hints and duplicate-URL hygiene compare through `urlKey` (ADR-0136), as the collector and cache already do. Capture dates, blank-Raw table-date fallback, same-day ledger ordering, the same-ID exclusion and the two-recorded-fetch refresh rule retain their existing roles. Captured URL spellings remain visible; query strings and path case remain distinct. | ADR-0004, ADR-0026, ADR-0087, ADR-0136 | | `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 | diff --git a/docs/adr/0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md b/docs/adr/0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md new file mode 100644 index 00000000..d2d29b43 --- /dev/null +++ b/docs/adr/0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md @@ -0,0 +1,74 @@ +# ADR-0161 — The WARC export names the block's own hash, and leaves out an entry it cannot date + +- **Date:** 2026-10-17 +- **Status:** accepted +- **Area:** `research-kit/lib/warc.mjs`, `research-kit/bin/export-warc.mjs` (ADR-0090). A fix + to the export's own integrity, not a new capability: it adds no command, flag, transport, + provider, configuration key or check. +- **Freeze:** lifts the feature freeze (ADR-0117) for the one `metadata` field + `body-sha256`, by the precedent of ADR-0140, which lifted it for the `.source.html` + file and two ledger fields. Nothing else is new: the exit code and the re-read are fixes to + an existing command's result, not a new command. +- **Refines:** ADR-0090 (the corpus as WARC 1.1). Its decision stands. + +## Context + +The 2026-10-03 gap audit (`docs/review-2026-10-03-gap-audit.md`, rank 7, VERIFIED) found +these defects in the export ADR-0090 introduced; the 2026-10-04 gap audit +(`docs/review-2026-10-04-gap-audit.md`, rank 15, which re-verified rank 7) restated them and +added one: + +- The `metadata` record's `capture-sha256` is the hash of the **whole capture file** — the + value the ledger holds — but the `resource` record's block is the body **after** the + front matter, so a consumer cannot check a block against the field that names its file. +- A ledger `scrape` entry with no `at` writes an empty `WARC-Date`. WARC 1.1 makes + `WARC-Date` mandatory; the clause is captured in the extraction audit's copy of the + specification (`docs/decisions/2026-10-04-gap-audit-extraction/research/raw/`), which the + audit cites as C:E-08. +- A capture left out is named on stdout, but the command exits 0, so a caller cannot tell a + partial export from a whole one. +- The file written is never read back: the bytes split in memory are the ones in hand, not + the bytes on disk, so a write that landed corrupted is reported as an archive. + +## Decision + +The `metadata` record gains `body-sha256`, the hash of the block the `resource` record +carries, beside `capture-sha256`, the whole file the ledger hashed. A ledger `scrape` entry +the export cannot date is left out and named like any other entry it cannot use, so no +record carries an empty mandatory `WARC-Date`. The command exits 2 (`INCOMPLETE`) when +anything was left out and 0 otherwise, and re-reads the file it wrote and splits it again +before it reports success. + +The exit code is 2, not 1: `README.md` maps `PASS` to 0, `FAIL`/`REOPEN` to 1 and +`INCOMPLETE` to 2, and a partial export is INCOMPLETE — the command did what it could and +says so with its own code, distinct from both a whole export and a failed one. The first +implementation of this ADR exited 1 (the 2026-10-03 audit's wording); the re-verification +says 2, and the kit's own mapping agrees, so 2 stands. + +## Rejected alternatives + +- **Write `capture-sha256` over the block it accompanies.** Then the field no longer equals + the hash the ledger holds, so it stops tying the archive to the evidence — the one thing it + is for. Two hashes, each naming its own object, keep both checks. +- **Keep writing the empty `WARC-Date` and warn.** `WARC-Date` is mandatory, so the record is + not conformant whatever is printed beside it; a capture that cannot be dated is a corpus + defect the caller should be able to act on, not a warning next to a written file. +- **Abort the whole export on an undated entry.** The export already names and continues past + a capture it cannot use (missing, edited, outside the project). An undated entry is the same + kind of per-entry defect; a partial archive is more useful than none, with exit 2 saying it + is partial. +- **Exit 1 for a partial export.** `README.md` reserves 1 for a `FAIL`/`REOPEN` verdict and 2 + for `INCOMPLETE`; the export produced something, incompletely, which is INCOMPLETE. Exit 1 + was the 2026-10-03 audit's first wording and was superseded by the re-verification. +- **Trust the in-memory bytes and skip the re-read.** `writeBytes` writes a scratch file and + renames it, so the window is small, but the on-disk file is the product; splitting the bytes + in hand says nothing about the bytes that landed. + +## Consequences + +`body-sha256` is an extension field in the `metadata` record's `application/warc-fields` +block. WARC 1.1 says a reader ignores a field name it does not recognize, so an archive tool +that does not know it is unaffected. A consumer can now tie a block to the ledger entry +(`body-sha256` beside `ledger-seq` and `capture-sha256`) and check it against the resource +record's own `WARC-Block-Digest`. The freeze lift is for `body-sha256` alone: it is not +permission to add further fields to the export without their own ADR. diff --git a/docs/adr/README.md b/docs/adr/README.md index f8af6d78..39e1ee1e 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -176,5 +176,6 @@ directory and are treated as ADR-equivalent. | [0159](0159-the-22-line-starts-at-22-22-1-because-earlier-builds-lose-windows-1252.md) | **The 22 line starts at 22.22.1, because 22.13.0 to 22.22.0 lose windows-1252.** Those builds' `TextDecoder` reads `windows-1252` as Latin-1 (nodejs/node#60888), so a page's euro signs and curly quotes are captured as control characters with no warning; `nodeLine` reports them as `fail` naming the defect and the first good build, the README says 22.22.1+ or 24+, the hard floor every command refuses below stays at 22. Rejected: a table decoder for 0x80-0x9F behind a runtime probe, refusing those builds in `checkNode`, documenting only. | | [0160](0160-package-approval-is-rederived-with-a-portable-baseline.md) | Claimed package approval is re-derived from its checked corpus with an unpromoted portable gate; false claims remain false, and major-1 early stamps require their actual historical input hash and a missing gate field. | +| [0161](0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md) | A WARC export names the resource block's own hash (`body-sha256`) beside the whole capture file's, leaves out a ledger entry it cannot date (WARC-Date is mandatory) instead of writing an empty one, exits 2 (`INCOMPLETE`) on a partial export, and re-reads the written file (rank 7, 2026-10-03 gap audit; rank 15, 2026-10-04 gap audit) | Domain vocabulary lives in [`CONTEXT.md`](../../CONTEXT.md). diff --git a/research-kit/README.md b/research-kit/README.md index 483b1a98..d2c4fd75 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -568,7 +568,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -1861 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +1864 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/bin/export-warc.mjs b/research-kit/bin/export-warc.mjs index 40c5bd95..44e0db9a 100644 --- a/research-kit/bin/export-warc.mjs +++ b/research-kit/bin/export-warc.mjs @@ -1,6 +1,7 @@ #!/usr/bin/env node // bin/export-warc.mjs - the corpus as one .warc.gz, a copy for archive tools (ADR-0090). +import fs from 'node:fs'; import { parseFlags, refuseUnknownFlags, kitCommand, operatorPath, writeBytes, writeFailure } from '../lib/core.mjs'; import { exportWarc, readWarc } from '../lib/warc.mjs'; import { isGated } from '../lib/gate.mjs'; @@ -18,8 +19,10 @@ if (flags.help) { One warcinfo record, then per fetched capture a resource record (the page text the kit kept - never an HTTP exchange it did not store) and a metadata record (transport, -completeness, the ledger's hash and sequence number). A capture changed since its fetch is -left out and named. The corpus and its ledger remain the evidence; this is a copy. +completeness, the capture file's hash, the block's own hash and the ledger's sequence +number). A capture changed since its fetch, or a ledger entry with no date, is left out and +named, and the command exits 2 (INCOMPLETE); a clean export exits 0. The corpus and its +ledger remain the evidence; this is a copy. `); process.exit(0); } @@ -46,6 +49,21 @@ try { process.stderr.write(`export-warc: ${why}\n`); process.exit(2); } +// The bytes on disk are re-read, not assumed from the ones in hand (rank 15, 2026-10-04 gap +// audit): a write that landed corrupted is refused, never reported as an archive. writeBytes +// writes a scratch file and renames it, so the window is small; the check is cheap and the +// file is the product. +try { + readWarc(fs.readFileSync(out)); +} catch (err) { + process.stderr.write(`export-warc: the written file could not be read back as WARC (${err.message})\n`); + process.exit(2); +} const plural = (n, word) => `${n} ${word}${n === 1 ? '' : 's'}`; process.stdout.write(`export-warc: wrote ${plural(records.length, 'record')} (${plural(captures, 'capture')}) to ${out}\n`); for (const s of skipped) process.stdout.write(` left out ${s.file} (ledger seq ${s.seq}): ${s.reason}\n`); +// Rank 7 (2026-10-03 gap audit; rank 15, 2026-10-04 gap audit): a capture left out used to +// exit 0, so a caller could not tell a partial export from a whole one. A partial export is +// INCOMPLETE, and README.md maps INCOMPLETE to 2 (FAIL/REOPEN is 1). The file it could write +// is still written. +process.exit(skipped.length ? 2 : 0); diff --git a/research-kit/lib/warc.mjs b/research-kit/lib/warc.mjs index 4bbd684b..3ce36d54 100644 --- a/research-kit/lib/warc.mjs +++ b/research-kit/lib/warc.mjs @@ -95,6 +95,13 @@ export function exportWarc(root) { }[problem] }); continue; } + // WARC-Date is mandatory (WARC 1.1). An entry the export cannot date is left out and + // named, never written as a record with an empty date (rank 7, 2026-10-03 gap audit). + const dated = String(entry.at ?? '').trim(); + if (!dated) { + skipped.push({ file: entry.raw, seq: entry.seq, reason: 'the ledger entry carries no date, and WARC-Date is mandatory' }); + continue; + } let raw = null; try { raw = fs.readFileSync(abs); } catch { /* named below */ } if (raw === null) { skipped.push({ file: entry.raw, seq: entry.seq, reason: 'the capture could not be read' }); continue; } @@ -106,22 +113,26 @@ export function exportWarc(root) { // between the two - a refresh, a concurrent collector - was exported with the new text // under the ledger's hash of the old (found 2026-09-30, break-test). const { front, body } = parseCapture(raw.toString('utf8')); + // The block the resource record carries, and its own hash. `capture-sha256` is the whole + // file the ledger hashed; `body-sha256` lets a consumer verify the block itself + // (rank 7, 2026-10-03 gap audit). + const block = Buffer.from(body, 'utf8'); const seed = entry.entrySha256 || `${entry.seq}:${entry.url}:${entry.bodySha256}`; const resourceId = uuidFrom(`resource:${seed}`); const metaId = uuidFrom(`metadata:${seed}`); plain.push({ id: resourceId, bytes: warcRecord({ - type: 'resource', id: resourceId, date: entry.at, + type: 'resource', id: resourceId, date: dated, fields: [['WARC-Target-URI', entry.url], ['WARC-Warcinfo-ID', infoId]], contentType: 'text/markdown; charset=utf-8', - block: Buffer.from(body, 'utf8'), + block, }), }); plain.push({ id: metaId, bytes: warcRecord({ - type: 'metadata', id: metaId, date: entry.at, + type: 'metadata', id: metaId, date: dated, fields: [['WARC-Target-URI', entry.url], ['WARC-Concurrent-To', resourceId], ['WARC-Warcinfo-ID', infoId]], contentType: 'application/warc-fields', block: warcFields([ @@ -131,6 +142,7 @@ export function exportWarc(root) { ['evidence-type', entry.type || ''], ['capture-file', entry.raw], ['capture-sha256', entry.bodySha256 || sha256(raw)], + ['body-sha256', sha256(block)], ['ledger-seq', entry.seq], ['ledger-entry-sha256', entry.entrySha256 || ''], ]), diff --git a/research-kit/test/warc.test.mjs b/research-kit/test/warc.test.mjs index d876b87d..e8075887 100644 --- a/research-kit/test/warc.test.mjs +++ b/research-kit/test/warc.test.mjs @@ -201,3 +201,48 @@ test('an export carries the bytes it verified, not a second read of the file', ( const text = zlib.gunzipSync(out.bytes).toString('utf8'); assert.doesNotMatch(text, /REPLACED_AFTER_VERIFY/, 'the export carried text the ledger hash never covered'); }); + +// Rank 7 of the 2026-10-03 gap audit (docs/review-2026-10-03-gap-audit.md, VERIFIED): the +// resource block is the capture after its front matter, so `capture-sha256` (the whole +// file, the value the ledger holds) cannot verify a block. `body-sha256` names the block. +test('the metadata names the block\'s own hash beside the capture file\'s', () => { + const dir = makePassingProject(); + const read = readWarc(exportWarc(dir).bytes); + const [, resource, meta] = read; + const scrape = readLedger(dir).entries.find((e) => e.op === 'scrape'); + const fields = meta.block.toString('utf8'); + assert.match(fields, new RegExp(`body-sha256: ${sha256(resource.block)}\\r\\n`)); + assert.match(fields, new RegExp(`capture-sha256: ${scrape.bodySha256}\\r\\n`)); + assert.equal(resource.headers['WARC-Block-Digest'], `sha256:${sha256(resource.block)}`); + assert.notEqual(sha256(resource.block), scrape.bodySha256, + 'the fixture needs the block and the file to differ, or this proves nothing'); +}); + +// WARC 1.1 makes WARC-Date mandatory (C:E-08). An entry the export cannot date must be left +// out and named, never written as a record with an empty WARC-Date. +test('a ledger entry with no date is left out, so no record carries an empty WARC-Date', () => { + const dir = makePassingProject(); + const entry = readLedger(dir).entries.find((e) => e.op === 'scrape' && e.raw); + fs.appendFileSync(path.join(dir, PATHS.ledger), + `${JSON.stringify({ seq: 99, op: 'scrape', url: 'https://x.invalid/undated', raw: entry.raw })}\n`); + const out = exportWarc(dir); + const named = out.skipped.filter((s) => s.file === entry.raw); + assert.equal(named.length, 1, JSON.stringify(out.skipped)); + assert.match(named[0].reason, /date/); + for (const r of readWarc(out.bytes)) assert.ok(r.headers['WARC-Date'], `${r.headers['WARC-Type']} carries an empty WARC-Date`); +}); + +// Rank 7 of the 2026-10-03 gap audit, restated as rank 15 of the 2026-10-04 gap audit: a +// capture left out used to exit 0, so a caller could not tell a partial export from a whole +// one. A partial export is INCOMPLETE, which README.md maps to 2 (FAIL/REOPEN is 1); the +// file it could write is still written. +test('the CLI exits 2 (INCOMPLETE) when a capture is left out, naming it, and writes what it could', () => { + const bin = fileURLToPath(new URL('../bin/export-warc.mjs', import.meta.url)); + const dir = makePassingProject(); + const scrape = readLedger(dir).entries.find((e) => e.op === 'scrape'); + fs.appendFileSync(path.join(dir, scrape.raw), '\nchanged after fetch\n'); + const r = spawnSync(process.execPath, [bin], { cwd: dir, encoding: 'utf8' }); + assert.equal(r.status, 2, `${r.stdout}${r.stderr}`); + assert.match(r.stdout, /left out .*changed since it was fetched/); + assert.equal(readWarc(fs.readFileSync(path.join(dir, 'research-corpus.warc.gz'))).length, 1); +}); From c92a59793c797b232c93243c692c38b35df5d4d9 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli Date: Sat, 17 Oct 2026 06:48:27 +0300 Subject: [PATCH 02/12] docs: record rank 7's outcome with exit INCOMPLETE and the corrected commit What changed: the rank-7 outcome row in `docs/review-2026-10-03-gap-audit.md` section 9 now says a partial export exits 2 (`INCOMPLETE`), that the written file is re-read from disk, and names the commit `8ee1be4`. Why: rank 15 of the 2026-10-04 gap audit re-verifies rank 7 and says exit 2, and `research-kit/README.md` maps `INCOMPLETE` to 2 (`FAIL`/`REOPEN` is 1). The first row recorded exit 1. What you verified: the commit `8ee1be4` holds the fix (8 files); `research-kit/README.md` still reads "1864 tests, offline"; a docs-only commit runs no suite (ADR-0120). What you got wrong: the row recorded exit 1 before the second audit was read; corrected to 2. Nothing is pushed. --- docs/review-2026-10-03-gap-audit.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/review-2026-10-03-gap-audit.md b/docs/review-2026-10-03-gap-audit.md index f0b5b14b..639cfde2 100644 --- a/docs/review-2026-10-03-gap-audit.md +++ b/docs/review-2026-10-03-gap-audit.md @@ -109,15 +109,16 @@ Documentation drift found by the delivery explorer and verified by the lead: `re Every external claim above rests on a ledgered capture in one of the two nested projects (preflight PASS, chains verify); the kit's own code is read directly. C:E-01, E-02, E-13, E-15, E-16, E-23, E-24 and K:E-06, E-07 are uncited search-return rows, kept as the record of what the searches returned. One claim rests on an older capture of the root corpus: Firecrawl format pricing (root E-02, 2026-09-13, type S). One claim is CONTESTED (Firecrawl scrape and robots.txt). One researcher used the session's Firecrawl MCP search once to locate pages, then fetched them through the kit; the located pages are ledgered, the locating search is not. Spend: 1,175 to 1,061 credits. -## 9. Outcome (ranks 1 to 3, authorised 2026-10-03) +## 9. Outcome (ranks 1 to 3 authorised 2026-10-03; rank 7 fixed 2026-10-17) | Rank | Change | Commit | |---|---|---| | 1 | `parseTable` records a `table-split` problem for every row-shaped line between a table's end and the next heading; `hygiene` fails it; a second blank-separated table is not one. | 9ca9fc1 | | 2 | Every Firecrawl scrape passes `--max-age 0`; a cached answer's `cacheState`/`cachedAt`, if sent, go into the capture's front matter and the ledger entry (ADR-0139). Verified live: the vendor served a previous day's copy as today's scrape; `--max-age 0` fetched live at the same credit. | c6c5377 | | 3 | Every transport returns the text it converted as `source`; the collector keeps it LF-folded as `.source.html`, the ledger names and hashes it, `verifyLedger` checks it like the capture, `readCaptures` never indexes it (ADR-0140, lifts the freeze for that file and two ledger fields). Firecrawl's `rawHtml` verified live at no extra credit. | bed6bc0 | +| 7 | The `metadata` record carries `body-sha256`, the block's own hash, beside `capture-sha256` (unchanged - the whole file, which is what the ledger holds), so a consumer can check a block against the export; a ledger `scrape` entry with no `at` is left out and named instead of writing a record with an empty mandatory `WARC-Date`; a partial export exits 2 (`INCOMPLETE`, rank 15 of the 2026-10-04 gap audit); and the written file is re-read from disk before success. ADR-0161 (lifts the freeze for `body-sha256`). | 8ee1be4 | -Ranks 4 to 15 remain as recorded above, not implemented. +Ranks 4 to 6 and 8 to 15 remain as recorded above, not implemented. ### Review of the three commits (breaker and mutation auditor, 2026-10-04) From 3abf0b8fdd269a34888f08bd440bdc138bedec29 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli Date: Sat, 17 Oct 2026 06:58:40 +0300 Subject: [PATCH 03/12] quotes: one passage, read at any bracket depth, the floor measured per fragment What changed: `research-kit/lib/quotes.mjs` reads a `[quote: ...]` marker by counting brackets instead of matching one level, so any nesting is parsed; `anchorFound` requires each fragment to start within a new `MAX_QUOTE_GAP` (300 compared characters) of the previous one and tries a placement at every occurrence; a new `quoteTooShort(anchor)` measures the short-quote floor over EACH fragment. `research-kit/lib/checks.mjs` uses `quoteTooShort` and its message now says no fragment reaches the floor. Why: rank 4 of the 2026-10-03 gap audit (`docs/review-2026-10-03-gap-audit.md`) and rank 4 of the 2026-10-04 gap audit, which re-verified it. A passage two levels deep was not a marker at all, so the quote went unchecked and nothing said so; the fragments of one quote could be arbitrarily far apart, so `[quote: free ... requests ... credits]` anchored three scattered words; and the floor was measured over the joined fragments, so those three words read as three "words" and cleared it. What it touched: `research-kit/lib/quotes.mjs`, `research-kit/lib/checks.mjs`, `research-kit/test/quotes.test.mjs` (three tests), `docs/ARCHITECTURE.md` (the `quotes.mjs` and `checks.mjs` rows), `research-kit/README.md` (the test count), `CHANGELOG.md`, and the new `docs/adr/0162-a-quote-is-one-passage-read-at-any-bracket-depth.md` with its index row. No freeze-lift is owed: this changes how an existing check reads, and adds no command, flag, file format or check. What you verified: - verified: the quotes group is green on this tree - 13 passed, 0 failed (`selftest.mjs quotes`). - verified: red-first - before the fix, exactly the three new tests failed (10 passed, 3 failed). - verified: the `measure` and `checks` groups pass (`selftest.mjs measure checks`, green). - verified: the `repo-hygiene` group passes with the new ADR and its index row. - verified: the root corpus still passes its gate (`preflight.mjs`: PASS, 0 blocking). - verified: the three nested decision projects that hold an ellipsis quote still pass their own gates - 2026-10-07-break-test-external-facts, 2026-10-04-gap-audit-citation and 2026-10-06-kit-vs-prompt-evaluation, each PASS with 0 blocking - so the new gap bound does not turn a real quote into a false failure. - expected: the commit gate's full offline suite is green; it runs for this commit because the commit stages `research-kit/`, so a red suite would block it rather than land it. - expected: the test count moves from 1864 to 1867 (three tests added), which the kit README now states. What you got wrong and fixed: - mistake: the first edit of `lib/checks.mjs` left `quoted += 1;` twice in the citation loop, so every quote was counted twice. - where: `research-kit/lib/checks.mjs`, the `citations` check's quote loop (the two counters sat together). - impact: the citations pass message read "2 quotes found" for one quote, and two existing tests failed at once ("the pass message counts the anchored quotes"). Nothing shipped - the red was caught before the commit. - cause: the replacement block was written from the comment line down but included the counter line above it, which the matched text did not contain. - fix: removed the duplicate line. - verified: `selftest.mjs quotes` green afterwards (13 passed, 0 failed), including the two tests that caught it. --- CHANGELOG.md | 5 + docs/ARCHITECTURE.md | 4 +- ...s-one-passage-read-at-any-bracket-depth.md | 74 +++++++++++++++ docs/adr/README.md | 1 + research-kit/README.md | 2 +- research-kit/lib/checks.mjs | 9 +- research-kit/lib/quotes.mjs | 95 ++++++++++++++----- research-kit/test/quotes.test.mjs | 38 ++++++++ 8 files changed, 198 insertions(+), 30 deletions(-) create mode 100644 docs/adr/0162-a-quote-is-one-passage-read-at-any-bracket-depth.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 30239cba..4b5b447b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,11 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in writing a record with an empty mandatory `WARC-Date`, a partial export exits 2 (`INCOMPLETE`), and the file it wrote is re-read (rank 7, `docs/review-2026-10-03-gap-audit.md`; rank 15, `docs/review-2026-10-04-gap-audit.md`). +- The `citations` check reads a `[quote: ...]` marker however deeply its brackets nest (a + Markdown link whose own text held brackets was not a marker at all, so the quote went + unchecked), holds every fragment within one passage instead of anchoring words scattered + over a page, and measures the short-quote floor over each fragment rather than the joined + text (rank 4, `docs/review-2026-10-03-gap-audit.md`). - `collect-remote` refuses empty repository, topic, client-ref, prior and prefer values by name before creating an output directory or reading a token (#287). - Invalid `--snapshot-max-age-ms` values are usage errors with exit 2. Library callers diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index d4dddb59..e1e6f96a 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -223,11 +223,11 @@ is an open decision (2026-09-30): | `decompose.mjs` | Phase 0 coordination: recipe parsing, docs-host discovery, material gathering (scrape-budget accounting via the corpus's `cacheDecision` — a cache hit is not an attempt), draft MAP writing. Contains **no judgment** — it seeds the checklist and gathers material; it never writes a status. **A failed search reaches the map and the terminal** (2026-09-26): MAP.md lists each failure (query, provider, one scrubbed line of the vendor's error, and any fallback) and tells three empty maps apart - a dry run, a quiet topic, an outage - and `searchSummary` gives the CLI its count. Until then the failures were kept in the result (F20) and shown nowhere: after every search failed, the map told the operator to stop dry-running. The error is **scrubbed** because MAP.md is committed and vendor error text has echoed request URLs with keys in them. **Its searches are metered** (2026-09-27): phase 0 runs four searches and now writes a usage row like research-run's (`command: 'decompose'`, `searchesUsed` summed from what each provider reports), so `research.mjs --status` counts them. Until then two runs made 8 SerpAPI searches that the vendor's meter showed and the local one - labelled "this box", though it is one project on this machine - did not. A dry run still seeds the map (its documented purpose) and now logs each search a real run would make and the provider whose meter it would spend, as research's dry run does (2026-09-27): it had previewed none of the four searches. The CLI's closing steps name `research/plan.json`. **`resolveTopic` decides what is mapped** (2026-09-27, ADR-0056): `plan.json`'s topic when `--topic` is omitted, and a refusal naming both when `--topic` differs (spacing and case aside). A project still called `UNTITLED_TOPIC` accepts any `--topic`. The CLI had rewritten the map's topic to whatever it was given, and the brief took its title from the map. `loadRecipe` names the recipes that exist when one is not found (2026-09-27): `--recipe api` had answered with only the folder it looked in. Its usage row carries `searchCreditsEstimate` too (ADR-0065), counted once per search including a fallback search (a duplicated line had counted fallbacks twice until 2026-09-27). With `--max-scrapes`, a candidate is scraped only if it carries the query that found it (`matchesQuery`, ADR-0068); every candidate still appears in the map. Under a **merged** search selection it asks every provider each query and interleaves by rank (`mergeByRank`, as research does; 2026-09-28): the CLI passed only the first while printing that both were searched, so phase 0 scraped what SerpAPI alone had ranked. The usage row carries `searchesOn`; a provider that missed a query another answered is listed in the map as `answered by the other provider` and is not counted lost. **A compound topic is searched part by part** (2026-09-28, ADR-0085): `topicQueries` reads `subject: part, part; part` as one search per part (a one- or two-word part carries the subject, and so does a part that refers back with a pronoun, ADR-0098; at most `MAX_TOPIC_PARTS`, 6) instead of the whole topic four times, and `interleaveByQuery` shares the candidate list one per part. Searched whole, MoonAliza's three-question topic mapped forum threads and a coffee-scale blog, and in search order its third question got none of the map's 20 candidates. A plain topic keeps its four searches. **Outlines** (2026-09-29, ADR-0091): after gathering, `outlineOf(body)` reads the `##`/`###` headings of every material page with a capture on disk (at most `MAX_OUTLINES` pages, 12 headings each; code blocks, links, zero-width marks, repeats and `OUTLINE_FURNITURE` dropped) into a `## Outlines seen in the material` section - STORM's table-of-contents step without its model, unjudged, no fetch. Furniture also covers GitHub's file-view chrome and Discourse per-post bylines (`OUTLINE_FURNITURE_PATTERNS`, 2026-09-29). The scrape budget follows `scrapeOrder`: likely-owner hosts first, by `docsHosts` score, then the rest (ADR-0094). Owners are counted by `ownerOf` (ADR-0096): on a shared host - github.com, gitlab.com, huggingface.co, medium.com and the like - the owner is the account (the segment after `orgs/` or `users/`; the host's own sections such as `topics` stay the host's), so five strangers' repositories no longer pool into one "likely owner" that outranks the vendor's docs. A page on the owners tying for the top score (at least `MIN_NAMED_OWNER_SCORE`, 3) that the relevance floor skips is listed in the map under "Skipped on the likely owner", with `plan.json` `urls` as the way in (ADR-0097): the owner's terse "OpenAI compatibility" page had been skipped with only a log line to show for it. The floor itself still holds for it - trusting the owner past it re-admitted RR-9's postgresql.org home page. A compound-topic part that opens with "the" carries the subject, as a pronoun part does (2026-09-30, ADR-0103): "the bot limiter", searched alone, filled a SearXNG map with audio mastering limiters. `resolveTopic` refuses a plan the corpus reader flagged for a duplicate object key (2026-09-30), in the reader's words: `JSON.parse` keeps the last `topic`, so every search was built from the second while the operator read the first. Its searches go through `searchPatiently` (one call, no retries), and a failed search degrades only to a fetch provider that can search - the browser cannot (2026-10-01). Only an http(s) search result with a host (`isWebUrl`) is taken into the material, so only such pages are listed or scraped (2026-10-01). **A damaged ledger is refused before any provider is asked** (2026-10-03, outside audit): `assertAppendable(root)` at the top of `decompose`, as `runResearch` has had since ADR-0122 - phase 0 spends too, and it made four searches and one fetch before `collectOne` threw. **Its searches go through the search session** (2026-10-03, ADR-0135), the same one `runResearch` asks: `decompose` builds it with no rate-limit patience and a sink that keeps each failure row for the map, refuses a `notReady` provider through `session.assertReady()` before the ledger check (until then only the CLI refused, so a library caller spent), asks `session.ask(query)` per query and keeps every http(s) result as material, and reads `session.meters()` into its usage row. With the move it gained the three lines `runResearch` already logged (`searched`, `search found N`, `search failed on every provider`), its usage row's `searchTransport` names the paying provider under a merge as research's does (it had joined every provider with `+`), its `searchFailures` counts every failed ask (a covered miss and a degraded first ask included) and its `degraded` counts only the degrades that answered - research's counts, under the ADR's one rule. **A bare `--limit` or `--max-scrapes` is refused** (2026-10-11, #269): both go through `checkFlagValues` (`int`, `min: 0`) at the top of the CLI with the other value flags, so a flag with no value fails as `-- needs a value` instead of being read as `true` - which the hand-written `Number(raw)` check it replaces turned into 1, so `--dry-run --max-scrapes` reported "scrape up to 1" and a real run paid for one page. Recipe loading refuses front matter with no dimensions and blank dimension labels, naming the file and physical line. The CLI presents INVALID_RECIPE as usage failure (exit 2) before map writes (#356). | ADR-0085, ADR-0091, ADR-0094, ADR-0096, ADR-0097, ADR-0098, ADR-0135 | | `dimensions.mjs` | The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (`coverageOfUniversals`). Recipes add on top of it; nothing replaces it. | — | | `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' `verifyLedger` reports a ledger-named capture it cannot read as `raw-unreadable` instead of throwing out of preflight (ADR-0077). It refuses a capture that is not a REGULAR FILE before it opens it, for the reason `readText` does (2026-09-28, Arena break test 12): a fifo named by the ledger hung every command that verifies a corpus, and a ledger symlinked to `dev/zero` ended them with `std::bad_alloc` rather than a verdict. A lock write that fails (a full disk) closes its descriptor and removes the empty lock it just created before the error reaches the caller; it leaked both (2026-09-28). | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). A source entry without `sourceSha256` is a `body-unmodified/unhashed` problem, never skipped (2026-10-04, break-test pass 4 F4): the no-hash skip was written for the first ledgers' captures, and a source never had that history, so a rechained entry naming an unverifiable sibling is refused. | ADR-0003 (corpus-adjacent), ADR-0020 `verifyLedger` does not read a capture that lies outside the project (2026-09-30, ADR-0076); it reports `raw-outside`, which preflight fails. A chain naming `../elsewhere` or a link leading out of `research/raw` had verified a file the corpus does not hold. `verifyLedger` reads the path through `projectFile` (2026-09-30, ADR-0109). `acquire` throws `LOCK_STUCK` when a lock it judged stale cannot be removed for any reason but ENOENT (already gone is the race the loop retries): the failed removal fell through to `continue`, skipping the deadline and the sleep, so a read-only research/raw holding a dead pid's lock, or a lock path that is a directory, spun a whole core forever and printed nothing - unreachable by `--report-on-signal`, since a synchronous loop never returns to the event loop (break-test 2026-10-01). The wait-out refusal is coded `LOCK_HELD`; `withLock(root, fn, { waitMs })` is the test seam for its 120 s wait, in the spirit of `materializeIndex`'s injectable git. Evicting on age stays rejected (ADR-0025). **`assertAppendable(root)`** (ADR-0122, 2026-10-02) is the one function that establishes a ledger can be appended to: an unparsed line (`LEDGER_DAMAGED`), a torn tail (`LEDGER_TORN_TAIL`), and a chain whose `seq`/`prev`/`entrySha256` no longer link (`LEDGER_CHAIN_BROKEN`, named by line and rule, remedy `git checkout -- research/raw/.fetches.jsonl` - the kit never rewrites a hash). `chainProblems(entries)` is the link check alone, shared with `verifyLedger`; `appendFetch` asks `assertAppendable` under the lock, and `runResearch` asks it before the first fetch. Until then a parsed-but-broken chain was appended to, and the paid-for fetch was refused by handoff with the rest. `isLineEndingRewrite` keeps its own CRLF-only fold, deliberately not core's `foldLineEndings` (2026-10-03): it asks whether autocrlf explains a mismatch, and autocrlf never makes a lone CR. Tail repair recognizes the leading BOM on a one-entry ledger as the reader does, so a valid first entry is never dropped as a torn tail; repairing a later append preserves the prefix (#296). | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not. The two sanctioned rewrites, `repairLedgerTail` and `rebuildLedger`, write through `writeText` (2026-09-28, ADR-0079): they emptied the ledger before writing it, so one cut short left a truncated ledger. Both look for the ledger **before** taking the lock (2026-09-28): the lock lives in `research/raw/`, a gate marker, so `doctor --fix-arity` in a folder with no ledger made it a gated project missing its scaffold. A fetch entry names its source sibling (`source`, `sourceSha256`, ADR-0140, 2026-10-03), and `verifyLedger` checks every file an entry names the same way - the capture and the source - so an edited source is `body-unmodified` and a missing one `raw-missing`, and the commit gate refuses them as it refuses an edited capture. `sourceOmitted` rides on an entry whose source was given and not kept (2026-10-04). -| `quotes.mjs` | **Quote anchors** (2026-09-28, ADR-0087): `quoteAnchors(text)` reads the `[quote: ...]` markers of a Finding cell, split into fragments on an ellipsis - a marker may hold balanced brackets one level deep, a code subscript or a Markdown link (2026-09-29: it ended at the first `]`, and the stub it kept was what got checked); `normalizeForMatch` is the comparison-only form (NFKC, Markdown link text, quotes, dashes, emphasis, escapes, whitespace, case); `anchorFound(fragments, body)` requires at least one nonempty normalized fragment and matches every nonempty fragment in order, exact after normalization, with no fuzzy fallback; `MIN_QUOTE_WORDS` is 3, and `MIN_QUOTE_CHARS` is 40 (2026-10-02, ADR-0126): a quote is short only under both, measured by `quoteWords` and `quoteChars` across the fragments - two rows of the kit's own decision project quoted a 130-character dependency map, one whitespace-separated "word", and were warned as anchoring almost nothing. Pure; `checks.mjs` reads the capture and reports. `quoteAnchors` takes one pair of quotation marks around the whole passage off before matching (`unwrap`): `[quote: "the sentence"]` was quote-not-found for a sentence the capture held word for word (2026-10-01). `normalizeForMatch` also drops Unicode's format characters (`\p{Cf}`: zero-width space, soft hyphen, word joiner, the joiners and directional marks) after NFKC, on both sides: `## [\u200B](...#bundled-skills) Bundled skills` in this repository's own corpus made `[quote: ## Bundled skills]` unfound, and 32 of its 188 captures hold such characters (break-test 2026-10-01). An anchor that normalizes entirely to empty cannot count as found: raw formatting characters or three formatting-only words used to clear the short-quote floor while every search needle was skipped, falsely passing citations and inflating the measurement count (2026-10-08). Formatting-only fragments alongside real text remain ignored; nonempty fragments still require exact ordered matches. | ADR-0087 | +| `quotes.mjs` | **Quote anchors** (2026-09-28, ADR-0087): `quoteAnchors(text)` reads the `[quote: ...]` markers of a Finding cell, split into fragments on an ellipsis - a marker opens at `[quote:` and closes at the `]` that balances its brackets, so it may hold a code subscript, a Markdown link, or a passage nested any number of levels deep (2026-09-29: it ended at the first `]`, and the stub it kept was what got checked; rank 4 of the 2026-10-03 and 2026-10-04 gap audits: a regex bounded to ONE level did not read a two-level passage as a marker at all, so the quote was not checked); `normalizeForMatch` is the comparison-only form (NFKC, Markdown link text, quotes, dashes, emphasis, escapes, whitespace, case); `anchorFound(fragments, body)` requires at least one nonempty normalized fragment and matches every nonempty fragment in order, exact after normalization, no fuzzy fallback, each fragment starting within `MAX_QUOTE_GAP` (300 compared characters) of the one before it, and a placement tried at each occurrence rather than only the first, so a word earlier in the page does not hide the passage the quote meant (rank 4, 2026-10-03 gap audit: `[quote: free ... requests ... credits]` anchored three words scattered over a pricing page); `MIN_QUOTE_WORDS` is 3, and `MIN_QUOTE_CHARS` is 40 (2026-10-02, ADR-0126): a quote is short only when EVERY fragment is under both, measured by `quoteTooShort` (rank 4, 2026-10-03 gap audit: over the JOINED fragments, `[quote: free ... requests ... credits]` read as three "words" and cleared a floor none of its fragments reaches alone) - two rows of the kit's own decision project quoted a 130-character dependency map, one whitespace-separated "word", and were warned as anchoring almost nothing. Pure; `checks.mjs` reads the capture and reports. `quoteAnchors` takes one pair of quotation marks around the whole passage off before matching (`unwrap`): `[quote: "the sentence"]` was quote-not-found for a sentence the capture held word for word (2026-10-01). `normalizeForMatch` also drops Unicode's format characters (`\p{Cf}`: zero-width space, soft hyphen, word joiner, the joiners and directional marks) after NFKC, on both sides: `## [\u200B](...#bundled-skills) Bundled skills` in this repository's own corpus made `[quote: ## Bundled skills]` unfound, and 32 of its 188 captures hold such characters (break-test 2026-10-01). An anchor that normalizes entirely to empty cannot count as found: raw formatting characters or three formatting-only words used to clear the short-quote floor while every search needle was skipped, falsely passing citations and inflating the measurement count (2026-10-08). Formatting-only fragments alongside real text remain ignored; nonempty fragments still require exact ordered matches. | ADR-0087 | | `measure.mjs` | **How a corpus's citations hold up, without a judge** (2026-09-28, ADR-0089): `measureCorpus(root)` - link works (capture on disk, in the ledger, unmodified, not an error page), quote-anchored rows and quotes found (`quotes.mjs`), full captures, closed unknowns on a `P` row and on two or more hosts, known unknowns, ledger verifies; `factCheck` is named and never computed. `renderMeasure` for people; `bin/measure.mjs` prints either, `--json` for machines, refuses outside a project, and never gates. | ADR-0089 | | `warc.mjs` | **The corpus as WARC 1.1** (2026-09-28, ADR-0090): `exportWarc(root)` - a `warcinfo` record, then per ledger `scrape` entry whose capture is on disk and unchanged a `resource` record (the text after the front matter, `text/markdown`) and a `metadata` record (`WARC-Concurrent-To`; transport, completeness, the capture file's hash, the block's own `body-sha256` and the ledger's seq); `WARC-Block-Digest: sha256:`, record IDs derived from the ledger entry (`uuidFrom`, RFC 9562 v8), each record its own gzip member; an edited capture is left out and named in `skipped`, as is a ledger entry the export cannot date (WARC-Date is mandatory; rank 7, 2026-10-03 gap audit, ADR-0161). `readWarc` splits a stream back into records and throws on bad framing. Validated with warcio on every corpus. `exportWarc` reads a ledger-named capture only if it is inside the project (`isInside` and `realInside`, ADR-0076) and a regular file (2026-09-30); anything else is `skipped` with its reason. A ledger naming `../` or a link out of `research/raw` had packed that outside file into the archive, and a FIFO would have hung the export. `exportWarc` reads the path through `projectFile` (2026-09-30, ADR-0109). The record is parsed from the bytes whose hash passed, never a second read of the file (2026-09-30). | ADR-0090 | | `prior.mjs` | **The registered prior: what was expected BEFORE the evidence, and the proof that it was.** `registerPrior` writes `research/PRIOR.md` and chains an `op: 'prior'` entry carrying its sha256, so two things stop being editable at once - the text, through `body-unmodified`, and its POSITION before the first scrape, through the chain itself. Three dishonest paths, each measured rather than argued: revising the prediction once the answer is known fails the chain; registering late is refused by the tool; and chaining one late BY HAND with every hash correct is caught by `provenance/prior-precedes-collection`, which exists because `appendFetch` is a public export and the gate does not get to trust that the CLI was used. **Nothing grades the prediction** (ADR-0013) - not its specificity, not its reasonableness, and above all not whether it turned out right, since a prior that was wrong has done its job and a gate that preferred correct ones would teach the opposite habit. The single content rule is a length floor, stated as a floor. Absence is SILENT at the gate: a prior is optional, and a warning on every project that skipped one is a warning nobody reads. `bin/prior.mjs` registers and reads it back; `bin/research.mjs` prints a one-line nudge before a spending run, because after the first page lands there is no second chance. **It also reaches the REMOTE path, and that half is what makes the rest useful**: the collector workflow scaffolds its own project on the runner with an empty ledger, so a prior registered on the operator machine cannot reach the corpus that comes back - and every corpus collected since `collect-remote` shipped came from that route. The prediction therefore travels as a dispatch input and is registered on the runner between the map and the collection, through `env` and an argv array like the topic (ADR-0020), with the topic's measured disclosure profile (ADR-0035). A test pins that the registration step precedes `collect`, because if it ever drifts below it, `bin/prior.mjs` refuses and the feature stops working silently on a workflow CI never runs. | ADR-0013, ADR-0020, ADR-0039 | -| `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. `partial-render` applies only to a capture behind an EVIDENCE row some unknown cites (ADR-0099): on a context-only row its instruction had no cited text to confirm. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. The same duty covers what a PASS claims: `gate-integrity` used to pass with "no override in effect" when what it measured was that no override was RECORDED - and `git commit --no-verify`, the escape hatch this kit documents most prominently, records nothing. It now says what it counted and names the bypass it cannot see. Found by sweeping all thirteen pass messages for claims wider than their measurement; it was the only one. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. `collection-attempts` counts an attempt through the rows a KNOWN-UNKNOWN cites, not only a URL typed into it (2026-09-26): an E-row is a capture of its URL, so citing one is the stronger proof of having reached for it. Until then a KNOWN-UNKNOWN citing the page it tried was warned as never attempted - found by the `2026-09-26-actions-sept-changes` project's own gate, and no test exercised its verdict (only the registry-order test named it). `citations/url-mismatch` (2026-09-27) fails a row whose URL is not, by `urlKey`, the page its capture was fetched from: a row could name another page and pass, and the URL cell is what a reader is shown as the source. `provenance/ledger-unparsed` names its repair (2026-09-27): a torn last line gets `doctor --fix-arity`, which drops exactly that; a broken line inside the chain is restored from git. It gave only the JSON error. A row is never compared with itself (2026-09-27): `supersededRows` skips a same-ID pair and `hygiene/duplicate-url` leaves the same ID to `duplicate-id`; a row pasted twice failed with "E-01 has been superseded by E-01". `hygiene/future-date` fails a Retrieved date more than a day ahead (2026-09-27): 2030-01-01 had passed, hiding the row's age from every freshness check. `hygiene/duplicate-id` fails rather than warns (2026-09-27, ADR-0054): two rows under one ID leave every citation, COVERED cell and brief reference to it ambiguous, because lookups keep the last row. A contract with two U-1 rows had passed. `corroboration` weighs each unknown ID once, so it no longer prints the same single-source warning twice. `hygiene/brief-stale` warns when a stamped brief's inputs hash no longer matches the corpus (2026-09-27, ADR-0055) and names the command that redrafts it; `hygiene/brief-unstamped` warns on a drafted brief with no stamp (2026-10-03, ADR-0138) - its currency cannot be checked and it keeps its approval, so the gate says so and names the redraft that stamps it; a template or a hand-written legacy brief gets no such warning. `hygiene/duplicate-url` treats two same-day rows for one URL as a refresh when the ledger records a fetch for each (2026-09-27, ADR-0060); a pasted row, with one fetch behind two rows, is still a duplicate. `discovery-contract/build-intent` tells a missing heading from an empty one, from the corpus's `intentHeading` (2026-09-27): a renamed heading read as "is empty". **`citations` checks quote anchors** (2026-09-28, ADR-0087): each `[quote: ...]` in a Finding must occur in the row's capture (`quote-not-found`, blocking) and carry at least three words or 40 characters (`quote-too-short`, warn; the character floor is ADR-0126, 2026-10-02); the matching lives in `quotes.mjs`. `citations/raw-error-status` warns when a row cites an HTTP 4xx/5xx capture from before that refusal, unless the row names the status itself (this repository's E-15 keeps a 404 as the record of a failed guess). **`unknown-closure/secondary-only`** (2026-09-28) warns when a CLOSED unknown cites no `P` row and at least one `S` row: the registry promised "a primary row" and only `L` was checked, so three corpora closed everything on `S` rows with no warning (found by the first measurement). Retrieval dates compare by `compareText` (2026-10-01). `runChecks` appends each check's findings one at a time (2026-10-01, break-test PR #188): `push(...check.run())` passed every finding as an argument, and past the engine's limit - 60,000 to 130,000 findings from one check, by Node line - the verdict died with a raw RangeError; preflight, doctor, brief and audit crashed, and the commit gate answered internal error. Reachable from about 66,000 evidence rows. The other spread sites (`priorOrder`, doctor's `gateHealth`) are bounded by the contract's unknown count and left as they are. **`hygiene` fails a malformed ID** (2026-10-03, audit G2): each `malformed-id` problem the corpus reader records becomes a `fail` beside `duplicate-id`, for the same reason - the ID is how every check finds a row; `corpus-shape` skips the kind so one typo is not reported twice. A `table-split` problem fails the same way (2026-10-03, gap audit rank 1): the row no check reads is named with its file and line, and `corpus-shape` does not repeat it. **Freshness is judged by the capture's own date** (2026-10-03, audit G8): `unknown-closure` ages a cited row by `freshnessDate` - the capture's front-matter `retrieved`, written by the collector at the fetch and under the ledger's hash, with the table's cell as the fallback for a row whose capture carries no parseable date - and names the judging date when the two differ; `hygiene` warns `date-mismatch` when a row's cell and its capture name different calendar days (`retrievedDay`), with the remedy of writing the capture's date in the row. Editing E-19's cell alone had made its stale warning disappear while the capture on disk still said 2026-09-20. The capture judged is the one the row NAMES (`ownCapture`, review of G8 the same day): `captureOf` answers a blank Raw cell with the URL's latest capture, and judged by that a row with no Raw cell borrowed a later fetch's date, lost its stale warning and drew a date-mismatch telling the operator to write the later date in; a row naming no capture is judged by its cell and has nothing to reconcile against. A future date in the capture fails `future-date` as one in the cell does, naming the capture, and the stale hint's "fresher capture of the same URL" is found by capture date (`freshnessDay`), not by the cells. **Same-day supersession follows the ledger** (2026-10-03, audit G4): `supersededRows` orders two rows of one URL by the day freshness judges them by (`freshnessDay` - the capture's date; ordered by the table cell, one edit to an older row's date had failed the gate on the genuinely newest row and told the operator to cite the older one; review of G4, the same day) and then by the ledger position of each row's cited capture (`ledgerRank`, the rule `newer` applies for the collector); a capture no fetch recorded ranks below every fetched one, and only among those does the table's order still break the tie. With A fetched, then B, then A again, the collector had held A current while the check said B superseded it, and swapping two evidence rows swapped the verdict. `uncited-capture` says "was collected" only of a file a ledger scrape entry names; a file under research/raw no fetch produced is named as such (2026-10-04, break-test pass 4 F5a). `discovery-contract/contract-missing` states that the contract is missing or could not be read (2026-10-08): the reader uses the same null fallback for absence and refused reads, so that finding cannot establish absence. Both states keep the existing failing rule; no filesystem-state API or new check is introduced. **Evidence URL comparisons use the adopted page identity** (2026-10-09, audit rank 14): supersession groups, stale-citation fresher-row hints and duplicate-URL hygiene compare through `urlKey` (ADR-0136), as the collector and cache already do. Capture dates, blank-Raw table-date fallback, same-day ledger ordering, the same-ID exclusion and the two-recorded-fetch refresh rule retain their existing roles. Captured URL spellings remain visible; query strings and path case remain distinct. | ADR-0004, ADR-0026, ADR-0087, ADR-0136 | +| `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. `partial-render` applies only to a capture behind an EVIDENCE row some unknown cites (ADR-0099): on a context-only row its instruction had no cited text to confirm. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. The same duty covers what a PASS claims: `gate-integrity` used to pass with "no override in effect" when what it measured was that no override was RECORDED - and `git commit --no-verify`, the escape hatch this kit documents most prominently, records nothing. It now says what it counted and names the bypass it cannot see. Found by sweeping all thirteen pass messages for claims wider than their measurement; it was the only one. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. `collection-attempts` counts an attempt through the rows a KNOWN-UNKNOWN cites, not only a URL typed into it (2026-09-26): an E-row is a capture of its URL, so citing one is the stronger proof of having reached for it. Until then a KNOWN-UNKNOWN citing the page it tried was warned as never attempted - found by the `2026-09-26-actions-sept-changes` project's own gate, and no test exercised its verdict (only the registry-order test named it). `citations/url-mismatch` (2026-09-27) fails a row whose URL is not, by `urlKey`, the page its capture was fetched from: a row could name another page and pass, and the URL cell is what a reader is shown as the source. `provenance/ledger-unparsed` names its repair (2026-09-27): a torn last line gets `doctor --fix-arity`, which drops exactly that; a broken line inside the chain is restored from git. It gave only the JSON error. A row is never compared with itself (2026-09-27): `supersededRows` skips a same-ID pair and `hygiene/duplicate-url` leaves the same ID to `duplicate-id`; a row pasted twice failed with "E-01 has been superseded by E-01". `hygiene/future-date` fails a Retrieved date more than a day ahead (2026-09-27): 2030-01-01 had passed, hiding the row's age from every freshness check. `hygiene/duplicate-id` fails rather than warns (2026-09-27, ADR-0054): two rows under one ID leave every citation, COVERED cell and brief reference to it ambiguous, because lookups keep the last row. A contract with two U-1 rows had passed. `corroboration` weighs each unknown ID once, so it no longer prints the same single-source warning twice. `hygiene/brief-stale` warns when a stamped brief's inputs hash no longer matches the corpus (2026-09-27, ADR-0055) and names the command that redrafts it; `hygiene/brief-unstamped` warns on a drafted brief with no stamp (2026-10-03, ADR-0138) - its currency cannot be checked and it keeps its approval, so the gate says so and names the redraft that stamps it; a template or a hand-written legacy brief gets no such warning. `hygiene/duplicate-url` treats two same-day rows for one URL as a refresh when the ledger records a fetch for each (2026-09-27, ADR-0060); a pasted row, with one fetch behind two rows, is still a duplicate. `discovery-contract/build-intent` tells a missing heading from an empty one, from the corpus's `intentHeading` (2026-09-27): a renamed heading read as "is empty". **`citations` checks quote anchors** (2026-09-28, ADR-0087): each `[quote: ...]` in a Finding must occur in the row's capture (`quote-not-found`, blocking) and carry at least three words or 40 characters in EVERY fragment (`quote-too-short`, warn; the character floor is ADR-0126, 2026-10-02; per-fragment measurement and the one-passage bound are rank 4 of the 2026-10-03 gap audit); the matching lives in `quotes.mjs`. `citations/raw-error-status` warns when a row cites an HTTP 4xx/5xx capture from before that refusal, unless the row names the status itself (this repository's E-15 keeps a 404 as the record of a failed guess). **`unknown-closure/secondary-only`** (2026-09-28) warns when a CLOSED unknown cites no `P` row and at least one `S` row: the registry promised "a primary row" and only `L` was checked, so three corpora closed everything on `S` rows with no warning (found by the first measurement). Retrieval dates compare by `compareText` (2026-10-01). `runChecks` appends each check's findings one at a time (2026-10-01, break-test PR #188): `push(...check.run())` passed every finding as an argument, and past the engine's limit - 60,000 to 130,000 findings from one check, by Node line - the verdict died with a raw RangeError; preflight, doctor, brief and audit crashed, and the commit gate answered internal error. Reachable from about 66,000 evidence rows. The other spread sites (`priorOrder`, doctor's `gateHealth`) are bounded by the contract's unknown count and left as they are. **`hygiene` fails a malformed ID** (2026-10-03, audit G2): each `malformed-id` problem the corpus reader records becomes a `fail` beside `duplicate-id`, for the same reason - the ID is how every check finds a row; `corpus-shape` skips the kind so one typo is not reported twice. A `table-split` problem fails the same way (2026-10-03, gap audit rank 1): the row no check reads is named with its file and line, and `corpus-shape` does not repeat it. **Freshness is judged by the capture's own date** (2026-10-03, audit G8): `unknown-closure` ages a cited row by `freshnessDate` - the capture's front-matter `retrieved`, written by the collector at the fetch and under the ledger's hash, with the table's cell as the fallback for a row whose capture carries no parseable date - and names the judging date when the two differ; `hygiene` warns `date-mismatch` when a row's cell and its capture name different calendar days (`retrievedDay`), with the remedy of writing the capture's date in the row. Editing E-19's cell alone had made its stale warning disappear while the capture on disk still said 2026-09-20. The capture judged is the one the row NAMES (`ownCapture`, review of G8 the same day): `captureOf` answers a blank Raw cell with the URL's latest capture, and judged by that a row with no Raw cell borrowed a later fetch's date, lost its stale warning and drew a date-mismatch telling the operator to write the later date in; a row naming no capture is judged by its cell and has nothing to reconcile against. A future date in the capture fails `future-date` as one in the cell does, naming the capture, and the stale hint's "fresher capture of the same URL" is found by capture date (`freshnessDay`), not by the cells. **Same-day supersession follows the ledger** (2026-10-03, audit G4): `supersededRows` orders two rows of one URL by the day freshness judges them by (`freshnessDay` - the capture's date; ordered by the table cell, one edit to an older row's date had failed the gate on the genuinely newest row and told the operator to cite the older one; review of G4, the same day) and then by the ledger position of each row's cited capture (`ledgerRank`, the rule `newer` applies for the collector); a capture no fetch recorded ranks below every fetched one, and only among those does the table's order still break the tie. With A fetched, then B, then A again, the collector had held A current while the check said B superseded it, and swapping two evidence rows swapped the verdict. `uncited-capture` says "was collected" only of a file a ledger scrape entry names; a file under research/raw no fetch produced is named as such (2026-10-04, break-test pass 4 F5a). `discovery-contract/contract-missing` states that the contract is missing or could not be read (2026-10-08): the reader uses the same null fallback for absence and refused reads, so that finding cannot establish absence. Both states keep the existing failing rule; no filesystem-state API or new check is introduced. **Evidence URL comparisons use the adopted page identity** (2026-10-09, audit rank 14): supersession groups, stale-citation fresher-row hints and duplicate-URL hygiene compare through `urlKey` (ADR-0136), as the collector and cache already do. Capture dates, blank-Raw table-date fallback, same-day ledger ordering, the same-ID exclusion and the two-recorded-fetch refresh rule retain their existing roles. Captured URL spellings remain visible; query strings and path case remain distinct. | ADR-0004, ADR-0026, ADR-0087, ADR-0136 | | `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 | | `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 | | `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, **what phase 1 may commit** (ADR-0048: `research/`, the root `SCAFFOLDING` - `.gitattributes`, `.gitignore`, `AGENTS.md`, `START_HERE.md` - and `docs/ARCHITECTURE.md` while `isEmptyArchitectureMap` says its staged bytes, read by `readStagedText`, are still the template's; until 2026-09-27 only `research/`, so a new project's first `git add -A` was refused and its `.gitattributes` could not travel), and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **Staged paths remain data, with checked acquisition** (ADR-0154): the default commit CLI lazily calls `readStagedPaths` after the ungated and `GATE_OFF` returns. One actual `git diff --cached --name-only --no-renames -z` supplies paths only on successful complete output; the existing 64 MiB capture rejects nonzero exit and buffer failure, and the reader rejects malformed NUL data. Successful zero bytes means `[]`; unreadable scope means null and raises the existing architecture-map breach even after green preflight, ordinary exit 1 under either posture. Explicit arrays/null and the existing stdin/repeated `--staged` options retain their meaning; edits never acquire a list. **One deliberate exception to "no CLI parsing"** remains for explicitly supplied stdin data (`stagedPathsFromStdin` / `splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"**, preserving unknown scope rather than assuming an empty set. No list read claims an atomic snapshot with later index materialization. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). The architecture-map block's fix says to **update** the map, then `git add` it (2026-09-27): a bare `git add docs/ARCHITECTURE.md` stages nothing while the map is unchanged, so the printed fix left the commit blocked. `bin/gate.mjs` no longer closes that block with "Phase 1 is not done until preflight prints PASS" - a map block happens only when the gate passes. `isPhaseOneEdit(root, file)` answers the same question for the edit hook, about one file. The empty-index verdict — a gated project whose `research/` is untracked, every fresh project's first commit — removes the scratch dir `materializeIndex` made before it learned the index was empty (2026-09-28): it returned before `scratch` was assigned, so the `finally` that removes it never ran, and every such verdict leaked one temp directory. **Integrity before scope** (ADR-0093): a failing verdict whose findings include an `INTEGRITY_RULES` provenance rule blocks the commit even when it is confined to `research/` - the ADR-0048 allowance covers completeness findings only. `stdinIsReadable(stdin = process.stdin)` takes the stream to ask about, so the hook's terminal case is testable (2026-10-01). **The suite rule** (ADR-0120, 2026-10-01): `suiteBreach(root, stagedPaths, { run })` runs after the map rule, in the commit gate only, and only where `isKitCheckout` finds the kit's four markers and a staged path is under `research-kit/` (or the list is unknown); `runSuiteHere` spawns the checkout's `bin/selftest.mjs` with `RESEARCH_KIT_RESULT_FILE` pointed at a scratch file and `RESEARCH_KIT_ALLOW_UNSUP=1` (unsupported is not red, ADR-0108), git's hook environment stripped (`GIT_INDEX_FILE`, `GIT_DIR`, `GIT_PREFIX` and kin - the same `REPOSITORY_LOCATION` list as the edit hook, else every scratch repository's git in the suite would act on the index being committed), and reads the file, never stdout: failures block naming the counts, no result blocks as a crashed runner, a run past its budget is stopped and blocks as unfinished, the fix naming `RESEARCH_KIT_GATE_TIMEOUT` before the override, and a green run whose runner exited non-zero (the README count check) blocks too. `suiteOwed(root, stagedPaths)` is the predicate alone; `suiteBreach`'s `announce` (`evaluate`'s `announceSuite`) is called right before the run, so `bin/gate.mjs` says the suite is running only when it is - not when the map rule or preflight refused the commit first. `evaluate` takes `runSuite` so the verdict is testable without a suite. **The suite's budget follows the gate's watchdog** (ADR-0142, 2026-10-05): `suiteBudgetMs(env)` is `SUITE_TIMEOUT_MS` (65 min, twice the slowest honest run measured, 1825 s on the operator's Windows PC) unless `RESEARCH_KIT_GATE_TIMEOUT` is set to a positive duration `gateTimeoutSeconds` reads (seconds, an optional fraction, an optional `s`/`m`/`h`/`d`, at most nine digits before the point - the grammar the hook's `gate_timeout_readable` checks too; `0` and anything outside it leave the default), and then that watchdog less `GATE_REST_S` (120 s, the hook's default watchdog for a gate that runs no suite), never more than half a short one, at least 1 ms and at most 2^31 - 1 - so the gate stops the suite and blocks before the hook's watchdog fires, whenever its own work fits what is left. `runSuiteHere` stops the runner with SIGKILL: the harness's SIGTERM listener kept a synchronously blocked runner alive past its budget on POSIX; a stopped run's scratch is left in the temp folder. The budget had been 20 minutes, which that PC's green suite met on every kit commit, and the variable had reached only the watchdog: set below the budget, the watchdog killed the gate first and a fail-open machine allowed a commit whose suite never reported. **The fix lines no longer say `--no-verify` is recorded** (2026-10-03, outside audit): git skips this hook, so nothing can record it; `checks.mjs` had said so since ADR-0035 while the gate's own fix text, the README, AGENTS.md and CONTEXT.md said the opposite. **Red suite diagnostics** (2026-10-08): `suiteBreach` forwards up to five `failed` identifiers, each limited to 160 characters with controls normalized, in its existing detail string; omitted or clipped data is named, and legacy missing labels are explicit. Counts and verdict keys are unchanged. `runSuiteHere` still removes its scratch result; no result retention system is added. | ADR-0002, ADR-0007, ADR-0008, ADR-0020, ADR-0093, ADR-0120, ADR-0142, ADR-0154 | diff --git a/docs/adr/0162-a-quote-is-one-passage-read-at-any-bracket-depth.md b/docs/adr/0162-a-quote-is-one-passage-read-at-any-bracket-depth.md new file mode 100644 index 00000000..655c8139 --- /dev/null +++ b/docs/adr/0162-a-quote-is-one-passage-read-at-any-bracket-depth.md @@ -0,0 +1,74 @@ +# ADR-0162 - A quote is one passage, read at any bracket depth, its floor measured per fragment + +- **Date:** 2026-10-17 +- **Status:** accepted +- **Area:** `research-kit/lib/quotes.mjs`, `research-kit/lib/checks.mjs` (ADR-0087). A fix to + the quote check's own reading, not a new capability: it adds no command, flag, transport, + provider, configuration key or check. +- **Refines:** ADR-0087 (a `[quote: ...]` anchor must occur in the capture) and ADR-0126 (the + short-quote floor is words AND characters). Both decisions stand. + +## Context + +The 2026-10-03 gap audit (`docs/review-2026-10-03-gap-audit.md`, rank 4, VERIFIED) found three +defects in the `[quote: ...]` check; the 2026-10-04 gap audit +(`docs/review-2026-10-04-gap-audit.md`, rank 4, which re-verified it) restated them: + +- The marker regex read **one** level of brackets. A passage two levels deep - a Markdown link + whose own text held brackets - was not read as a marker at all: `matchAll` found nothing, so + the quote was not checked, and nothing said so. +- The fragments of one quote could be **arbitrarily far apart**. `anchorFound` searched each + fragment from the end of the previous match with no upper bound, so + `[quote: free ... requests ... credits]` anchored three words scattered over a pricing page + and passed as verified. +- The short-quote floor was measured over the **joined** fragments. Three one-word fragments + read as three "words" and cleared a floor none of them reaches alone. + +The second audit named a fourth item in the same rank - "a quote that normalises to nothing is +counted as found". That was closed on 2026-10-08 (an anchor that normalises entirely to empty +cannot count as found; formatting-only fragments beside real text are still ignored). It is not +re-done here, and its behaviour is kept. + +## Decision + +The marker reader counts brackets instead of matching a fixed depth: a marker opens at +`[quote:` and closes at the `]` that balances it, so a passage may hold a code subscript, a +Markdown link, or a link whose own text holds brackets, with no depth limit. + +A quote is one passage. `anchorFound` requires each fragment to start within `MAX_QUOTE_GAP` +(300 characters of the compared, normalised text, about a paragraph) of the end of the one +before it. Each fragment's placement is tried at every occurrence, not only the first: with a +bound, a first occurrence out of context would otherwise reject the passage the quote meant. + +The short-quote floor is measured **per fragment**, by `quoteTooShort`: the quote is short only +when EVERY fragment is under both floors. A single long or multi-word fragment is enough for the +quote to be specific. + +## Rejected alternatives + +- **Keep the one-level regex and name an unreadable marker instead** (`quote-unparsed`). The + audit allows either. Refused: a Finding may legitimately discuss the marker syntax - this + repository's own decision projects write `[quote: ...]` as an example - and a warning on an + empty or unterminated marker would fire on that prose. Counting brackets reads the real + passage and does not fire there. +- **Bound the gap by "the same paragraph".** `normalizeForMatch` collapses runs of whitespace, + so paragraph breaks are already gone from the compared text; checking them would need a second + normalised form and a second matcher for one rule. The character bound is simpler and testable. +- **Bound the gap relative to the quote's own length.** A short quote would then get the + tightest bound, where the scattered-word case lives, and it would read as more precise while + being less predictable. A single documented constant is clearer. +- **Fuzzy-match within the bound.** ADR-0087 refused fuzzy matching (a 0.92 similarity accepts + an invented word inside a real sentence). Nothing here reopens it. +- **Warn rather than fail when fragments are far apart.** Each fragment still occurs, so the + outcome is not the same as a refuted quote. Kept as the existing `quote-not-found` fail: the + quote AS A PASSAGE does not occur, which is what the check claims. + +## Consequences + +`MAX_QUOTE_GAP` and `quoteTooShort` are exported; `anchorFound` takes `{ maxGap }` so a caller +can tighten the bound. `measure.mjs` counts quotes through `anchorFound`, so its count follows +the same rule - a scattered quote is no longer counted as found there either. `quoteWords` and +`quoteChars` remain for measurement and orientation, but the check no longer uses them. + +A quote whose fragments genuinely lie in one passage is unaffected: the repository's own +decision corpora hold four ellipsis quotes, and all three projects still pass their gates. diff --git a/docs/adr/README.md b/docs/adr/README.md index 39e1ee1e..50eb1e31 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -177,5 +177,6 @@ directory and are treated as ADR-equivalent. | [0160](0160-package-approval-is-rederived-with-a-portable-baseline.md) | Claimed package approval is re-derived from its checked corpus with an unpromoted portable gate; false claims remain false, and major-1 early stamps require their actual historical input hash and a missing gate field. | | [0161](0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md) | A WARC export names the resource block's own hash (`body-sha256`) beside the whole capture file's, leaves out a ledger entry it cannot date (WARC-Date is mandatory) instead of writing an empty one, exits 2 (`INCOMPLETE`) on a partial export, and re-reads the written file (rank 7, 2026-10-03 gap audit; rank 15, 2026-10-04 gap audit) | +| [0162](0162-a-quote-is-one-passage-read-at-any-bracket-depth.md) | A `[quote: ...]` marker is read at any bracket depth (a two-level passage was not a marker at all, so it went unchecked), its fragments must lie in one passage (`MAX_QUOTE_GAP`, tried at each occurrence), and the short-quote floor is measured per fragment rather than over the joined text (rank 4, 2026-10-03 gap audit) | Domain vocabulary lives in [`CONTEXT.md`](../../CONTEXT.md). diff --git a/research-kit/README.md b/research-kit/README.md index d2c4fd75..4269af1c 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -568,7 +568,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -1864 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +1867 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/lib/checks.mjs b/research-kit/lib/checks.mjs index 1cf2c047..09f01858 100644 --- a/research-kit/lib/checks.mjs +++ b/research-kit/lib/checks.mjs @@ -13,7 +13,7 @@ import { documentGroups, closestPair } from './similarity.mjs'; import { coverageOfUniversals } from './dimensions.mjs'; import { verifyLedger } from './provenance.mjs'; import { readPrior, PRIOR_PATH } from './prior.mjs'; -import { quoteAnchors, anchorFound, quoteWords, quoteChars, MIN_QUOTE_WORDS, MIN_QUOTE_CHARS } from './quotes.mjs'; +import { quoteAnchors, anchorFound, quoteTooShort, MIN_QUOTE_WORDS, MIN_QUOTE_CHARS } from './quotes.mjs'; import { draftStamp, briefInputsHash, briefState } from './brief.mjs'; const VALID_STATUSES = ['CLOSED', 'KNOWN-UNKNOWN']; @@ -147,10 +147,11 @@ function citations(corpus) { const body = parseCapture(readText(resolve(corpus.root, capture.file)) ?? '').body; for (const anchor of anchors) { quoted += 1; - // Short in words AND in characters: a long token of one "word" anchors one exact line. - if (quoteWords(anchor) < MIN_QUOTE_WORDS && quoteChars(anchor) < MIN_QUOTE_CHARS) { + // Short in words AND in characters, per fragment: a long token of one "word" anchors one + // exact line, and an ellipsis does not turn three one-word fragments into a sentence. + if (quoteTooShort(anchor)) { out.push(finding('warn', 'citations', 'quote-too-short', - `${row.id} quotes "${anchor.quote}" - under ${MIN_QUOTE_WORDS} words and ${MIN_QUOTE_CHARS} characters anchors almost nothing; quote the sentence the claim rests on`, + `${row.id} quotes "${anchor.quote}" - no fragment reaches ${MIN_QUOTE_WORDS} words or ${MIN_QUOTE_CHARS} characters, so it anchors almost nothing; quote the sentence the claim rests on`, { row: row.id, line: row.line })); continue; } diff --git a/research-kit/lib/quotes.mjs b/research-kit/lib/quotes.mjs index 434b47a2..7e987126 100644 --- a/research-kit/lib/quotes.mjs +++ b/research-kit/lib/quotes.mjs @@ -19,10 +19,32 @@ export const MIN_QUOTE_WORDS = 3; */ export const MIN_QUOTE_CHARS = 40; -// The marker may hold balanced brackets one level deep - a code subscript (`calls[i].id`) or a -// Markdown link (`[v0.4.7](url)`). It ended at the first `]` until 2026-09-29, which cut such a -// quote short and then checked only the stub. -const QUOTE_RE = /\[quote:\s*((?:[^[\]]|\[[^[\]]*\])*)\]/gi; +/** + * The `[quote: ...]` markers in a cell, each `{ start, end, content }`. A marker opens at + * `[quote:` and closes at the `]` that brings the bracket depth back to zero. + * + * It ended at the first `]` until 2026-09-29, which cut a code subscript (`calls[i].id`) or a + * Markdown link (`[v0.4.7](url)`) to a stub and then checked only the stub. A regex bounded to + * ONE level of brackets did not even read a passage two levels deep - a link whose own text + * held brackets - so such a marker was not a marker at all, and a quote the gate cannot read is + * a quote it does not check (rank 4, 2026-10-03 gap audit). Counting brackets has no limit. + */ +function markersOf(s) { + const starts = [...s.matchAll(/\[quote:/gi)].map((m) => m.index); + const out = []; + for (const start of starts) { + if (out.length && start < out[out.length - 1].end) continue; // inside a marker already taken + let depth = 0; + let end = -1; + for (let j = start; j < s.length; j += 1) { + const ch = s[j]; + if (ch === '[') depth += 1; + else if (ch === ']') { depth -= 1; if (depth === 0) { end = j; break; } } + } + if (end >= 0) out.push({ start, end, content: s.slice(start + '[quote:'.length, end) }); + } + return out; +} /** * The quote markers in a Finding cell, each split into fragments on an ellipsis (`...` or @@ -32,8 +54,8 @@ const QUOTE_RE = /\[quote:\s*((?:[^[\]]|\[[^[\]]*\])*)\]/gi; */ export function quoteAnchors(text) { const out = []; - for (const match of String(text ?? '').matchAll(QUOTE_RE)) { - const quote = unwrap(match[1].trim()); + for (const marker of markersOf(String(text ?? ''))) { + const quote = unwrap(marker.content.trim()); const fragments = quote.split(/\s*(?:\.\.\.|…)\s*/).map((f) => f.trim()).filter(Boolean); if (fragments.length) out.push({ quote, fragments }); } @@ -91,31 +113,58 @@ export function normalizeForMatch(text) { } /** - * Whether every fragment occurs in the body, in order. Exact after normalization, with no - * fuzzy fallback: a 0.92 similarity accepts an invented word inside a real sentence, which - * is the very failure this exists to catch (the brief's contradiction section). + * How far apart an ellipsis may leave two fragments, in the compared (normalized) text, which + * has its runs of whitespace collapsed: an ellipsis stands for text left out of ONE passage, not + * for a jump across the page. `[quote: free ... requests ... credits]` anchored three words + * scattered over a pricing page and passed as a verified quote (rank 4, 2026-10-03 gap audit). + * The bound is about a paragraph; a page's length is what made the scatter look plausible. + */ +export const MAX_QUOTE_GAP = 300; + +/** + * Whether every fragment occurs in the body, in order, each starting no more than `maxGap` + * characters after the previous one ends. Exact after normalization, with no fuzzy fallback: a + * 0.92 similarity accepts an invented word inside a real sentence, which is the very failure + * this exists to catch (the brief's contradiction section). A fragment that normalizes to + * nothing is skipped, and an anchor with no fragment left is not found (2026-10-08). A placement + * is TRIED at each occurrence, not only at the first: a word that appears earlier out of context + * must not hide the passage the quote meant, which a bounded first-occurrence match would. */ -export function anchorFound(fragments, body) { +export function anchorFound(fragments, body, { maxGap = MAX_QUOTE_GAP } = {}) { const haystack = normalizeForMatch(body); - let from = 0; - let matched = false; - for (const fragment of fragments) { - const needle = normalizeForMatch(fragment); - if (!needle) continue; - const at = haystack.indexOf(needle, from); - if (at < 0) return false; - from = at + needle.length; - matched = true; - } - return matched; + const parts = fragments.map((f) => normalizeForMatch(f)).filter(Boolean); + if (!parts.length) return false; + const place = (i, from, first) => { + if (i === parts.length) return true; + const needle = parts[i]; + for (let at = haystack.indexOf(needle, from); at >= 0; at = haystack.indexOf(needle, at + 1)) { + // Occurrences are increasing, so once one is past the bound every later one is too. + if (!first && at - from > maxGap) return false; + if (place(i + 1, at + needle.length, false)) return true; + } + return false; + }; + return place(0, 0, true); +} + +/** + * Whether a quote anchors almost nothing, measured over EACH fragment rather than the joined + * text. An ellipsis joining three one-word fragments (`free ... requests ... credits`) has three + * "words" between them and cleared a floor none of the fragments reaches alone (rank 4, + * 2026-10-03 gap audit). A single long or multi-word fragment is enough for the quote to be + * specific; only when EVERY fragment is under both floors is the whole quote too short. + */ +export function quoteTooShort(anchor) { + return anchor.fragments.every((f) => f.split(/\s+/).filter(Boolean).length < MIN_QUOTE_WORDS + && f.length < MIN_QUOTE_CHARS); } -/** Words in a quote, across its fragments: what MIN_QUOTE_WORDS is measured against. */ +/** Words in a quote, across its fragments: what MIN_QUOTE_WORDS was measured against. */ export function quoteWords(anchor) { return anchor.fragments.join(' ').split(/\s+/).filter(Boolean).length; } -/** Characters in a quote, across its fragments: what MIN_QUOTE_CHARS is measured against. */ +/** Characters in a quote, across its fragments: what MIN_QUOTE_CHARS was measured against. */ export function quoteChars(anchor) { return anchor.fragments.join(' ').length; } diff --git a/research-kit/test/quotes.test.mjs b/research-kit/test/quotes.test.mjs index d3e69e35..c4014c24 100644 --- a/research-kit/test/quotes.test.mjs +++ b/research-kit/test/quotes.test.mjs @@ -215,3 +215,41 @@ test('citations refuse a formatting-only quote that clears a raw length or word .filter((f) => f.rule === 'quote-too-short' || f.rule === 'quote-not-found'), [], 'a found forty-character token keeps its existing exemption'); }); + +// Rank 4 of the 2026-10-03 gap audit (restated as rank 4 of the 2026-10-04 gap audit): the +// marker regex read ONE level of brackets, so a passage holding a Markdown link whose own text +// held brackets was not read as a marker at all - and a quote the gate cannot read is a quote it +// does not check. A scanner that counts brackets has no depth limit. +test('a quote may hold brackets more than one level deep, and is actually checked', () => { + const deep = 'see [the docs](https://x.invalid/a) and [the list [1]] here'; + assert.deepEqual(quoteAnchors(`[quote: ${deep}]`).map((q) => q.quote), [deep]); + assert.equal(anchorFound(quoteAnchors(`[quote: ${deep}]`)[0].fragments, `We ${deep} today.`), true); + assert.deepEqual(quoteAnchors('[quote: a [b [c] d] e] and [quote: x]').map((q) => q.quote), ['a [b [c] d] e', 'x']); +}); + +// Rank 4: a quote's "fragments may be arbitrarily far apart", so `[quote: free ... requests ... +// credits]` anchored three scattered words and passed as verified. An ellipsis stands for text +// left out of ONE passage, not for a jump across the page. +test('an ellipsis does not bridge a page: the fragments must lie in one passage', () => { + const page = `The free plan allows 10 requests. ${'x'.repeat(4000)} A note about credits.`; + assert.equal(anchorFound(quoteAnchors('[quote: The free plan ... A note about credits]')[0].fragments, page), false, + 'an ellipsis bridged 4000 characters'); + const passage = 'The free plan allows 10 requests per minute, and the paid plan grants credits.'; + assert.equal(anchorFound(quoteAnchors('[quote: The free plan allows ... grants credits]')[0].fragments, passage), true, + 'fragments within one passage still anchor'); + // A word earlier in the page, out of context, must not hide the passage the quote meant. + const twice = `The free plan starts at $0. ${'y'.repeat(1000)} The free plan grants credits.`; + assert.equal(anchorFound(quoteAnchors('[quote: The free plan ... grants credits]')[0].fragments, twice), true, + 'the placement is searched, not the first occurrence only'); +}); + +// Rank 4: the length floor was measured over the JOINED fragments, so three one-word fragments +// read as three "words" and cleared a floor none of them reaches alone. Measured per fragment, +// only a quote whose every fragment is weak is short. +test('the short-quote floor is measured over each fragment, not the joined text', () => { + const scattered = quoted('Pricing tiers. [quote: free ... requests ... credits]'); + assert.equal(scattered.find((f) => f.rule === 'quote-too-short')?.severity, 'warn', JSON.stringify(scattered)); + const mixed = quoted('Pricing tiers. [quote: free ... includes 1,000 credits]'); + assert.equal(mixed.find((f) => f.rule === 'quote-too-short'), undefined, JSON.stringify(mixed)); + assert.equal(mixed.find((f) => f.rule === 'quote-not-found'), undefined, JSON.stringify(mixed)); +}); From 1d1303bb7ba15a1667deb976e333f827edf5daf2 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli Date: Sat, 17 Oct 2026 07:22:57 +0300 Subject: [PATCH 04/12] docs: record rank 4's outcome in the first gap audit What changed: `docs/review-2026-10-03-gap-audit.md` section 9 gains the rank 4 row, the heading says ranks 4 and 7 were fixed on 2026-10-17, and the count line drops rank 4 from the open set. Why: rank 4 was fixed in `3abf0b8`; the audit's outcome table is where its ranks are closed, and the first audit now records where rank 4 stands. What you verified: the three edits read back correctly; `3abf0b8` is the commit that made the change, and its own report carries the verification (the quotes, measure, checks and repo-hygiene groups green, the root corpus preflight PASS, and the three ellipsis-quote decision projects' preflights PASS). A docs-only change owes no suite run, and the commit gate runs none. What you got wrong: nothing to report. --- docs/review-2026-10-03-gap-audit.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/review-2026-10-03-gap-audit.md b/docs/review-2026-10-03-gap-audit.md index 639cfde2..e7904a8e 100644 --- a/docs/review-2026-10-03-gap-audit.md +++ b/docs/review-2026-10-03-gap-audit.md @@ -109,16 +109,17 @@ Documentation drift found by the delivery explorer and verified by the lead: `re Every external claim above rests on a ledgered capture in one of the two nested projects (preflight PASS, chains verify); the kit's own code is read directly. C:E-01, E-02, E-13, E-15, E-16, E-23, E-24 and K:E-06, E-07 are uncited search-return rows, kept as the record of what the searches returned. One claim rests on an older capture of the root corpus: Firecrawl format pricing (root E-02, 2026-09-13, type S). One claim is CONTESTED (Firecrawl scrape and robots.txt). One researcher used the session's Firecrawl MCP search once to locate pages, then fetched them through the kit; the located pages are ledgered, the locating search is not. Spend: 1,175 to 1,061 credits. -## 9. Outcome (ranks 1 to 3 authorised 2026-10-03; rank 7 fixed 2026-10-17) +## 9. Outcome (ranks 1 to 3 authorised 2026-10-03; ranks 4 and 7 fixed 2026-10-17) | Rank | Change | Commit | |---|---|---| | 1 | `parseTable` records a `table-split` problem for every row-shaped line between a table's end and the next heading; `hygiene` fails it; a second blank-separated table is not one. | 9ca9fc1 | | 2 | Every Firecrawl scrape passes `--max-age 0`; a cached answer's `cacheState`/`cachedAt`, if sent, go into the capture's front matter and the ledger entry (ADR-0139). Verified live: the vendor served a previous day's copy as today's scrape; `--max-age 0` fetched live at the same credit. | c6c5377 | | 3 | Every transport returns the text it converted as `source`; the collector keeps it LF-folded as `.source.html`, the ledger names and hashes it, `verifyLedger` checks it like the capture, `readCaptures` never indexes it (ADR-0140, lifts the freeze for that file and two ledger fields). Firecrawl's `rawHtml` verified live at no extra credit. | bed6bc0 | +| 4 | A `[quote: ...]` marker is read by counting brackets, so a passage nested two deep (a Markdown link whose own text held brackets) is a marker and is checked, where the one-level regex read it as no marker at all; `anchorFound` requires every fragment to start within `MAX_QUOTE_GAP` (300 compared characters) of the previous one, trying a placement at each occurrence, so `[quote: free ... requests ... credits]` can no longer anchor words scattered over a page; and the short-quote floor is measured over EACH fragment by `quoteTooShort`, not over the joined text (ADR-0162). The empty-normalised-needle case was already closed 2026-10-08. | 3abf0b8 | | 7 | The `metadata` record carries `body-sha256`, the block's own hash, beside `capture-sha256` (unchanged - the whole file, which is what the ledger holds), so a consumer can check a block against the export; a ledger `scrape` entry with no `at` is left out and named instead of writing a record with an empty mandatory `WARC-Date`; a partial export exits 2 (`INCOMPLETE`, rank 15 of the 2026-10-04 gap audit); and the written file is re-read from disk before success. ADR-0161 (lifts the freeze for `body-sha256`). | 8ee1be4 | -Ranks 4 to 6 and 8 to 15 remain as recorded above, not implemented. +Ranks 5, 6 and 8 to 15 remain as recorded above, not implemented. ### Review of the three commits (breaker and mutation auditor, 2026-10-04) From dd5026fe0f7a7ab4b2add4f8d5f083fe7217f5b9 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli Date: Sat, 17 Oct 2026 07:27:52 +0300 Subject: [PATCH 05/12] collect: stamp a fetch with the moment it happened, not midnight of its day What changed: `research-kit/lib/collect.mjs` writes the scrape ledger entry's `at` as `nowIso(now)` - the moment of the fetch, on the operation's injected clock - instead of `${date}T00:00:00.000Z`. `research-kit/lib/timeline.mjs`'s When-column comment now says a fetch carries a real time; the rendering rule itself is unchanged (a bare day still renders as a date, and every corpus collected before this commit has one). Why: rank 9 of the 2026-10-04 gap audit (`docs/review-2026-10-04-gap-audit.md`): a scrape's `at` was a day written as a millisecond instant while a failed fetch already recorded the real time (`appendFetch` defaults `at` to `nowIso()`, `lib/provenance.mjs:402`), so the two sorted against each other wrongly. The day the capture was retrieved stays in `retrieved`, on the capture and in the evidence row. Alternative set aside: keep day precision and lean on `seq` for same-day order - the ledger is a provenance record, and a generation time is what proves a hash existed before a time, so a real value is correct rather than a convenience. No ADR: a bug fix with one obvious remedy (standing protocol, rule 3). What it touched: `research-kit/lib/collect.mjs` (the import list and the fetch's ledger append), `research-kit/lib/timeline.mjs` (the When-column comment), `research-kit/test/collect.test.mjs` (one test), `docs/ARCHITECTURE.md` (the `collect.mjs` and `timeline.mjs` rows), `CHANGELOG.md` and `research-kit/README.md` (the test count). No new command, flag, transport, provider, configuration key, file format or check, so the freeze (ADR-0117) is not in question. `test/harness.mjs` and `test/stdout-flush.test.mjs` keep their explicit day-precision `at`: they are deterministic fixture writers, not the collector, and a corpus whose `at` is a midnight placeholder is one they should still be able to build - every corpus collected before this commit has that shape. What you verified: - verified: red-first - against the unmodified library the new test failed and its siblings passed (`selftest.mjs collect`: 135 passed, 1 failed, 2 unsupported). - verified: after the fix the groups are green (`selftest.mjs collect brief provenance`: 190 passed, 0 failed, 2 unsupported - the two are the host's UNSUP, ADR-0108, not a red suite). - verified: `selftest.mjs warc` is green (11 passed, 0 failed): the WARC export reads `at` for `WARC-Date` and is unaffected in shape. - verified: the root corpus still passes its gate (`preflight.mjs`: PASS, 0 blocking). - verified: both production callers pass `date` and `now` together (`lib/decompose.mjs:520-527`, `lib/research-run.mjs:542-550`), no `bin/` command exposes a date flag, and the default `now = new Date()` is the real clock - so `at` and `retrieved` cannot disagree. - expected: the commit gate's full offline suite is green; it runs for this commit because the commit stages `research-kit/`. - expected: the test count moves from 1867 to 1868 (one test added), which the kit README now states. What you got wrong and fixed: nothing to report. --- CHANGELOG.md | 4 ++++ docs/ARCHITECTURE.md | 4 ++-- research-kit/README.md | 2 +- research-kit/lib/collect.mjs | 7 +++++-- research-kit/lib/timeline.mjs | 9 +++++---- research-kit/test/collect.test.mjs | 20 ++++++++++++++++++++ 6 files changed, 37 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b5b447b..f07e0634 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,10 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in ## Unreleased +- The collector stamps a fetch's ledger `at` with the moment it happened instead of midnight + of its day; a failed fetch already did, so the two sorted against each other wrongly, and the + day the capture was retrieved is unchanged in `retrieved` (rank 9, + `docs/review-2026-10-04-gap-audit.md`). - The WARC export's metadata record names the resource block's own hash (`body-sha256`) beside the capture file's, a ledger `scrape` entry with no date is left out instead of writing a record with an empty mandatory `WARC-Date`, a partial export exits 2 diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index e1e6f96a..994cfdfe 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -199,7 +199,7 @@ is an open decision (2026-09-30): | `http-transport.mjs` | The **keyless adapter**, and the second one - which is what makes the transport seam real rather than hypothetical. Pure Node ESM, no dependencies: built-in `fetch` driven synchronously through a child-process rendezvous (it re-execs itself with a JSON job on stdin), a word-density main-content extractor, HTML→markdown, DuckDuckGo-lite search, same-domain link mapping. Exposes the **same seven-function shape** as the Firecrawl adapter (pinned by test), never reads an API key, never charges a credit, stamps `transport: 'http-keyless'`, and grades its own `completeness` (`full` only ≥1500 chars of markdown, else `partial` — an honest self-assessment the capture-completeness check then judges). A fetch past its timeout says "no answer within Ns", not `spawnSync ... ETIMEDOUT` (2026-09-27). **It honours a configured proxy** (2026-09-26): `fetchEnv` (in `runtime.mjs` since ADR-0047, shared with `serpapi.mjs`) adds `NODE_USE_ENV_PROXY=1` to the fetch child's environment when HTTPS_PROXY/HTTP_PROXY is set, because Node's built-in fetch ignores those variables without it; an operator's own NODE_USE_ENV_PROXY, even `0`, is left alone. Until then the fetch went around the proxy it was given: in the container where this was found, GitHub answered with its unauthenticated rate-limit 403, because the bypass lost the proxy's authentication (corrected 2026-09-27; the first version said "proxy-only egress"). The flag needs Node 22.21+ or 24+, and `doctor` warns on older (ADR-0046). **It grades by content type** (2026-09-26): the fetch child reports `Content-Type`, and `bodyKind` routes HTML (or no type) through extraction and the length bar as before; JSON, XML, CSV, plain text and Markdown are kept verbatim and graded `full` unless empty, because nothing was extracted or dropped; a binary type (PDF, images) is never `full`, and the reason names the type. A complete 748-character GitHub API response had been graded `partial` by the HTML bar. **A page's own `
`/`
` is believed** (2026-09-27, `declaredContent`): the largest one with at least 100 words is the content, found on its own because the div-block search is lazy and non-overlapping, so a
nested in wrapper divs was never a candidate. On a GitHub Docs page that kept 143 characters of a 17,385-character article; now 5,921. Outside declared content, link-heavy blocks (breadcrumbs, sidebars) are navigation, not dropped content; prose outside it is still reported, and the reason says "outside the page's main content". `search` reports DuckDuckGo's bot check (anomaly modal, no result links) as a failed search with its reason, not as zero results (2026-09-27, ADR-0066); the user agent stays honest. `decodeEntities` reads a numeric reference to 0, a surrogate or past U+10FFFF as U+FFFD, as HTML does (2026-09-28): `String.fromCodePoint` threw on the last, and one malformed entity crashed the scrape. `search` and `map` decode an href before reading it as a URL, and `map` drops a fragment from the page it names (2026-09-28): taken raw, `?a=1&b=2` became a parameter named "amp;b", and map's pattern refused any href holding a `#`, dropping `&` and every link to a section of another page. The child reads a body through `boundedText` (runtime.mjs), refusing one past `MAX_PAGE_BYTES` (16 MiB) whether declared or streamed, where `response.text()` held any body whole (2026-09-28, ADR-0082). **A run of sections is the content** (2026-09-28): when the densest block holds under a quarter of a page's words and its `
`s hold at least half, `mainContent` takes the sections together (`basis: 'sections'`), and link-heavy navigation around them is not counted as lost - nodejs.org/api/fs.html, 314 sections and no `
`, went from 1,095 kept characters to 253,654. A **binary** response (PDF, image, archive) is refused by name and kept as no capture (2026-09-30, ADR-0105): read as text it was unrecoverable and could hold no quote, and one 10-page PDF had put 2.3 MB of it into a corpus. The refusal names `--transport firecrawl-cli`, which converts documents. Tag pairs are found by `tagPairs` (and `replacePairs`, `stripTags`) in linear time, remembering the next `>` and the next closing tag as the scan moves, where the lazy pair regexes were quadratic on unclosed tags - 9.4 s for 391 KB of `
`; the result links of a keyless search page are `resultLinks`. `test/fixtures/regex-extract.mjs` keeps the regex extractor as the oracle the linear one is compared against (2026-09-30). Redirects are followed one hop at a time, and a hop into an internal address (loopback, private, link-local, `localhost`; `internalTarget`) is refused unless the URL asked for was internal itself; `RESEARCH_KIT_ALLOW_INTERNAL_REDIRECTS=1` lifts it (ADR-0110, 2026-09-30). The fetch child wraps `dns.lookup` with `guardedLookup`, so every connection its fetch opens is judged where it is opened: an internal answer is refused (`EINTERNAL`), unless the URL asked for was internal or the host is the proxy (ADR-0114, DNS rebinding). `isInternal` is exported for the browser guard (ADR-0118). A refusal names the server's own reason beside the status (`serverReason`, 2026-10-02): a JSON `message`, an HTML title or the first line of a text body, one line and bounded (`REASON_MAX`), so `HTTP 403 - the server said: "..."` is what the ledger and the terminal record; a github.com 403 had been recorded as `HTTP 403` alone while its body named the refuser - this sandbox's proxy, not GitHub (`docs/decisions/2026-10-02-github-plain-fetch-refusal`). `scrape`, `search` and `map` all go through it. **Entities are decoded once, at the end of the conversion** (2026-10-03, output-reliability audit G3): the block passes - headings, list items, table cells - convert their bodies with `inline(body, { decode: false })`, because the page-level pass strips every tag that is left and a `<` decoded earlier was read there as the start of one; a list item holding `x < 5 and y > 2` had come out as `x 2`, a cell and a heading the same, and a link label inside an item lost everything from its `<` to the item's end, while a paragraph survived. A link's label is never decoded on its own either, wherever the link stands: the G3 fix had left the page-level pass decoding it inside the link pass, so a paragraph's `limit < 10 and burst > 2` still came out `[limit 2](/x)` and `&lt;` was decoded twice (found 2026-10-03, review of G3); the label's emptiness is judged on its decoded text, so a ` `-only link is dropped in a list item as in a paragraph; and `decodeEntities` decodes every reference kind in one pass, so `&lt;` is the text `<`. The regex oracle in `test/fixtures/regex-extract.mjs` carries the same rules, so the 3,000-page equivalence test still proves the scanner changed cost and nothing else. The keyless child reads its page through `boundedBody` and puts the decoder's `decodeFallback` sentence on its JSON line; the parent grades such a capture `partial` with the sentence as the reason (`withDecodeFallback`, `verbatim`), because a body read in a charset the decoder did not know is not known to be the page (2026-10-03, audit G7). A charset declared only in a `` tag is still not read, but such a page's bytes are not valid UTF-8 and the decoder names that, so the capture is graded partial rather than silently wrong (review of G7, 2026-10-03). **Under an explicit refusal a name asked for is judged before any fetch** (ADR-0155, 2026-10-11): with `allowInternalRedirects: false` the fetch child judges a NAME asked for with `internalTarget` - `localhost`, or a name resolving inward - and refuses it against the URL asked for, before fetching; the guarded lookup still judges it at connection time. A literal address the caller spelled out is the caller's, and the default (`undefined`) still treats an internal URL the operator asked for as theirs (ADR-0110). | ADR-0005 | An HTML answer's decoded body is returned as `source` beside the Markdown it was converted into (ADR-0140, 2026-10-03); a verbatim text or JSON answer returns none, being its own source. | `browser-transport.mjs` | **The `browser` transport** (2026-09-28, ADR-0088): renders a page with a local Chromium, `--headless --disable-gpu --virtual-time-budget --dump-dom ` as an argv array, and grades the DOM with the keyless extractor (`mainContent`, `htmlToMarkdown`, `gradeCompleteness`). `findBrowser` (config `browserPath`, `RESEARCH_KIT_BROWSER`, usual paths), `browserArgs` (`--proxy-server` from the proxy environment, `--no-sandbox` only as root, never a TLS-weakening flag), `chromeErrorOf` (Chromium's own error page, which `--dump-dom` prints with exit 0). Fetch only: `CONTRACTS = ['fetch']`, `map` is the keyless map, and `transport.mjs` searches keyless for it. A signal other than the timeout's is reported as `killed by ` with Chromium's own cause - the last FATAL stderr line, else the last ERROR line, else the last line - never as the timeout (2026-09-29). On Linux it prefers Chrome stable (`/opt/google/chrome/chrome`), the binary Ubuntu 23.10+'s AppArmor lets sandbox itself, and a "No usable sandbox" crash names the remedies that keep the sandbox - never `--no-sandbox` for non-root (ADR-0092). Its recorded command is `browser ` (2026-09-30): it had used the keyless formatter whole, so every browser capture's ledger entry read `http-keyless browser `. `scrape` renders through `renderGuarded`, which runs the guard child (`browser-guard.mjs`) under spawnSync and reads back the DOM, Chromium's exit and the guard's refusals; a page that landed on a refusal is reported by name and not captured (ADR-0118, superseding the ADR-0115 pre-check). The refusal named is the one the dump holds - the guard's refusal page carries its host and reason - and only a dump that is Chromium's own error page falls back to the last recorded (2026-10-02): a dead third-party host refused after the navigation, or on a no-network host the browser's own www.google.com, had been named as what the page reached for. `browserArgs` never names a proxy: the child adds the guard's, and switches Chromium's background traffic off - which Chromium 141 ignores for its own services, so the guard drops those by name (ADR-0124). A timeout is judged by what Chromium printed, not by its exit (2026-10-01, CI, three times in one afternoon on Windows and Ubuntu): a whole dump after the timeout is the render, because Chromium prints the DOM once when its own budget says the page is done and a hang at exit does not unrender it; a dump holding the guard's refusal is the refusal; no dump at all is the timeout, and the message now carries the last thing Chromium said on stderr, which the timeout branch used to drop. Chromium is given its own deadline, `--timeout`, ten seconds under the transport's kill timeout and never below five (2026-10-01): with the virtual-time budget alone, one resource that never answers froze the budget - virtual time advances only when the page is idle - so headless never dumped, printed nothing, and was killed with nothing to show, which is what CI's four hangs that afternoon looked like and what a page with a never-answering image reproduces here. At the deadline Chromium dumps the DOM it has and exits, so the page is captured as it stands. A capture is conservatively graded `partial` when the guard still has pending loads at process completion or process elapsed minus startup reaches the configured deadline threshold. `cutAtDeadline` reports the configured loading deadline and the guard's request state **at process completion** (2026-10-08), rather than claiming an observed DOM-dump time or assigning the historical wait to the browser: neither a late process end nor the final pending snapshot proves those facts. The same note rides a refusal's message. A timeout's message says what the launch cost - "took N s to make its first request" or "never made a request in N s" (ADR-0119). **The origin's HTTP status comes from Chromium's own net log** (2026-10-03, ADR-0137, output-reliability audit G5): `--dump-dom` reports no status, so every browser capture carried `statusCode: ''` and the collector's >= 400 rule, which reads a number, never fired - an origin's 403 page was graded `full`. `documentStatusFromNetLog(log, requested)` reads the log the guard child had Chromium write (`--log-net-log`): event and source names are looked up through the log's own `constants`, never by number; the document is the `URL_REQUEST` source whose first `URL_REQUEST_START_JOB` matches the requested URL (compared in canonical form without the fragment, a `main frame` request winning); each later start on that source is a redirect, and the status is the `HTTP/x NNN` line of the last `HTTP_TRANSACTION_READ_RESPONSE_HEADERS` in the final job. `scrape` then fails a non-2xx answer the way the keyless transport does (`HTTP N - the server said: ""`), refuses a final URL that is internal by spelling and not the requested host:port (the guard's own exemption), carries `statusCode` as a number and the post-redirect URL, and grades a render whose log gave no status `partial` with `omitted` saying the origin's status was not observed (rule 2 of the ADR) - never guessed from the page's text. A JavaScript navigation is a new source, not a redirect, and is not followed. The guard child's `EINTERNAL` - a name asked for refused before any launch under an explicit refusal - is reported as that refusal, not as a browser that could not start (ADR-0155, 2026-10-11). `scrape` and `status` given no `config` read the machine's with `loadConfig(env)` (2026-10-11): the collector calls `runScrape(url)` with no options, and the empty default meant `browserPath` was never consulted. | ADR-0088, ADR-0092 | The rendered DOM is returned as `source` beside the Markdown (ADR-0140, 2026-10-03), so the collector keeps the text the capture was converted from. **`guardChildTimeout(timeout)`** (2026-10-04, break-test pass 6 F1): the parent waits `2 × timeout + 10 s` for the guard child - the launch allowance, the render budget (each `timeout`, ADR-0119) and 10 s for the child to kill the browser and report. At `timeout + 10 s`, as before, the parent was itself the timeout for any render whose launch took more than the grace, and the child's record of where the time went died with it; the verdict read "never made a request in 0 s". A child the parent gave up on is now named as such, with the time it was given; the result carries `gaveUp: true`, and the transport reads that flag, never the message's wording (GPT's review, 2026-10-04). | `browser-guard.mjs` | **The guard proxy Chromium renders through, and the child that runs both** (ADR-0118). `startGuard` listens on a loopback port and judges every plain request and CONNECT with `judge`: the keyless rule (`isInternal`), a name by the addresses it resolves to, the connection pinned to the address judged; only the `host:port` asked for is exempt, `allowInternal` is the operator's own decision, and a configured proxy is the upstream (absolute URLs, CONNECT passed through, `Proxy-Authorization` from the URL). A refusal is a 403 carrying `REFUSAL_MARKER` and is recorded. `renderThroughGuard` runs the browser with `--proxy-server` for the guard and `--proxy-bypass-list=<-loopback>`, its own process group on POSIX (killed whole on a timeout), the DOM capped at `MAX_PAGE_BYTES`, the exit reported after a short grace even when a forked helper holds stdout open. Run as a process it takes one job on stdin and prints one JSON line. The guard keeps a record of every request it carried - `seen`, each with its outcome (a status, `refused`, `tunnel`, an error, or `unanswered` once the guard closes) - and `pending()` lists the ones still unanswered, with how long they have waited; the child reports `requests`, `pending` and `elapsedMs` beside the dump, and `seen` as a timeline from the launch - each request's offset, duration and outcome - because a Windows run then stalled 28 s with nothing pending at the guard, so where the time went became the question (2026-10-01). The child keeps the first 30 lines of the browser's stderr as well as the last 60, with the omission counted between them, and stamps the launch (`startedAt`): a hung Windows launch's tail showed profile creation 43 s after the launch, so the wait was in the head the tail could not show (2026-10-01). The render timeout starts at the browser's first request through the guard, and the launch is allowed as long again (ADR-0119): the child arms the kill at launch and re-arms it on the first request, and reports `startupMs`: three Windows CI runs in a row ran the first launch to its deadline and nothing said on what, and the guard is the one place every load passes through. **The browser's own service traffic is dropped** (2026-10-02, ADR-0124): `BROWSER_SERVICE_HOSTS` names Chromium's service hosts - `clients*.google.com`, `*.clients.google.com`, `mtalk.google.com`, `accounts.google.com`, `www.google.com`, the update, safe-browsing, variations and autofill endpoints on `googleapis.com`, `*.gvt1.com`, `dl.google.com` - and `isBrowserService` answers for one; a request there is answered 403 with `DROP_MARKER`, logged `dropped`, and is neither judged, carried nor counted as a refusal, the page asked for exempt by host and port. Rendering a plain loopback page, Chromium 141 had made nine such connections per render, carried as external, and on a no-network host the last of them - `www.google.com`, unresolvable - became the verdict of the LIVE test about an internal address. **A request before the page's own is the browser's** (ADR-0125): with `pageFirst`, which `renderThroughGuard` passes, every request is dropped until the host and port asked for has been requested - CI's Chrome stable preconnected to `www.gstatic.com` 256 ms into a render, before the page, and that host is one real pages use too; the order settles what a list cannot. The CONNECT socket gets its error listener before any answer is written (2026-10-02): a browser resets a CONNECT it has given up on with the 403's body unread, and an ECONNRESET on the handed-over socket with no listener was an uncaught exception that killed the guard child mid-render - an empty report, read by the transport as "the browser exited null", on three CI legs at random points of the render. **The child has Chromium write its net log** (2026-10-03, ADR-0137): `renderThroughGuard` makes a private folder (`rk-netlog-`), adds `--log-net-log=<dir>/net-log.json` before `--dump-dom`, reads the file after the render (a log over `MAX_PAGE_BYTES` is dropped as empty) and deletes the folder on every way out; a folder that cannot be made renders without the flag, which the transport grades as status not observed. The child also reports `allowInternal`, the internal-address decision it made by resolving the requested URL, so the transport judges the final URL by the same rule. The log travels in full through the child's JSON stdout and is never written to the corpus. **A name asked for is judged by what it resolves to, exempt or not** (ADR-0155, 2026-10-11): `judge` refuses a name resolving inward, or `localhost`, unless the caller allowed this machine's network, whether or not it is the host and port asked for; the exemption covers only the literal-address judgment. Under `allowInternalRedirects: false` the child judges a name asked for before launching and answers `EINTERNAL` with no render. | ADR-0118, ADR-0124, ADR-0125 | -| `collect.mjs` | One URL's journey: `collectOne` (cache, force, fail entries, evidence/source rows) and `writeRaw` (the capture writer, which hands back the corpus's own index entry). Consumes the `runScrape` seam; stamps `transport` and `completeness` from what the adapter declared — invents neither. The Finding cell a row is born with is `lib/finding.mjs`'s: the collector **imports** `firstFinding` and does not re-export it, so the extractor has one address. A fetch reports **why it happened in words** (`whyFetched`: first capture, refreshed with the age, or --force), not `cacheDecision`'s code - verbatim, a fresh fetch logged as "collected <url> - not-collected" (found in a live-collection log, 2026-09-26). **`writeRaw` never overwrites a capture with different bytes** (2026-09-27, ADR-0058): a same-day re-collection of an unchanged page reuses the file, and a changed page is written as `<name>.r2.md` (then `.r3`...), which sorts after the original so `readCaptures` still takes it as the day's latest. The earlier reading had been replaced, and its ledger hash broken. **An error page is a failed fetch** (2026-09-28): a result whose `statusCode` is 400 or more is recorded as a `fail` ledger entry and writes no capture and no EVIDENCE row - Firecrawl answered ok with Google's "Error 503" page and the kit had filed it as evidence; the keyless transport already refused non-2xx answers. `recentlyGone` (ADR-0100) answers `gone` without fetching when the ledger's last entry for the URL is a 404 or 410 inside the refresh window. writeRaw refuses (`OUTSIDE_PROJECT`, a named write refusal in core) a link at the capture's generated name - dangling or not - and a capture folder whose real path leaves the project, before it reads or writes; a project living under a link is judged by real path and still writes (2026-10-01). A front-matter value with a line break is written on one line, so an adapter title or error cannot add a field; the body is written verbatim (2026-10-01). A page whose text exceeds `CAPTURE_MAX_BYTES` is a **failed fetch**, named with its size and the limit, before anything is written: the reader refuses such a file as `capture-too-large` and the gate blocks on it, so a 40 MB answer had become a capture, a ledger entry and a row the corpus could not hold (2026-10-02, break-test pass 3). `refreshCaptures`, the re-read inside the lock before every fetch, asks `readCaptures` only for files the snapshot does not hold and for no sketches (2026-10-02): per-capture cost at 400 rows went from 74 ms to 13 ms. A forced re-fetch that comes back byte-identical reuses the capture (`writeRaw` never overwrites) and lets the row that cites it stand - the fetch is in the ledger, the reason says `the page is unchanged, so E-01 stands`; it had appended a second row that superseded the first, so every unknown citing unchanged content had to move (break-test pass 3, 2026-10-02). A changed page still goes to `.r2` with a row of its own. `writeRaw` writes the body with LF line endings only, whatever the transport received (2026-10-02): the scaffold's `.gitattributes` makes git store a capture as LF, and a keyless capture written with the CR bytes copyright.gov served hashed one way on the collector and checked out another way everywhere else, failing handoff on a capture nobody had touched. The one such capture already committed (a licence decision project, E-18) is pinned byte for byte with a `-text` attribute in that project. `refreshCaptures` reads the new captures with the ledger's rank (2026-10-03): one ordering rule with the reopen, so a capture another collector wrote since the snapshot is current only if its fetch came later. The collector remembers its own fetch as `latest`, and `refreshCaptures` decides every known capture the ledger names again with the rank, since a reused file's position moves while its bytes do not (2026-10-03). | ADR-0016 | **The text a capture was converted from is kept beside it** (2026-10-03, ADR-0140, gap audit rank 3): `writeSource(root, captureFile, source, markdown)` writes `<capture>.source.html` (`sourcePathFor`), LF-folded like the body, and returns its path and hash for `collectOne` to hand to `appendFetch`; nothing is kept for a verbatim answer (the source is the capture), a source over `CAPTURE_MAX_BYTES`, or a sibling already on disk with other content, which is never overwritten. The converted Markdown was all the corpus kept, so G3's conversion defect was permanent and the ledger certified it. When a source was given and is not kept, `writeSource` returns `{ omitted }` with the reason - over `CAPTURE_MAX_BYTES`, a sibling already on disk with other content, a sibling name that cannot be written - and the ledger entry carries it as `sourceOmitted` (breaker review, 2026-10-04): a link planted at the sibling's name had made `assertCapturePath` throw after the capture was written, leaving an orphan capture with no ledger entry that the next run took for a cache hit; the write is now inside the try, and the fetch is ledgered without a source. +| `collect.mjs` | One URL's journey: `collectOne` (cache, force, fail entries, evidence/source rows) and `writeRaw` (the capture writer, which hands back the corpus's own index entry). Consumes the `runScrape` seam; stamps `transport` and `completeness` from what the adapter declared — invents neither. The Finding cell a row is born with is `lib/finding.mjs`'s: the collector **imports** `firstFinding` and does not re-export it, so the extractor has one address. A fetch reports **why it happened in words** (`whyFetched`: first capture, refreshed with the age, or --force), not `cacheDecision`'s code - verbatim, a fresh fetch logged as "collected <url> - not-collected" (found in a live-collection log, 2026-09-26). **`writeRaw` never overwrites a capture with different bytes** (2026-09-27, ADR-0058): a same-day re-collection of an unchanged page reuses the file, and a changed page is written as `<name>.r2.md` (then `.r3`...), which sorts after the original so `readCaptures` still takes it as the day's latest. The earlier reading had been replaced, and its ledger hash broken. **An error page is a failed fetch** (2026-09-28): a result whose `statusCode` is 400 or more is recorded as a `fail` ledger entry and writes no capture and no EVIDENCE row - Firecrawl answered ok with Google's "Error 503" page and the kit had filed it as evidence; the keyless transport already refused non-2xx answers. `recentlyGone` (ADR-0100) answers `gone` without fetching when the ledger's last entry for the URL is a 404 or 410 inside the refresh window. **A fetch's ledger `at` is the moment it happened** (`nowIso(now)`), not midnight of its day (rank 9, 2026-10-04 gap audit): the failed-fetch path already stamped the real time and a scrape did not, so the two sorted against each other wrongly; the day the capture was retrieved stays in its `retrieved` front matter and in the evidence row. writeRaw refuses (`OUTSIDE_PROJECT`, a named write refusal in core) a link at the capture's generated name - dangling or not - and a capture folder whose real path leaves the project, before it reads or writes; a project living under a link is judged by real path and still writes (2026-10-01). A front-matter value with a line break is written on one line, so an adapter title or error cannot add a field; the body is written verbatim (2026-10-01). A page whose text exceeds `CAPTURE_MAX_BYTES` is a **failed fetch**, named with its size and the limit, before anything is written: the reader refuses such a file as `capture-too-large` and the gate blocks on it, so a 40 MB answer had become a capture, a ledger entry and a row the corpus could not hold (2026-10-02, break-test pass 3). `refreshCaptures`, the re-read inside the lock before every fetch, asks `readCaptures` only for files the snapshot does not hold and for no sketches (2026-10-02): per-capture cost at 400 rows went from 74 ms to 13 ms. A forced re-fetch that comes back byte-identical reuses the capture (`writeRaw` never overwrites) and lets the row that cites it stand - the fetch is in the ledger, the reason says `the page is unchanged, so E-01 stands`; it had appended a second row that superseded the first, so every unknown citing unchanged content had to move (break-test pass 3, 2026-10-02). A changed page still goes to `.r2` with a row of its own. `writeRaw` writes the body with LF line endings only, whatever the transport received (2026-10-02): the scaffold's `.gitattributes` makes git store a capture as LF, and a keyless capture written with the CR bytes copyright.gov served hashed one way on the collector and checked out another way everywhere else, failing handoff on a capture nobody had touched. The one such capture already committed (a licence decision project, E-18) is pinned byte for byte with a `-text` attribute in that project. `refreshCaptures` reads the new captures with the ledger's rank (2026-10-03): one ordering rule with the reopen, so a capture another collector wrote since the snapshot is current only if its fetch came later. The collector remembers its own fetch as `latest`, and `refreshCaptures` decides every known capture the ledger names again with the rank, since a reused file's position moves while its bytes do not (2026-10-03). | ADR-0016 | **The text a capture was converted from is kept beside it** (2026-10-03, ADR-0140, gap audit rank 3): `writeSource(root, captureFile, source, markdown)` writes `<capture>.source.html` (`sourcePathFor`), LF-folded like the body, and returns its path and hash for `collectOne` to hand to `appendFetch`; nothing is kept for a verbatim answer (the source is the capture), a source over `CAPTURE_MAX_BYTES`, or a sibling already on disk with other content, which is never overwritten. The converted Markdown was all the corpus kept, so G3's conversion defect was permanent and the ledger certified it. When a source was given and is not kept, `writeSource` returns `{ omitted }` with the reason - over `CAPTURE_MAX_BYTES`, a sibling already on disk with other content, a sibling name that cannot be written - and the ledger entry carries it as `sourceOmitted` (breaker review, 2026-10-04): a link planted at the sibling's name had made `assertCapturePath` throw after the capture was written, leaving an orphan capture with no ledger entry that the next run took for a cache hit; the write is now inside the try, and the fetch is ledgered without a source. | `finding.mjs` | The auto-extracted **Finding** cell: `firstFinding(markdown, fallback)` — page markdown in, one line of prose out — over the heuristics behind it (boilerplate/tracking/nav/campaign regex families, the sentence splitter, the table reader, the tier-hero block skip, the graded score, the 220-character cap). Two arguments over ~460 lines: a deep module that was buried in the middle of another one until ADR-0016. Pure by construction — no filesystem, no corpus, no adapter — the one module a caller can exercise with a string. **`findingWithContext` is the real entry point** and returns `{ finding, excerpt, heading, signals, score }`: the same selection, plus where on the page it came from, the surrounding paragraph bounded to `EXCERPT_CHARS`, and which signals fired. `firstFinding` is a thin wrapper over it, so the sentence the collector writes and the excerpt a reviewer reads cannot be chosen by different code. **Ranking is one declared table, `RULES`** — 21 named rules, each with a weight and the reason for it, and `explainScore` returns the rules that fired alongside the total. Weights used to be two separate runs of `value += n`, one in `score()` and one in the selector, so the priority order could not be stated without executing the code and the number a reviewer saw could differ from the number that chose the sentence; a test now asserts no weight is applied outside `RULES`, and that the reported signals sum to the score. Heading context is an **ancestry stack**, not the last heading seen, so a claim under `#### Capacity` still knows it sits inside `## Pricing`. The policy is evidence-first: a measured quantity or a metering rule (**how** usage is counted, which often carries no digit) is strongest; where a sentence sits counts as much as what it says, so a terms-bearing heading and an FAQ answer both score while blog and careers copy is pushed down; and things that only look concrete — one row of a pricing comparison matrix, a number inside a sign-up prompt — are penalised enough to lose to plain prose. The `comparison-row` rule's tier test is three linear scans (`tierThenPeriodThenDigit`, per line, since `.` never crossed a newline) instead of `\b(plan|tier)\b.*\b(month|year|user)\b.*\d`, which was quadratic in the row - every tier word on a non-matching row scanned to its end before failing - and cost 15 s on a 309 KB row, the size of a line this corpus already holds, twice per row (break-test 2026-10-01). Prose candidates retain the physical paragraph start; excerpt recovery runs forward and stops once the existing 420-character cap is exceeded. Blank/heading boundaries, including ones inside skipped fences, remain the same; no backward array insertion makes a long unbroken capture quadratic (#334). | ADR-0016 (superseding ADR-0009) An underscore inside a word is the word (2026-10-02): `stripMarkdown` dropped every `_` as emphasis, so a drafted finding named `return_run_details` as "returnrundetails" and `check_suite_id` as "checksuiteid" - parameters no page carries, and a quote copied from the draft could never be found; emphasis marks at a word's edge still go. | | `evidence-context.mjs` | Raw cells pass projectFile for both project and capture-folder boundaries before any content is read; private project files, traversal and links leaving research/raw are refused, and directories are named as non-captures (#351, ADR-0109 trigger). One unknown, everything it rests on, in one read-only answer: the unknown's wording and why it blocks, every evidence row it cites, each row's source title / URL / type / retrieval date, the Finding, a bounded excerpt of the capture chosen for overlap with that Finding (and, in a block longer than the bound, windowed where the most distinct tokens of the Finding fall together - 2026-09-27: a JSON capture is one block, and the excerpt was its first 420 characters whatever the Finding quoted), rows that supersede it, and other rows for the same URLs. **It decides nothing**: it writes no status, offers no verdict, and reports a gap rather than filling it — a test asserts the rendering contains no conclusion and says plainly that the judgement is the reader's, and structural tests assert it imports no verdict machinery and contains no write call at all - the architectural property, which a phrase-based check cannot give. The recorded status is printed LAST and labelled as a recording, because a verdict shown before the evidence anchors the reader. Where the extractor would now pick a different sentence than the row records, that is shown beside it, labelled a suggestion. Reads no environment variable and spawns nothing, so it cannot use a credential. A Raw cell outside the project is reported ("is outside the project, so it is not read") and never read: it printed the outside file into an agent's context (2026-09-28, ADR-0076). `bin/evidence-context.mjs --all` over a contract with no unknowns yet answers "no unknowns" (or `[]` with `--json`) and exits 0 (2026-09-29, break-test): it exited 2 asking for the `--all` it was given. | — | | `credits.mjs` | **What a run does when the paying provider runs dry** — ADR-0086's one switch to the fallback, ADR-0129's stop-and-report. The one owner of a run's exhaustion state: `fellBack` / `stopped`, the `uncollected` and `skippedSearches` accounting, and the single `credits-exhausted` row the failure log keeps per run. The coordinator sequences and asks it which provider serves each call (`exhausted`, `live`, `onExhaustion`, `askLive`) and reads the state back for the summary; before the extraction the state lived as closures inside `runResearch` and `stopped.uncollected` was incremented from three loop sites. | ADR-0086, ADR-0129 | @@ -243,7 +243,7 @@ is an open decision (2026-09-30): | `scaffold.mjs` | Project shape: `LAYOUT` (12 entries), the **four** `GATE_MARKERS` (pinned by ADR-0001 and its test), template rendering, placeholder validation, hook executability. `.gitattributes` is a LAYOUT entry but **not** a gate marker: a project without it still gates, it just breaks on a Windows builder (ADR-0020). `UNTITLED_TOPIC` names the placeholder topic, so `decompose` can tell a project without a topic from one that has one (ADR-0056). A folder created while scaffolding a brand-new project is not counted as `repaired` (2026-09-27): only a folder missing from a project that already existed is a repair. **`MERGED_FILES` (`.gitignore`, `.gitattributes`) are merged into an operator's existing copy** (2026-09-27, ADR-0064): `missingKitLines` finds the kit's rule lines it lacks (whitespace collapsed) and they are appended once under a marked comment, so `.env` stays out and the ledger stays in. `validateProject` warns `kit-rules-missing` and `agents-md-foreign` (an AGENTS.md without the kit's heading, kept but named). A created folder counts as `repaired` only when `research/` already existed (2026-09-27): the operator's own `.gitignore` had made a fresh folder read as a project. | ADR-0001, ADR-0020 | | `installer.mjs` | Installs the git hook (records/restores prior `hooksPath`) and the edit-time gate (backs up, repairs known corruption, refuses unfamiliar states). The hook was renamed (A10.4): the installer recognises a registration left at a retired name, replaces it rather than adding a second, and reports the repair — `retiredRepairNote`. **And whether the deployment is still the kit you have** (`deployedDrift`, `driftNote`). `deploy` is a full mirror, so drift is exact rather than heuristic: every source file should be present at the destination with the same bytes, and nothing else should be there. Doctor measures the drift against the home the install state recorded (the one the hooks point at; it had measured `KIT_HOME` and named the recorded one), and run from the deployed kit itself - `node ~/.agents/research-kit/bin/doctor.mjs`, the copy an agent runs - it says so as `info`, with the command that does measure, instead of `pass ... matches this tree`: a tree compared with itself always matches, and a line appended to the deployed `lib/core.mjs` had passed that way (break-test pass 3, 2026-10-02). This exists because doctor's `deploy` check read the install STATE and stopped - "kit deployed <date>", pass, READY - and recorded is not current. Measured on this machine 2026-09-22, that passed over a deployment **missing 119 files and carrying 38 stale ones**, including a skill describing a protocol without the step shipped that morning. An agent invoking the skill runs the DEPLOYED copy, so every fix since the last install existed only in the repository while the machine reported ready. The skill is reported separately from the kit because it drifts independently and is the file that tells an agent what the protocol IS. **Reported, never repaired**: `install.mjs` acts, and a health check that quietly rewrote what it measured would erase the evidence of the problem it found - pinned by a test that asserts the missing file is still missing afterwards. `removeEditGate` drops the `PreToolUse` list and `hooks` object its entry leaves empty, and removes a settings file with nothing left (2026-09-27): `--uninstall` on a machine with no settings file left `{"hooks": {"PreToolUse": []}}` behind. `removeCommitGate` restores the recorded `hooksPath` only while the current value is still the kit's own folder (2026-09-27, ADR-0063), and otherwise leaves it and names it: an uninstall had unset a machine's `/opt/myhooks`, which the kit never set. `installEditGate` refuses, as the commit gate already did, when `hooks/edit-gate.mjs` is not deployed (2026-09-27): it had registered a command naming a missing file, so every Edit would run a hook that crashed. `deployedDrift` asks `isDirectory`, not `exists`, of the kit home and each skill location, and `listTree` guards its `readdirSync`/`statSync` (break-test, 2026-09-29): `RESEARCH_KIT_HOME` naming a regular file sent `readdirSync` at it, and `doctor` died with a raw ENOTDIR stack; a file there now reads as not deployed. `listTree` walks each real folder once (2026-09-29): a folder link back to an ancestor re-listed the tree on every lap until ELOOP, about 40 phantom "extra" files per real one. `deploy({ dryRun })` lists what the deploy would actually prune - retired files that exist, plus the mirror's extras (2026-09-29, Arena break test): it listed every retired name whether or not it was there, so a machine never installed read "would prune" three files, and it never named a stale file the mirror would remove. `readSettings` also refuses a `hooks.PreToolUse` node of a type the walkers do not handle (`knownHookShape`, 2026-09-30): `[null]`, a number, a string, or an entry whose `hooks` is not a list of objects reads as `unfamiliar`, and the file is not rewritten. It had reached the walkers and thrown a raw `TypeError` out of `install-hooks.mjs`. An entry with no `hooks` key is still the known shape. `knownHookShape` also requires `hooks` itself, each entry and each hook to be plain objects (2026-09-30): `"hooks": "none"`, `null`, a list or a number had passed as the known shape, and the install spread the string into the file. **Project binding is checked before deployment writes** (2026-10-08, ADR-0006): `deploy({ into })` refuses a missing path, a regular file, or a directory with none of `.git`, `AGENTS.md`, or `research/`; `.git` may be a file or directory so worktrees remain eligible, `AGENTS.md` must be a file, and `research/` must be a directory. The same refusal precedes the mirror, personal/project skill copies and install-state update on the real run and the preview; `install.mjs --into` reports it with exit 2. The guard had been absent, so a typo could deploy the home copy, prune its old files and bind skills into an unrelated folder. Config readers retain their separate last-good snapshot policy. `deploy` refuses, touching nothing, a `RESEARCH_KIT_HOME` that holds files and is neither a kit (`lib/core.mjs`, `bin/gate.mjs` and `bin/preflight.mjs`, all three: one alone took a project with a `lib/core.mjs` for a kit) nor the recorded deployment, `--dry-run` included; `install` exits 2 (ADR-0111, 2026-10-01). The mirror had deleted a folder's own files and reported them as retired. The refusal reads only the first few files it names (2026-10-01): counting all of them took 1.6 s for /usr. `installCommitGate` records the hooks folder it replaced as `research-kit.previousHooksPath` in git's global config, never the kit's own, and `removeCommitGate` clears it (ADR-0113). **A dry run answers what the real run answers** (2026-10-01, arena #196): `installCommitGate` checks that `githooks/pre-commit` is deployed before it previews, and `installEditGate`'s check no longer reads `!dryRun && !exists(hook)` - on a machine with nothing deployed `--dry-run` promised both gates and exited 0 while the real run refused both and exited 1. Both previews carry `ok: true`, one result shape for every caller, and `bin/install-hooks.mjs` tests `ok` before `dryRun`: the first cut reordered the CLI without that key and a healthy deployed machine printed `commit gate: undefined`, which a CLI test over a deployed kit now pins. **A repository-local `core.hooksPath` is reported at install time** (2026-10-03, ADR-0131): `installCommitGate` takes `cwd`, probes it with `localHooksPathOverride`, and returns `displaced: { cwd, local }` on the real run and the dry run alike; `bin/install-hooks.mjs` says it on stderr with the remedy and exits 0, and the install goes through - the gate is machine state. The installer had read and written only the global path and said "installed" from inside a husky repository where the gate would never run (outside review, 2026-10-02, three rounds). **A dry run writes nothing on any path** (2026-10-03, outside audit): `removeCommitGate`, `removeEditGate` and `uninstall` take `dryRun` and answer before any write, and `bin/install-hooks.mjs` reads `--dry-run` FIRST - it was read after the role, the posture and the uninstall had run, so `--dry-run --role builder --fail-closed` saved both and `--dry-run --uninstall` unset the installed commit gate. Both tree walks skip `__pycache__` (2026-10-04, break-test pass 4 F1): CPython's cache from the conformance runners was deployed with the kit and then counted as drift on every machine that had run the suite. **The skill set** (2026-10-06, ADR-0146): `SKILL_SET_DIR` (`skills/`) and `shippedSkills(from)` - every directory there holding a `SKILL.md`, sorted; `deploy` mirrors each to `<root>/<name>` in every `skillRoots()` root and, with `into`, beside the bound research-first skill (`boundSet`); `deployedDrift` measures each against every root where research-first is deployed, so a root installed before the set existed is drift, reported, never repaired. **A same-named skill that is not the kit's is left as it is** (2026-10-06, review): a set skill is written only where its folder is absent or its SKILL.md is, byte for byte, the one the kit last deployed there (`skillSetHashes` in the install state, kept across deploys) or the one it would deploy now - the content decides, never a recorded path, which outlives the kit's copy; any other is returned as `skillConflicts` (the dry run included), printed by `install.mjs`, and skipped by drift. The first cut overwrote a user's own `brainstorming/SKILL.md` and left its other files mixed with the kit's. A set skill the kit owns is **mirrored** (`copyTree` with `mirror`) and its extras count as drift (`orphaned`), unlike research-first's (PR review, 2026-10-06): the deploy had copied over the folder, so a reference a later kit dropped stayed and could still steer the skill, and doctor called it current. **`handOnState`** lives here since 2026-10-04 (ADR-0141; doctor re-exports it): what the hooks hand on to - none, missing, kit, other - and `installCommitGate` records no folder it classifies as `kit` (`handsOnTo`), on a first install or a re-install from the state file. doctor's one-line remedy for that hand-on (ADR-0134) had lasted one session on the operator's machine: the installer kept the old kit's `githooks/` in the install state and the next `install-hooks` run wrote it back into git config, and the old gate blocked commits again on its outdated checks. The state file and git config agree once an install has run; a `missing` folder stays recorded, for the uninstall restore. **Read-only deployed files remain replaceable** (2026-10-08, break-test pass 7 correction): `copyTree` restores write permission on an existing regular-file destination before copying, independently of the incoming source's mode, and explicitly applies the source's ordinary permission bits with owner-write after copying. This covers repeated read-only-source deployment and replacement of earlier read-only output from a normal writable source. Windows chmod changes write permission only; the copied file's read-only attribute is cleared, while POSIX ordinary execute bits are preserved and `githooks` keeps its final `HOOK_MODE` (E-04/E-05 in `docs/decisions/2026-10-07-break-test-external-facts`). Source-mode propagation was observed in the original Linux probe rather than promised by the copy API; directory, ownership and ACL failures remain separate limits. | ADR-0012, ADR-0131, ADR-0141 | | `render.mjs` | Terminal tables and Markdown findings, with no policy or I/O. `renderTable` sizes each column with a **fold, not a spread** (2026-09-28, Arena break test 12): `Math.max(header, ...rows.map(...))` passes one ARGUMENT per row, and V8 exhausts the stack somewhere past 100,000 of them - measured, 50,000 rows rendered and 200,000 threw `RangeError: Maximum call stack size exceeded`, which is a raw stack trace where a report belongs. Nothing in the kit produces a report that large today; a renderer that dies on a large input is still a renderer with an undocumented ceiling. **`renderGateWarnings(verdict)`** presents an existing evaluation in Markdown for the brief, audits and package reading guides (ADR-0143). It preserves every warning reason, remedy and supplied row/unknown target, labels whole-corpus evaluation time, and distinguishes unavailable observations from an evaluated zero. Presentation only: it performs no I/O or policy judgment. | ADR-0143 | -| `timeline.mjs` | TIMELINE.md generation and the diagnostics-log appender the gate CLI records through. Its regenerate command is spelled by `documentCommand`, with the project's recorded `kitPath` first (ADR-0084). The When column writes a day as a date and a real time as `YYYY-MM-DD HH:MM UTC` (2026-09-27): it had mixed evidence dates with the ledger's day-precision `T00:00:00.000Z` stamps. A migration row gives its recorded reason or note and a prior row its file, not `url -> raw` ("undefined -> undefined", fixed 2026-09-29). Events sort by `compareText` on their timestamp (2026-10-01). Timeline details fold whitespace before escaping table pipes, keeping multiline fetch errors in a single Markdown row; the root timeline is regenerated from its intact corpus (#354, 2026-10-13). | ADR-0051 | +| `timeline.mjs` | TIMELINE.md generation and the diagnostics-log appender the gate CLI records through. Its regenerate command is spelled by `documentCommand`, with the project's recorded `kitPath` first (ADR-0084). The When column writes a day as a date and a real time as `YYYY-MM-DD HH:MM UTC` (2026-09-27): it had mixed evidence dates with the ledger's day-precision stamps (a scrape's `at` was midnight of its day until rank 9 of the 2026-10-04 gap audit wrote the real moment). A migration row gives its recorded reason or note and a prior row its file, not `url -> raw` ("undefined -> undefined", fixed 2026-09-29). Events sort by `compareText` on their timestamp (2026-10-01). Timeline details fold whitespace before escaping table pipes, keeping multiline fetch errors in a single Markdown row; the root timeline is regenerated from its intact corpus (#354, 2026-10-13). | ADR-0051 | **Not a module, but part of the map:** `examples/release-evidence/` (added 2026-09-20) is six synthetic release packages plus `build.mjs` (regenerates them, keeping their hashes diff --git a/research-kit/README.md b/research-kit/README.md index 4269af1c..4c4a800d 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -568,7 +568,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -1867 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +1868 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/lib/collect.mjs b/research-kit/lib/collect.mjs index 7427aff4..edfa78f9 100644 --- a/research-kit/lib/collect.mjs +++ b/research-kit/lib/collect.mjs @@ -10,7 +10,7 @@ import fs from 'node:fs'; import path from 'node:path'; import { - PATHS, HEADERS, resolve, today, sha256, titleFromUrl, hostOf, urlDigest, writeText, readText, exists, + PATHS, HEADERS, resolve, today, nowIso, sha256, titleFromUrl, hostOf, urlDigest, writeText, readText, exists, sleepSync, urlKey, ageInDays, realInside, foldLineEndings, } from './core.mjs'; import { rateLimitWaitMs } from './firecrawl.mjs'; @@ -331,7 +331,10 @@ export function collectOne(root, url, { completeness: entry.completeness, omitted: entry.omitted, cmd: entry.command, - at: `${date}T00:00:00.000Z`, + // The moment the fetch happened, not midnight of its day (rank 9, 2026-10-04 gap audit): + // a failed fetch already recorded the real time, so the two sorted against each other + // wrongly. The day stays in `retrieved` on the capture and in the evidence row. + at: nowIso(now), cacheState: result.cacheState, cachedAt: result.cachedAt, }); diff --git a/research-kit/lib/timeline.mjs b/research-kit/lib/timeline.mjs index f39e682e..8f3a2ab3 100644 --- a/research-kit/lib/timeline.mjs +++ b/research-kit/lib/timeline.mjs @@ -46,10 +46,11 @@ export function buildTimeline(root, { corpus = null } = {}) { } /** - * One readable form per moment. The collector stamps fetches at day precision - * (`T00:00:00.000Z`), evidence rows carry a date, and overrides and diagnostics carry a real - * time - the column mixed all three (found 2026-09-27). A day is written as a date, a real - * time as `YYYY-MM-DD HH:MM UTC`. Sorting still uses the raw value. + * One readable form per moment. The collector stamps a fetch with the real time it happened + * (`nowIso(now)`; it wrote midnight of the day until rank 9 of the 2026-10-04 gap audit fixed + * that), evidence rows carry a date, and overrides and diagnostics carry a real time - the + * column once mixed day-precision and real-time values (found 2026-09-27). A bare day is + * written as a date, a real time as `YYYY-MM-DD HH:MM UTC`. Sorting still uses the raw value. */ function when(at) { const text = String(at ?? ''); diff --git a/research-kit/test/collect.test.mjs b/research-kit/test/collect.test.mjs index aad3bbd1..d76e64d4 100644 --- a/research-kit/test/collect.test.mjs +++ b/research-kit/test/collect.test.mjs @@ -1522,6 +1522,26 @@ test('force recollects a search-found fresh capture without changing URL identit } }); +// Rank 9 of the 2026-10-04 gap audit: the ledger's `at` for a scrape was `T00:00:00.000Z` - a +// day written as a millisecond instant - while a failed fetch used the real time, so the two +// sorted against each other wrongly. The moment the fetch happened belongs in `at`; the day +// stays in the capture's `retrieved` and in the evidence row. +test('a scrape is stamped with the moment it happened, not midnight of its day', () => { + const dir = makeProject(); + const when = new Date('2026-10-08T12:34:56.789Z'); + const markdown = `# Limits\n\n${'The free plan allows 10 requests per minute. '.repeat(40)}`; + const outcome = collectOne(dir, 'https://x.invalid/limits', { + runScrape: (u) => ({ ok: true, url: u, title: 'Limits', markdown, statusCode: 200, + transport: 'stub-transport', completeness: 'full', omitted: '', cmd: `stub scrape ${u}` }), + corpus: readCorpus(dir), transportName: 'stub-transport', date: '2026-10-08', now: when, + }); + assert.equal(outcome.status, 'collected', outcome.reason); + const entry = readCorpus(dir).ledger.entries.at(-1); + assert.equal(entry.at, '2026-10-08T12:34:56.789Z', 'the ledger holds the moment, not a midnight placeholder'); + const retrieved = readText(resolve(dir, outcome.entry.file)).match(/^retrieved: (.+)$/m); + assert.equal(retrieved && retrieved[1], '2026-10-08', 'the day the capture was retrieved stays the day'); +}); + test('fresh search hits leave selection room while same-run and plan aliases stay deduplicated', () => { const fresh = searchCacheFixture({ seedDate: '2026-10-08' }); assert.equal(fresh.before.hit, true); From bbd3297386a77e7004fe2d7f1355270545e3f0f1 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <stepenkoanatoli@gmail.com> Date: Sat, 17 Oct 2026 07:52:55 +0300 Subject: [PATCH 06/12] docs: record where the second gap audit's ranks stand at HEAD What changed: `docs/review-2026-10-04-gap-audit.md` gains a section 10 that re-probes every rank against `dd5026f` and records which are closed upstream (1, 2, 10), which one is now fixed (9), which has no open defect as written (6), and which remain open. Why: the audit describes the code at its base `a4ae8f6`, and 253 commits landed since; the outcome table is where its ranks are closed, and the second audit had none. What you verified: each row is a read of the current code - `renderGateWarnings` in `lib/render.mjs` and its call at `lib/brief.mjs:301`; `lib/artifact-validator.mjs:418-421` and `lib/artifact-state.mjs:131-132` for rank 2; `lib/checks.mjs:50-58` for rank 6; `lib/gate.mjs:613` for the integrity set; and `git log -S` naming `d997b85` for rank 10. No code changed, so no suite run is owed and the commit gate runs none. What you got wrong: **mistake** - I planned the bug lane as ranks 9, 10, 11 in order, reading the audit's ranks as open; **where** - the plan made after unit 2, from `docs/review-2026-10-04-gap-audit.md` section 3 recommendation 3; **impact** - nothing was built from it, so no delivered work is affected, but a whole unit would have been spent re-implementing rank 10, which `d997b85` already fixed; **cause** - I trusted the audit's ranks and line numbers without re-probing them against HEAD; **fix** - this re-probe, which establishes the closed and open sets and records them; **verified** - `git log -S "urlKey(e.url) === urlKey(row.url)" -- research-kit/lib/checks.mjs` names `d997b85`, and the open set is now in section 10. --- docs/review-2026-10-04-gap-audit.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/review-2026-10-04-gap-audit.md b/docs/review-2026-10-04-gap-audit.md index 24362711..faf43d5a 100644 --- a/docs/review-2026-10-04-gap-audit.md +++ b/docs/review-2026-10-04-gap-audit.md @@ -272,3 +272,28 @@ question), 3 second (cheapest, two classes, prepares 2). Not to touch, with the evidence: the thirteen-check registry and the one verdict function (do not add a fourteenth check for rank 3; grade at collection); the ledger; ADR-0037's refusal of a thin-capture heuristic; the transport seam's pinned shape; the freeze. + +## 10. Outcome (rank 9 fixed 2026-10-17; the rest re-probed 2026-10-17) + +A rank's line numbers describe the code it was written against, so every rank below was +re-probed against `dd5026f` - thirteen days and 253 commits after this audit's base +(`a4ae8f6`). A rank is closed when the probe finds the defect gone, and a rank the audit +left UNRESOLVED stays unresolved until its named observation is made. + +| Rank | State at `dd5026f` | Evidence | +|---|---|---| +| 1 | **closed upstream** - the drafted brief names the warnings and their count | `renderGateWarnings` (`lib/render.mjs`) writes the count and each warned rule with its row or unknown; `lib/brief.mjs:301` calls it, and the PASS line (`lib/brief.mjs:276-277`) says disclosed unknowns and gate warnings still apply | +| 2 | **closed upstream** - approval is re-derived from the checked corpus, and its blank-Raw half resolves through `captureOf` | `lib/artifact-validator.mjs:418-421` derives `buildAuthorized: status === 'PASS' && claimed` (ADR-0160, `dfaf34c`); `lib/artifact-state.mjs:131-132` counts a row it cannot resolve as not reviewed rather than skipping it | +| 3, 5, 7, 8, 11, 12, 13, 14, 16, 17 | **open** by probe - the token the named remedy would have introduced is absent from `lib/` and `bin/` | no `suspect-capture`, `numPages`/`totalPages`, `page_age`/`sourceDate`, `table-header-renamed` or `pathToFileURL`; `INTEGRITY_RULES` (`lib/gate.mjs:613`) still omits `raw-missing`, `raw-outside`, `raw-unreadable` | +| 6 | **not open as written** - its named remedy is contradicted by a later decision | `lib/checks.mjs:50-58` judges freshness by the capture the row NAMES (`ownCapture`) and deliberately does not fall back to `captureOf`: judged by the URL's latest capture, a row with a blank Raw cell "borrowed a fresh fetch's date, lost its stale warning, and drew a date-mismatch telling the operator to write the later date into the row" (found 2026-10-03, review of G8). Doing what rank 6 asks would restore that defect | +| 9 | **fixed** | `dd5026f`: a fetch's ledger `at` is the moment it happened (`nowIso(now)`), not midnight of its day; the `retrieved` day is unchanged | +| 10 | **closed upstream** | `d997b85 fix(checks): compare evidence URLs by adopted page identity` keys supersession, the duplicate-url rule and the fresher-row hint on `urlKey` | + +Two of these need a decision before they can be built, and neither is a bug fix of a decided +design: rank 3's remedy is the `suspect-capture` hygiene rule, a new kind and so an ADR under +ADR-0117, grading on what the page or the vendor says and never on word counts (ADR-0037); +and rank 5 stays UNRESOLVED because whether the Firecrawl scrape endpoint's JSON carries +`numPages`/`totalPages` (not only the parse endpoint's) is unverified. + +This is a probe of the ranks, not a re-audit: it has re-read the named code, not re-run the +audit's explorers. From 8b4a661b41cdd3919c6b0c7a5075b29b4d93e719 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <stepenkoanatoli@gmail.com> Date: Sat, 17 Oct 2026 07:57:24 +0300 Subject: [PATCH 07/12] docs: record the re-probed audit ranks per clause, not per rank What changed: `docs/review-2026-10-04-gap-audit.md` section 10 records rank 13 as partly closed - the clause that is done (`lib/mcp.mjs:420-427` type-checks array arguments and their items, and the FAIL path already omits the link and sets `isError`), the clauses still open (`resourceLink` munges a string into `file:///` instead of `pathToFileURL(resolve(file))`; an unapproved PASS returns `isError: false`), and a line saying why the rows are recorded per clause. Why: `bbd3297` recorded rank 13 as wholly open from one token probe, but a rank can be half done and reading `lib/mcp.mjs` shows its first clause is already fixed. What you verified: the cited lines were read at `bbd3297` - `lib/mcp.mjs:203-212`, `:368-379`, `:394`, `:420-427`. A docs-only change owes no suite run and the commit gate runs none. What you got wrong: **mistake** - I recorded rank 13 as wholly open; **where** - section 10 of `docs/review-2026-10-04-gap-audit.md`, added by `bbd3297`; **impact** - a reader (or I) would have spent a unit re-fixing a clause already closed; no delivered code is affected; **cause** - I probed for one token the remedy would introduce (`pathToFileURL`) and generalised it to the whole rank without reading the cited lines; **fix** - this row plus the rule that a bundled rank is recorded per clause; **verified** - the lines are cited above and `git log -1` shows this as the correction commit. --- docs/review-2026-10-04-gap-audit.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/review-2026-10-04-gap-audit.md b/docs/review-2026-10-04-gap-audit.md index faf43d5a..4f9c11c9 100644 --- a/docs/review-2026-10-04-gap-audit.md +++ b/docs/review-2026-10-04-gap-audit.md @@ -284,11 +284,17 @@ left UNRESOLVED stays unresolved until its named observation is made. |---|---|---| | 1 | **closed upstream** - the drafted brief names the warnings and their count | `renderGateWarnings` (`lib/render.mjs`) writes the count and each warned rule with its row or unknown; `lib/brief.mjs:301` calls it, and the PASS line (`lib/brief.mjs:276-277`) says disclosed unknowns and gate warnings still apply | | 2 | **closed upstream** - approval is re-derived from the checked corpus, and its blank-Raw half resolves through `captureOf` | `lib/artifact-validator.mjs:418-421` derives `buildAuthorized: status === 'PASS' && claimed` (ADR-0160, `dfaf34c`); `lib/artifact-state.mjs:131-132` counts a row it cannot resolve as not reviewed rather than skipping it | -| 3, 5, 7, 8, 11, 12, 13, 14, 16, 17 | **open** by probe - the token the named remedy would have introduced is absent from `lib/` and `bin/` | no `suspect-capture`, `numPages`/`totalPages`, `page_age`/`sourceDate`, `table-header-renamed` or `pathToFileURL`; `INTEGRITY_RULES` (`lib/gate.mjs:613`) still omits `raw-missing`, `raw-outside`, `raw-unreadable` | +| 3, 5, 7, 8, 11, 12, 14, 16, 17 | **open** by probe - the token the named remedy would have introduced is absent from `lib/` and `bin/` | no `suspect-capture`, `numPages`/`totalPages`, `page_age`/`sourceDate`, `table-header-renamed` or `pathToFileURL`; `INTEGRITY_RULES` (`lib/gate.mjs:613`) still omits `raw-missing`, `raw-outside`, `raw-unreadable` | +| 13 | **partly closed** - the first clause is done and the third is half done; the link is still built by hand | `lib/mcp.mjs:420-427` type-checks array arguments AND their items, and the FAIL path (`lib/mcp.mjs:368-379`) already omits the link and sets `isError`; still open: `resourceLink` munges a string into `file:///` rather than `pathToFileURL(resolve(file))` (`lib/mcp.mjs:203-212` - no percent-encoding, no absolute-ising), and an unapproved PASS returns `isError: false` (`lib/mcp.mjs:394`) | | 6 | **not open as written** - its named remedy is contradicted by a later decision | `lib/checks.mjs:50-58` judges freshness by the capture the row NAMES (`ownCapture`) and deliberately does not fall back to `captureOf`: judged by the URL's latest capture, a row with a blank Raw cell "borrowed a fresh fetch's date, lost its stale warning, and drew a date-mismatch telling the operator to write the later date into the row" (found 2026-10-03, review of G8). Doing what rank 6 asks would restore that defect | | 9 | **fixed** | `dd5026f`: a fetch's ledger `at` is the moment it happened (`nowIso(now)`), not midnight of its day; the `retrieved` day is unchanged | | 10 | **closed upstream** | `d997b85 fix(checks): compare evidence URLs by adopted page identity` keys supersession, the duplicate-url rule and the fresher-row hint on `urlKey` | +A row that bundles clauses is recorded per clause, because a rank can be half done: rank 13's +first clause was already closed when this probe began, and a probe looking only for the token the +remedy would have introduced could not see that. The rows above were read at the cited lines +after that lesson. + Two of these need a decision before they can be built, and neither is a bug fix of a decided design: rank 3's remedy is the `suspect-capture` hygiene rule, a new kind and so an ADR under ADR-0117, grading on what the page or the vendor says and never on word counts (ADR-0037); From 268967838a23ba7fc759b6d17f9786ae9417feaa Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <stepenkoanatoli@gmail.com> Date: Sat, 17 Oct 2026 08:07:00 +0300 Subject: [PATCH 08/12] similarity: fingerprint a page in any script, not only Latin ASCII MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit What changed: `lib/similarity.mjs`'s `shingles` folds a page to words with `[^\p{L}\p{N}\p{M}\s]` instead of `[^a-z0-9\s]`, and composes the text to NFC before that. `test/similarity.test.mjs` gains four tests - a Cyrillic page is fingerprinted and two copies of it are one document, an accented word keeps its letters, canonically equivalent text is one word, and ASCII text shingles exactly as the old rule did. `docs/ARCHITECTURE.md`'s `similarity.mjs` row, `CHANGELOG.md`, `research-kit/README.md`'s test count (1868 -> 1872) and ADR-0163 with its index row carry the decision. Why: rank 17 of `docs/review-2026-10-04-gap-audit.md` (rank 14 of the 2026-10-03 audit). The class was Latin ASCII, so a page in Russian, Arabic, Hebrew, Greek or Devanagari lost every letter to a space: it shingled to nothing, `sketch` returned null for it, and `documentGroups` reads null as "cannot tell" - so a republished copy of such a page was reported as an INDEPENDENT witness, which is the one inversion this module exists to prevent, applied to every corpus not written in the Latin alphabet. Accented Latin was hashed as its ASCII skeleton, and canonically equivalent text (`é` vs `e` + U+0301) was two words spelled two ways. What it touched: `research-kit/lib/similarity.mjs`, `research-kit/test/similarity.test.mjs`, `docs/ARCHITECTURE.md` (the `similarity.mjs` row, in this commit as the standing protocol requires), `docs/adr/0163-a-shingle-is-a-word-in-any-script.md`, `docs/adr/README.md`, `CHANGELOG.md`, `research-kit/README.md`. ADR-0163 records the alternatives set aside: character n-grams now (the threshold and the pinned Jaccard were measured on the word basis, so changing the basis is its own decision with its own measurement), per-script segmentation, NFKC (which folds distinct characters together), and transliterating to ASCII. What I verified: - red-first, verified: with `lib/similarity.mjs` restored to HEAD, `selftest.mjs similarity` reports 11 passed, 3 failed - the three behavioural tests fail and the ASCII guard passes; with the fix, 14 passed, 0 failed. - verified: `selftest.mjs similarity checks corpus` -> 142 passed, 0 failed, 1 unsupported (SYMLINK-NOT-PERMITTED, this host's known limitation, not red per ADR-0108). - verified: root `preflight.mjs` PASS 0 blocking, 1 warning (the pre-existing hygiene/brief-unstamped); `handoff.mjs` exit 0. - expected: no ASCII page's sketch moves - after `toLowerCase` the old class kept exactly a-z, 0-9 and whitespace, which is what these classes keep; the fourth test pins that against the old rule, so `MIRROR_THRESHOLD` and its 2026-09-21 measurement still describe the same corpus. - verified: the full offline suite runs in the conventional commit gate for a `research-kit/` change (~35 min on this machine). What I got wrong and fixed: 1. **mistake** - the first version of the Cyrillic test used `слово0`, `слово1`, ...; **where** - the new non-Latin test in `test/similarity.test.mjs`; **impact** - the ASCII DIGITS in those words survived the old `[^a-z0-9]` class, so the page shingled into `0 1 2 ...`, the assertions all passed, and the test PASSED against the very code it was written to fail on - a vacuous test that would have shipped the fix unproven; no delivered code was affected, because nothing had shipped; **cause** - I chose test data containing the one character class the old rule kept; **fix** - the words are now built from a Cyrillic alphabet with no ASCII anywhere, and the test asserts that no Latin letter or digit appears in the page; **verified** - with the old shingler in place it fails with "every window of a page this long must shingle", and with the fix it passes. - mistake: the replacement data indexed a 30-letter alphabet modulo 33. where: the same test. impact: three indices were `undefined`, so `${undefined}` put the ASCII word `undefined` into a page the test claims is non-Latin; and the guard I wrote for it, `/[\x00-\x7F]/`, is satisfied by the space that `join(' ')` inserts, so the guard failed against the fix and told me nothing about the data. cause: I wrote the modulo from memory instead of from `CYR.length`, and I asserted "ASCII" instead of the class the old rule actually kept. fix: indices come from `CYR.length`, the guard tests `[A-Za-z0-9]`, and the words' distinctness is asserted. verified: `selftest.mjs similarity` -> 14 passed, 0 failed, and re-running against the old shingler shows exactly the three intended failures. - mistake: ADR-0163 was first written with PowerShell `Set-Content`, which added a UTF-8 BOM and CRLF, and the shell channel turned its `é` into a replacement character. where: `docs/adr/0163-a-shingle-is-a-word-in-any-script.md`, first write. impact: an ADR that differed byte-for-byte from every other one in `docs/adr/` and read U+FFFD where it names U+00E9. cause: I wrote a UTF-8 document through a shell string instead of the file tool. fix: written with the file tool. verified: the file is LF, BOM-less, its `é` is U+00E9 and there is no U+FFFD in it. --- CHANGELOG.md | 8 +++ docs/ARCHITECTURE.md | 2 +- .../0163-a-shingle-is-a-word-in-any-script.md | 69 +++++++++++++++++++ docs/adr/README.md | 2 + research-kit/README.md | 2 +- research-kit/lib/similarity.mjs | 16 ++++- research-kit/test/similarity.test.mjs | 60 +++++++++++++++- 7 files changed, 155 insertions(+), 4 deletions(-) create mode 100644 docs/adr/0163-a-shingle-is-a-word-in-any-script.md diff --git a/CHANGELOG.md b/CHANGELOG.md index f07e0634..a4a7c748 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,14 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in ## Unreleased +- Near-duplicate detection reads a **word in any script**: the Latin-ASCII `[^a-z0-9]` class + stripped every non-Latin letter, so a page in Russian, Arabic, Hebrew, Greek or Devanagari + shingled to nothing, `sketch` returned null for it, and a republished copy of it was reported + as an independent witness - the one finding this module exists to make, inverted. Accented + Latin no longer collapses to its ASCII skeleton, and canonically equivalent text is one word + (rank 17, `docs/review-2026-10-04-gap-audit.md`; rank 14 of `docs/review-2026-10-03-gap-audit.md`). + ASCII input is unchanged, so `MIRROR_THRESHOLD` still rests on the corpus it was measured + against (ADR-0163). - The collector stamps a fetch's ledger `at` with the moment it happened instead of midnight of its day; a failed fetch already did, so the two sorted against each other wrongly, and the day the capture was retrieved is unchanged in `retrieved` (rank 9, diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 994cfdfe..7b2d43f2 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -228,7 +228,7 @@ is an open decision (2026-09-30): | `warc.mjs` | **The corpus as WARC 1.1** (2026-09-28, ADR-0090): `exportWarc(root)` - a `warcinfo` record, then per ledger `scrape` entry whose capture is on disk and unchanged a `resource` record (the text after the front matter, `text/markdown`) and a `metadata` record (`WARC-Concurrent-To`; transport, completeness, the capture file's hash, the block's own `body-sha256` and the ledger's seq); `WARC-Block-Digest: sha256:`, record IDs derived from the ledger entry (`uuidFrom`, RFC 9562 v8), each record its own gzip member; an edited capture is left out and named in `skipped`, as is a ledger entry the export cannot date (WARC-Date is mandatory; rank 7, 2026-10-03 gap audit, ADR-0161). `readWarc` splits a stream back into records and throws on bad framing. Validated with warcio on every corpus. `exportWarc` reads a ledger-named capture only if it is inside the project (`isInside` and `realInside`, ADR-0076) and a regular file (2026-09-30); anything else is `skipped` with its reason. A ledger naming `../` or a link out of `research/raw` had packed that outside file into the archive, and a FIFO would have hung the export. `exportWarc` reads the path through `projectFile` (2026-09-30, ADR-0109). The record is parsed from the bytes whose hash passed, never a second read of the file (2026-09-30). | ADR-0090 | | `prior.mjs` | **The registered prior: what was expected BEFORE the evidence, and the proof that it was.** `registerPrior` writes `research/PRIOR.md` and chains an `op: 'prior'` entry carrying its sha256, so two things stop being editable at once - the text, through `body-unmodified`, and its POSITION before the first scrape, through the chain itself. Three dishonest paths, each measured rather than argued: revising the prediction once the answer is known fails the chain; registering late is refused by the tool; and chaining one late BY HAND with every hash correct is caught by `provenance/prior-precedes-collection`, which exists because `appendFetch` is a public export and the gate does not get to trust that the CLI was used. **Nothing grades the prediction** (ADR-0013) - not its specificity, not its reasonableness, and above all not whether it turned out right, since a prior that was wrong has done its job and a gate that preferred correct ones would teach the opposite habit. The single content rule is a length floor, stated as a floor. Absence is SILENT at the gate: a prior is optional, and a warning on every project that skipped one is a warning nobody reads. `bin/prior.mjs` registers and reads it back; `bin/research.mjs` prints a one-line nudge before a spending run, because after the first page lands there is no second chance. **It also reaches the REMOTE path, and that half is what makes the rest useful**: the collector workflow scaffolds its own project on the runner with an empty ledger, so a prior registered on the operator machine cannot reach the corpus that comes back - and every corpus collected since `collect-remote` shipped came from that route. The prediction therefore travels as a dispatch input and is registered on the runner between the map and the collection, through `env` and an argv array like the topic (ADR-0020), with the topic's measured disclosure profile (ADR-0035). A test pins that the registration step precedes `collect`, because if it ever drifts below it, `bin/prior.mjs` refuses and the feature stops working silently on a workflow CI never runs. | ADR-0013, ADR-0020, ADR-0039 | | `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. `partial-render` applies only to a capture behind an EVIDENCE row some unknown cites (ADR-0099): on a context-only row its instruction had no cited text to confirm. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. The same duty covers what a PASS claims: `gate-integrity` used to pass with "no override in effect" when what it measured was that no override was RECORDED - and `git commit --no-verify`, the escape hatch this kit documents most prominently, records nothing. It now says what it counted and names the bypass it cannot see. Found by sweeping all thirteen pass messages for claims wider than their measurement; it was the only one. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. `collection-attempts` counts an attempt through the rows a KNOWN-UNKNOWN cites, not only a URL typed into it (2026-09-26): an E-row is a capture of its URL, so citing one is the stronger proof of having reached for it. Until then a KNOWN-UNKNOWN citing the page it tried was warned as never attempted - found by the `2026-09-26-actions-sept-changes` project's own gate, and no test exercised its verdict (only the registry-order test named it). `citations/url-mismatch` (2026-09-27) fails a row whose URL is not, by `urlKey`, the page its capture was fetched from: a row could name another page and pass, and the URL cell is what a reader is shown as the source. `provenance/ledger-unparsed` names its repair (2026-09-27): a torn last line gets `doctor --fix-arity`, which drops exactly that; a broken line inside the chain is restored from git. It gave only the JSON error. A row is never compared with itself (2026-09-27): `supersededRows` skips a same-ID pair and `hygiene/duplicate-url` leaves the same ID to `duplicate-id`; a row pasted twice failed with "E-01 has been superseded by E-01". `hygiene/future-date` fails a Retrieved date more than a day ahead (2026-09-27): 2030-01-01 had passed, hiding the row's age from every freshness check. `hygiene/duplicate-id` fails rather than warns (2026-09-27, ADR-0054): two rows under one ID leave every citation, COVERED cell and brief reference to it ambiguous, because lookups keep the last row. A contract with two U-1 rows had passed. `corroboration` weighs each unknown ID once, so it no longer prints the same single-source warning twice. `hygiene/brief-stale` warns when a stamped brief's inputs hash no longer matches the corpus (2026-09-27, ADR-0055) and names the command that redrafts it; `hygiene/brief-unstamped` warns on a drafted brief with no stamp (2026-10-03, ADR-0138) - its currency cannot be checked and it keeps its approval, so the gate says so and names the redraft that stamps it; a template or a hand-written legacy brief gets no such warning. `hygiene/duplicate-url` treats two same-day rows for one URL as a refresh when the ledger records a fetch for each (2026-09-27, ADR-0060); a pasted row, with one fetch behind two rows, is still a duplicate. `discovery-contract/build-intent` tells a missing heading from an empty one, from the corpus's `intentHeading` (2026-09-27): a renamed heading read as "is empty". **`citations` checks quote anchors** (2026-09-28, ADR-0087): each `[quote: ...]` in a Finding must occur in the row's capture (`quote-not-found`, blocking) and carry at least three words or 40 characters in EVERY fragment (`quote-too-short`, warn; the character floor is ADR-0126, 2026-10-02; per-fragment measurement and the one-passage bound are rank 4 of the 2026-10-03 gap audit); the matching lives in `quotes.mjs`. `citations/raw-error-status` warns when a row cites an HTTP 4xx/5xx capture from before that refusal, unless the row names the status itself (this repository's E-15 keeps a 404 as the record of a failed guess). **`unknown-closure/secondary-only`** (2026-09-28) warns when a CLOSED unknown cites no `P` row and at least one `S` row: the registry promised "a primary row" and only `L` was checked, so three corpora closed everything on `S` rows with no warning (found by the first measurement). Retrieval dates compare by `compareText` (2026-10-01). `runChecks` appends each check's findings one at a time (2026-10-01, break-test PR #188): `push(...check.run())` passed every finding as an argument, and past the engine's limit - 60,000 to 130,000 findings from one check, by Node line - the verdict died with a raw RangeError; preflight, doctor, brief and audit crashed, and the commit gate answered internal error. Reachable from about 66,000 evidence rows. The other spread sites (`priorOrder`, doctor's `gateHealth`) are bounded by the contract's unknown count and left as they are. **`hygiene` fails a malformed ID** (2026-10-03, audit G2): each `malformed-id` problem the corpus reader records becomes a `fail` beside `duplicate-id`, for the same reason - the ID is how every check finds a row; `corpus-shape` skips the kind so one typo is not reported twice. A `table-split` problem fails the same way (2026-10-03, gap audit rank 1): the row no check reads is named with its file and line, and `corpus-shape` does not repeat it. **Freshness is judged by the capture's own date** (2026-10-03, audit G8): `unknown-closure` ages a cited row by `freshnessDate` - the capture's front-matter `retrieved`, written by the collector at the fetch and under the ledger's hash, with the table's cell as the fallback for a row whose capture carries no parseable date - and names the judging date when the two differ; `hygiene` warns `date-mismatch` when a row's cell and its capture name different calendar days (`retrievedDay`), with the remedy of writing the capture's date in the row. Editing E-19's cell alone had made its stale warning disappear while the capture on disk still said 2026-09-20. The capture judged is the one the row NAMES (`ownCapture`, review of G8 the same day): `captureOf` answers a blank Raw cell with the URL's latest capture, and judged by that a row with no Raw cell borrowed a later fetch's date, lost its stale warning and drew a date-mismatch telling the operator to write the later date in; a row naming no capture is judged by its cell and has nothing to reconcile against. A future date in the capture fails `future-date` as one in the cell does, naming the capture, and the stale hint's "fresher capture of the same URL" is found by capture date (`freshnessDay`), not by the cells. **Same-day supersession follows the ledger** (2026-10-03, audit G4): `supersededRows` orders two rows of one URL by the day freshness judges them by (`freshnessDay` - the capture's date; ordered by the table cell, one edit to an older row's date had failed the gate on the genuinely newest row and told the operator to cite the older one; review of G4, the same day) and then by the ledger position of each row's cited capture (`ledgerRank`, the rule `newer` applies for the collector); a capture no fetch recorded ranks below every fetched one, and only among those does the table's order still break the tie. With A fetched, then B, then A again, the collector had held A current while the check said B superseded it, and swapping two evidence rows swapped the verdict. `uncited-capture` says "was collected" only of a file a ledger scrape entry names; a file under research/raw no fetch produced is named as such (2026-10-04, break-test pass 4 F5a). `discovery-contract/contract-missing` states that the contract is missing or could not be read (2026-10-08): the reader uses the same null fallback for absence and refused reads, so that finding cannot establish absence. Both states keep the existing failing rule; no filesystem-state API or new check is introduced. **Evidence URL comparisons use the adopted page identity** (2026-10-09, audit rank 14): supersession groups, stale-citation fresher-row hints and duplicate-URL hygiene compare through `urlKey` (ADR-0136), as the collector and cache already do. Capture dates, blank-Raw table-date fallback, same-day ledger ordering, the same-ID exclusion and the two-recorded-fetch refresh rule retain their existing roles. Captured URL spellings remain visible; query strings and path case remain distinct. | ADR-0004, ADR-0026, ADR-0087, ADR-0136 | -| `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 | +| `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped, and **a word means a word in any script**: `\p{L}\p{N}\p{M}` and NFC composition, not the Latin-ASCII `[^a-z0-9]` class, which stripped every non-Latin letter and left a Russian, Arabic, Greek or accented page with NO shingle at all - so `sketch` returned null for it and a republished copy counted as an independent witness, rank 17 of the 2026-10-04 gap audit; ASCII input is byte-identical, so the measurement below still holds) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 | | `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 | | `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, **what phase 1 may commit** (ADR-0048: `research/`, the root `SCAFFOLDING` - `.gitattributes`, `.gitignore`, `AGENTS.md`, `START_HERE.md` - and `docs/ARCHITECTURE.md` while `isEmptyArchitectureMap` says its staged bytes, read by `readStagedText`, are still the template's; until 2026-09-27 only `research/`, so a new project's first `git add -A` was refused and its `.gitattributes` could not travel), and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **Staged paths remain data, with checked acquisition** (ADR-0154): the default commit CLI lazily calls `readStagedPaths` after the ungated and `GATE_OFF` returns. One actual `git diff --cached --name-only --no-renames -z` supplies paths only on successful complete output; the existing 64 MiB capture rejects nonzero exit and buffer failure, and the reader rejects malformed NUL data. Successful zero bytes means `[]`; unreadable scope means null and raises the existing architecture-map breach even after green preflight, ordinary exit 1 under either posture. Explicit arrays/null and the existing stdin/repeated `--staged` options retain their meaning; edits never acquire a list. **One deliberate exception to "no CLI parsing"** remains for explicitly supplied stdin data (`stagedPathsFromStdin` / `splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"**, preserving unknown scope rather than assuming an empty set. No list read claims an atomic snapshot with later index materialization. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). The architecture-map block's fix says to **update** the map, then `git add` it (2026-09-27): a bare `git add docs/ARCHITECTURE.md` stages nothing while the map is unchanged, so the printed fix left the commit blocked. `bin/gate.mjs` no longer closes that block with "Phase 1 is not done until preflight prints PASS" - a map block happens only when the gate passes. `isPhaseOneEdit(root, file)` answers the same question for the edit hook, about one file. The empty-index verdict — a gated project whose `research/` is untracked, every fresh project's first commit — removes the scratch dir `materializeIndex` made before it learned the index was empty (2026-09-28): it returned before `scratch` was assigned, so the `finally` that removes it never ran, and every such verdict leaked one temp directory. **Integrity before scope** (ADR-0093): a failing verdict whose findings include an `INTEGRITY_RULES` provenance rule blocks the commit even when it is confined to `research/` - the ADR-0048 allowance covers completeness findings only. `stdinIsReadable(stdin = process.stdin)` takes the stream to ask about, so the hook's terminal case is testable (2026-10-01). **The suite rule** (ADR-0120, 2026-10-01): `suiteBreach(root, stagedPaths, { run })` runs after the map rule, in the commit gate only, and only where `isKitCheckout` finds the kit's four markers and a staged path is under `research-kit/` (or the list is unknown); `runSuiteHere` spawns the checkout's `bin/selftest.mjs` with `RESEARCH_KIT_RESULT_FILE` pointed at a scratch file and `RESEARCH_KIT_ALLOW_UNSUP=1` (unsupported is not red, ADR-0108), git's hook environment stripped (`GIT_INDEX_FILE`, `GIT_DIR`, `GIT_PREFIX` and kin - the same `REPOSITORY_LOCATION` list as the edit hook, else every scratch repository's git in the suite would act on the index being committed), and reads the file, never stdout: failures block naming the counts, no result blocks as a crashed runner, a run past its budget is stopped and blocks as unfinished, the fix naming `RESEARCH_KIT_GATE_TIMEOUT` before the override, and a green run whose runner exited non-zero (the README count check) blocks too. `suiteOwed(root, stagedPaths)` is the predicate alone; `suiteBreach`'s `announce` (`evaluate`'s `announceSuite`) is called right before the run, so `bin/gate.mjs` says the suite is running only when it is - not when the map rule or preflight refused the commit first. `evaluate` takes `runSuite` so the verdict is testable without a suite. **The suite's budget follows the gate's watchdog** (ADR-0142, 2026-10-05): `suiteBudgetMs(env)` is `SUITE_TIMEOUT_MS` (65 min, twice the slowest honest run measured, 1825 s on the operator's Windows PC) unless `RESEARCH_KIT_GATE_TIMEOUT` is set to a positive duration `gateTimeoutSeconds` reads (seconds, an optional fraction, an optional `s`/`m`/`h`/`d`, at most nine digits before the point - the grammar the hook's `gate_timeout_readable` checks too; `0` and anything outside it leave the default), and then that watchdog less `GATE_REST_S` (120 s, the hook's default watchdog for a gate that runs no suite), never more than half a short one, at least 1 ms and at most 2^31 - 1 - so the gate stops the suite and blocks before the hook's watchdog fires, whenever its own work fits what is left. `runSuiteHere` stops the runner with SIGKILL: the harness's SIGTERM listener kept a synchronously blocked runner alive past its budget on POSIX; a stopped run's scratch is left in the temp folder. The budget had been 20 minutes, which that PC's green suite met on every kit commit, and the variable had reached only the watchdog: set below the budget, the watchdog killed the gate first and a fail-open machine allowed a commit whose suite never reported. **The fix lines no longer say `--no-verify` is recorded** (2026-10-03, outside audit): git skips this hook, so nothing can record it; `checks.mjs` had said so since ADR-0035 while the gate's own fix text, the README, AGENTS.md and CONTEXT.md said the opposite. **Red suite diagnostics** (2026-10-08): `suiteBreach` forwards up to five `failed` identifiers, each limited to 160 characters with controls normalized, in its existing detail string; omitted or clipped data is named, and legacy missing labels are explicit. Counts and verdict keys are unchanged. `runSuiteHere` still removes its scratch result; no result retention system is added. | ADR-0002, ADR-0007, ADR-0008, ADR-0020, ADR-0093, ADR-0120, ADR-0142, ADR-0154 | | `doctor.mjs` | Read-only diagnostics: machine + project + gate + chain health in one report. Consumes the verdict context (`verdictContext` / `readGateState`) for the chain-verified corpus and the gate-state triple; adds one machine read of its own (the git-resolved `hooksPathEffective`) for report. **Role-aware** (ADR-0010): the Firecrawl findings are informational on a `builder`, and the handoff findings (ADR-0011) are blockers there — the collection entrypoints' refusal is what makes that split honest. The handoff findings come from the owner, not from a second composition: doctor passes the corpus it already read to `verifyHandoff(root, { corpus })`, so the arrival question has one implementation and one set of pins. One deliberate write: the repository-local hooksPath override log (the third override is silent by nature — the bypassed hook cannot report it). `gateHealth` and `runDoctor` accept explicit Git/config/settings paths and a child environment so diagnostics can run against disposable machine fixtures without consulting host state. **`commitGateState`** asks WHOSE pre-commit is at `core.hooksPath` - `current` / `foreign` / `unusable` - because executable is not the same as ours. It used to ask only whether some executable hook was there, the same existence-for-correspondence mistake the `deploy` check made; `settingsState` has asked the question of the EDIT gate since a registration here named our hook inside a different checkout, and `core.hooksPath` is a single machine-wide setting that husky, lefthook and pre-commit all rewrite. Pure and separate from `gateHealth`, which reads real git config - a rule testable only by fishing a finding out of an array on a host-dependent path is a rule nobody tests (ADR-0004), which is why the older test for it asserts the severity is one of three. **Ownership is decided before executability**, because the first version asked about the mode first and so reported a non-executable FOREIGN hook as `unusable` with `chmod +x` beside it - advice to repair somebody else's gate while leaving this one just as absent. That ordering bug was invisible on Windows, where `hookExecutability` returns ok for any file that exists, and surfaced on Linux in CI. On a **collector**, handoff findings are `info` in the collector's words ("does not exist yet - nothing has been collected here; a builder handed this corpus would be refused"), since 2026-09-26: they were graded `pass` with the builder's wording, so a fresh project printed "pass ... the corpus arrived without its ledger". On a builder they remain blockers, worded as before. It reports the **Node line** and, when a proxy variable is set, whether this Node sends the kit's requests through it A collector without the Firecrawl CLI is told the exact install command, `npm install -g firecrawl-cli@<tested>` from `cliInstallSpec()` (2026-09-27; it said "install the Firecrawl CLI" and nothing anywhere said how). (`proxy`: keyless pages, search requests and remote collection; a warning naming the version to move to - or, for a value Node cannot parse, the spelling it needs - since the Firecrawl CLI has its own proxy handling) - measured on 22.20.0, where the fetch went around the proxy (ADR-0046, ADR-0047). A collector that explicitly chose a keyless or browser transport passes the Firecrawl checks; nothing chosen still fails (ADR-0095). The Firecrawl fixes (`firecrawl-cli`, `firecrawl-auth`) name the no-account route beside the install or login - `"transport": "http-keyless"` in the machine config, the setting that makes them pass (ADR-0095) - where the CLI fix had offered only a per-run flag and the login fix nothing (2026-09-29). `editGateState` reads an entry whose `hooks` is not a list, or a hook that is `null`, as naming nothing (2026-09-30); `[null]` had thrown a `TypeError`. `scanForSecrets` catches a folder it cannot read where it reads it, and skips a folder it has already walked by real path (2026-09-30): one try/catch around the whole walk had ended the scan at the first unreadable folder, leaving every file after it unscanned and reporting nothing found. `scanForSecrets` records a cap stop separately from the scanned count (2026-10-08, bounded-scan audit repair): it keeps the 2000-text-file limit and reports incomplete coverage only when that cap actually prevents further traversal; ending at the final file's boundary is not itself incomplete. `runDoctor` reports a capped scan as a `secret-scan` warning, beside any unchanged critical `secret` hits, instead of reporting a capped no-hit scan as passing. It names no unobserved remaining-file count and makes no whole-repository clearance claim. `bin/doctor.mjs` reports `KIT_VERSION`, as `kitVersion` in `--json` and on its last line (ADR-0116). Reports each ill-typed posture key as a `config-ill-typed` warning naming the key, the values the kit reads, and the state it resolved to (ADR-0121). The `firecrawl-cli` finding judges the installed version with the run's own rule (`cliCompatibility`): an unsupported major is a **fail** with the remedy on a collector that needs the CLI and information elsewhere, a newer minor passes and says what it was tested against - doctor had printed `pass 2.0.0` for a CLI the run then refused (2026-10-02, break-test pass 3). `scanForSecrets` skips a binary file and counts it (`skippedBinary`, named in the coverage line) instead of matching credential patterns against an executable's string table (ADR-0128, 2026-10-02): run on a Windows home folder it reported fourteen committed keys, every one the PEM marker OpenSSH and libssh2 parse, or random bytes spelling `sk-`. **`handOnState`** (2026-10-03, ADR-0134; since 2026-10-04 defined in `installer.mjs` and re-exported here, ADR-0141, so the install judges what it records by the same rule): what the hooks hand on to - none, missing, kit, other - and `gateHealth` warns on kit and missing; the maintainer's machine handed every commit on to the previous implementation's gate, which crashed and failed open while doctor said READY. `handOnState` resolves a relative folder from the repository's top level (`base`), where git runs its hooks from; doctor had resolved `.custom-hooks` against its own working directory and from a subdirectory called a working hand-on missing (fourth audit round, 2026-10-03). `scanForSecrets` skips `research/raw` at ANY depth (2026-10-04): the skip matched the relative path `research/raw`, so only the root's captures were outside the scan and a nested decision project's were read as the project's own files - CI tripped `google-api-key` on a Google page's source sibling (ADR-0140), Google's own public keys; a capture is page content, and the folder exists wherever a nested project does (ADR-0030). The shared unusable-proxy check also rejects proxy URLs carrying a non-root path, query or fragment, so those configurations warn instead of promising proxy use (#380, 2026-10-12). `machineHealth` uses the shared `envProxyChoice` with injectable `execArgv` (#381, 2026-10-12). An explicit disabled choice warns that requests go direct and names how to enable the proxy, rather than promising proxy use from Node version support alone; the raw variable value is never echoed. | ADR-0010, ADR-0011, ADR-0095, ADR-0134 | diff --git a/docs/adr/0163-a-shingle-is-a-word-in-any-script.md b/docs/adr/0163-a-shingle-is-a-word-in-any-script.md new file mode 100644 index 00000000..d26b7fd6 --- /dev/null +++ b/docs/adr/0163-a-shingle-is-a-word-in-any-script.md @@ -0,0 +1,69 @@ +# ADR-0163 - A shingle is a word in any script, and text is compared as it was written + +- **Date:** 2026-10-17 +- **Status:** accepted +- **Area:** `research-kit/lib/similarity.mjs` +- **Supersedes nothing.** The word basis, the sketch size and `MIRROR_THRESHOLD` are kept as + they were decided and measured; what changes is which characters count as a word. + +## Context + +`shingles` folded a page to words with `[^a-z0-9\s]` -> space. That class is Latin ASCII. So: + +- A page in **any other script** - Russian, Arabic, Hebrew, Greek, Devanagari - lost every + letter to a space. The page shingled to **nothing**, `sketch` returned `null` for it, and + `documentGroups` reads `null` as "cannot tell" rather than "different", so a republished + copy of such a page counted as an **independent witness**. That is the exact inversion this + module exists to prevent (rank 17, `docs/review-2026-10-04-gap-audit.md`; rank 14 of + `docs/review-2026-10-03-gap-audit.md`), and it applied to every corpus not written in the + Latin alphabet. +- **Accented Latin** collapsed to its ASCII skeleton, so a page's words were hashed as + something the page did not say - the accent was not noise, it was dropped. +- **Canonically equivalent text** was two words spelled two ways: `é` (U+00E9) and `e` + U+0301 + are the same character and shingled differently. + +## Decision + +A word is a maximal run of `\p{L}` (letters), `\p{N}` (numbers) and `\p{M}` (combining marks - +they carry the vowels of Devanagari and Arabic), in any script; everything else is a separator, +as punctuation always was. Text is composed to **NFC** before that, so canonically equivalent +input is one word rather than two. + +ASCII input is untouched, and that is load-bearing rather than lucky: after `toLowerCase`, the +old class kept exactly `a-z`, `0-9` and whitespace, which is what these keep too. No ASCII +page's shingles move, so `MIRROR_THRESHOLD = 0.25` and the pinned exact Jaccard (0.3879 for the +known stale mirror against `nodejs.org`, 0.0275 for the highest genuinely different cross-host +pair, both measured 2026-09-21) still describe the corpus they were measured on. +`test/similarity.test.mjs` pins that equivalence against the old rule. + +## Rejected alternatives + +- **Shingle over characters (character n-grams) instead of words.** This is the robust, + language-agnostic basis and it would also fingerprint a script that does not separate words + with spaces at all - Chinese, Japanese, Thai - where a whole page is one token and still + shingles to nothing. Refused **for now**: the threshold and the pinned Jaccard were measured + on the word basis, so changing the basis changes what those numbers mean and it needs its own + measurement over real non-Latin pairs, not a ride-along in a bug fix. +- **Segment scripts without word spaces** (split CJK ideographs, dictionary-segment Thai). A + per-script rule this module cannot measure from the corpus it has; and it would silently give + those pages a sensitivity nobody has checked, which is the failure mode this module's + measured constants exist to prevent. +- **NFKC normalisation.** It folds **distinct** characters together (`①` into `1`), inventing an + equivalence the page did not have. NFC composes only canonically equivalent sequences, so it + cannot create a false match - it can only recover a true one. +- **Transliterate to ASCII, or write the limitation down and keep it.** That is what the code + did. It loses the text: a page in another script was not merely unjudgeable, it was reported + as a second independent witness, which is the best grade in the registry for one document. + +## Consequences + +A page in a space-separated script is now fingerprinted, and two copies of one are one document. +A page written in a script with **no word separators and no punctuation** still shingles to +nothing and stays unjudgeable. That is recorded, not hidden, as a KNOWN-UNKNOWN: it needs a +measured basis of its own, and the trigger to take it is a corpus in such a script that needs +duplicate detection. + +A page whose text is accented Latin may move slightly, because its words are now themselves +rather than their ASCII skeleton; `mirror` is a finding a reviewer reads, never a gate failure +(ADR-0108). The root corpus and every project under `docs/decisions/` keep their grades, +because their captures are ASCII. diff --git a/docs/adr/README.md b/docs/adr/README.md index 50eb1e31..76a0bc89 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -179,4 +179,6 @@ directory and are treated as ADR-equivalent. | [0161](0161-a-warc-block-is-verifiable-beside-the-capture-that-names-it.md) | A WARC export names the resource block's own hash (`body-sha256`) beside the whole capture file's, leaves out a ledger entry it cannot date (WARC-Date is mandatory) instead of writing an empty one, exits 2 (`INCOMPLETE`) on a partial export, and re-reads the written file (rank 7, 2026-10-03 gap audit; rank 15, 2026-10-04 gap audit) | | [0162](0162-a-quote-is-one-passage-read-at-any-bracket-depth.md) | A `[quote: ...]` marker is read at any bracket depth (a two-level passage was not a marker at all, so it went unchecked), its fragments must lie in one passage (`MAX_QUOTE_GAP`, tried at each occurrence), and the short-quote floor is measured per fragment rather than over the joined text (rank 4, 2026-10-03 gap audit) | +| [0163](0163-a-shingle-is-a-word-in-any-script.md) | A shingle is a **word in any script**: `\p{L}\p{N}\p{M}` and NFC replace the Latin-ASCII `[^a-z0-9]` class, so a page in Russian, Arabic, Hebrew, Greek or accented Latin is fingerprinted instead of shingling to nothing - which made a republished copy of it report as an independent witness, the exact inversion this module exists to prevent (rank 17, 2026-10-04 gap audit; rank 14, 2026-10-03). ASCII input is byte-identical, so `MIRROR_THRESHOLD` and its measurement still hold. Rejected: character n-grams now, per-script segmentation, NFKC, transliterating. | + Domain vocabulary lives in [`CONTEXT.md`](../../CONTEXT.md). diff --git a/research-kit/README.md b/research-kit/README.md index 4c4a800d..16d9bd9c 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -568,7 +568,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -1868 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +1872 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/lib/similarity.mjs b/research-kit/lib/similarity.mjs index dcf8fb72..b8911e54 100644 --- a/research-kit/lib/similarity.mjs +++ b/research-kit/lib/similarity.mjs @@ -20,8 +20,22 @@ export function shingles(text, n = SHINGLE_WORDS) { const words = String(text ?? '') .toLowerCase() + // Canonically equivalent text is the SAME text: `é` and `e` + U+0301 are one word spelled + // once or twice, and composing first is what keeps the combining marks below from making + // one word into two different ones. NFC only - NFKC is deliberately NOT used, because it + // folds characters a page did not equate (`①` into `1`), and this measurement must not + // invent an equivalence the text does not have. + .normalize('NFC') .replace(/https?:\/\/\S+/g, ' ') - .replace(/[^a-z0-9\s]+/g, ' ') + // A WORD, in any script (rank 17, 2026-10-04 gap audit). This class was `[^a-z0-9]`, so a + // page in Russian, Arabic, Hebrew, Greek or Devanagari lost every letter to a space: the + // page shingled to nothing, `sketch` returned null for it, and `documentGroups` read a + // republished copy of it as an independent witness - the one finding this module exists to + // make. `\p{L}` letters, `\p{N}` numbers, `\p{M}` the combining marks that carry the vowels + // of Devanagari and Arabic. ASCII input is untouched, and that is not luck: after + // `toLowerCase` the old class kept exactly a-z, 0-9 and whitespace, which is what these + // keep, so no ASCII page's sketch moves (test/similarity.test.mjs pins that). + .replace(/[^\p{L}\p{N}\p{M}\s]+/gu, ' ') .split(/\s+/) .filter(Boolean); const out = new Set(); diff --git a/research-kit/test/similarity.test.mjs b/research-kit/test/similarity.test.mjs index 94f15438..2c8f4b0e 100644 --- a/research-kit/test/similarity.test.mjs +++ b/research-kit/test/similarity.test.mjs @@ -16,7 +16,7 @@ import { test, describe, assert, fs, path } from './harness.mjs'; import { parseCapture } from '../lib/corpus.mjs'; import { sketch, similarity, shingles, documentGroups, isMirror, closestPair, - MIRROR_THRESHOLD, MIN_SHINGLES, + MIRROR_THRESHOLD, MIN_SHINGLES, SHINGLE_WORDS, } from '../lib/similarity.mjs'; describe('similarity'); @@ -137,3 +137,61 @@ test('closestPair is REPORTED, not acted on - the verdict does not move with it' // Two distinct documents stay two distinct documents whatever the number says. assert.equal(documentGroups([canonical, other]).length, 2); }); + +// Rank 17 of the 2026-10-04 gap audit. The shingler kept `[a-z0-9]`, so a page in any other +// script lost every letter to a space, came back with NO shingles, and `sketch` returned null +// for it - which every caller reads as "cannot tell", so a republished copy of a Russian or +// Arabic page counted as an independent witness. That is the one finding this module exists +// to make, inverted for every corpus not written in the Latin alphabet. +test('a page in a script that is not Latin is fingerprinted, and two copies of it are one document', () => { + // No ASCII character anywhere in this page. An earlier version of this test used `слово0`, + // `слово1` - and the ASCII DIGITS survived the old `[^a-z0-9]` class, so the page shingled + // into `0 1 2 ...` and the test passed against the very code it was written to fail on. A + // version after that indexed a 30-letter alphabet modulo 33, so `${undefined}` put the ASCII + // word `undefined` into the page. Hence the assertion below, which makes the vacuous case a + // failure instead of a pass. + const CYR = 'абвгдежзийклмнопрстуфхцчшщыэюя'; + const words = Array.from({ length: 80 }, (_, i) => `${CYR[i % CYR.length]}${CYR[Math.floor(i / CYR.length)]}${CYR[(i * 5) % CYR.length]}`); + const page = words.join(' '); + assert.ok(!/[A-Za-z0-9]/.test(page), 'no Latin letter or digit may appear anywhere in this page, or the old [^a-z0-9] class would have had something to hash and the test would prove nothing'); + assert.equal(new Set(words).size, 80, 'the words must be distinct, or the shingle count below is wrong'); + assert.equal(shingles(page).size, 80 - SHINGLE_WORDS + 1, 'every window of a page this long must shingle'); + const copy = words.slice(); + copy[40] = 'изменение'; + const a = sketch(page); + const b = sketch(copy.join(' ')); + assert.notEqual(a, null, 'a Cyrillic page must be judgeable at all'); + assert.equal(isMirror(a, b), true, 'two copies of one page are one document, not two witnesses'); +}); + +test('an accented word keeps its letters instead of collapsing to its ASCII skeleton', () => { + const words = Array.from({ length: 30 }, (_, i) => `café${i} über${i} naïve${i}`); + const set = shingles(words.join(' ')); + assert.ok(set.size > 0, 'accented Latin text must shingle'); + assert.ok([...set].some((s) => s.includes('café0')), 'the word must survive with its accent'); + assert.ok(![...set].some((s) => s.includes('caf0')), 'the stripped form must not be what is hashed'); +}); + +test('canonically equivalent text is one word, composed or decomposed', () => { + const composed = Array.from({ length: 60 }, (_, i) => `café${i}`).join(' '); + const decomposed = Array.from({ length: 60 }, (_, i) => `cafe\u0301${i}`).join(' '); + assert.notEqual(sketch(composed), null); + assert.equal(similarity(sketch(composed), sketch(decomposed)), 1, 'é and e + U+0301 are the same word'); +}); + +// The guard for the two fixes above. Every page that is plain ASCII must shingle EXACTLY as the +// ASCII-only rule did, because MIRROR_THRESHOLD and the pinned Jaccard in this file were +// measured on that: after `toLowerCase`, `[^a-z0-9\s]` kept exactly a-z, 0-9 and whitespace, +// which is what the Unicode class keeps too. If this test ever fails, the threshold no longer +// rests on the corpus it was measured against. +test('ASCII text shingles exactly as the ASCII-only rule did', () => { + const sample = 'The Free Plan allows 10 requests per minute - see https://x.invalid/limits for the exact wording of the limit.'; + const oldRule = String(sample).toLowerCase() + .replace(/https?:\/\/\S+/g, ' ') + .replace(/[^a-z0-9\s]+/g, ' ') + .split(/\s+/) + .filter(Boolean); + const expected = new Set(); + for (let i = 0; i + SHINGLE_WORDS <= oldRule.length; i += 1) expected.add(oldRule.slice(i, i + SHINGLE_WORDS).join(' ')); + assert.deepEqual([...shingles(sample)].sort(), [...expected].sort()); +}); From 0d29248ef8b2ae76f97e6a6de62c7ff938fa0082 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <stepenkoanatoli@gmail.com> Date: Sat, 17 Oct 2026 08:31:31 +0300 Subject: [PATCH 09/12] docs: record rank 17's outcome in both gap audits What changed: `docs/review-2026-10-04-gap-audit.md` section 10 records rank 17 as partly fixed - the non-Latin shingle clause in `2689678`, with the clauses still open (the 23-entry suffix list, lenient `Date.parse`, a string `maxAgeDays`, and the gate's integrity set) - and its heading says so; `docs/review-2026-10-03-gap-audit.md` section 9 gains the row for its rank 14, the same defect, and its closing line no longer lists 14 as unimplemented. Why: the outcome tables are where a rank is closed, and both audits carry this defect - rank 17 of the 2026-10-04 audit, rank 14 of the 2026-10-03 one. What you verified: the commit it records is `2689678` (read from `git log`), whose gate ran the full offline suite and allowed it; the clauses still open were read at HEAD. Docs only, so no suite run is owed and the commit gate runs none. What you got wrong: **mistake** - when I split rank 13 out of section 10's lumped row in `8b4a661`, I left `pathToFileURL` in that row's evidence cell; **where** - `docs/review-2026-10-04-gap-audit.md` section 10, the row listing the open ranks; **impact** - the row cited rank 13's remedy token for a set of ranks that no longer contained 13, so a reader would look for a link defect among the wrong ranks; **cause** - I edited the rank numbers in the first cell and did not re-read the third; **fix** - the cell now lists only what is absent from the ranks in the row, and rank 17 has its own row (whose own evidence cell carries the gate's integrity set, which is 17's clause); **verified** - the table was re-read after the edit, and this commit's diff shows the corrected cells. --- docs/review-2026-10-03-gap-audit.md | 6 ++++-- docs/review-2026-10-04-gap-audit.md | 5 +++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/review-2026-10-03-gap-audit.md b/docs/review-2026-10-03-gap-audit.md index e7904a8e..62fc5119 100644 --- a/docs/review-2026-10-03-gap-audit.md +++ b/docs/review-2026-10-03-gap-audit.md @@ -109,7 +109,7 @@ Documentation drift found by the delivery explorer and verified by the lead: `re Every external claim above rests on a ledgered capture in one of the two nested projects (preflight PASS, chains verify); the kit's own code is read directly. C:E-01, E-02, E-13, E-15, E-16, E-23, E-24 and K:E-06, E-07 are uncited search-return rows, kept as the record of what the searches returned. One claim rests on an older capture of the root corpus: Firecrawl format pricing (root E-02, 2026-09-13, type S). One claim is CONTESTED (Firecrawl scrape and robots.txt). One researcher used the session's Firecrawl MCP search once to locate pages, then fetched them through the kit; the located pages are ledgered, the locating search is not. Spend: 1,175 to 1,061 credits. -## 9. Outcome (ranks 1 to 3 authorised 2026-10-03; ranks 4 and 7 fixed 2026-10-17) +## 9. Outcome (ranks 1 to 3 authorised 2026-10-03; ranks 4, 7 and 14 fixed 2026-10-17) | Rank | Change | Commit | |---|---|---| @@ -119,7 +119,9 @@ Every external claim above rests on a ledgered capture in one of the two nested | 4 | A `[quote: ...]` marker is read by counting brackets, so a passage nested two deep (a Markdown link whose own text held brackets) is a marker and is checked, where the one-level regex read it as no marker at all; `anchorFound` requires every fragment to start within `MAX_QUOTE_GAP` (300 compared characters) of the previous one, trying a placement at each occurrence, so `[quote: free ... requests ... credits]` can no longer anchor words scattered over a page; and the short-quote floor is measured over EACH fragment by `quoteTooShort`, not over the joined text (ADR-0162). The empty-normalised-needle case was already closed 2026-10-08. | 3abf0b8 | | 7 | The `metadata` record carries `body-sha256`, the block's own hash, beside `capture-sha256` (unchanged - the whole file, which is what the ledger holds), so a consumer can check a block against the export; a ledger `scrape` entry with no `at` is left out and named instead of writing a record with an empty mandatory `WARC-Date`; a partial export exits 2 (`INCOMPLETE`, rank 15 of the 2026-10-04 gap audit); and the written file is re-read from disk before success. ADR-0161 (lifts the freeze for `body-sha256`). | 8ee1be4 | -Ranks 5, 6 and 8 to 15 remain as recorded above, not implemented. +| 14 | A page's words are read in any script: `shingles` folds text to words with `\p{L}\p{N}\p{M}` and NFC composition instead of the Latin-ASCII `[^a-z0-9]`, so a page in Russian, Arabic, Hebrew, Greek or accented Latin is fingerprinted where it used to shingle to nothing - `sketch` returned null for it, and a republished copy of it was reported as an independent witness, the one inversion this module exists to prevent. ASCII input is byte-identical, so `MIRROR_THRESHOLD` still rests on its own measurement (ADR-0163; rank 17 of the 2026-10-04 gap audit, whose other clauses remain open). | 2689678 | + +Ranks 5, 6, 8 to 13 and 15 remain as recorded above, not implemented. ### Review of the three commits (breaker and mutation auditor, 2026-10-04) diff --git a/docs/review-2026-10-04-gap-audit.md b/docs/review-2026-10-04-gap-audit.md index 4f9c11c9..6623af39 100644 --- a/docs/review-2026-10-04-gap-audit.md +++ b/docs/review-2026-10-04-gap-audit.md @@ -273,7 +273,7 @@ evidence: the thirteen-check registry and the one verdict function (do not add a for rank 3; grade at collection); the ledger; ADR-0037's refusal of a thin-capture heuristic; the transport seam's pinned shape; the freeze. -## 10. Outcome (rank 9 fixed 2026-10-17; the rest re-probed 2026-10-17) +## 10. Outcome (re-probed 2026-10-17; ranks 9 and 17 since fixed or partly fixed) A rank's line numbers describe the code it was written against, so every rank below was re-probed against `dd5026f` - thirteen days and 253 commits after this audit's base @@ -284,7 +284,8 @@ left UNRESOLVED stays unresolved until its named observation is made. |---|---|---| | 1 | **closed upstream** - the drafted brief names the warnings and their count | `renderGateWarnings` (`lib/render.mjs`) writes the count and each warned rule with its row or unknown; `lib/brief.mjs:301` calls it, and the PASS line (`lib/brief.mjs:276-277`) says disclosed unknowns and gate warnings still apply | | 2 | **closed upstream** - approval is re-derived from the checked corpus, and its blank-Raw half resolves through `captureOf` | `lib/artifact-validator.mjs:418-421` derives `buildAuthorized: status === 'PASS' && claimed` (ADR-0160, `dfaf34c`); `lib/artifact-state.mjs:131-132` counts a row it cannot resolve as not reviewed rather than skipping it | -| 3, 5, 7, 8, 11, 12, 14, 16, 17 | **open** by probe - the token the named remedy would have introduced is absent from `lib/` and `bin/` | no `suspect-capture`, `numPages`/`totalPages`, `page_age`/`sourceDate`, `table-header-renamed` or `pathToFileURL`; `INTEGRITY_RULES` (`lib/gate.mjs:613`) still omits `raw-missing`, `raw-outside`, `raw-unreadable` | +| 3, 5, 7, 8, 11, 12, 14, 16 | **open** by probe - the token the named remedy would have introduced is absent from `lib/` and `bin/` | no `suspect-capture`, `numPages`/`totalPages`, `page_age`/`sourceDate` or `table-header-renamed` anywhere in `lib/` or `bin/` | +| 17 | **partly fixed** - the non-Latin shingle clause is fixed (`2689678`), the other clauses are open | fixed: `shingles` reads a word in any script (`\p{L}\p{N}\p{M}` and NFC) where the Latin-ASCII `[^a-z0-9]` left a Russian, Arabic, Greek or accented page with no shingle at all, so `sketch` returned null for it and a republished copy of it was reported as an independent witness (ADR-0163). Still open: the 23-entry suffix list for "same site" (`lib/core.mjs:768-786`); lenient `Date.parse` (`lib/core.mjs:714-719`); a string `maxAgeDays` silently becoming 180 (`lib/machine.mjs:170`); and `INTEGRITY_RULES` (`lib/gate.mjs:613`) still omitting `raw-missing`, `raw-outside`, `raw-unreadable` | | 13 | **partly closed** - the first clause is done and the third is half done; the link is still built by hand | `lib/mcp.mjs:420-427` type-checks array arguments AND their items, and the FAIL path (`lib/mcp.mjs:368-379`) already omits the link and sets `isError`; still open: `resourceLink` munges a string into `file:///` rather than `pathToFileURL(resolve(file))` (`lib/mcp.mjs:203-212` - no percent-encoding, no absolute-ising), and an unapproved PASS returns `isError: false` (`lib/mcp.mjs:394`) | | 6 | **not open as written** - its named remedy is contradicted by a later decision | `lib/checks.mjs:50-58` judges freshness by the capture the row NAMES (`ownCapture`) and deliberately does not fall back to `captureOf`: judged by the URL's latest capture, a row with a blank Raw cell "borrowed a fresh fetch's date, lost its stale warning, and drew a date-mismatch telling the operator to write the later date into the row" (found 2026-10-03, review of G8). Doing what rank 6 asks would restore that defect | | 9 | **fixed** | `dd5026f`: a fetch's ledger `at` is the moment it happened (`nowIso(now)`), not midnight of its day; the `retrieved` day is unchanged | From 52ec30d68058bd55ad45b79522651ad34519a940 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <stepenkoanatoli@gmail.com> Date: Sat, 17 Oct 2026 08:36:16 +0300 Subject: [PATCH 10/12] mcp: build the fetch_corpus link with pathToFileURL, not by concatenation What changed: `lib/mcp.mjs`'s `resourceLink` returns `pathToFileURL(path.resolve(String(file))).href` instead of pasting `file:///` onto a forward-slashed path, and the module imports `node:path` and `pathToFileURL` from `node:url`. `test/mcp.test.mjs`'s Windows-path test now asserts the properties that matter (no backslash, no raw space, and a URI that round-trips through the platform's own decoder to the same file), and a new test covers a file name holding `#` and `?`. `docs/ARCHITECTURE.md`'s `mcp.mjs` row, `CHANGELOG.md` and `research-kit/README.md`'s test count (1872 -> 1873) carry it. Why: rank 13 of `docs/review-2026-10-04-gap-audit.md`, whose remedy the audit names itself. The link is the only handle a client gets - the spec says a link a tool returns "is not guaranteed to appear in the results of a `resources/list` request", so there is nothing to rediscover it by - and building it by concatenation made a `#` in a package name start a fragment and a `?` start a query (a link to a file that does not exist), left a space unencoded, and anchored a RELATIVE path at the filesystem root. Measured, at HEAD and with the fix: `C:\tmp\q#1\corpus?x.zip` was `file:///C:/tmp/q#1/corpus?x.zip` and `relative\out\corpus.zip` was `file:///relative/out/corpus.zip`; they are now `file:///C:/tmp/q%231/corpus%3Fx.zip` and `file:///C:/Users/PC/Desktop/gittoken/research-kit/relative/out/corpus.zip`. What it touched: `research-kit/lib/mcp.mjs`, `research-kit/test/mcp.test.mjs`, `docs/ARCHITECTURE.md` (the `mcp.mjs` row, in this commit as rule 1 requires), `CHANGELOG.md`, `research-kit/README.md`. No ADR (rule 3, ADR-0075): this is a bug fix with one obvious remedy, which the audit names - `pathToFileURL` - so the alternatives set aside are named here instead. They were: hand-escaping the path (re-implementing percent-encoding, and `#`, `?` and a space are precisely the cases people get wrong by hand), keeping the string and documenting the limitation (the link would stay wrong for a file the tool had already written), and leaving a relative path to the caller (the function is handed a path, and resolving it is the function's job). What I verified: - red-first, verified: with `lib/mcp.mjs` restored to HEAD, `selftest.mjs mcp` reports 50 passed, 2 failed - "a space is not legal unencoded in a URI, so it must be percent-encoded" and "a raw # ends the path" - and with the fix it reports 52 passed, 0 failed. - verified: `selftest.mjs mcp dispatch` -> 89 passed, 0 failed, 1 unsupported (the known host limitation of ADR-0108, not red). - verified: the availability-and-remedy probe above, run against HEAD's module and then against the fixed one, printing the four URIs quoted in **why**. - expected: the full offline suite runs in the commit gate, which a `research-kit/` change owes (ADR-0120); its verdict is this commit's gate output. - expected: nothing else reads the URI's shape. `resourceLink` has one call site (`lib/mcp.mjs:389`) and the literal `file:///` appeared nowhere else in `lib/` or `bin/` (grepped). - expected (on ubuntu; verified on this Windows host): the assertions hold on both platforms, because they test properties rather than a string. See the first entry below for why that matters. What I got wrong and fixed: 1. **mistake** - the obvious replacement for the pinned-string test would assert the exact URI, e.g. `file:///C:/tmp/a%20b/corpus.zip`; **where** - the Windows-path test in `test/mcp.test.mjs`; **impact** - it would have passed here and failed on ubuntu, where `C:\tmp\a b\corpus.zip` is not a path but a file NAME, so `path.resolve` yields `<cwd>/C:\tmp\a b\corpus.zip` and the platform escapes the backslashes; that is 3 of the 7 required CI jobs red on a one-line fix; nothing delivered was affected, because it was caught before the test was written; **cause** - the OLD implementation was pure string munging and therefore platform-independent, so the test it replaced was too, and I first reached for the same shape of assertion on a function that now delegates to the platform; **fix** - the test asserts what is true on any platform: no raw `#`, `?`, space or backslash in the URI, and `fileURLToPath(uri)` equal to `path.resolve(input)`; **verified** - `selftest.mjs mcp` -> 52 passed, 0 failed on this host, and the red-first run fails the same two tests, so the assertion discriminates rather than merely passing on one platform (expected: green on ubuntu, for the reason in the fix). 2. **mistake** - my first evidence probe was a `node -e` one-liner whose raw template literal was mangled by the shell into a syntax error, and the second imported the module by Windows path (`C:/...`), which the ESM loader refuses (`ERR_UNSUPPORTED_ESM_URL_SCHEME`); **where** - the ad-hoc probe used to print the before/after URIs, twice; **impact** - none on delivered code; the cost was two failed commands and, worse, a temptation to quote the probe's expected output from memory instead of seeing it; **cause** - quoting a JS template literal through PowerShell, then assuming a Windows path is importable as an ESM specifier; **fix** - wrote the probe to a file and imported through `pathToFileURL(...).href`; **verified** - the probe then printed the four URIs quoted in **why**, at HEAD and with the fix. --- CHANGELOG.md | 5 +++++ docs/ARCHITECTURE.md | 2 +- research-kit/README.md | 2 +- research-kit/lib/mcp.mjs | 12 ++++++++++-- research-kit/test/mcp.test.mjs | 22 ++++++++++++++++++++-- 5 files changed, 37 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a4a7c748..0d23c8f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,11 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in ## Unreleased +- The MCP `fetch_corpus` link is built with the platform's own encoder + (`pathToFileURL(path.resolve(file))`) instead of pasting `file:///` onto the path. A `#` in a + package name became a fragment and a `?` a query, so the link addressed a file that does not + exist; a space was left unencoded; and a relative path was anchored at the filesystem root + instead of the working directory (rank 13, `docs/review-2026-10-04-gap-audit.md`). - Near-duplicate detection reads a **word in any script**: the Latin-ASCII `[^a-z0-9]` class stripped every non-Latin letter, so a page in Russian, Arabic, Hebrew, Greek or Devanagari shingled to nothing, `sketch` returned null for it, and a republished copy of it was reported diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7b2d43f2..ae4042a0 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -187,7 +187,7 @@ is an open decision (2026-09-30): | `artifact-validator.mjs` | Ledger verification follows actual archive capture entries and parsed evidence rows, so relabelling roles or declaring zero captures cannot disable provenance checking; an empty failed-collection scaffold still passes (#358). **Is this package valid, and does it authorize a build** — two questions, kept apart. `validateArtifact({ file, expectedClientRef })` → `{ status, buildAuthorized, packageId, clientRef, workflowRunId, state, errors, warnings }`, where `status: 'PASS'` means internally consistent and says *nothing* about permission. Offline and read-only by construction: no transport imported, `process.env` never read, nothing from the package executed, no shell, and the only write is a `fs.mkdtemp` copy of `project/` removed in a `finally` — needed because the chain and handoff checks are the kit's existing code (`readCorpus`, `verifyHandoff`) and reuse beats a second implementation of provenance verification nobody tested. Stable error codes (`ZIP-*`, `MANIFEST-*`, `FILE-*`, `LEDGER-*`, `CAPTURE-*`, `AUTHORIZATION-INCONSISTENT`, `SECRET-DETECTED`, `FORMAT-UNSUPPORTED`, `CLIENT-REF-MISMATCH`) and **never a secret value in an error** — the report names the pattern and the line. `authorizationProblems()` is the one implementation of the build rule; the schema encodes it a second time and when the schema catches it first the rule is still reported in its own words. An unsupported format **major** is `INCOMPLETE`, not invalid. A crash is `BLOCKED`, never `PASS`. Its report carries `reviewedBy` (`review.by`, or `undeclared` for a 1.0.0 package) and judges nothing by it (ADR-0074). Report metadata is surfaced only after the manifest passes schema validation; malformed state and identity fields are null, and the CLI escapes control characters in every text diagnostic so hostile strings cannot forge lines. Reads format majors **1 and 2** (2026-09-30, ADR-0107): a 1.x `HUMAN_REVIEW_REQUIRED` is reported as `REVIEW_REQUIRED`, and `eraProblems` refuses a value from the other major by name - a 2.x package with the old state or `review.by: "human"`, or a 1.x one with the new state. A binary body (NUL in its first 8000 bytes) is not matched against the credential patterns, as a body over 512 KiB is not (ADR-0128, 2026-10-02): `docs/` travels whole, and one vendored DLL carrying the PEM marker it parses would have failed a package with no secret in it. Missing, ill-typed, malformed and below-oldest format versions are MANIFEST-SCHEMA/FAIL with regeneration as the remedy; only a numeric format major newer than this reader is FORMAT-UNSUPPORTED/INCOMPLETE (#361). Claimed approvals are re-derived from checked bytes with the portable unpromoted pluralist, 180-day baseline; warnings name that baseline and format-1 legacy compatibility. | ADR-0032, ADR-0160 | | `artifact.mjs` | Raw-cell review reads and explicit ledger reads use projectFile under ADR-0109; Raw reads stay inside the capture folder, and a non-file ledger is refused before reading (#351). **Packaging a project without changing it; shared authorization derivation lives in `artifact-state.mjs`.** `createArtifact` → `{ bytes, name, manifest, entries, derived }`; `writeArtifact` writes then **re-validates with the consumer's own validator** before reporting success; its bytes go through `writeBytes`, so re-creating a package at a name that already holds one cannot trade a good package for a truncated one (2026-09-28). There is no `--build-authorized` and no option that reaches an authorization field: `deriveState(root)` runs the real gate with `env` defaulting to `{}` so a packaging shell's override cannot become a property of a package that outlives it. `mapClassified` and `briefReviewed` have honest machine answers; `findingsReviewState` answers the third **negatively** — a `Finding` byte-identical to what `firstFinding` re-extracts from the capture was demonstrably never rewritten, which detects the unreviewed case without grading the reviewed one (ADR-0013's line, uncrossed). It repeats the collector's own call exactly - the page body after the front-matter, falling back to the page's `title` or `url` from that front-matter - since 2026-09-27: run over the whole file, the extractor read the front-matter as the page, and a URL-only corpus nobody had touched (one JSON capture, run 36287211468) was packaged `findingsReviewed: true`. `packageName` never carries the topic (an artifact listing is world-readable on a public repository) and falls back to `run-<workflowRunId>`; `checkClientRef` refuses rather than sanitises. Inventory sorted by ZIP path, manifest and digest appended **last**, digest over the exact manifest bytes. README-FIRST in a package that is not yet approved prints the preflight and brief commands and **one `create` line carrying the package's own identity** (`repackageArgs`, 2026-09-27): the manifest is written once, and the package gave its reviewer no way to get one that says `APPROVED_BRIEF`. Commands are spelled by `homeCommand` (ADR-0050). `createArtifact` checks the identity against the manifest schema's own `source` rules before deriving or writing anything (`sourceProblem`, 2026-09-27): `create --commit abc` wrote a package, then reported it did not validate, and left it on disk. **`review.by`** (format 1.1.0, ADR-0074) is the reviewer's declaration read from `BRIEF.md` by `reviewedBy`: `agent` or `undeclared` (`human` was retired in format 2.0.0, ADR-0107). Not an approval condition. README-FIRST no longer says "a human reviewed it"; it names the declared reviewer, or says none was declared. `collectProjectFiles` and the explicit ledger add refuse (`OUTSIDE_PROJECT`) any path whose real location is outside the project: it followed links and packed outside files, a private key included (ADR-0076). The same walk asks the consumer's own `checkEntryName` about every name it is about to store and refuses the run before a byte is written (`UNPACKAGEABLE_NAME`, 2026-09-28): `entryName` translates a backslash into a path separator, so a file named `a\b.md` travelled as `a/b.md` - a directory invented out of a name - while the manifest kept the original, and only the read-back caught the contradiction, after the package had been written. The same refusal covers two files differing only in case (2026-09-28), with the reader's lowercase comparison: the reader's ZIP-CASE-COLLISION fired only after the write. Format **2.0.0** (2026-09-30, ADR-0107): a collected package waiting for review says `REVIEW_REQUIRED`, and README-FIRST gives the review to the agent alone. collectProjectFiles refuses (`UNPACKAGEABLE_FILE`) what packaging cannot read to an end - a FIFO, socket or device, and a folder link back into its own ancestry (judged by real path, so a link to a sibling folder inside the project still packages) - before a byte is read (2026-10-01). **A reviewed brief is complete and current** (2026-10-03, output-reliability audit G1): `deriveState` approves only when every judged section is present and answered (`judgedSection`) and a stamped brief's inputs hash matches the corpus (`draftStamp`, `briefInputsHash`); `briefState` alone read a missing section as answered, so a drafted brief with no review sections derived APPROVED_BRIEF on the repository's own corpus, and a brief whose evidence changed after the draft stayed authorized beside hygiene's brief-stale warning. A brief with no stamp predates ADR-0055 - the root's own and ten decision briefs - and keeps its approval: its currency cannot be checked, `hygiene/brief-unstamped` says so (ADR-0138), a redraft with --force stamps it. **README-FIRST in every state and the collection summary preserve the producer evaluation warnings** through `renderGateWarnings` (ADR-0143). Authorization fields and machine package formats are unchanged. `bin/artifact.mjs` refuses a value flag given no value by name, exit 3, before anything is read or written (2026-10-11, #280): `create` had asked only whether a required flag was present, so a bare `--ref` went into the manifest as "true", a bare `--run-id` as 1 and a bare `--output` wrote a package named `true` that re-validated PASS; a bare `validate --file` tried to open a file called `true`. Caller references use checkClientRef before token lookup, output-folder creation or dispatch (#375, 2026-10-13); CR/LF are refused by the shared checker, and diagnostics are redacted. The artifact CLI awaits exitAfterFlush for its report, help and refusal endings, preserving queued stdout/stderr and the existing exit status on asynchronous POSIX pipes (#359). | ADR-0032, ADR-0031, ADR-0052, ADR-0143 | | `artifact-state.mjs` | The one approval derivation shared by producer and validator: real gate and handoff, classified map, reviewed findings, authored/current brief; consumers rederive claimed approval under the explicit portable baseline and exact format-1 pre-0069 input hash (ADR-0160). Finding review resolves a blank Raw cell through the shared capture resolver and refuses unresolved captures; every capture read retains project and raw-folder path containment (#351, #357). | ADR-0032, ADR-0160 | -| `mcp.mjs` | **The collector as a Model Context Protocol server**, and the first module in this repository built from an APPROVED BRIEF rather than from a plan - every design choice cites an E-row in `docs/decisions/2026-09-21-agent-interface/`. `handle(message, deps)` is a pure function of a JSON-RPC message plus injected transport, so the whole surface is tested offline. **stdio, not Streamable HTTP**: semantics are identical on both, so the difference is operational, and remote costs the entire OAuth 2.1 chain. **Two tools, not one** - `collect` returns a run id immediately, `fetch_corpus` returns a link - because a server cannot initiate a request, so the alternative is one call held open for minutes and an agent that loses a paid run when its process dies. **A `resource_link`, never embedded bytes**: the spec defines embedding but states no maximum size, so a link is the defensible option. **No tool takes a token** and a test asserts no parameter is credential-shaped. `PROTOCOL_VERSION` is pinned and a mismatch answers `-32022` with `data.supported`, the same discipline as pinning `X-GitHub-Api-Version`. `createStdioLoop` serialises chunk processing through one promise chain - overlapping `data` events were mutating the buffer concurrently, which a test caught. **DUAL-ERA**: `SUPPORTED_VERSIONS` is `['2026-07-28', '2025-11-25', '2025-06-18']` (2025-06-18 since 2026-09-27, ADR-0049: a client on the official SDK up to 1.24.0 asks for it and refuses 2025-11-25, and nothing 2025-11-25 changed touches this server; 2025-03-26 is not served, because the results carry `structuredContent` and `resource_link`) and `initialize` is implemented beside `server/discover`, because the specification is ahead of every shipped client - the official SDK at v1.30.0 declares `2025-11-25` as its latest and answered the modern-only first version with `-32601: unknown method initialize`. `initialize` **echoes** a serveable requested version and falls back to **legacy** rather than to this server's preference, because a client throws on a version it does not support. **Arguments that fail a tool's schema are a tool result** (`isError: true`, the reason as text) since 2026-09-27 - SEP-1303 in 2025-11-25 says input validation errors should be Tool Execution Errors, so the model can correct itself; they were JSON-RPC -32602, which a client raises as an exception. An unknown tool stays -32602. `collect`'s `topic` carries `minLength: 1`, and `validateArgs` enforces `minLength` after trimming (2026-09-27): an empty topic had passed and reached GitHub, which answered HTTP 422. `fetch_corpus` checks `out_dir` with `usableOutDir` before it reads the run (2026-09-27), so an unusable folder is an `OUT_DIR` tool error, not `UNKNOWN`. `fetch_corpus` returns `reviewedBy`, and its text says review is for an agent or a person (ADR-0074). `createStdioLoop` bounds a line at `MAX_LINE` (4 MiB, 2026-09-28): a line past it is refused with -32600 as soon as it passes the cap and dropped through its newline, and the next message is served - the buffer had grown without limit until a newline came, so a client that sent none could grow the server until it died. **No request is lost at end of input** (break-test PR #145, 2026-09-29): `exitWhenSettled` waits for every call already read before exiting (bounded by `END_GRACE_MS`), and a final request without a trailing newline is served on `end` - both had been dropped with exit 0. `SERVER_INFO.version` is `KIT_VERSION` (ADR-0116); it had announced a fixed 1.0.0. **The approved fetch response directs the builder to README-FIRST and the brief for warnings and known unknowns** (ADR-0143). It forwards no new machine warning field and derives no new authorization. `validateArgs` checks allowed keys by own membership and enforces the declared arrays of strings: malformed query/URL values and inherited Object property names become schema tool errors before credential lookup or dispatch. Omitted or empty arrays, and valid string arrays, keep their existing normalization. The shared `queriesInput` refusal prevents CR/LF inside individual queries or URLs from dispatching extra runner work (#376, 2026-10-12). Rejected query/URL input passes through the existing credential redactor before it becomes tool-result text (#378, 2026-10-12), including caller credentials absent from the server environment. Caller references use checkClientRef before token lookup, output-folder creation or dispatch (#375, 2026-10-13); CR/LF are refused by the shared checker, and diagnostics are redacted. Output folders resolve to absolute paths before access checks; the MCP caller passes that same path to fetchCorpus so the write and resource link identify the same file (#372, 2026-10-13). Non-PASS corpus downloads return only a redacted diagnostic with validator codes and remedies, never a corpus link or review instructions; buildAuthorized stays false (#371, 2026-10-13). | ADR-0034, ADR-0049, ADR-0143 | +| `mcp.mjs` | **The collector as a Model Context Protocol server**, and the first module in this repository built from an APPROVED BRIEF rather than from a plan - every design choice cites an E-row in `docs/decisions/2026-09-21-agent-interface/`. `handle(message, deps)` is a pure function of a JSON-RPC message plus injected transport, so the whole surface is tested offline. **stdio, not Streamable HTTP**: semantics are identical on both, so the difference is operational, and remote costs the entire OAuth 2.1 chain. **Two tools, not one** - `collect` returns a run id immediately, `fetch_corpus` returns a link - because a server cannot initiate a request, so the alternative is one call held open for minutes and an agent that loses a paid run when its process dies. **A `resource_link`, never embedded bytes**: the spec defines embedding but states no maximum size, so a link is the defensible option, and the URI comes from the platform's own encoder - `pathToFileURL(path.resolve(file))` - where pasting `file:///` onto the path turned a `#` in a package name into a fragment and a `?` into a query (a link to a file that does not exist), left a space unencoded, and anchored a relative path at the filesystem root rather than the working directory (rank 13, `docs/review-2026-10-04-gap-audit.md`). **No tool takes a token** and a test asserts no parameter is credential-shaped. `PROTOCOL_VERSION` is pinned and a mismatch answers `-32022` with `data.supported`, the same discipline as pinning `X-GitHub-Api-Version`. `createStdioLoop` serialises chunk processing through one promise chain - overlapping `data` events were mutating the buffer concurrently, which a test caught. **DUAL-ERA**: `SUPPORTED_VERSIONS` is `['2026-07-28', '2025-11-25', '2025-06-18']` (2025-06-18 since 2026-09-27, ADR-0049: a client on the official SDK up to 1.24.0 asks for it and refuses 2025-11-25, and nothing 2025-11-25 changed touches this server; 2025-03-26 is not served, because the results carry `structuredContent` and `resource_link`) and `initialize` is implemented beside `server/discover`, because the specification is ahead of every shipped client - the official SDK at v1.30.0 declares `2025-11-25` as its latest and answered the modern-only first version with `-32601: unknown method initialize`. `initialize` **echoes** a serveable requested version and falls back to **legacy** rather than to this server's preference, because a client throws on a version it does not support. **Arguments that fail a tool's schema are a tool result** (`isError: true`, the reason as text) since 2026-09-27 - SEP-1303 in 2025-11-25 says input validation errors should be Tool Execution Errors, so the model can correct itself; they were JSON-RPC -32602, which a client raises as an exception. An unknown tool stays -32602. `collect`'s `topic` carries `minLength: 1`, and `validateArgs` enforces `minLength` after trimming (2026-09-27): an empty topic had passed and reached GitHub, which answered HTTP 422. `fetch_corpus` checks `out_dir` with `usableOutDir` before it reads the run (2026-09-27), so an unusable folder is an `OUT_DIR` tool error, not `UNKNOWN`. `fetch_corpus` returns `reviewedBy`, and its text says review is for an agent or a person (ADR-0074). `createStdioLoop` bounds a line at `MAX_LINE` (4 MiB, 2026-09-28): a line past it is refused with -32600 as soon as it passes the cap and dropped through its newline, and the next message is served - the buffer had grown without limit until a newline came, so a client that sent none could grow the server until it died. **No request is lost at end of input** (break-test PR #145, 2026-09-29): `exitWhenSettled` waits for every call already read before exiting (bounded by `END_GRACE_MS`), and a final request without a trailing newline is served on `end` - both had been dropped with exit 0. `SERVER_INFO.version` is `KIT_VERSION` (ADR-0116); it had announced a fixed 1.0.0. **The approved fetch response directs the builder to README-FIRST and the brief for warnings and known unknowns** (ADR-0143). It forwards no new machine warning field and derives no new authorization. `validateArgs` checks allowed keys by own membership and enforces the declared arrays of strings: malformed query/URL values and inherited Object property names become schema tool errors before credential lookup or dispatch. Omitted or empty arrays, and valid string arrays, keep their existing normalization. The shared `queriesInput` refusal prevents CR/LF inside individual queries or URLs from dispatching extra runner work (#376, 2026-10-12). Rejected query/URL input passes through the existing credential redactor before it becomes tool-result text (#378, 2026-10-12), including caller credentials absent from the server environment. Caller references use checkClientRef before token lookup, output-folder creation or dispatch (#375, 2026-10-13); CR/LF are refused by the shared checker, and diagnostics are redacted. Output folders resolve to absolute paths before access checks; the MCP caller passes that same path to fetchCorpus so the write and resource link identify the same file (#372, 2026-10-13). Non-PASS corpus downloads return only a redacted diagnostic with validator codes and remedies, never a corpus link or review instructions; buildAuthorized stays false (#371, 2026-10-13). | ADR-0034, ADR-0049, ADR-0143 | | `disclosure.mjs` | **What a public run actually exposes, measured rather than asserted.** Written because the opposite claim was documented first: the collector's header said dispatch inputs are visible to anyone with read access, and checking it found the topic in none of the readable responses. `probeRun` sends **unauthenticated** requests - there is no token parameter, and a probe that quietly authenticated would answer a different question while looking like this one. `summarise` separates `exposedNames` (true on any public repo, and the reason the topic is kept out of run and artifact names) from `exposedContent` and `exposedSubject`. `needleChecked` exists so a report can never let "false" read as "no": without a topic to search for, the rendering says **not checked**. An unreachable endpoint is a refusal to conclude, never a clean result - **enforced since 2026-09-27**: until then that sentence was true of the findings and false of the verdict, and with no network every probe was unreachable and the command exited 0, "only the shape is public". `summarise` now reports `unmeasured` (probes with status 0) and `conclusive`; `exitCode(report)` keeps a measured exposure (2, 1) whatever else failed, and otherwise needs every probe answered for 0, else 3; the rendering says UNREACHABLE and "not tried" instead of "refused", shows `not measured` for a false it could not establish, and closes NOT MEASURED instead of the shape paragraph. An unreachable probe names the cause of the failed fetch through `fetchFailure` (2026-09-30), not the bare `fetch failed`. `probeRun` takes the artifact and job ids from the listings its probes already read (2026-09-30); it had fetched both again inside one try/catch, so a transient failure on the second request left the artifact download and the log untried. | ADR-0035 | | `dispatch.mjs` | **Asking the collector to run, from outside GitHub** - the seam an agent or a desktop app uses. `dispatchCollection` (pins `X-GitHub-Api-Version: 2026-03-10` on the REQUEST, not merely in a comment), `waitForRun`, `listArtifacts`, `downloadArtifact`, `unwrapArtifact`, plus `tokenFromEnv` and `redact`. **No token may be passed as an argument** - `tokenFromEnv` is the only way in, because a credential on a command line reaches the shell history, `ps`, and any log that echoes its own command; a test asserts the source contains no token flag. `redact` is PATTERN-based rather than "remove the token we hold", so it also catches a credential the process was never given - one echoed back by a server, a proxy, a redirect. A **204** is a named `NO_RUN_ID` failure, never an empty success: "the run started and I cannot tell which one" is the failure a caller is least able to diagnose from a status code. **Both routes to a run id are requested**, not just the pinned version: the `X-GitHub-Api-Version: 2026-03-10` header AND `return_run_details: true`, the optional parameter GitHub added 2026-02-19 that returns the run id on `2022-11-28` as well. Measured before it was trusted - four calls on 2026-09-21 established that the old version honours the parameter and the new version accepts rather than rejects it, so sending both is safe and a server that strips or downgrades the header still answers with a run id. That retired ADR-0031's original fallback, which was to poll `/actions/runs` and correlate by `created_at` - a race the design never actually needed. `NO_RUN_ID` now means BOTH routes were ignored, which is a far narrower diagnosis. `waiting` is reported as a status, not a stall - it means a protection rule is holding the job. `unwrapArtifact` opens GitHub's outer ZIP through the same distrustful reader, and passes a hostile or ambiguous container through untouched for the validator to refuse rather than guessing at packaging depth. Every request goes through an injectable `fetch`, so the whole module is tested offline. **Pages by URL** (2026-09-27): `queriesInput` builds the one `queries` input from search queries and pages to fetch, one per line, and refuses a URL that is not http(s) before anything is dispatched. `collect.yml` fetches a line that is an http(s) URL instead of searching it, and with only URLs it searches nothing. `collect-remote --url` and the MCP `collect` tool's `urls` share this function, so all three agree. Until then the collector could only search, and a page the caller already knew had to be collected locally. **`usableOutDir` checks the folder before anything is spent** (2026-09-27): it creates the folder and tests write access, and an unusable one is a named `OUT_DIR` failure with a remedy. `fetchCorpus` calls it before it lists artifacts, the MCP `fetch_corpus` tool before it reads the run, and `collect-remote` before it dispatches. The folder had been checked only after the download, as a raw ENOTDIR the MCP server could report only as `UNKNOWN`, and for `collect-remote` that came after a paid run and up to half an hour of waiting. `fetchCorpus` writes the package it downloaded through `writeBytes` (2026-09-28, ADR-0079): `fs.writeFileSync` emptied a package already at that name when the write failed. `bin/collect-remote.mjs` checks `--max-pages` (1..25), `--depth`, `--runner`, `--search-transport` and `--timeout` against the ranges and choices `collect.yml` enforces, before it asks for a token (2026-09-29, Arena break test): `--max-pages 8o` was dispatched as typed, and the paid run failed on the runner. A dispatch that never left names the cause of the failed fetch through `fetchFailure` (2026-09-30), still passed through `redact`; so does `.github/scripts/check-action-pins.mjs` for a pin it could not check. Both had printed the bare `fetch failed`. `getRun` and `listArtifacts` read their body through `readJsonBody` (2026-09-30): a 200 whose body is not JSON (a maintenance page, a proxy) is `DispatchError` `BAD_BODY` with the status and a remedy, where it had thrown a raw `SyntaxError` that `collect-remote` could only call `UNKNOWN`. `getRun`, `listArtifacts` and `downloadArtifact` reach GitHub through `callApi` (2026-09-30), so a request that never left is `DispatchError` `NETWORK` with its cause (`fetchFailure`, redacted) and a remedy, as a dispatch already was; they had thrown the raw `fetch failed`, which `collect-remote` could only call `UNKNOWN`. fetchCorpus treats names from the API as data: a wrapper entry is flattened to its file name, an artifact name holding a separator, `:`, `..` or a control character is refused (`ARTIFACT_NAME`) and nothing is written (2026-10-01). unwrapArtifact hands on a wrapper whose inner package cannot be decompressed unchanged, for the validator to judge, instead of throwing ZIP-READ (2026-10-01). `waitForRun` retries a failed poll that may pass on its own - no connection, a 429 or 5xx, a body that is not JSON - up to `MAX_POLL_RETRIES` (5) in a row, reset by a good poll and never past the deadline; 401, 403 and 404 stay fatal (2026-10-01). Every failed poll had ended a watch of up to 30 minutes with RUN_FAILED while the run went on. `getRun` gives a 401, 403 or 404 its own remedy (a missing run, a refused token), and the timeout remedy names `--run-id` (2026-10-01). `splitRepository` admits only two nonempty `[A-Za-z0-9._-]+` components other than `.` and `..`, so dispatch, run lookup, artifact listing and download refuse route-changing delimiters and control characters before their first request (#379, 2026-10-12). `queriesInput` refuses CR/LF in either list before trimming and serialization (#376, 2026-10-12), retaining existing empty-input and one-line normalization. A declared item cannot become additional workflow lines. `dispatchCollection` also uses `readJsonBody`: a successful non-JSON response is `BAD_BODY` with the proxy/maintenance-page remedy; valid JSON missing the run ID retains `NO_RUN_ID` (#377, 2026-10-12). Output folders resolve to absolute paths before access checks; the MCP caller passes that same path to fetchCorpus so the write and resource link identify the same file (#372, 2026-10-13). Default corpus downloads use a fresh private mkdtemp folder per call; existing or dangling symbolic links at the package name are refused as OUT_DIR before the write, and MCP describes the private default (#373). Dispatch and run reads classify a rate-limited 403 from headers or body as RATE_LIMITED, retaining Retry-After seconds and the UTC reset time; permission-only 403 remedies remain unchanged (#374). | ADR-0033, ADR-0032 | | `brief.mjs` | The phase-1 → phase-2 handoff: renders `research/BRIEF.md` from the corpus, and **owns the brief's shape** — `BRIEF_SECTIONS` (six sections, the two that are judged marked), the markers that distinguish *still the scaffold* from *the drafter wrote this* (`briefState` → `template \| legacy \| draft \| authored`), and the section reader (`briefSection` / `judgedSection`) that `lib/audit.mjs` consumes instead of regexes of its own. The renderer writes headings **from** that definition, so the writer and its readers cannot drift; the scaffold ships `BRIEF_FILE_MARKER`, which the substitution pass leaves alone — the alternative, matching the template's prose, made a sentence of English load-bearing. The verified table has **one row per claim and source, naming the unknowns it closes** (2026-09-26): two unknowns leading with the same E-row used to print its finding twice, word for word - three committed briefs carried such duplicates. **Every E-row an unknown cites** gets a row, not only the first (2026-09-29): a MoonAliza closure resting on E-03 and E-06 showed E-03 alone, and the row it most depended on never reached the builder. With no KNOWN-UNKNOWN declared, the unknowns section reports that status and names any CLOSED unknown citing no `P` row: the earlier primary-source assurance was removed in 2026-09-27 and the unconditional closure assurance in ADR-0143. Its commands are spelled by `documentCommand`, with the project's recorded `kitPath` first (ADR-0084). `--force` over a draft, authored or legacy brief keeps it as `research/BRIEF.md.bak-<date>` (numbered on a second force that day) and says so (2026-09-27): it restored the TODOs and kept nothing of the answers it replaced. **Every draft is stamped** (2026-09-27, ADR-0055) with a hash of its own text, a hash of what it was drafted from (`briefInputsHash`: topic, intent, unknowns, evidence rows, and since ADR-0069 the map rows and the ledger and capture counts), and the verdict it was drafted under (`gate=`), which `bin/preflight.mjs` compares with its own verdict and notes when they differ. `draftStamp` reads the stamp back, wherever it stands (the last one, if several), and text after it is an edit like text before it (2026-10-03, ADR-0138): anchored to the end of the file, a line appended after the stamp - a reviewer's note, the `Reviewed by: agent` line - had made the brief read as unstamped, so a stale brief re-approved the build and `brief-stale` fell silent. An unedited draft is redrafted without `--force` and without a backup, and any edit keeps the refusal. A draft written while the gate failed had kept saying "Gate: FAIL" after PASS. A known unknown's day-one step drops a leading "Day one:" before it is labelled (2026-09-27), so a cell written that way is not rendered "Day-one verification: Day one: ...". **`reviewedBy(text)`** (ADR-0074) reads the brief's `Reviewed by: agent|human` line; the drafted brief carries a placeholder that reads as `undeclared`. A line naming both roles, or two lines that disagree, also read as `undeclared`; a line inside a code block is not a declaration, and an indented one is. A known unknown's Evidence cell that states what is known and then labels its day-one step mid-sentence renders as "Known so far" plus one "Day-one verification" line, never the label twice (2026-09-29). A cell the author wrote as the brief prints it, starting "Known so far:", keeps that label once too. `reviewedBy` reads only `agent` (2026-09-30, ADR-0107): a `Reviewed by: human` line reads as `undeclared`, and the drafted placeholder names only the agent. **PASS reports configured checks, not complete closure or factual correctness** (ADR-0143). The preamble preserves the evaluated warning reasons through `renderGateWarnings`; the known-unknown section reports declared status without claiming all unknowns are closed, including on FAIL. The contradictions placeholder asks for corroboration rather than asserting agreement. On FAIL or without an evaluation, the verified section explicitly labels its CLOSED rows as declarations whose verification has not succeeded or has not been evaluated. Six sections are retained. | ADR-0051, ADR-0084, ADR-0143 | diff --git a/research-kit/README.md b/research-kit/README.md index 16d9bd9c..bb19ca08 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -568,7 +568,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -1872 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +1873 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/lib/mcp.mjs b/research-kit/lib/mcp.mjs index 1f5df771..fbb8a205 100644 --- a/research-kit/lib/mcp.mjs +++ b/research-kit/lib/mcp.mjs @@ -43,6 +43,8 @@ // discipline as pinning `X-GitHub-Api-Version` on the GitHub side, for the same reason: // this protocol has already broken compatibility once. +import path from 'node:path'; +import { pathToFileURL } from 'node:url'; import { dispatchCollection, getRunSummary, fetchCorpus, tokenFromEnv, redact, DispatchError, queriesInput, usableOutDir } from './dispatch.mjs'; import { KIT_VERSION } from './core.mjs'; import { checkClientRef } from './artifact.mjs'; @@ -199,12 +201,18 @@ const text = (value) => ({ type: 'text', text: typeof value === 'string' ? value * the path it is handed. The spec warns that a link a tool returns "is not guaranteed to * appear in the results of a `resources/list` request", so the URI is the handle - there * is nothing to rediscover it by. + * + * The URI is built by the platform's own encoder rather than by concatenation (rank 13, + * `docs/review-2026-10-04-gap-audit.md`). Pasting `file:///` onto the path made a `#` end the + * path and start a fragment (a package at `q#1/corpus.zip` produced a link to `q`) and a `?` + * start a query, left a space unencoded, and anchored a RELATIVE path at the filesystem root + * instead of the working directory. `pathToFileURL` escapes exactly what a URI must escape and + * resolves against the working directory, so the link addresses the file the tool wrote. */ export function resourceLink(file, { name, description }) { - const posix = String(file).split('\\').join('/'); return { type: 'resource_link', - uri: `file:///${posix.replace(/^\/+/, '')}`, + uri: pathToFileURL(path.resolve(String(file))).href, name, description, mimeType: 'application/zip', diff --git a/research-kit/test/mcp.test.mjs b/research-kit/test/mcp.test.mjs index 842ebe8e..c493b177 100644 --- a/research-kit/test/mcp.test.mjs +++ b/research-kit/test/mcp.test.mjs @@ -9,6 +9,7 @@ import { PassThrough } from 'node:stream'; import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; import { test, describe, assert, tempDir, cleanup, fs, path, KIT_ROOT, Unsupported } from './harness.mjs'; import { handle, versionProblem, validateArgs, resourceLink, createStdioLoop, exitWhenSettled, @@ -400,10 +401,27 @@ test('the corpus comes back as a LINK, never as embedded bytes', async () => { assert.equal(link.mimeType, 'application/zip'); }); -test('a Windows path becomes a usable file URI', () => { +test('a path becomes a usable file URI: escaped, resolved, and round-trippable', () => { const link = resourceLink('C:\\tmp\\a b\\corpus.zip', { name: 'corpus.zip', description: 'd' }); - assert.equal(link.uri, 'file:///C:/tmp/a b/corpus.zip', + assert.ok(!link.uri.includes('\\'), 'backslashes are not path separators in a URI, and a client given one cannot open the file'); + assert.ok(!link.uri.includes(' '), 'a space is not legal unencoded in a URI, so it must be percent-encoded'); + // Compared through the platform's own decoder instead of against a string, on purpose: + // `C:\tmp\...` is a path on Windows and a file NAME on POSIX, and only this platform knows + // which. The property that matters is that the URI addresses the file the tool wrote, and a + // hand-built URI fails it. + assert.equal(fileURLToPath(link.uri), path.resolve('C:\\tmp\\a b\\corpus.zip')); +}); + +test('a file name holding # or ? is a file name, not a fragment or a query', () => { + // The link is the only handle a client gets for the package. A raw `#` ends the path and + // starts a fragment, so the client opens (or fails to open) a file that does not exist - and + // the tool reported success. + const file = 'C:\\tmp\\q#1\\corpus?x.zip'; + const link = resourceLink(file, { name: 'corpus?x.zip', description: 'd' }); + assert.ok(!link.uri.includes('#'), 'a raw # ends the path: everything after it is a fragment'); + assert.ok(!link.uri.includes('?'), 'a raw ? starts a query: the file name is cut off there'); + assert.equal(fileURLToPath(link.uri), path.resolve(file), 'the URI must address the file the tool wrote'); }); test('a relative output folder gives a link to the actual absolute download path', async () => { From a887c3afa261eabe8097614c4e36894cb8ae5095 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <stepenkoanatoli@gmail.com> Date: Mon, 19 Oct 2026 09:28:21 +0300 Subject: [PATCH 11/12] quotes: a quote marker the gate cannot read blocks, and measure counts it unfound (#319) what changed lib/quotes.mjs gains unreadableQuoteMarkers(text): each `[quote` (any case, with or without a colon, closed or not) that opens no marker quoteAnchors reads, with its first 40 characters. A `[quote` inside a read marker, a code span or a URL, or the text of a Markdown link, is not an attempt. The bracket matching markersOf used is now closeOf, shared by both. The citations check reports each attempt as quote-unreadable, fail, naming the row and both repairs (write `[quote: the passage]`, or drop the marker and paraphrase). measure counts each as a quote not found, and its row as anchored. why #319: after ADR-0162 read brackets at any depth, three of the issue's four shapes were still no marker at all - no closing bracket, `[Quote :`, and `[quote the sentence]` - so the row passed as one with no quote and measure counted it unanchored. An invented sentence in one of those shapes passed where the same sentence written `[quote: ...]` failed, which is ADR-0087's case of quotations nothing checked. No parser reads those shapes without guessing where the passage starts and ends, so they are named instead. ADR-0175 records the decision, the exclusions, and why ADR-0162's reason to stay silent (prose that discusses the syntax) does not hold on measurement. what it touched research-kit/lib/quotes.mjs, lib/checks.mjs (citations), lib/measure.mjs; research-kit/test/quotes.test.mjs (two tests: the parser's attempts and exclusions, and the gate's verdict on each shape) and test/measure.test.mjs (one test); the new docs/adr/0175-a-quote-marker-the-gate-cannot-read-blocks.md and its index row; CHANGELOG.md Unreleased; docs/ARCHITECTURE.md (quotes.mjs, checks.mjs and measure.mjs rows); research-kit/README.md (2054 -> 2057 tests). A new finding under the feature freeze, hence ADR-0175 (ADR-0117). what you verified verified: red-first - with this commit's quotes.mjs and tests but the previous commit's checks.mjs and measure.mjs, selftest.mjs quotes measure gives 18 passed, 2 failed: exactly the gate test and the measure test. verified: selftest.mjs repo-hygiene quotes checks measure architecture-map: 139 passed, 0 failed (cwd rk-wt/gap, RESEARCH_KIT_TEST_TIMEOUT=180000). verified: no false positive on this repository - over its 41 research projects (the root corpus and every docs/decisions project), 417 EVIDENCE rows and 517 readable anchors, unreadableQuoteMarkers finds 0 attempts. verified: 2057 is the exact number of tests the tree registers (count-tests.mjs). verified: the full suite passed in this commit's gate, run on a working tree equal to the index - the gate's suite reads the working tree, so this commit exists only if that run was green. expected: GitHub Actions on the PR. untested: no live collection or credit spend. what you got wrong and fixed nothing to report Fixes #319 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- CHANGELOG.md | 5 ++ docs/ARCHITECTURE.md | 6 +- ...uote-marker-the-gate-cannot-read-blocks.md | 82 +++++++++++++++++++ docs/adr/README.md | 1 + research-kit/README.md | 2 +- research-kit/lib/checks.mjs | 9 +- research-kit/lib/measure.mjs | 8 +- research-kit/lib/quotes.mjs | 44 ++++++++-- research-kit/test/measure.test.mjs | 12 +++ research-kit/test/quotes.test.mjs | 33 +++++++- 10 files changed, 187 insertions(+), 15 deletions(-) create mode 100644 docs/adr/0175-a-quote-marker-the-gate-cannot-read-blocks.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 44066572..4cc5aaea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,11 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in unchecked), holds every fragment within one passage instead of anchoring words scattered over a page, and measures the short-quote floor over each fragment rather than the joined text (rank 4, `docs/review-2026-10-03-gap-audit.md`). +- A `[quote` the gate cannot read - no closing bracket, a space before the colon, no colon - is + `citations/quote-unreadable`, blocking: it was not a marker at all, so an invented sentence + in that shape passed where the same sentence written `[quote: ...]` failed, and `measure` + counted its row as unanchored. `measure` now counts it as a quote not found. A `[quote` in a + code span, a URL or Markdown link text is not an attempt (#319, ADR-0175). - `collect-remote` refuses empty repository, topic, client-ref, prior and prefer values by name before creating an output directory or reading a token (#287). - Invalid `--snapshot-max-age-ms` values are usage errors with exit 2. Library callers diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 1478f2a1..dd52e41d 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -231,11 +231,11 @@ is an open decision (2026-09-30): | `decompose.mjs` | Phase 0 coordination: recipe parsing, docs-host discovery, material gathering (scrape-budget accounting via the corpus's `cacheDecision` — a cache hit is not an attempt), draft MAP writing. Contains **no judgment** — it seeds the checklist and gathers material; it never writes a status. **A failed search reaches the map and the terminal** (2026-09-26): MAP.md lists each failure (query, provider, one scrubbed line of the vendor's error, and any fallback) and tells three empty maps apart - a dry run, a quiet topic, an outage - and `searchSummary` gives the CLI its count. Until then the failures were kept in the result (F20) and shown nowhere: after every search failed, the map told the operator to stop dry-running. The error is **scrubbed** because MAP.md is committed and vendor error text has echoed request URLs with keys in them. **Its searches are metered** (2026-09-27): phase 0 runs four searches and now writes a usage row like research-run's (`command: 'decompose'`, `searchesUsed` summed from what each provider reports), so `research.mjs --status` counts them. Until then two runs made 8 SerpAPI searches that the vendor's meter showed and the local one - labelled "this box", though it is one project on this machine - did not. A dry run still seeds the map (its documented purpose) and now logs each search a real run would make and the provider whose meter it would spend, as research's dry run does (2026-09-27): it had previewed none of the four searches. The CLI's closing steps name `research/plan.json`. **`resolveTopic` decides what is mapped** (2026-09-27, ADR-0056): `plan.json`'s topic when `--topic` is omitted, and a refusal naming both when `--topic` differs (spacing and case aside). A project still called `UNTITLED_TOPIC` accepts any `--topic`. The CLI had rewritten the map's topic to whatever it was given, and the brief took its title from the map. `loadRecipe` names the recipes that exist when one is not found (2026-09-27): `--recipe api` had answered with only the folder it looked in. Its usage row carries `searchCreditsEstimate` too (ADR-0065), counted once per search including a fallback search (a duplicated line had counted fallbacks twice until 2026-09-27). With `--max-scrapes`, a candidate is scraped only if it carries the query that found it (`matchesQuery`, ADR-0068); every candidate still appears in the map. Under a **merged** search selection it asks every provider each query and interleaves by rank (`mergeByRank`, as research does; 2026-09-28): the CLI passed only the first while printing that both were searched, so phase 0 scraped what SerpAPI alone had ranked. The usage row carries `searchesOn`; a provider that missed a query another answered is listed in the map as `answered by the other provider` and is not counted lost. **A compound topic is searched part by part** (2026-09-28, ADR-0085): `topicQueries` reads `subject: part, part; part` as one search per part (a one- or two-word part carries the subject, and so does a part that refers back with a pronoun, ADR-0098; at most `MAX_TOPIC_PARTS`, 6) instead of the whole topic four times, and `interleaveByQuery` shares the candidate list one per part. Searched whole, MoonAliza's three-question topic mapped forum threads and a coffee-scale blog, and in search order its third question got none of the map's 20 candidates. A plain topic keeps its four searches. **Outlines** (2026-09-29, ADR-0091): after gathering, `outlineOf(body)` reads the `##`/`###` headings of every material page with a capture on disk (at most `MAX_OUTLINES` pages, 12 headings each; code blocks, links, zero-width marks, repeats and `OUTLINE_FURNITURE` dropped) into a `## Outlines seen in the material` section - STORM's table-of-contents step without its model, unjudged, no fetch. Furniture also covers GitHub's file-view chrome and Discourse per-post bylines (`OUTLINE_FURNITURE_PATTERNS`, 2026-09-29). The scrape budget follows `scrapeOrder`: likely-owner hosts first, by `docsHosts` score, then the rest (ADR-0094). Owners are counted by `ownerOf` (ADR-0096): on a shared host - github.com, gitlab.com, huggingface.co, medium.com and the like - the owner is the account (the segment after `orgs/` or `users/`; the host's own sections such as `topics` stay the host's), so five strangers' repositories no longer pool into one "likely owner" that outranks the vendor's docs. A page on the owners tying for the top score (at least `MIN_NAMED_OWNER_SCORE`, 3) that the relevance floor skips is listed in the map under "Skipped on the likely owner", with `plan.json` `urls` as the way in (ADR-0097): the owner's terse "OpenAI compatibility" page had been skipped with only a log line to show for it. The floor itself still holds for it - trusting the owner past it re-admitted RR-9's postgresql.org home page. A compound-topic part that opens with "the" carries the subject, as a pronoun part does (2026-09-30, ADR-0103): "the bot limiter", searched alone, filled a SearXNG map with audio mastering limiters. `resolveTopic` refuses a plan the corpus reader flagged for a duplicate object key (2026-09-30), in the reader's words: `JSON.parse` keeps the last `topic`, so every search was built from the second while the operator read the first. Its searches go through `searchPatiently` (one call, no retries), and a failed search degrades only to a fetch provider that can search - the browser cannot (2026-10-01). Only an http(s) search result with a host (`isWebUrl`) is taken into the material, so only such pages are listed or scraped (2026-10-01). **A damaged ledger is refused before any provider is asked** (2026-10-03, outside audit): `assertAppendable(root)` at the top of `decompose`, as `runResearch` has had since ADR-0122 - phase 0 spends too, and it made four searches and one fetch before `collectOne` threw. **Its searches go through the search session** (2026-10-03, ADR-0135), the same one `runResearch` asks: `decompose` builds it with no rate-limit patience and a sink that keeps each failure row for the map, refuses a `notReady` provider through `session.assertReady()` before the ledger check (until then only the CLI refused, so a library caller spent), asks `session.ask(query)` per query and keeps every http(s) result as material, and reads `session.meters()` into its usage row. With the move it gained the three lines `runResearch` already logged (`searched`, `search found N`, `search failed on every provider`), its usage row's `searchTransport` names the paying provider under a merge as research's does (it had joined every provider with `+`), its `searchFailures` counts every failed ask (a covered miss and a degraded first ask included) and its `degraded` counts only the degrades that answered - research's counts, under the ADR's one rule. **A bare `--limit` or `--max-scrapes` is refused** (2026-10-11, #269): both go through `checkFlagValues` (`int`, `min: 0`) at the top of the CLI with the other value flags, so a flag with no value fails as `--<name> needs a value` instead of being read as `true` - which the hand-written `Number(raw)` check it replaces turned into 1, so `--dry-run --max-scrapes` reported "scrape up to 1" and a real run paid for one page. Recipe loading refuses front matter with no dimensions and blank dimension labels, naming the file and physical line. The CLI presents INVALID_RECIPE as usage failure (exit 2) before map writes (#356). **A dry run keeps a drafted map** (#305): over a map with subtopic rows, candidate material or non-template text it still previews its searches and the rows it would seed (a recipe's own rows named), then returns `{ written: false, preview: true, reason }` with the row counts - `--force` or not - and the CLI prints the reason and exits 0; only an absent, empty or template map gets the seeded one. A real run still refuses judged rows without `--force`. `--force` over judged rows keeps the judged map as `MAP.md.bak-<date>` (`backup` in the result, printed by the CLI) before redrafting (#301, 2026-10-17). | ADR-0085, ADR-0091, ADR-0094, ADR-0096, ADR-0097, ADR-0098, ADR-0135 | | `dimensions.mjs` | The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (`coverageOfUniversals`). Recipes add on top of it; nothing replaces it. | — | | `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' `verifyLedger` reports a ledger-named capture it cannot read as `raw-unreadable` instead of throwing out of preflight (ADR-0077). It refuses a capture that is not a REGULAR FILE before it opens it, for the reason `readText` does (2026-09-28, Arena break test 12): a fifo named by the ledger hung every command that verifies a corpus, and a ledger symlinked to `dev/zero` ended them with `std::bad_alloc` rather than a verdict. A lock write that fails (a full disk) closes its descriptor and removes the empty lock it just created before the error reaches the caller; it leaked both (2026-09-28). | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). A source entry without `sourceSha256` is a `body-unmodified/unhashed` problem, never skipped (2026-10-04, break-test pass 4 F4): the no-hash skip was written for the first ledgers' captures, and a source never had that history, so a rechained entry naming an unverifiable sibling is refused. | ADR-0003 (corpus-adjacent), ADR-0020 `verifyLedger` does not read a capture that lies outside the project (2026-09-30, ADR-0076); it reports `raw-outside`, which preflight fails. A chain naming `../elsewhere` or a link leading out of `research/raw` had verified a file the corpus does not hold. `verifyLedger` reads the path through `projectFile` (2026-09-30, ADR-0109). `acquire` throws `LOCK_STUCK` when a lock it judged stale cannot be removed for any reason but ENOENT (already gone is the race the loop retries): the failed removal fell through to `continue`, skipping the deadline and the sleep, so a read-only research/raw holding a dead pid's lock, or a lock path that is a directory, spun a whole core forever and printed nothing - unreachable by `--report-on-signal`, since a synchronous loop never returns to the event loop (break-test 2026-10-01). The wait-out refusal is coded `LOCK_HELD`; `withLock(root, fn, { waitMs })` is the test seam for its 120 s wait, in the spirit of `materializeIndex`'s injectable git. Evicting on age stays rejected (ADR-0025). **`assertAppendable(root)`** (ADR-0122, 2026-10-02) is the one function that establishes a ledger can be appended to: an unparsed line (`LEDGER_DAMAGED`), a torn tail (`LEDGER_TORN_TAIL`), and a chain whose `seq`/`prev`/`entrySha256` no longer link (`LEDGER_CHAIN_BROKEN`, named by line and rule, remedy `git checkout -- research/raw/.fetches.jsonl` - the kit never rewrites a hash). `chainProblems(entries)` is the link check alone, shared with `verifyLedger`; `appendFetch` asks `assertAppendable` under the lock, and `runResearch` asks it before the first fetch. Until then a parsed-but-broken chain was appended to, and the paid-for fetch was refused by handoff with the rest. `isLineEndingRewrite` keeps its own CRLF-only fold, deliberately not core's `foldLineEndings` (2026-10-03): it asks whether autocrlf explains a mismatch, and autocrlf never makes a lone CR. Tail repair recognizes the leading BOM on a one-entry ledger as the reader does, so a valid first entry is never dropped as a torn tail; repairing a later append preserves the prefix (#296). | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not. The two sanctioned rewrites, `repairLedgerTail` and `rebuildLedger`, write through `writeText` (2026-09-28, ADR-0079): they emptied the ledger before writing it, so one cut short left a truncated ledger. Both look for the ledger **before** taking the lock (2026-09-28): the lock lives in `research/raw/`, a gate marker, so `doctor --fix-arity` in a folder with no ledger made it a gated project missing its scaffold. A fetch entry names its source sibling (`source`, `sourceSha256`, ADR-0140, 2026-10-03), and `verifyLedger` checks every file an entry names the same way - the capture and the source - so an edited source is `body-unmodified` and a missing one `raw-missing`, and the commit gate refuses them as it refuses an edited capture. `sourceOmitted` rides on an entry whose source was given and not kept (2026-10-04). -| `quotes.mjs` | **Quote anchors** (2026-09-28, ADR-0087): `quoteAnchors(text)` reads the `[quote: ...]` markers of a Finding cell, split into fragments on an ellipsis - a marker opens at `[quote:` and closes at the `]` that balances its brackets, so it may hold a code subscript, a Markdown link, or a passage nested any number of levels deep (2026-09-29: it ended at the first `]`, and the stub it kept was what got checked; rank 4 of the 2026-10-03 and 2026-10-04 gap audits: a regex bounded to ONE level did not read a two-level passage as a marker at all, so the quote was not checked); `normalizeForMatch` is the comparison-only form (NFKC, Markdown link text, quotes, dashes, emphasis, escapes, whitespace, case); `anchorFound(fragments, body)` requires at least one nonempty normalized fragment and matches every nonempty fragment in order, exact after normalization, no fuzzy fallback, each fragment starting within `MAX_QUOTE_GAP` (300 compared characters) of the one before it, and a placement tried at each occurrence rather than only the first, so a word earlier in the page does not hide the passage the quote meant (rank 4, 2026-10-03 gap audit: `[quote: free ... requests ... credits]` anchored three words scattered over a pricing page); `MIN_QUOTE_WORDS` is 3, and `MIN_QUOTE_CHARS` is 40 (2026-10-02, ADR-0126): a quote is short only when EVERY fragment is under both, measured by `quoteTooShort` (rank 4, 2026-10-03 gap audit: over the JOINED fragments, `[quote: free ... requests ... credits]` read as three "words" and cleared a floor none of its fragments reaches alone) - two rows of the kit's own decision project quoted a 130-character dependency map, one whitespace-separated "word", and were warned as anchoring almost nothing. Pure; `checks.mjs` reads the capture and reports. `quoteAnchors` takes one pair of quotation marks around the whole passage off before matching (`unwrap`): `[quote: "the sentence"]` was quote-not-found for a sentence the capture held word for word (2026-10-01). `normalizeForMatch` also drops Unicode's format characters (`\p{Cf}`: zero-width space, soft hyphen, word joiner, the joiners and directional marks) after NFKC, on both sides: `## [\u200B](...#bundled-skills) Bundled skills` in this repository's own corpus made `[quote: ## Bundled skills]` unfound, and 32 of its 188 captures hold such characters (break-test 2026-10-01). An anchor that normalizes entirely to empty cannot count as found: raw formatting characters or three formatting-only words used to clear the short-quote floor while every search needle was skipped, falsely passing citations and inflating the measurement count (2026-10-08). Formatting-only fragments alongside real text remain ignored; nonempty fragments still require exact ordered matches. | ADR-0087 | -| `measure.mjs` | **How a corpus's citations hold up, without a judge** (2026-09-28, ADR-0089): `measureCorpus(root)` - link works (capture on disk, in the ledger, unmodified, not an error page), quote-anchored rows and quotes found (`quotes.mjs`), full captures, closed unknowns on a `P` row and on two or more hosts, known unknowns, ledger verifies; `factCheck` is named and never computed. `renderMeasure` for people; `bin/measure.mjs` prints either, `--json` for machines, refuses outside a project, and never gates. | ADR-0089 | +| `quotes.mjs` | **Quote anchors** (2026-09-28, ADR-0087): `quoteAnchors(text)` reads the `[quote: ...]` markers of a Finding cell, split into fragments on an ellipsis - a marker opens at `[quote:` and closes at the `]` that balances its brackets, so it may hold a code subscript, a Markdown link, or a passage nested any number of levels deep (2026-09-29: it ended at the first `]`, and the stub it kept was what got checked; rank 4 of the 2026-10-03 and 2026-10-04 gap audits: a regex bounded to ONE level did not read a two-level passage as a marker at all, so the quote was not checked); `normalizeForMatch` is the comparison-only form (NFKC, Markdown link text, quotes, dashes, emphasis, escapes, whitespace, case); `anchorFound(fragments, body)` requires at least one nonempty normalized fragment and matches every nonempty fragment in order, exact after normalization, no fuzzy fallback, each fragment starting within `MAX_QUOTE_GAP` (300 compared characters) of the one before it, and a placement tried at each occurrence rather than only the first, so a word earlier in the page does not hide the passage the quote meant (rank 4, 2026-10-03 gap audit: `[quote: free ... requests ... credits]` anchored three words scattered over a pricing page); `MIN_QUOTE_WORDS` is 3, and `MIN_QUOTE_CHARS` is 40 (2026-10-02, ADR-0126): a quote is short only when EVERY fragment is under both, measured by `quoteTooShort` (rank 4, 2026-10-03 gap audit: over the JOINED fragments, `[quote: free ... requests ... credits]` read as three "words" and cleared a floor none of its fragments reaches alone) - two rows of the kit's own decision project quoted a 130-character dependency map, one whitespace-separated "word", and were warned as anchoring almost nothing. Pure; `checks.mjs` reads the capture and reports. `quoteAnchors` takes one pair of quotation marks around the whole passage off before matching (`unwrap`): `[quote: "the sentence"]` was quote-not-found for a sentence the capture held word for word (2026-10-01). `normalizeForMatch` also drops Unicode's format characters (`\p{Cf}`: zero-width space, soft hyphen, word joiner, the joiners and directional marks) after NFKC, on both sides: `## [\u200B](...#bundled-skills) Bundled skills` in this repository's own corpus made `[quote: ## Bundled skills]` unfound, and 32 of its 188 captures hold such characters (break-test 2026-10-01). An anchor that normalizes entirely to empty cannot count as found: raw formatting characters or three formatting-only words used to clear the short-quote floor while every search needle was skipped, falsely passing citations and inflating the measurement count (2026-10-08). Formatting-only fragments alongside real text remain ignored; nonempty fragments still require exact ordered matches. `unreadableQuoteMarkers(text)` returns each `[quote` (any case, with or without a colon, closed or not) that opens no marker `quoteAnchors` reads, with its first 40 characters (#319, ADR-0175); not an attempt: a `[quote` inside a read marker, a code span or a URL, or the text of a Markdown link. Over the repository's 41 research projects it finds none. | ADR-0087, ADR-0162, ADR-0175 | +| `measure.mjs` | **How a corpus's citations hold up, without a judge** (2026-09-28, ADR-0089): `measureCorpus(root)` - link works (capture on disk, in the ledger, unmodified, not an error page), quote-anchored rows and quotes found (`quotes.mjs`; a marker the gate cannot read counts as a quote not found and its row as anchored, never as a row without a quote, #319), full captures, closed unknowns on a `P` row and on two or more hosts, known unknowns, ledger verifies; `factCheck` is named and never computed. `renderMeasure` for people; `bin/measure.mjs` prints either, `--json` for machines, refuses outside a project, and never gates. | ADR-0089, ADR-0175 | | `warc.mjs` | **The corpus as WARC 1.1** (2026-09-28, ADR-0090): `exportWarc(root)` - a `warcinfo` record, then per ledger `scrape` entry whose capture is on disk and unchanged a `resource` record (the text after the front matter, `text/markdown`) and a `metadata` record (`WARC-Concurrent-To`; transport, completeness, the capture file's hash, the block's own `body-sha256` and the ledger's seq); `WARC-Block-Digest: sha256:`, record IDs derived from the ledger entry (`uuidFrom`, RFC 9562 v8), each record its own gzip member; an edited capture is left out and named in `skipped`, as is a ledger entry the export cannot date (WARC-Date is mandatory; rank 7, 2026-10-03 gap audit, ADR-0161). `readWarc` splits a stream back into records and throws on bad framing. Validated with warcio on every corpus. `exportWarc` reads a ledger-named capture only if it is inside the project (`isInside` and `realInside`, ADR-0076) and a regular file (2026-09-30); anything else is `skipped` with its reason. A ledger naming `../` or a link out of `research/raw` had packed that outside file into the archive, and a FIFO would have hung the export. `exportWarc` reads the path through `projectFile` (2026-09-30, ADR-0109). The record is parsed from the bytes whose hash passed, never a second read of the file (2026-09-30). | ADR-0090 | | `prior.mjs` | **The registered prior: what was expected BEFORE the evidence, and the proof that it was.** `registerPrior` writes `research/PRIOR.md` and chains an `op: 'prior'` entry carrying its sha256, so two things stop being editable at once - the text, through `body-unmodified`, and its POSITION before the first scrape, through the chain itself. Three dishonest paths, each measured rather than argued: revising the prediction once the answer is known fails the chain; registering late is refused by the tool; and chaining one late BY HAND with every hash correct is caught by `provenance/prior-precedes-collection`, which exists because `appendFetch` is a public export and the gate does not get to trust that the CLI was used. **Nothing grades the prediction** (ADR-0013) - not its specificity, not its reasonableness, and above all not whether it turned out right, since a prior that was wrong has done its job and a gate that preferred correct ones would teach the opposite habit. The single content rule is a length floor, stated as a floor. The decision to proceed is taken inside the same section of the lock as the append, on the ledger read there (#352); a refusal is decided before the lock too, since the collection refusals only accumulate. A ledger that records no scrape beside captures or evidence rows on disk is a lost or emptied one (`unrecordedCollection`, #353): registering is refused, and `readPrior` reports it as `unrecorded`, so `prior-precedes-collection`, the read-back and the brief's prior section all say the order is unknown (`precedes` is their one verdict), naming the files and both remedies. A stray capture beside scrapes on record is `hygiene/uncited-capture`'s, not the prior's. A prior is bounded at `CAPTURE_MAX_BYTES`, and `readPriorSource` checks a `--file` before reading it, because the brief quotes the prior whole (#355). Absence is SILENT at the gate: a prior is optional, and a warning on every project that skipped one is a warning nobody reads. `bin/prior.mjs` registers and reads it back; `bin/research.mjs` prints a one-line nudge before a spending run, because after the first page lands there is no second chance. **It also reaches the REMOTE path, and that half is what makes the rest useful**: the collector workflow scaffolds its own project on the runner with an empty ledger, so a prior registered on the operator machine cannot reach the corpus that comes back - and every corpus collected since `collect-remote` shipped came from that route. The prediction therefore travels as a dispatch input and is registered on the runner between the map and the collection, through `env` and an argv array like the topic (ADR-0020), with the topic's measured disclosure profile (ADR-0035). A test pins that the registration step precedes `collect`, because if it ever drifts below it, `bin/prior.mjs` refuses and the feature stops working silently on a workflow CI never runs. | ADR-0013, ADR-0020, ADR-0039 | -| `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. `partial-render` applies only to a capture behind an EVIDENCE row some unknown cites (ADR-0099): on a context-only row its instruction had no cited text to confirm. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. The same duty covers what a PASS claims: `gate-integrity` used to pass with "no override in effect" when what it measured was that no override was RECORDED - and `git commit --no-verify`, the escape hatch this kit documents most prominently, records nothing. It now says what it counted and names the bypass it cannot see. Found by sweeping all thirteen pass messages for claims wider than their measurement; it was the only one. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. `collection-attempts` counts an attempt through the rows a KNOWN-UNKNOWN cites, not only a URL typed into it (2026-09-26): an E-row is a capture of its URL, so citing one is the stronger proof of having reached for it. Until then a KNOWN-UNKNOWN citing the page it tried was warned as never attempted - found by the `2026-09-26-actions-sept-changes` project's own gate, and no test exercised its verdict (only the registry-order test named it). `citations/url-mismatch` (2026-09-27) fails a row whose URL is not, by `urlKey`, the page its capture was fetched from: a row could name another page and pass, and the URL cell is what a reader is shown as the source. `provenance/ledger-unparsed` names its repair (2026-09-27): a torn last line gets `doctor --fix-arity`, which drops exactly that; a broken line inside the chain is restored from git. It gave only the JSON error. A row is never compared with itself (2026-09-27): `supersededRows` skips a same-ID pair and `hygiene/duplicate-url` leaves the same ID to `duplicate-id`; a row pasted twice failed with "E-01 has been superseded by E-01". `hygiene/future-date` fails a Retrieved date more than a day ahead (2026-09-27): 2030-01-01 had passed, hiding the row's age from every freshness check. `hygiene/duplicate-id` fails rather than warns (2026-09-27, ADR-0054): two rows under one ID leave every citation, COVERED cell and brief reference to it ambiguous, because lookups keep the last row. A contract with two U-1 rows had passed. `corroboration` weighs each unknown ID once, so it no longer prints the same single-source warning twice. `hygiene/brief-stale` warns when a stamped brief's inputs hash no longer matches the corpus (2026-09-27, ADR-0055) and names the command that redrafts it; `hygiene/brief-unstamped` warns on a drafted brief with no stamp (2026-10-03, ADR-0138) - its currency cannot be checked and it keeps its approval, so the gate says so and names the redraft that stamps it; a template or a hand-written legacy brief gets no such warning. `hygiene/duplicate-url` treats two same-day rows for one URL as a refresh when the ledger records a fetch for each (2026-09-27, ADR-0060); a pasted row, with one fetch behind two rows, is still a duplicate. `discovery-contract/build-intent` tells a missing heading from an empty one, from the corpus's `intentHeading` (2026-09-27): a renamed heading read as "is empty". **`citations` checks quote anchors** (2026-09-28, ADR-0087): each `[quote: ...]` in a Finding must occur in the row's capture (`quote-not-found`, blocking) and carry at least three words or 40 characters in EVERY fragment (`quote-too-short`, warn; the character floor is ADR-0126, 2026-10-02; per-fragment measurement and the one-passage bound are rank 4 of the 2026-10-03 gap audit); the matching lives in `quotes.mjs`. `citations/raw-error-status` warns when a row cites an HTTP 4xx/5xx capture from before that refusal, unless the row names the status itself (this repository's E-15 keeps a 404 as the record of a failed guess). **`unknown-closure/secondary-only`** (2026-09-28) warns when a CLOSED unknown cites no `P` row and at least one `S` row: the registry promised "a primary row" and only `L` was checked, so three corpora closed everything on `S` rows with no warning (found by the first measurement). Retrieval dates compare by `compareText` (2026-10-01). `runChecks` appends each check's findings one at a time (2026-10-01, break-test PR #188): `push(...check.run())` passed every finding as an argument, and past the engine's limit - 60,000 to 130,000 findings from one check, by Node line - the verdict died with a raw RangeError; preflight, doctor, brief and audit crashed, and the commit gate answered internal error. Reachable from about 66,000 evidence rows. The other spread sites (`priorOrder`, doctor's `gateHealth`) are bounded by the contract's unknown count and left as they are. **`hygiene` fails a malformed ID** (2026-10-03, audit G2): each `malformed-id` problem the corpus reader records becomes a `fail` beside `duplicate-id`, for the same reason - the ID is how every check finds a row; `corpus-shape` skips the kind so one typo is not reported twice. A `table-split` problem fails the same way (2026-10-03, gap audit rank 1): the row no check reads is named with its file and line, and `corpus-shape` does not repeat it. **Freshness is judged by the capture's own date** (2026-10-03, audit G8): `unknown-closure` ages a cited row by `freshnessDate` - the capture's front-matter `retrieved`, written by the collector at the fetch and under the ledger's hash, with the table's cell as the fallback for a row whose capture carries no parseable date - and names the judging date when the two differ; `hygiene` warns `date-mismatch` when a row's cell and its capture name different calendar days (`retrievedDay`), with the remedy of writing the capture's date in the row. Editing E-19's cell alone had made its stale warning disappear while the capture on disk still said 2026-09-20. The capture judged is the one the row NAMES (`ownCapture`, review of G8 the same day): `captureOf` answers a blank Raw cell with the URL's latest capture, and judged by that a row with no Raw cell borrowed a later fetch's date, lost its stale warning and drew a date-mismatch telling the operator to write the later date in; a row naming no capture is judged by its cell and has nothing to reconcile against. A future date in the capture fails `future-date` as one in the cell does, naming the capture, and the stale hint's "fresher capture of the same URL" is found by capture date (`freshnessDay`), not by the cells. **Same-day supersession follows the ledger** (2026-10-03, audit G4): `supersededRows` orders two rows of one URL by the day freshness judges them by (`freshnessDay` - the capture's date; ordered by the table cell, one edit to an older row's date had failed the gate on the genuinely newest row and told the operator to cite the older one; review of G4, the same day) and then by the ledger position of each row's cited capture (`ledgerRank`, the rule `newer` applies for the collector); a capture no fetch recorded ranks below every fetched one, and only among those does the table's order still break the tie. With A fetched, then B, then A again, the collector had held A current while the check said B superseded it, and swapping two evidence rows swapped the verdict. `uncited-capture` says "was collected" only of a file a ledger scrape entry names; a file under research/raw no fetch produced is named as such (2026-10-04, break-test pass 4 F5a). `discovery-contract/contract-missing` states that the contract is missing or could not be read (2026-10-08): the reader uses the same null fallback for absence and refused reads, so that finding cannot establish absence. Both states keep the existing failing rule; no filesystem-state API or new check is introduced. **Evidence URL comparisons use the adopted page identity** (2026-10-09, audit rank 14): supersession groups, stale-citation fresher-row hints and duplicate-URL hygiene compare through `urlKey` (ADR-0136), as the collector and cache already do. Capture dates, blank-Raw table-date fallback, same-day ledger ordering, the same-ID exclusion and the two-recorded-fetch refresh rule retain their existing roles. Captured URL spellings remain visible; query strings and path case remain distinct. | ADR-0004, ADR-0026, ADR-0087, ADR-0136 | +| `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. `partial-render` applies only to a capture behind an EVIDENCE row some unknown cites (ADR-0099): on a context-only row its instruction had no cited text to confirm. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. The same duty covers what a PASS claims: `gate-integrity` used to pass with "no override in effect" when what it measured was that no override was RECORDED - and `git commit --no-verify`, the escape hatch this kit documents most prominently, records nothing. It now says what it counted and names the bypass it cannot see. Found by sweeping all thirteen pass messages for claims wider than their measurement; it was the only one. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. `collection-attempts` counts an attempt through the rows a KNOWN-UNKNOWN cites, not only a URL typed into it (2026-09-26): an E-row is a capture of its URL, so citing one is the stronger proof of having reached for it. Until then a KNOWN-UNKNOWN citing the page it tried was warned as never attempted - found by the `2026-09-26-actions-sept-changes` project's own gate, and no test exercised its verdict (only the registry-order test named it). `citations/url-mismatch` (2026-09-27) fails a row whose URL is not, by `urlKey`, the page its capture was fetched from: a row could name another page and pass, and the URL cell is what a reader is shown as the source. `provenance/ledger-unparsed` names its repair (2026-09-27): a torn last line gets `doctor --fix-arity`, which drops exactly that; a broken line inside the chain is restored from git. It gave only the JSON error. A row is never compared with itself (2026-09-27): `supersededRows` skips a same-ID pair and `hygiene/duplicate-url` leaves the same ID to `duplicate-id`; a row pasted twice failed with "E-01 has been superseded by E-01". `hygiene/future-date` fails a Retrieved date more than a day ahead (2026-09-27): 2030-01-01 had passed, hiding the row's age from every freshness check. `hygiene/duplicate-id` fails rather than warns (2026-09-27, ADR-0054): two rows under one ID leave every citation, COVERED cell and brief reference to it ambiguous, because lookups keep the last row. A contract with two U-1 rows had passed. `corroboration` weighs each unknown ID once, so it no longer prints the same single-source warning twice. `hygiene/brief-stale` warns when a stamped brief's inputs hash no longer matches the corpus (2026-09-27, ADR-0055) and names the command that redrafts it; `hygiene/brief-unstamped` warns on a drafted brief with no stamp (2026-10-03, ADR-0138) - its currency cannot be checked and it keeps its approval, so the gate says so and names the redraft that stamps it; a template or a hand-written legacy brief gets no such warning. `hygiene/duplicate-url` treats two same-day rows for one URL as a refresh when the ledger records a fetch for each (2026-09-27, ADR-0060); a pasted row, with one fetch behind two rows, is still a duplicate. `discovery-contract/build-intent` tells a missing heading from an empty one, from the corpus's `intentHeading` (2026-09-27): a renamed heading read as "is empty". **`citations` checks quote anchors** (2026-09-28, ADR-0087): each `[quote: ...]` in a Finding must occur in the row's capture (`quote-not-found`, blocking) and carry at least three words or 40 characters in EVERY fragment (`quote-too-short`, warn; the character floor is ADR-0126, 2026-10-02; per-fragment measurement and the one-passage bound are rank 4 of the 2026-10-03 gap audit), and a `[quote` it cannot read as a marker - no closing bracket, a space before the colon, no colon - is `quote-unreadable`, blocking, naming the row and both repairs (#319, ADR-0175): it was no marker at all, so an invented sentence in that shape passed where the same sentence written `[quote: ...]` failed; the matching lives in `quotes.mjs`. `citations/raw-error-status` warns when a row cites an HTTP 4xx/5xx capture from before that refusal, unless the row names the status itself (this repository's E-15 keeps a 404 as the record of a failed guess). **`unknown-closure/secondary-only`** (2026-09-28) warns when a CLOSED unknown cites no `P` row and at least one `S` row: the registry promised "a primary row" and only `L` was checked, so three corpora closed everything on `S` rows with no warning (found by the first measurement). Retrieval dates compare by `compareText` (2026-10-01). `runChecks` appends each check's findings one at a time (2026-10-01, break-test PR #188): `push(...check.run())` passed every finding as an argument, and past the engine's limit - 60,000 to 130,000 findings from one check, by Node line - the verdict died with a raw RangeError; preflight, doctor, brief and audit crashed, and the commit gate answered internal error. Reachable from about 66,000 evidence rows. The other spread sites (`priorOrder`, doctor's `gateHealth`) are bounded by the contract's unknown count and left as they are. **`hygiene` fails a malformed ID** (2026-10-03, audit G2): each `malformed-id` problem the corpus reader records becomes a `fail` beside `duplicate-id`, for the same reason - the ID is how every check finds a row; `corpus-shape` skips the kind so one typo is not reported twice. A `table-split` problem fails the same way (2026-10-03, gap audit rank 1): the row no check reads is named with its file and line, and `corpus-shape` does not repeat it. **Freshness is judged by the capture's own date** (2026-10-03, audit G8): `unknown-closure` ages a cited row by `freshnessDate` - the capture's front-matter `retrieved`, written by the collector at the fetch and under the ledger's hash, with the table's cell as the fallback for a row whose capture carries no parseable date - and names the judging date when the two differ; `hygiene` warns `date-mismatch` when a row's cell and its capture name different calendar days (`retrievedDay`), with the remedy of writing the capture's date in the row. Editing E-19's cell alone had made its stale warning disappear while the capture on disk still said 2026-09-20. The capture judged is the one the row NAMES (`ownCapture`, review of G8 the same day): `captureOf` answers a blank Raw cell with the URL's latest capture, and judged by that a row with no Raw cell borrowed a later fetch's date, lost its stale warning and drew a date-mismatch telling the operator to write the later date in; a row naming no capture is judged by its cell and has nothing to reconcile against. A future date in the capture fails `future-date` as one in the cell does, naming the capture, and the stale hint's "fresher capture of the same URL" is found by capture date (`freshnessDay`), not by the cells. **Same-day supersession follows the ledger** (2026-10-03, audit G4): `supersededRows` orders two rows of one URL by the day freshness judges them by (`freshnessDay` - the capture's date; ordered by the table cell, one edit to an older row's date had failed the gate on the genuinely newest row and told the operator to cite the older one; review of G4, the same day) and then by the ledger position of each row's cited capture (`ledgerRank`, the rule `newer` applies for the collector); a capture no fetch recorded ranks below every fetched one, and only among those does the table's order still break the tie. With A fetched, then B, then A again, the collector had held A current while the check said B superseded it, and swapping two evidence rows swapped the verdict. `uncited-capture` says "was collected" only of a file a ledger scrape entry names; a file under research/raw no fetch produced is named as such (2026-10-04, break-test pass 4 F5a). `discovery-contract/contract-missing` states that the contract is missing or could not be read (2026-10-08): the reader uses the same null fallback for absence and refused reads, so that finding cannot establish absence. Both states keep the existing failing rule; no filesystem-state API or new check is introduced. **Evidence URL comparisons use the adopted page identity** (2026-10-09, audit rank 14): supersession groups, stale-citation fresher-row hints and duplicate-URL hygiene compare through `urlKey` (ADR-0136), as the collector and cache already do. Capture dates, blank-Raw table-date fallback, same-day ledger ordering, the same-ID exclusion and the two-recorded-fetch refresh rule retain their existing roles. Captured URL spellings remain visible; query strings and path case remain distinct. | ADR-0004, ADR-0026, ADR-0087, ADR-0136 | | `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped, and **a word means a word in any script**: `\p{L}\p{N}\p{M}` and NFC composition, not the Latin-ASCII `[^a-z0-9]` class, which stripped every non-Latin letter and left a Russian, Arabic, Greek or accented page with NO shingle at all - so `sketch` returned null for it and a republished copy counted as an independent witness, rank 17 of the 2026-10-04 gap audit; ASCII input is byte-identical, so the measurement below still holds) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 | | `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 | | `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, **what phase 1 may commit** (ADR-0048: `research/`, the root `SCAFFOLDING` - `.gitattributes`, `.gitignore`, `AGENTS.md`, `START_HERE.md` - and `docs/ARCHITECTURE.md` while `isEmptyArchitectureMap` says its staged bytes, read by `readStagedText`, are still the template's; until 2026-09-27 only `research/`, so a new project's first `git add -A` was refused and its `.gitattributes` could not travel), and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **Staged paths remain data, with checked acquisition** (ADR-0154): the default commit CLI lazily calls `readStagedPaths` after the ungated and `GATE_OFF` returns. One actual `git diff --cached --name-only --no-renames -z` supplies paths only on successful complete output; the existing 64 MiB capture rejects nonzero exit and buffer failure, and the reader rejects malformed NUL data. Successful zero bytes means `[]`; unreadable scope means null and raises the existing architecture-map breach even after green preflight, ordinary exit 1 under either posture. Explicit arrays/null and the existing stdin/repeated `--staged` options retain their meaning; edits never acquire a list. **One deliberate exception to "no CLI parsing"** remains for explicitly supplied stdin data (`splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"**, preserving unknown scope rather than assuming an empty set. `bin/gate.mjs --staged-stdin` reads stdin itself (`stdinIsReadable`, then `fs.readFileSync(0)` into `splitPathList`) and refuses a terminal by name rather than guessing an empty set; the unused `stagedPathsFromStdin` helper, which answered that case with a silent null, was removed (#297, 2026-10-17). No list read claims an atomic snapshot with later index materialization. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). The architecture-map block's fix says to **update** the map, then `git add` it (2026-09-27): a bare `git add docs/ARCHITECTURE.md` stages nothing while the map is unchanged, so the printed fix left the commit blocked. `bin/gate.mjs` no longer closes that block with "Phase 1 is not done until preflight prints PASS" - a map block happens only when the gate passes. `isPhaseOneEdit(root, file)` answers the same question for the edit hook, about one file. The empty-index verdict — a gated project whose `research/` is untracked, every fresh project's first commit — removes the scratch dir `materializeIndex` made before it learned the index was empty (2026-09-28): it returned before `scratch` was assigned, so the `finally` that removes it never ran, and every such verdict leaked one temp directory. **Integrity before scope** (ADR-0093): a failing verdict whose findings include an `INTEGRITY_RULES` provenance rule blocks the commit even when it is confined to `research/` - the ADR-0048 allowance covers completeness findings only. `stdinIsReadable(stdin = process.stdin)` takes the stream to ask about, so the hook's terminal case is testable (2026-10-01). **The suite rule** (ADR-0120, 2026-10-01): `suiteBreach(root, stagedPaths, { run })` runs after the map rule, in the commit gate only, and only where `isKitCheckout` finds the kit's four markers and a staged path is under `research-kit/` (or the list is unknown); `runSuiteHere` spawns the checkout's `bin/selftest.mjs` with `RESEARCH_KIT_RESULT_FILE` pointed at a scratch file and `RESEARCH_KIT_ALLOW_UNSUP=1` (unsupported is not red, ADR-0108), git's hook environment stripped (`GIT_INDEX_FILE`, `GIT_DIR`, `GIT_PREFIX` and kin - the same `REPOSITORY_LOCATION` list as the edit hook, else every scratch repository's git in the suite would act on the index being committed), and reads the file, never stdout: failures block naming the counts, no result blocks as a crashed runner, a run past its budget is stopped and blocks as unfinished, the fix naming `RESEARCH_KIT_GATE_TIMEOUT` before the override, and a green run whose runner exited non-zero (the README count check) blocks too. `suiteOwed(root, stagedPaths)` is the predicate alone; `suiteBreach`'s `announce` (`evaluate`'s `announceSuite`) is called right before the run, so `bin/gate.mjs` says the suite is running only when it is - not when the map rule or preflight refused the commit first. `evaluate` takes `runSuite` so the verdict is testable without a suite. **The suite's budget follows the gate's watchdog** (ADR-0142, 2026-10-05): `suiteBudgetMs(env)` is `SUITE_TIMEOUT_MS` (65 min, twice the slowest honest run measured, 1825 s on the operator's Windows PC) unless `RESEARCH_KIT_GATE_TIMEOUT` is set to a positive duration `gateTimeoutSeconds` reads (seconds, an optional fraction, an optional `s`/`m`/`h`/`d`, at most nine digits before the point - the grammar the hook's `gate_timeout_readable` checks too; `0` and anything outside it leave the default), and then that watchdog less `GATE_REST_S` (120 s, the hook's default watchdog for a gate that runs no suite), never more than half a short one, at least 1 ms and at most 2^31 - 1 - so the gate stops the suite and blocks before the hook's watchdog fires, whenever its own work fits what is left. `runSuiteHere` stops the runner with SIGKILL: the harness's SIGTERM listener kept a synchronously blocked runner alive past its budget on POSIX; a stopped run's scratch is left in the temp folder. The budget had been 20 minutes, which that PC's green suite met on every kit commit, and the variable had reached only the watchdog: set below the budget, the watchdog killed the gate first and a fail-open machine allowed a commit whose suite never reported. **The fix lines no longer say `--no-verify` is recorded** (2026-10-03, outside audit): git skips this hook, so nothing can record it; `checks.mjs` had said so since ADR-0035 while the gate's own fix text, the README, AGENTS.md and CONTEXT.md said the opposite. **Red suite diagnostics** (2026-10-08): `suiteBreach` forwards up to five `failed` identifiers, each limited to 160 characters with controls normalized, in its existing detail string; omitted or clipped data is named, and legacy missing labels are explicit. Counts and verdict keys are unchanged. `runSuiteHere` still removes its scratch result; no result retention system is added. **A project below the repository root** (#309, ADR-0182): git runs the hook at the top level, and an ungated top level answered `not-gated` before reading a staged path. `bin/gate.mjs` now, at an ungated cwd, reads the staged list (supplied, or `readStagedPaths`) and `stagedProjects(top, paths, { contracts })` assigns each path to its nearest project below the top. Below the top a project is a folder with `research/DISCOVERY.md` and either `research/raw/` or a contract `new-project` rendered (`isProjectBelowTop`, `isKitContract`: the kit's heading, no `{{TOPIC}}` token; any one of the four markers had judged data folders and the vendored `template/` in repositories without the kit, and raw/ alone did not travel to a clone of a repository that ignores `raw/`, review 2026-10-18): `trackedContracts` lists the tracked contracts with one `git ls-files` (its glob kept a glob under `GIT_LITERAL_PATHSPECS`), so folders are string lookups (30,000 paths in 9,060 folders had cost 25 s at four stats each), a candidate without raw/ has its contract read from the index (`contractText`; HEAD's for a deletion), and a contract or raw path the staged list names - a deletion - counts too; without the listing each folder is asked of the disk. Each project is `evaluate`d with its own paths relative to it and `top` set, so `materializeIndex`, `readStagedText` and `stagedRawDeletions` run git at the top level and name the project's files by their path from there (`materializeIndex` returns the project's scratch `root`). Any project that blocks blocks, named in the output ("BLOCKED in packages/a", the fix run from the project - `cd <project> && ...`, quoted by `shellWord` - and the project's own `research/GATE_OFF` as the override) and as `project` in `--json`; a list git cannot read at an ungated top level is not judged, as before, and is said on stderr. Diagnostics are recorded only into a `research/raw/` that exists: creating it makes a folder gated. **Removing evidence HEAD holds is an integrity block** (#312): when the commit gate judged the index and a staged path is under `research/raw/` (or the list is unknown), `evidenceRemoval` runs BEFORE the pass branch - a capture deleted with the ledger line naming it leaves a chain that verifies. It blocks when HEAD's ledger (`readHeadLedger`, `git show HEAD:...`) is being deleted; when an entry HEAD records is gone from the staged ledger, matched by seq and file so the sanctioned `rebuildLedger` migration passes; and when `stagedRawDeletions` (`git diff --cached --diff-filter=D --no-renames`, so a rename's deleting half counts) removes a capture or source that HEAD's or the staged ledger names - each naming the file or seq and `git checkout HEAD -- <file>`, whatever else is staged. A capture never in HEAD (mid-collection) or not staged (part-staged) stays committable, as ADR-0093 says; a deletion list git cannot read blocks. The first version read only the staged ledger and ran only on a failing verdict, so `git rm -r research/raw`, or a capture removed with its ledger line, committed (review 2026-10-18). | ADR-0002, ADR-0007, ADR-0008, ADR-0020, ADR-0093, ADR-0120, ADR-0142, ADR-0154, ADR-0182 | diff --git a/docs/adr/0175-a-quote-marker-the-gate-cannot-read-blocks.md b/docs/adr/0175-a-quote-marker-the-gate-cannot-read-blocks.md new file mode 100644 index 00000000..a80e0021 --- /dev/null +++ b/docs/adr/0175-a-quote-marker-the-gate-cannot-read-blocks.md @@ -0,0 +1,82 @@ +# ADR-0175 - A quote marker the gate cannot read blocks, as a quote it refutes does + +- **Date:** 2026-10-19 +- **Status:** accepted +- **Area:** `research-kit/lib/quotes.mjs`, `research-kit/lib/checks.mjs` (`citations`), + `research-kit/lib/measure.mjs`. One more rule in a check that already owns two quote rules + (`quote-not-found`, `quote-too-short`); no command, flag, format or configuration key. A new + finding under the feature freeze (ADR-0117), hence this record. +- **Refines:** ADR-0087 (a `[quote: ...]` anchor must occur in the capture) and ADR-0162 (a + marker is read at any bracket depth). Both stand. + +## Context + +Issue #319 measured, on a copy of this repository's corpus, six Finding endings against the gate +and `measure`. Two were read: a real passage passed, an invented one failed `quote-not-found`. +Four were not markers at all, so the row passed as one with no quote and `measure` counted it +among the unanchored rows: + +- `[quote: calls[a[0]].id is read on billing]` - brackets two levels deep; +- `[quote: Billing is credit-based` - no closing bracket; +- `[Quote : Billing is credit-based]` - a space before the colon; +- `[quote Billing is credit-based]` - no colon. + +From the writer's side each is a quotation that was written down and never checked: an invented +sentence in any of these shapes passed, where the same sentence in the exact shape failed. That +is ADR-0087's own case - a report whose six quotations, attached to real citations, occurred +nowhere in the corpus - reproduced by a typo an agent writing the marker by hand makes. + +ADR-0162 closed the first shape by counting brackets, and refused to name an unreadable marker +instead, because a Finding may discuss the marker syntax and a warning on an empty or +unterminated marker would fire on that prose. Counting brackets cannot close the other three: no +reading of `[Quote :` or `[quote the sentence]` recovers the passage without guessing where it +starts and ends. + +## Decision + +`unreadableQuoteMarkers(text)` in `quotes.mjs` returns every `[quote` (any case, any spacing +after the bracket, with or without a colon, closed or not) that does not open a marker +`quoteAnchors` reads. `citations` reports each as `quote-unreadable`, **fail**, naming the row, +the first 40 characters of the attempt, and both repairs: write it as `[quote: the passage]`, or +drop the marker and paraphrase. It blocks for the reason `quote-not-found` blocks: in both cases +the quotation is unverified, and a reader of the brief cannot tell. + +`measure` counts each as a quote, not found, and its row as anchored: the row carries a quotation, +and the quotation was not found in the capture. + +Not attempts, so not reported: + +- a `[quote` inside a marker already read (a passage that quotes the syntax); +- a `[quote` inside a code span or a URL, the one false-positive shape #319 names; +- a Markdown link whose text starts with the word (`[Quote of the day](https://...)`), which is + a link, not a marker; +- `[quotes]`, `[quoted]` and the like (`\bquote\b`). + +ADR-0162's concern is answered by measurement rather than by staying silent: over the 41 +research projects in this repository (the root corpus and every `docs/decisions/*` project), +417 EVIDENCE rows and their 517 readable anchors, the rule fires 0 times. The prose that writes +the syntax as an example writes `[quote: ...]`, which is a read marker, or puts it in backticks. + +## Rejected alternatives + +- **Read the typo shapes as if they were well-formed** (accept `[Quote :` and `[quote `, or close + an unterminated marker at the cell's end). A tolerant parser guesses the passage: without a + colon, where does the claim's own text stop and the quote start, and without a closing bracket, + is the rest of the cell part of it? A guessed passage that happens to match proves nothing, and + one that does not produces a `quote-not-found` on text the writer never quoted. Naming the + attempt and the one shape that is read is exact. +- **Warn instead of fail.** A warning leaves the brief passing with a quotation nothing checked, + which is the case this check exists to stop; `quote-not-found` is blocking for the same reason. + The repair is one character, and the detail names it. +- **Count it as an unanchored row in `measure`** (the old behaviour, unchanged). The row is not + without a quote - its writer wrote one - and leaving it out of the quotes raised the "quotes + found" share with every quotation written wrongly. +- **Report it in `quoteAnchors`' return value** (an `unreadable` field beside the anchors). Every + caller of `quoteAnchors` would have to learn to look; a separate function keeps the anchors' + shape, and the two callers that need the attempts ask for them. + +## Consequences + +`unreadableQuoteMarkers` is exported. `citations` holds three quote rules: `quote-not-found` and +`quote-unreadable` (fail), `quote-too-short` (warn). A brief that passed before can now fail, only +by holding a `[quote` the gate did not read - which, measured on this repository, none does. diff --git a/docs/adr/README.md b/docs/adr/README.md index 0a457c9e..cbd5edb4 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -182,6 +182,7 @@ directory and are treated as ADR-equivalent. | [0172](0172-the-mcp-stdio-loop-answers-pings-while-a-call-is-in-flight.md) | **The MCP stdio loop frames synchronously and answers pings while a call is in flight** (supersedes ADR-0034's one-chain consequence). Each `data` event cuts, caps and parses its chunk with no await, so the 4 MiB cap holds during a call (#346); `tools/call` keeps its own chain, one at a time in order, and `ping`, `tools/list` and the rest go on a second chain that never waits for a call (#347). Rejected: every answer concurrent, `pause()`/`resume()` back-pressure, unordered non-call answers. | | [0173](0173-a-topic-request-waits-for-an-approved-topics-folder.md) | **A topic request waits for an approved topics folder.** With no folder approved, auto-collect reports a topic request `waiting` with the reason - no result, no commit - and runs it unchanged on the first cycle after the operator sets the folder, as it already does while paused or over today's cap (#323). Amends ADR-0148 decision 4's last sentence; content and target-safety refusals are unchanged. Rejected: keeping the final refusal, refusing then re-queuing on a settings change, retrying every refusal. | | [0174](0174-a-page-the-kit-refused-for-what-it-is-is-not-fetched-again.md) | **A page the kit refused for what it is is not fetched again within the refresh window** (extends ADR-0100). An 11 MB page, a PDF and a 403 were scraped and paid for on every run; `recentlyGone` now also remembers the size cap and the binary refusal as `refused` on any route, a standing 4xx (not 405/407/408/429, never a rate limit) for the route that got it, and reads a 404/410 in the transports' wording too. 5xx, timeouts, rate limits and transport errors are still retried; `--force` retries anything. Rejected: remembering every failure, leaving the 4xx out, remembering 408/429 or a proxy's 405/407, a structured ledger field, keying all or none of the memory on the transport. | +| [0175](0175-a-quote-marker-the-gate-cannot-read-blocks.md) | **A quote marker the gate cannot read blocks** (`citations/quote-unreadable`, fail): a `[quote` with no closing bracket, a space before the colon or no colon was not a marker at all, so an invented sentence in that shape passed where the same sentence written `[quote: ...]` failed; `measure` counts it as a quote not found, not as an unanchored row (#319). Not attempts: inside a read marker, a code span, a URL, or Markdown link text. 0 hits over the repository's 41 projects. Rejected: a tolerant parser that guesses the passage, warn-only, counting it as unanchored. | | [0180](0180-config-ill-typed-covers-every-typed-config-key.md) | **`config-ill-typed` covers every typed config key.** A `transport`, `searchTransport`, `serpapiKey`, `searxngUrl`, `browserPath`, `projectSkillDir`, `maxAgeDays`, `editGate.settingsPath` or `skillRoots` that is present and not of its type keeps its default and is named, with what it should hold; a `skillRoots` list that loses an entry is named too; an `editGate` that is not an object is `editGate.mode` ill-typed, resolved to `hard-block` (#310). Extends ADR-0121, which named the posture keys only; rejected: posture keys only, a second finding name, an unreadable config on a wrong type, `ask` for a non-object `editGate`. | | [0182](0182-the-gates-judge-a-path-by-its-nearest-gated-project.md) | **The gates judge a path by the project it belongs to in its repository** (#309). git runs the commit hook at the top level, so a project below it was never judged; at an ungated top level each staged path now goes to its nearest project below - a folder whose tracked `research/DISCOVERY.md` new-project rendered, or that holds `research/raw/`, so data folders and the vendored template are not projects and a clone that never received raw/ still is - judged from the index and named in the verdict. The edit gate judges a named file the same way, asks a gated top level alone, and keeps the raw/ and GATE_OFF refusals for every project above the file; doctor says where the commit gate runs. Rejected: judging by the cwd, any one of the four markers below the top, DISCOVERY.md or kit.json alone, raw/ on disk as the only second signal, disk-only markers, an unbounded walk up, judging nested projects under a gated root (deferred), a doctor warning alone. | diff --git a/research-kit/README.md b/research-kit/README.md index 400d06a1..b33598d4 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -568,7 +568,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -2054 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +2057 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/lib/checks.mjs b/research-kit/lib/checks.mjs index e63c9454..69c81686 100644 --- a/research-kit/lib/checks.mjs +++ b/research-kit/lib/checks.mjs @@ -13,7 +13,7 @@ import { documentGroups, closestPair } from './similarity.mjs'; import { coverageOfUniversals } from './dimensions.mjs'; import { verifyLedger } from './provenance.mjs'; import { readPrior, onDiskOf, unrecordedDetail, PRIOR_PATH } from './prior.mjs'; -import { quoteAnchors, anchorFound, quoteTooShort, MIN_QUOTE_WORDS, MIN_QUOTE_CHARS } from './quotes.mjs'; +import { quoteAnchors, anchorFound, quoteTooShort, unreadableQuoteMarkers, MIN_QUOTE_WORDS, MIN_QUOTE_CHARS } from './quotes.mjs'; import { draftStamp, briefInputsHash, briefState } from './brief.mjs'; const VALID_STATUSES = ['CLOSED', 'KNOWN-UNKNOWN']; @@ -103,6 +103,13 @@ function citations(corpus) { if (!row.finding.trim()) { out.push(finding('fail', 'citations', 'row-finding', `${row.id} has an empty Finding cell`, { row: row.id, line: row.line })); } + // A quote marker the parser cannot read is a quotation nothing checks, so it blocks like a + // quote the capture refutes (#319, ADR-0175). + for (const attempt of unreadableQuoteMarkers(row.finding)) { + out.push(finding('fail', 'citations', 'quote-unreadable', + `${row.id} holds a quote marker the gate cannot read, "${attempt.text}${attempt.text.length < 40 ? '' : '...'}" - write it as [quote: the passage], the colon right after "quote" and the marker closed with "]", or drop the marker and paraphrase`, + { row: row.id, line: row.line })); + } if (!['P', 'S', 'L'].includes(row.type)) { out.push(finding('warn', 'citations', 'row-type', `${row.id} has source type "${row.type || '(blank)'}" - expected P, S, or L`, { row: row.id, line: row.line })); } diff --git a/research-kit/lib/measure.mjs b/research-kit/lib/measure.mjs index 3eba8c54..aaf2af8e 100644 --- a/research-kit/lib/measure.mjs +++ b/research-kit/lib/measure.mjs @@ -9,7 +9,7 @@ import { readCorpus, captureOf, parseCapture } from './corpus.mjs'; import { verifyLedger } from './provenance.mjs'; import { resolve, readText, siteOf, exists } from './core.mjs'; -import { quoteAnchors, anchorFound } from './quotes.mjs'; +import { quoteAnchors, anchorFound, unreadableQuoteMarkers } from './quotes.mjs'; const share = (part, whole) => (whole ? Math.round((part / whole) * 1000) / 10 : null); @@ -33,8 +33,12 @@ export function measureCorpus(root, { corpus = null } = {}) { if (works) linkWorks += 1; if (capture) cited.set(capture.file, capture); const anchors = quoteAnchors(row.finding); - if (!anchors.length) continue; + // A marker the gate cannot read was still a quote written down: it counts as a quote not + // found, never as a row without one (#319). + const unreadable = unreadableQuoteMarkers(row.finding).length; + if (!anchors.length && !unreadable) continue; anchoredRows += 1; + quotes += unreadable; const body = capture ? parseCapture(readText(resolve(root, capture.file)) ?? '').body : ''; for (const anchor of anchors) { quotes += 1; diff --git a/research-kit/lib/quotes.mjs b/research-kit/lib/quotes.mjs index 4b0888e1..1e65ba08 100644 --- a/research-kit/lib/quotes.mjs +++ b/research-kit/lib/quotes.mjs @@ -34,18 +34,48 @@ function markersOf(s) { const out = []; for (const start of starts) { if (out.length && start < out[out.length - 1].end) continue; // inside a marker already taken - let depth = 0; - let end = -1; - for (let j = start; j < s.length; j += 1) { - const ch = s[j]; - if (ch === '[') depth += 1; - else if (ch === ']') { depth -= 1; if (depth === 0) { end = j; break; } } - } + const end = closeOf(s, start); if (end >= 0) out.push({ start, end, content: s.slice(start + '[quote:'.length, end) }); } return out; } +/** The index of the `]` that brings the bracket depth opened at `start` back to zero, or -1. */ +function closeOf(s, start) { + let depth = 0; + for (let j = start; j < s.length; j += 1) { + const ch = s[j]; + if (ch === '[') depth += 1; + else if (ch === ']') { depth -= 1; if (depth === 0) return j; } + } + return -1; +} + +/** + * The attempts at a quote marker that `quoteAnchors` cannot read, each `{ start, text }` with the + * attempt's first 40 characters: a `[quote` (any case) that opens no marker - no closing bracket, + * a space before the colon (`[Quote :`), no colon at all (`[quote the sentence]`). Each read as a + * Finding with no quote, so an invented sentence in one of these shapes passed where the same + * sentence written `[quote: ...]` failed - ADR-0087's six unchecked quotations again (#319). + * Not attempts: a `[quote` inside a marker already read, inside a code span or a URL, and a + * Markdown link whose text starts with the word (`[Quote of the day](https://...)`). + */ +export function unreadableQuoteMarkers(text) { + const s = String(text ?? ''); + const spans = [ + ...markersOf(s).map((m) => [m.start, m.end + 1]), + ...[...s.matchAll(/(`+)[\s\S]*?\1/g), ...s.matchAll(/https?:\/\/\S+/gi)].map((m) => [m.index, m.index + m[0].length]), + ]; + const out = []; + for (const m of s.matchAll(/\[\s*quote\b/gi)) { + if (spans.some(([from, to]) => m.index >= from && m.index < to)) continue; + const end = closeOf(s, m.index); + if (end >= 0 && s[end + 1] === '(') continue; + out.push({ start: m.index, text: s.slice(m.index, m.index + 40) }); + } + return out; +} + /** * The quote markers in a Finding cell, each split into fragments on an ellipsis (`...` or * `…`), which stands for text the quoter left out. Prose in quotation marks is NOT a marker: diff --git a/research-kit/test/measure.test.mjs b/research-kit/test/measure.test.mjs index 16cdd816..a21f3138 100644 --- a/research-kit/test/measure.test.mjs +++ b/research-kit/test/measure.test.mjs @@ -38,6 +38,18 @@ test('quotes are counted found or not, and an edited capture stops the link from assert.equal(m.ledgerVerifies, false); }); +// #319: a marker the parser could not read was counted as a row with no quote at all. +test('a quote marker the gate cannot read counts as a quote not found, not as an unanchored row', () => { + const passage = 'The free plan allows 10 requests per minute'; + for (const marker of [`[quote: ${passage}`, `[Quote : ${passage}]`, `[quote ${passage}]`]) { + const dir = makePassingProject(); + corrupt(dir, PATHS.evidence, (t) => t.replace('includes 1,000 credits. |', `includes 1,000 credits. ${marker} |`)); + const m = measureCorpus(dir); + assert.deepEqual(m.anchoredRows, { count: 1, percent: 100 }, marker); + assert.deepEqual([m.quotesFound.count, m.quotesFound.of], [0, 1], marker); + } +}); + test('the CLI prints the report, --json prints it for machines, and a non-project is refused', () => { const bin = fileURLToPath(new URL('../bin/measure.mjs', import.meta.url)); const dir = makePassingProject(); diff --git a/research-kit/test/quotes.test.mjs b/research-kit/test/quotes.test.mjs index c4014c24..8343736b 100644 --- a/research-kit/test/quotes.test.mjs +++ b/research-kit/test/quotes.test.mjs @@ -5,7 +5,7 @@ import { test, describe, assert, makePassingProject, corrupt } from './harness.m import { PATHS } from '../lib/core.mjs'; import { readCorpus } from '../lib/corpus.mjs'; import { runCheck } from '../lib/checks.mjs'; -import { quoteAnchors, normalizeForMatch, anchorFound, quoteChars, MIN_QUOTE_CHARS } from '../lib/quotes.mjs'; +import { quoteAnchors, normalizeForMatch, anchorFound, quoteChars, unreadableQuoteMarkers, MIN_QUOTE_CHARS } from '../lib/quotes.mjs'; describe('quotes'); @@ -59,6 +59,37 @@ test('a quote found in its capture passes; a fabricated one blocks; a short one assert.equal(short.find((f) => f.rule === 'quote-too-short')?.severity, 'warn', JSON.stringify(short)); }); +// #319: a `[quote` the parser could not read was not a marker at all - no closing bracket, a +// space before the colon, no colon - so the cell read as one with no quote and nothing checked it. +test('the attempts at a marker the parser cannot read are found, and a readable one is still read', () => { + const at = (s) => unreadableQuoteMarkers(s).map((a) => a.text); + assert.deepEqual(at('[quote: no closing bracket'), ['[quote: no closing bracket']); + assert.deepEqual(at('Claim [Quote : a space before the colon]'), ['[Quote : a space before the colon]']); + assert.deepEqual(at('Claim [quote no colon at all]'), ['[quote no colon at all]']); + assert.deepEqual(at('[quote: read] and [QUOTE: read too] and [quote: a [b [c]] d]'), []); + assert.deepEqual(at('Write it as [quote: ...] or [quote: …] in the cell'), []); + assert.deepEqual(at('Not attempts: `[quote x`, https://x.invalid/?q=[quote and [Quote of the day](https://x.invalid/q), [quotes] and [quoted]'), []); + const cell = 'Two. [quote: the free plan allows ten] then [quote the paid plan allows more]'; + assert.deepEqual(quoteAnchors(cell).map((x) => x.quote), ['the free plan allows ten']); + assert.deepEqual(at(cell), ['[quote the paid plan allows more]']); +}); + +test('a quote marker the gate cannot read blocks, naming the row; a readable one does not', () => { + const invented = 'The free plan allows 100 requests per minute'; + for (const marker of [`[quote: ${invented}`, `[Quote : ${invented}]`, `[quote ${invented}]`]) { + const judged = quoted(`Free tier limits. ${marker}`); + const blocked = judged.find((f) => f.rule === 'quote-unreadable'); + assert.equal(blocked?.severity, 'fail', `${marker}: ${JSON.stringify(judged)}`); + assert.match(blocked.detail, /^E-01 holds a quote marker the gate cannot read/); + } + const real = quoted('Free tier limits. [quote: The free plan allows 10 requests per minute]'); + assert.equal(real.find((f) => f.rule === 'quote-unreadable'), undefined, JSON.stringify(real)); + // Two levels of brackets are read since ADR-0162, so this one is checked, and not found. + const nested = quoted('Free tier limits. [quote: calls[a[0]].id is read on billing]'); + assert.equal(nested.find((f) => f.rule === 'quote-unreadable'), undefined, JSON.stringify(nested)); + assert.equal(nested.find((f) => f.rule === 'quote-not-found')?.severity, 'fail', JSON.stringify(nested)); +}); + // Found 2026-10-02 (C1, the firecrawl-cli-1-25 decision project): a quote of a package // manifest's dependency map - 130 characters, one whitespace-separated "word" - was warned // "under 3 words anchors almost nothing". It anchors one exact line of one file. Words were From 598766f5edf5ccaf0414934685cc93f9695a9f94 Mon Sep 17 00:00:00 2001 From: StepenkoAnatoli <stepenkoanatoli@gmail.com> Date: Mon, 19 Oct 2026 09:59:04 +0300 Subject: [PATCH 12/12] similarity: a character is a token where a script writes no word spaces (#333) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit what changed lib/similarity.mjs: in Chinese, Japanese, Thai, Lao, Khmer and Burmese (Unicode Script_Extensions Han, Hiragana, Katakana, Bopomofo, Thai, Lao, Khmer, Myanmar) each character, with the combining marks written on it, is a token. From each token a shingle is the shortest run that weighs SHINGLE_WORDS words, where a word weighs CHARS_PER_WORD (new, 3) and such a character weighs 1: five words, fifteen characters, or a mix. sketch's floor counts MIN_SHINGLES in words of text (enoughText), so a page without word spaces needs as much text as any other. Text in no listed script has no match and shingles exactly as before. MIRROR_THRESHOLD's comment gains the re-run over this repository's captures. why #333: after ADR-0163, a page in a script without word spaces was one token per clause, or one per page with no punctuation, so it shingled to too little, sketch returned null, and a republished copy of it graded corroboration/independent, the inversion this module exists to prevent. ADR-0163 deferred character n-grams until they were measured; the operator decided on 2026-10-19 to take it. The issue's plain character 5-gram was measured and set aside: two different halves of one Thai UI catalog score 0.177 on it (English 0.015 on words) and two translations of one UI 0.243, within 0.007 of the threshold. Fifteen characters keep Chinese, Japanese and Thai on the scale MIRROR_THRESHOLD was measured on. ADR-0176 records the measurement and the alternatives set aside (a length per script, dictionary segmentation, character n-grams for every script, Hangul by character, a basis chosen per page, a floor counted in shingles). what it touched research-kit/lib/similarity.mjs; research-kit/test/similarity.test.mjs (five tests: an unpunctuated Chinese page and its copy, a Japanese page ended only by 。 and a Thai page, why fifteen and not five, the floor in every script, and a guard that every spaced script shingles as the word rule did); research-kit/test/checks.test.mjs (one: a Japanese page on two hosts grades corroboration/mirror); the new docs/adr/0176-a-shingle-in-a-script-without-word-spaces-is-fifteen-characters.md and its index row; ADR-0163's status (superseded in part); CHANGELOG.md Unreleased; docs/ARCHITECTURE.md (similarity.mjs row); research-kit/README.md (2057 -> 2063 tests). A gap ADR-0163 deferred whose trigger fired, so it is admitted under the feature freeze (ADR-0117). what you verified verified: red-first - with the previous commit's similarity.mjs and these tests, selftest.mjs similarity checks gives 86 passed, 2 failed: the Japanese corroboration test, and the similarity file, which imports CHARS_PER_WORD. With only that constant added to the old module, selftest.mjs similarity gives 16 passed, 3 failed: the three behaviour tests. The why-fifteen test and the guard pass on both, by design. verified: selftest.mjs similarity checks quotes measure on the tree holding #319 and this change: 125 passed, 1 failed (the Thai assertion, below); after the fix, selftest.mjs similarity: 19 passed, 0 failed. The fix touched that test file only. (cwd rk-wt/gap, RESEARCH_KIT_TEST_TIMEOUT=180000) verified: selftest.mjs similarity checks architecture-map repo-hygiene on this commit's tree, on top of #319's commit: 139 passed, 0 failed. verified: the shipped module against ADR-0163's rule over this repository's 417 captures (86,736 pairs): 391 shingle identically, the 26 holding 1 to 149 characters of these scripts move, and no pair changes its mirror verdict or its judgeability. verified: over 50 real texts on this host (vim 9.2 tutors and menus, mintty catalogs, Python 3.14's cjkencodings, the Chrome 154, Edge 154 and an Electron app's locale catalogs; 1,225 pairs), every pair sharing its text is a mirror and no pair of different texts is. The numbers are in ADR-0176. verified: 2063 is the exact number of tests the tree registers (count-tests.mjs). verified: the full suite passed in this commit's gate, run on a working tree equal to the index - the gate's suite reads the working tree, so this commit exists only if that run was green. expected: GitHub Actions on the PR. untested: real Lao, Khmer and Burmese text (none on this host; a KNOWN-UNKNOWN in ADR-0176). No live collection or credit spend. what you got wrong and fixed mistake: a test asserted that the FIRST Thai shingle holds the syllable ตั้. where: test/similarity.test.mjs, the Japanese and Thai test. impact: the focused run was red, 1 failed; nothing committed. cause: the first shingle is built from การ, ผู้ใช้ and จำนวน; ตั้ is in a later one. fix: the test asserts that some shingle holds the token ตั้, and that no mark stands alone as a token. verified: selftest.mjs similarity, 19 passed, 0 failed. mistake: the script that wrote the new code dropped a regex backslash. where: the UNSPACED_CHARACTER pattern and a comment in lib/similarity.mjs. impact: `p{M}*` would have matched the letters p, {, M and }, not the marks, so a Thai mark would have stood alone; caught on reading, before any run. cause: `\p` in a JavaScript template literal is `p`. fix: the pattern is built with String.raw, and the comment restored. verified: the Thai test's mark assertions pass (above). mistake: the first measurement run ran out of memory. where: the scratch reader of Edge's locale catalogs. impact: none on the repository; the run produced nothing. cause: Edge's .pak header widens its counts to 32 bits and its entries to 8 bytes, so the Chromium layout read garbage spans. fix: the reader detects Edge's layout. verified: every catalog's last offset equals its file's length. mistake: the ADR draft called the issue's example a 6 KB page of one 75-character sentence. where: docs/adr/0176, Consequences. impact: a wrong figure; corrected before commit. cause: I took the size from the issue and counted the sentence by eye. fix: 76 characters (72 letters and 4 punctuation marks) repeated forty times. verified: counted with Node: 76 characters, 72 letters. mistake: two code comments misstated the measurement. where: lib/similarity.mjs, the CHARS_PER_WORD and MIRROR_THRESHOLD comments. impact: read as written, the English split fell to 0.003 (that is Chinese; English stays 0.015, on words) and two versions of one text reached 0.69 (the top is 0.683); comments only, corrected before commit. cause: I wrote the figures from memory of the table, not from it. fix: the split figures are named Thai, Japanese and Chinese, and the range reads 0.57 to 0.68. verified: both comments match ADR-0176's table (0.021 / 0.010 / 0.003; 0.683 / 0.647 / 0.570). Fixes #333 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --- CHANGELOG.md | 8 + docs/ARCHITECTURE.md | 2 +- .../0163-a-shingle-is-a-word-in-any-script.md | 2 +- ...thout-word-spaces-is-fifteen-characters.md | 146 ++++++++++++++++++ docs/adr/README.md | 1 + research-kit/README.md | 2 +- research-kit/lib/similarity.mjs | 74 ++++++++- research-kit/test/checks.test.mjs | 16 ++ research-kit/test/similarity.test.mjs | 86 ++++++++++- 9 files changed, 329 insertions(+), 8 deletions(-) create mode 100644 docs/adr/0176-a-shingle-in-a-script-without-word-spaces-is-fifteen-characters.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4cc5aaea..bb17c603 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,14 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in (rank 17, `docs/review-2026-10-04-gap-audit.md`; rank 14 of `docs/review-2026-10-03-gap-audit.md`). ASCII input is unchanged, so `MIRROR_THRESHOLD` still rests on the corpus it was measured against (ADR-0163). +- Near-duplicate detection also reads Chinese, Japanese, Thai, Lao, Khmer and Burmese, the + scripts that put no space between their words. A page in them was one "word" per clause, or + one per page, so it shingled to too little, `sketch` returned null for it, and a republished + copy of it was an independent witness again. In those scripts a character is a token and a + shingle is fifteen of them: the length at which real Chinese, Japanese and Thai text scores + on the scale `MIRROR_THRESHOLD` was measured on, where the plain character 5-gram put two + different Thai texts at 0.18. `MIN_SHINGLES` asks the same amount of text of a page in every + script. Every other page shingles as before (#333, ADR-0176). - The collector stamps a fetch's ledger `at` with the moment it happened instead of midnight of its day; a failed fetch already did, so the two sorted against each other wrongly, and the day the capture was retrieved is unchanged in `retrieved` (rank 9, diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index dd52e41d..bb0e5cee 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -236,7 +236,7 @@ is an open decision (2026-09-30): | `warc.mjs` | **The corpus as WARC 1.1** (2026-09-28, ADR-0090): `exportWarc(root)` - a `warcinfo` record, then per ledger `scrape` entry whose capture is on disk and unchanged a `resource` record (the text after the front matter, `text/markdown`) and a `metadata` record (`WARC-Concurrent-To`; transport, completeness, the capture file's hash, the block's own `body-sha256` and the ledger's seq); `WARC-Block-Digest: sha256:`, record IDs derived from the ledger entry (`uuidFrom`, RFC 9562 v8), each record its own gzip member; an edited capture is left out and named in `skipped`, as is a ledger entry the export cannot date (WARC-Date is mandatory; rank 7, 2026-10-03 gap audit, ADR-0161). `readWarc` splits a stream back into records and throws on bad framing. Validated with warcio on every corpus. `exportWarc` reads a ledger-named capture only if it is inside the project (`isInside` and `realInside`, ADR-0076) and a regular file (2026-09-30); anything else is `skipped` with its reason. A ledger naming `../` or a link out of `research/raw` had packed that outside file into the archive, and a FIFO would have hung the export. `exportWarc` reads the path through `projectFile` (2026-09-30, ADR-0109). The record is parsed from the bytes whose hash passed, never a second read of the file (2026-09-30). | ADR-0090 | | `prior.mjs` | **The registered prior: what was expected BEFORE the evidence, and the proof that it was.** `registerPrior` writes `research/PRIOR.md` and chains an `op: 'prior'` entry carrying its sha256, so two things stop being editable at once - the text, through `body-unmodified`, and its POSITION before the first scrape, through the chain itself. Three dishonest paths, each measured rather than argued: revising the prediction once the answer is known fails the chain; registering late is refused by the tool; and chaining one late BY HAND with every hash correct is caught by `provenance/prior-precedes-collection`, which exists because `appendFetch` is a public export and the gate does not get to trust that the CLI was used. **Nothing grades the prediction** (ADR-0013) - not its specificity, not its reasonableness, and above all not whether it turned out right, since a prior that was wrong has done its job and a gate that preferred correct ones would teach the opposite habit. The single content rule is a length floor, stated as a floor. The decision to proceed is taken inside the same section of the lock as the append, on the ledger read there (#352); a refusal is decided before the lock too, since the collection refusals only accumulate. A ledger that records no scrape beside captures or evidence rows on disk is a lost or emptied one (`unrecordedCollection`, #353): registering is refused, and `readPrior` reports it as `unrecorded`, so `prior-precedes-collection`, the read-back and the brief's prior section all say the order is unknown (`precedes` is their one verdict), naming the files and both remedies. A stray capture beside scrapes on record is `hygiene/uncited-capture`'s, not the prior's. A prior is bounded at `CAPTURE_MAX_BYTES`, and `readPriorSource` checks a `--file` before reading it, because the brief quotes the prior whole (#355). Absence is SILENT at the gate: a prior is optional, and a warning on every project that skipped one is a warning nobody reads. `bin/prior.mjs` registers and reads it back; `bin/research.mjs` prints a one-line nudge before a spending run, because after the first page lands there is no second chance. **It also reaches the REMOTE path, and that half is what makes the rest useful**: the collector workflow scaffolds its own project on the runner with an empty ledger, so a prior registered on the operator machine cannot reach the corpus that comes back - and every corpus collected since `collect-remote` shipped came from that route. The prediction therefore travels as a dispatch input and is registered on the runner between the map and the collection, through `env` and an argv array like the topic (ADR-0020), with the topic's measured disclosure profile (ADR-0035). A test pins that the registration step precedes `collect`, because if it ever drifts below it, `bin/prior.mjs` refuses and the feature stops working silently on a workflow CI never runs. | ADR-0013, ADR-0020, ADR-0039 | | `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. `partial-render` applies only to a capture behind an EVIDENCE row some unknown cites (ADR-0099): on a context-only row its instruction had no cited text to confirm. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. The same duty covers what a PASS claims: `gate-integrity` used to pass with "no override in effect" when what it measured was that no override was RECORDED - and `git commit --no-verify`, the escape hatch this kit documents most prominently, records nothing. It now says what it counted and names the bypass it cannot see. Found by sweeping all thirteen pass messages for claims wider than their measurement; it was the only one. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. `collection-attempts` counts an attempt through the rows a KNOWN-UNKNOWN cites, not only a URL typed into it (2026-09-26): an E-row is a capture of its URL, so citing one is the stronger proof of having reached for it. Until then a KNOWN-UNKNOWN citing the page it tried was warned as never attempted - found by the `2026-09-26-actions-sept-changes` project's own gate, and no test exercised its verdict (only the registry-order test named it). `citations/url-mismatch` (2026-09-27) fails a row whose URL is not, by `urlKey`, the page its capture was fetched from: a row could name another page and pass, and the URL cell is what a reader is shown as the source. `provenance/ledger-unparsed` names its repair (2026-09-27): a torn last line gets `doctor --fix-arity`, which drops exactly that; a broken line inside the chain is restored from git. It gave only the JSON error. A row is never compared with itself (2026-09-27): `supersededRows` skips a same-ID pair and `hygiene/duplicate-url` leaves the same ID to `duplicate-id`; a row pasted twice failed with "E-01 has been superseded by E-01". `hygiene/future-date` fails a Retrieved date more than a day ahead (2026-09-27): 2030-01-01 had passed, hiding the row's age from every freshness check. `hygiene/duplicate-id` fails rather than warns (2026-09-27, ADR-0054): two rows under one ID leave every citation, COVERED cell and brief reference to it ambiguous, because lookups keep the last row. A contract with two U-1 rows had passed. `corroboration` weighs each unknown ID once, so it no longer prints the same single-source warning twice. `hygiene/brief-stale` warns when a stamped brief's inputs hash no longer matches the corpus (2026-09-27, ADR-0055) and names the command that redrafts it; `hygiene/brief-unstamped` warns on a drafted brief with no stamp (2026-10-03, ADR-0138) - its currency cannot be checked and it keeps its approval, so the gate says so and names the redraft that stamps it; a template or a hand-written legacy brief gets no such warning. `hygiene/duplicate-url` treats two same-day rows for one URL as a refresh when the ledger records a fetch for each (2026-09-27, ADR-0060); a pasted row, with one fetch behind two rows, is still a duplicate. `discovery-contract/build-intent` tells a missing heading from an empty one, from the corpus's `intentHeading` (2026-09-27): a renamed heading read as "is empty". **`citations` checks quote anchors** (2026-09-28, ADR-0087): each `[quote: ...]` in a Finding must occur in the row's capture (`quote-not-found`, blocking) and carry at least three words or 40 characters in EVERY fragment (`quote-too-short`, warn; the character floor is ADR-0126, 2026-10-02; per-fragment measurement and the one-passage bound are rank 4 of the 2026-10-03 gap audit), and a `[quote` it cannot read as a marker - no closing bracket, a space before the colon, no colon - is `quote-unreadable`, blocking, naming the row and both repairs (#319, ADR-0175): it was no marker at all, so an invented sentence in that shape passed where the same sentence written `[quote: ...]` failed; the matching lives in `quotes.mjs`. `citations/raw-error-status` warns when a row cites an HTTP 4xx/5xx capture from before that refusal, unless the row names the status itself (this repository's E-15 keeps a 404 as the record of a failed guess). **`unknown-closure/secondary-only`** (2026-09-28) warns when a CLOSED unknown cites no `P` row and at least one `S` row: the registry promised "a primary row" and only `L` was checked, so three corpora closed everything on `S` rows with no warning (found by the first measurement). Retrieval dates compare by `compareText` (2026-10-01). `runChecks` appends each check's findings one at a time (2026-10-01, break-test PR #188): `push(...check.run())` passed every finding as an argument, and past the engine's limit - 60,000 to 130,000 findings from one check, by Node line - the verdict died with a raw RangeError; preflight, doctor, brief and audit crashed, and the commit gate answered internal error. Reachable from about 66,000 evidence rows. The other spread sites (`priorOrder`, doctor's `gateHealth`) are bounded by the contract's unknown count and left as they are. **`hygiene` fails a malformed ID** (2026-10-03, audit G2): each `malformed-id` problem the corpus reader records becomes a `fail` beside `duplicate-id`, for the same reason - the ID is how every check finds a row; `corpus-shape` skips the kind so one typo is not reported twice. A `table-split` problem fails the same way (2026-10-03, gap audit rank 1): the row no check reads is named with its file and line, and `corpus-shape` does not repeat it. **Freshness is judged by the capture's own date** (2026-10-03, audit G8): `unknown-closure` ages a cited row by `freshnessDate` - the capture's front-matter `retrieved`, written by the collector at the fetch and under the ledger's hash, with the table's cell as the fallback for a row whose capture carries no parseable date - and names the judging date when the two differ; `hygiene` warns `date-mismatch` when a row's cell and its capture name different calendar days (`retrievedDay`), with the remedy of writing the capture's date in the row. Editing E-19's cell alone had made its stale warning disappear while the capture on disk still said 2026-09-20. The capture judged is the one the row NAMES (`ownCapture`, review of G8 the same day): `captureOf` answers a blank Raw cell with the URL's latest capture, and judged by that a row with no Raw cell borrowed a later fetch's date, lost its stale warning and drew a date-mismatch telling the operator to write the later date in; a row naming no capture is judged by its cell and has nothing to reconcile against. A future date in the capture fails `future-date` as one in the cell does, naming the capture, and the stale hint's "fresher capture of the same URL" is found by capture date (`freshnessDay`), not by the cells. **Same-day supersession follows the ledger** (2026-10-03, audit G4): `supersededRows` orders two rows of one URL by the day freshness judges them by (`freshnessDay` - the capture's date; ordered by the table cell, one edit to an older row's date had failed the gate on the genuinely newest row and told the operator to cite the older one; review of G4, the same day) and then by the ledger position of each row's cited capture (`ledgerRank`, the rule `newer` applies for the collector); a capture no fetch recorded ranks below every fetched one, and only among those does the table's order still break the tie. With A fetched, then B, then A again, the collector had held A current while the check said B superseded it, and swapping two evidence rows swapped the verdict. `uncited-capture` says "was collected" only of a file a ledger scrape entry names; a file under research/raw no fetch produced is named as such (2026-10-04, break-test pass 4 F5a). `discovery-contract/contract-missing` states that the contract is missing or could not be read (2026-10-08): the reader uses the same null fallback for absence and refused reads, so that finding cannot establish absence. Both states keep the existing failing rule; no filesystem-state API or new check is introduced. **Evidence URL comparisons use the adopted page identity** (2026-10-09, audit rank 14): supersession groups, stale-citation fresher-row hints and duplicate-URL hygiene compare through `urlKey` (ADR-0136), as the collector and cache already do. Capture dates, blank-Raw table-date fallback, same-day ledger ordering, the same-ID exclusion and the two-recorded-fetch refresh rule retain their existing roles. Captured URL spellings remain visible; query strings and path case remain distinct. | ADR-0004, ADR-0026, ADR-0087, ADR-0136 | -| `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped, and **a word means a word in any script**: `\p{L}\p{N}\p{M}` and NFC composition, not the Latin-ASCII `[^a-z0-9]` class, which stripped every non-Latin letter and left a Russian, Arabic, Greek or accented page with NO shingle at all - so `sketch` returned null for it and a republished copy counted as an independent witness, rank 17 of the 2026-10-04 gap audit; ASCII input is byte-identical, so the measurement below still holds) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 | +| `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped, and **a word means a word in any script**: `\p{L}\p{N}\p{M}` and NFC composition, not the Latin-ASCII `[^a-z0-9]` class, which stripped every non-Latin letter and left a Russian, Arabic, Greek or accented page with NO shingle at all - so `sketch` returned null for it and a republished copy counted as an independent witness, rank 17 of the 2026-10-04 gap audit; ASCII input is byte-identical, so the measurement below still holds; and in a script with **no word spaces** - Chinese, Japanese, Thai, Lao, Khmer, Burmese - a token is one character with its marks, and a shingle is the shortest run weighing five words, such a character weighing a third (`CHARS_PER_WORD = 3`, so fifteen characters): measured on real Chinese, Japanese and Thai text, where the plain character 5-gram put two different Thai texts at 0.18, #333, ADR-0176) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, counted in words of text so that a page without word spaces needs as much text as any other (ADR-0176), and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036, ADR-0163, ADR-0176 | | `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 | | `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, **what phase 1 may commit** (ADR-0048: `research/`, the root `SCAFFOLDING` - `.gitattributes`, `.gitignore`, `AGENTS.md`, `START_HERE.md` - and `docs/ARCHITECTURE.md` while `isEmptyArchitectureMap` says its staged bytes, read by `readStagedText`, are still the template's; until 2026-09-27 only `research/`, so a new project's first `git add -A` was refused and its `.gitattributes` could not travel), and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **Staged paths remain data, with checked acquisition** (ADR-0154): the default commit CLI lazily calls `readStagedPaths` after the ungated and `GATE_OFF` returns. One actual `git diff --cached --name-only --no-renames -z` supplies paths only on successful complete output; the existing 64 MiB capture rejects nonzero exit and buffer failure, and the reader rejects malformed NUL data. Successful zero bytes means `[]`; unreadable scope means null and raises the existing architecture-map breach even after green preflight, ordinary exit 1 under either posture. Explicit arrays/null and the existing stdin/repeated `--staged` options retain their meaning; edits never acquire a list. **One deliberate exception to "no CLI parsing"** remains for explicitly supplied stdin data (`splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"**, preserving unknown scope rather than assuming an empty set. `bin/gate.mjs --staged-stdin` reads stdin itself (`stdinIsReadable`, then `fs.readFileSync(0)` into `splitPathList`) and refuses a terminal by name rather than guessing an empty set; the unused `stagedPathsFromStdin` helper, which answered that case with a silent null, was removed (#297, 2026-10-17). No list read claims an atomic snapshot with later index materialization. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). The architecture-map block's fix says to **update** the map, then `git add` it (2026-09-27): a bare `git add docs/ARCHITECTURE.md` stages nothing while the map is unchanged, so the printed fix left the commit blocked. `bin/gate.mjs` no longer closes that block with "Phase 1 is not done until preflight prints PASS" - a map block happens only when the gate passes. `isPhaseOneEdit(root, file)` answers the same question for the edit hook, about one file. The empty-index verdict — a gated project whose `research/` is untracked, every fresh project's first commit — removes the scratch dir `materializeIndex` made before it learned the index was empty (2026-09-28): it returned before `scratch` was assigned, so the `finally` that removes it never ran, and every such verdict leaked one temp directory. **Integrity before scope** (ADR-0093): a failing verdict whose findings include an `INTEGRITY_RULES` provenance rule blocks the commit even when it is confined to `research/` - the ADR-0048 allowance covers completeness findings only. `stdinIsReadable(stdin = process.stdin)` takes the stream to ask about, so the hook's terminal case is testable (2026-10-01). **The suite rule** (ADR-0120, 2026-10-01): `suiteBreach(root, stagedPaths, { run })` runs after the map rule, in the commit gate only, and only where `isKitCheckout` finds the kit's four markers and a staged path is under `research-kit/` (or the list is unknown); `runSuiteHere` spawns the checkout's `bin/selftest.mjs` with `RESEARCH_KIT_RESULT_FILE` pointed at a scratch file and `RESEARCH_KIT_ALLOW_UNSUP=1` (unsupported is not red, ADR-0108), git's hook environment stripped (`GIT_INDEX_FILE`, `GIT_DIR`, `GIT_PREFIX` and kin - the same `REPOSITORY_LOCATION` list as the edit hook, else every scratch repository's git in the suite would act on the index being committed), and reads the file, never stdout: failures block naming the counts, no result blocks as a crashed runner, a run past its budget is stopped and blocks as unfinished, the fix naming `RESEARCH_KIT_GATE_TIMEOUT` before the override, and a green run whose runner exited non-zero (the README count check) blocks too. `suiteOwed(root, stagedPaths)` is the predicate alone; `suiteBreach`'s `announce` (`evaluate`'s `announceSuite`) is called right before the run, so `bin/gate.mjs` says the suite is running only when it is - not when the map rule or preflight refused the commit first. `evaluate` takes `runSuite` so the verdict is testable without a suite. **The suite's budget follows the gate's watchdog** (ADR-0142, 2026-10-05): `suiteBudgetMs(env)` is `SUITE_TIMEOUT_MS` (65 min, twice the slowest honest run measured, 1825 s on the operator's Windows PC) unless `RESEARCH_KIT_GATE_TIMEOUT` is set to a positive duration `gateTimeoutSeconds` reads (seconds, an optional fraction, an optional `s`/`m`/`h`/`d`, at most nine digits before the point - the grammar the hook's `gate_timeout_readable` checks too; `0` and anything outside it leave the default), and then that watchdog less `GATE_REST_S` (120 s, the hook's default watchdog for a gate that runs no suite), never more than half a short one, at least 1 ms and at most 2^31 - 1 - so the gate stops the suite and blocks before the hook's watchdog fires, whenever its own work fits what is left. `runSuiteHere` stops the runner with SIGKILL: the harness's SIGTERM listener kept a synchronously blocked runner alive past its budget on POSIX; a stopped run's scratch is left in the temp folder. The budget had been 20 minutes, which that PC's green suite met on every kit commit, and the variable had reached only the watchdog: set below the budget, the watchdog killed the gate first and a fail-open machine allowed a commit whose suite never reported. **The fix lines no longer say `--no-verify` is recorded** (2026-10-03, outside audit): git skips this hook, so nothing can record it; `checks.mjs` had said so since ADR-0035 while the gate's own fix text, the README, AGENTS.md and CONTEXT.md said the opposite. **Red suite diagnostics** (2026-10-08): `suiteBreach` forwards up to five `failed` identifiers, each limited to 160 characters with controls normalized, in its existing detail string; omitted or clipped data is named, and legacy missing labels are explicit. Counts and verdict keys are unchanged. `runSuiteHere` still removes its scratch result; no result retention system is added. **A project below the repository root** (#309, ADR-0182): git runs the hook at the top level, and an ungated top level answered `not-gated` before reading a staged path. `bin/gate.mjs` now, at an ungated cwd, reads the staged list (supplied, or `readStagedPaths`) and `stagedProjects(top, paths, { contracts })` assigns each path to its nearest project below the top. Below the top a project is a folder with `research/DISCOVERY.md` and either `research/raw/` or a contract `new-project` rendered (`isProjectBelowTop`, `isKitContract`: the kit's heading, no `{{TOPIC}}` token; any one of the four markers had judged data folders and the vendored `template/` in repositories without the kit, and raw/ alone did not travel to a clone of a repository that ignores `raw/`, review 2026-10-18): `trackedContracts` lists the tracked contracts with one `git ls-files` (its glob kept a glob under `GIT_LITERAL_PATHSPECS`), so folders are string lookups (30,000 paths in 9,060 folders had cost 25 s at four stats each), a candidate without raw/ has its contract read from the index (`contractText`; HEAD's for a deletion), and a contract or raw path the staged list names - a deletion - counts too; without the listing each folder is asked of the disk. Each project is `evaluate`d with its own paths relative to it and `top` set, so `materializeIndex`, `readStagedText` and `stagedRawDeletions` run git at the top level and name the project's files by their path from there (`materializeIndex` returns the project's scratch `root`). Any project that blocks blocks, named in the output ("BLOCKED in packages/a", the fix run from the project - `cd <project> && ...`, quoted by `shellWord` - and the project's own `research/GATE_OFF` as the override) and as `project` in `--json`; a list git cannot read at an ungated top level is not judged, as before, and is said on stderr. Diagnostics are recorded only into a `research/raw/` that exists: creating it makes a folder gated. **Removing evidence HEAD holds is an integrity block** (#312): when the commit gate judged the index and a staged path is under `research/raw/` (or the list is unknown), `evidenceRemoval` runs BEFORE the pass branch - a capture deleted with the ledger line naming it leaves a chain that verifies. It blocks when HEAD's ledger (`readHeadLedger`, `git show HEAD:...`) is being deleted; when an entry HEAD records is gone from the staged ledger, matched by seq and file so the sanctioned `rebuildLedger` migration passes; and when `stagedRawDeletions` (`git diff --cached --diff-filter=D --no-renames`, so a rename's deleting half counts) removes a capture or source that HEAD's or the staged ledger names - each naming the file or seq and `git checkout HEAD -- <file>`, whatever else is staged. A capture never in HEAD (mid-collection) or not staged (part-staged) stays committable, as ADR-0093 says; a deletion list git cannot read blocks. The first version read only the staged ledger and ran only on a failing verdict, so `git rm -r research/raw`, or a capture removed with its ledger line, committed (review 2026-10-18). | ADR-0002, ADR-0007, ADR-0008, ADR-0020, ADR-0093, ADR-0120, ADR-0142, ADR-0154, ADR-0182 | | `doctor.mjs` | Read-only diagnostics: machine + project + gate + chain health in one report. Consumes the verdict context (`verdictContext` / `readGateState`) for the chain-verified corpus and the gate-state triple; adds one machine read of its own (the git-resolved `hooksPathEffective`) for report. **Role-aware** (ADR-0010): the Firecrawl findings are informational on a `builder`, and the handoff findings (ADR-0011) are blockers there — the collection entrypoints' refusal is what makes that split honest. The handoff findings come from the owner, not from a second composition: doctor passes the corpus it already read to `verifyHandoff(root, { corpus })`, so the arrival question has one implementation and one set of pins. One deliberate write: the repository-local hooksPath override log (the third override is silent by nature — the bypassed hook cannot report it). `gateHealth` and `runDoctor` accept explicit Git/config/settings paths and a child environment so diagnostics can run against disposable machine fixtures without consulting host state. **`commitGateState`** asks WHOSE pre-commit is at `core.hooksPath` - `current` / `foreign` / `unusable` - because executable is not the same as ours. It used to ask only whether some executable hook was there, the same existence-for-correspondence mistake the `deploy` check made; `settingsState` has asked the question of the EDIT gate since a registration here named our hook inside a different checkout, and `core.hooksPath` is a single machine-wide setting that husky, lefthook and pre-commit all rewrite. Pure and separate from `gateHealth`, which reads real git config - a rule testable only by fishing a finding out of an array on a host-dependent path is a rule nobody tests (ADR-0004), which is why the older test for it asserts the severity is one of three. **Ownership is decided before executability**, because the first version asked about the mode first and so reported a non-executable FOREIGN hook as `unusable` with `chmod +x` beside it - advice to repair somebody else's gate while leaving this one just as absent. That ordering bug was invisible on Windows, where `hookExecutability` returns ok for any file that exists, and surfaced on Linux in CI. On a **collector**, handoff findings are `info` in the collector's words ("does not exist yet - nothing has been collected here; a builder handed this corpus would be refused"), since 2026-09-26: they were graded `pass` with the builder's wording, so a fresh project printed "pass ... the corpus arrived without its ledger". On a builder they remain blockers, worded as before. It reports the **Node line** and, when a proxy variable is set, whether this Node sends the kit's requests through it A collector without the Firecrawl CLI is told the exact install command, `npm install -g firecrawl-cli@<tested>` from `cliInstallSpec()` (2026-09-27; it said "install the Firecrawl CLI" and nothing anywhere said how). (`proxy`: keyless pages, search requests and remote collection; a warning naming the version to move to - or, for a value Node cannot parse, the spelling it needs - since the Firecrawl CLI has its own proxy handling) - measured on 22.20.0, where the fetch went around the proxy (ADR-0046, ADR-0047). A collector that explicitly chose a keyless or browser transport passes the Firecrawl checks; nothing chosen still fails (ADR-0095). The Firecrawl fixes (`firecrawl-cli`, `firecrawl-auth`) name the no-account route beside the install or login - `"transport": "http-keyless"` in the machine config, the setting that makes them pass (ADR-0095) - where the CLI fix had offered only a per-run flag and the login fix nothing (2026-09-29). `editGateState` reads an entry whose `hooks` is not a list, or a hook that is `null`, as naming nothing (2026-09-30); `[null]` had thrown a `TypeError`. `scanForSecrets` catches a folder it cannot read where it reads it, and skips a folder it has already walked by real path (2026-09-30): one try/catch around the whole walk had ended the scan at the first unreadable folder, leaving every file after it unscanned and reporting nothing found. `scanForSecrets` records a cap stop separately from the scanned count (2026-10-08, bounded-scan audit repair): it keeps the 2000-text-file limit and reports incomplete coverage only when that cap actually prevents further traversal; ending at the final file's boundary is not itself incomplete. `runDoctor` reports a capped scan as a `secret-scan` warning, beside any unchanged critical `secret` hits, instead of reporting a capped no-hit scan as passing. It names no unobserved remaining-file count and makes no whole-repository clearance claim. `bin/doctor.mjs` reports `KIT_VERSION`, as `kitVersion` in `--json` and on its last line (ADR-0116). Reports each ill-typed posture key as a `config-ill-typed` warning naming the key, the values the kit reads, and the state it resolved to (ADR-0121), and an unreadable `RESEARCH_KIT_ALLOW_INTERNAL_REDIRECTS` under the same finding, with the refusal it resolved to (#290); every other ill-typed key gets the same warning, naming what it should hold (`machine.mjs`'s `CONFIG_KEY_HOLDS`, with the transport names added here) and that its default is used (#310). The `machine-config` line names the resolved config file, and a second `machine-config` warning says a relative `RESEARCH_KIT_CONFIG` is read against the repository top level, so another repository or folder names another file (#367). An unreadable config is worded by its `problem`: JSON that parses to a non-object is not described as not parsing. The `firecrawl-cli` finding judges the installed version with the run's own rule (`cliCompatibility`): an unsupported major is a **fail** with the remedy on a collector that needs the CLI and information elsewhere, a newer minor passes and says what it was tested against - doctor had printed `pass 2.0.0` for a CLI the run then refused (2026-10-02, break-test pass 3). `scanForSecrets` skips a binary file and counts it (`skippedBinary`, named in the coverage line) instead of matching credential patterns against an executable's string table (ADR-0128, 2026-10-02): run on a Windows home folder it reported fourteen committed keys, every one the PEM marker OpenSSH and libssh2 parse, or random bytes spelling `sk-`. **`handOnState`** (2026-10-03, ADR-0134; since 2026-10-04 defined in `installer.mjs` and re-exported here, ADR-0141, so the install judges what it records by the same rule): what the hooks hand on to - none, missing, kit, other - and `gateHealth` warns on kit and missing; the maintainer's machine handed every commit on to the previous implementation's gate, which crashed and failed open while doctor said READY. `handOnState` resolves a relative folder from the repository's top level (`base`), where git runs its hooks from; doctor had resolved `.custom-hooks` against its own working directory and from a subdirectory called a working hand-on missing (fourth audit round, 2026-10-03). `scanForSecrets` skips `research/raw` at ANY depth (2026-10-04): the skip matched the relative path `research/raw`, so only the root's captures were outside the scan and a nested decision project's were read as the project's own files - CI tripped `google-api-key` on a Google page's source sibling (ADR-0140), Google's own public keys; a capture is page content, and the folder exists wherever a nested project does (ADR-0030). The shared unusable-proxy check also rejects proxy URLs carrying a non-root path, query or fragment, so those configurations warn instead of promising proxy use (#380, 2026-10-12). `machineHealth` uses the shared `envProxyChoice` with injectable `execArgv` (#381, 2026-10-12). An explicit disabled choice warns that requests go direct and names how to enable the proxy, rather than promising proxy use from Node version support alone; the raw variable value is never echoed. The `git` finding says "not on PATH" only when git cannot be started (`runGit` ENOENT), and otherwise "git cannot run here" with git's message (#363): a global gitconfig git cannot parse had been diagnosed as a missing git. Run inside a project bound with `install --into`, the `skill` check also compares that project's bound copies (`boundDrift`) and warns with `install.mjs --into .` when they differ (#328); it had checked only the personal root. From a gated project below its repository's top level (#309, ADR-0182), a current `gate-commit` names that top level, where git runs the hook: a pass saying the gate judges this project's staged paths, a warn when the folder is not a project by `isProjectBelowTop`, a warn naming `git add research/DISCOVERY.md` when its contract is not in the index (`git ls-files --error-unmatch`), or a warn when the top level is itself gated and its verdict is the only one asked. It had said a bare pass. | ADR-0010, ADR-0011, ADR-0095, ADR-0134, ADR-0182 | diff --git a/docs/adr/0163-a-shingle-is-a-word-in-any-script.md b/docs/adr/0163-a-shingle-is-a-word-in-any-script.md index d26b7fd6..54a18bf8 100644 --- a/docs/adr/0163-a-shingle-is-a-word-in-any-script.md +++ b/docs/adr/0163-a-shingle-is-a-word-in-any-script.md @@ -1,7 +1,7 @@ # ADR-0163 - A shingle is a word in any script, and text is compared as it was written - **Date:** 2026-10-17 -- **Status:** accepted +- **Status:** superseded in part by ADR-0176 (in a script without word spaces a character is a token, and a shingle is fifteen of them; the word rule for every other script stands) - **Area:** `research-kit/lib/similarity.mjs` - **Supersedes nothing.** The word basis, the sketch size and `MIRROR_THRESHOLD` are kept as they were decided and measured; what changes is which characters count as a word. diff --git a/docs/adr/0176-a-shingle-in-a-script-without-word-spaces-is-fifteen-characters.md b/docs/adr/0176-a-shingle-in-a-script-without-word-spaces-is-fifteen-characters.md new file mode 100644 index 00000000..ca19b586 --- /dev/null +++ b/docs/adr/0176-a-shingle-in-a-script-without-word-spaces-is-fifteen-characters.md @@ -0,0 +1,146 @@ +# ADR-0176 - A shingle in a script without word spaces is fifteen characters + +- **Date:** 2026-10-19 +- **Status:** accepted +- **Area:** `research-kit/lib/similarity.mjs`. How a page is cut into shingles, and how the floor + below which a page is unjudgeable is counted; no command, flag, format, configuration key or + finding. +- **Supersedes in part:** ADR-0163, in three places: its rejection of character n-grams "for + now", its rejection of splitting CJK ideographs, and the KNOWN-UNKNOWN it left for scripts with + no word separators. That record's trigger has fired: #333 asks for exactly this, and the + operator decided on 2026-10-19 to take it. ADR-0163's word rule for every other script, its + rejection of dictionary segmentation, the sketch size and `MIRROR_THRESHOLD` stand. +- **Refines:** ADR-0036. `MIN_SHINGLES` stays 40, now counted in words of text. + +## Context + +ADR-0163 made a word any run of letters, numbers and marks, in any script. That fixed every +script that puts spaces between its words. It did not fix the ones that put none: Chinese, +Japanese, Thai, Lao, Khmer and Burmese. A page in them was one token per clause, or one token for +the whole page when it had no punctuation. The issue's own measurement on `main`: its Japanese +example page gave 0 shingles. Under ADR-0163's rule, a Japanese page of thirty sentences ended by +`。` alone gives thirty tokens and 26 shingles. That is under `MIN_SHINGLES`, so `sketch` returns +null. A copy of that page on a second host then grades `corroboration/independent`, the check's +best grade. That is the inversion this module exists to prevent (#333). + +ADR-0163 deferred character n-grams for a measured reason. The threshold and its pinned numbers +were measured on words, so a new basis needs its own measurement over real pairs in those +scripts. This record is that measurement. + +## Decision + +1. **A token is a character** in the scripts that write no word spaces: Han, Hiragana, + Katakana, Bopomofo, Thai, Lao, Khmer and Myanmar, matched by Unicode Script_Extensions. The + combining marks written on a character stay with it, so a Thai consonant keeps its vowel and + tone signs. Script_Extensions rather than Script, so that the signs these scripts share count + with them: the long-vowel mark `ー` and the iteration mark `々`. Hangul is not in the list, + because Korean puts spaces between its words. +2. **A shingle weighs five words, and a character weighs a third of a word** + (`CHARS_PER_WORD = 3`). From each token, a shingle is the shortest run that weighs + `SHINGLE_WORDS` words: five words, fifteen characters, or a mix (`Vim は、このチュー...` is one + word and twelve characters). One rule for every position, so a page that mixes scripts is cut + the same way wherever it is read. +3. **`MIN_SHINGLES` is counted in words of text.** A shingle that starts on one of these + characters counts a third. Without this, a page with a token per character passed the floor of + forty on about 54 characters, a third of the text an English page needs. The floor exists so + that two short pages are not judged on the boilerplate they share. +4. **Every other page shingles exactly as before.** Text with no character from the list has no + match, every token weighs one word, and the window is five tokens. + `test/similarity.test.mjs` pins this against the word rule, for Cyrillic, Greek, Arabic, + Hebrew, Devanagari, Korean and accented Latin. + +## Measurement (2026-10-19, offline) + +All texts were already installed on the operator's PC. Nothing was fetched and no credit was +spent: + +- Git for Windows 2.55.0: the vim 9.2 tutors in English, Japanese, simplified and traditional + Chinese, Korean and Russian, the vim menu translations, and the mintty message catalogs (the + translated strings only); +- Python 3.14's `test/cjkencodings` samples; +- the Chromium UI string catalogs (`Locales/*.pak`) of Chrome 154.0.8037.98, of Edge + 154.0.4258.53 and 154.0.4258.62, and of an Electron app's older Chromium, in `en-US`, `th`, + `ja` and `zh-CN`. Each catalog is its unique strings in code-point order, because resource ids + differ between builds. + +Exact Jaccard. The English column is the word basis, which is unchanged; the others are the +character basis at one character per word (the plain 5-gram) and at three: + +| pair | English | Thai 1 / 3 | Japanese 1 / 3 | Chinese 1 / 3 | +|---|---|---|---|---| +| two halves of one catalog, split by resource id (different text, one vocabulary) | 0.015 | 0.177 / **0.021** | 0.109 / **0.010** | 0.039 / **0.003** | +| two versions of one catalog (Chrome 154, older Electron) | 0.649 | 0.757 / **0.683** | 0.733 / **0.647** | 0.706 / **0.570** | +| Chrome against Edge (Edge's strings are re-translated; only its English shares Chromium's source) | 0.203 | 0.243 / **0.059** | 0.166 / **0.040** | 0.076 / **0.012** | + +Two and four characters per word were also measured. At two, Thai halves score 0.053; at four, +the Chinese versions fall to 0.524. Three is the one length at which, in all three scripts, the +floor sits within about 1.5 times English and two versions of one text score within 0.08 of +English. That is the scale `MIRROR_THRESHOLD` was set on. + +The shipped module was then run against ADR-0163's rule on the same texts and on this +repository: + +- **This repository's captures:** 417 pages and 86,736 pairs. 391 pages shingle exactly as + before. The other 26 hold between 1 and 149 characters of these scripts, such as a language + name or a Chinese example in a standard. Their shingles move, but no pair changes its mirror + verdict or whether it can be judged. The known stale mirror is in English, so the 0.3879 + pinned beside `MIRROR_THRESHOLD` stands. +- **The texts above:** 50 documents and 1,225 pairs. Every pair that shares its text is a + mirror. One text in two encodings scores 1.0. That includes samples ADR-0163's rule could not + judge at all: Big5, GB2312, EUC-JP, Shift_JIS, ISO-2022-JP, and a Korean one that mixes in + Hanja. The Big5 and GB2312 samples sit whole inside the GBK and GB18030 ones and score 0.40 + to 0.52. Every catalog paired with its other version is a mirror. No pair of different texts + is. The closest pairs are English catalogs of Chrome, Edge and Electron, at 0.23 and 0.24, on + the word basis this record does not touch, since Edge's English strings are largely + Chromium's. On the character basis the closest is Chrome's Thai catalog against Edge's, at + 0.070 (exact 0.060). +- This run read the catalogs as stored, while the table's run inflated their few compressed + entries. The two agree within 0.004. + +## Rejected alternatives + +- **Plain character 5-grams**, one character per token in a five-token window. This was the + issue's proposal. Rejected on measurement: on two different halves of one Thai catalog it + scores 0.177, twelve times English. On two *different translations* of one UI it scores 0.243, + 0.007 under the threshold. `MIRROR_THRESHOLD` was set nine times above the strongest different + pair, and at 5-grams Thai keeps less than a 1.5x margin. A short window matches what two pages + share in vocabulary, not in text. +- **A length per script** (two characters per word for Chinese, three for Japanese, four for + Thai). Chinese matches English more closely at two (versions 0.629, halves 0.008). But a page + mixes scripts: Japanese is Han and kana in one sentence, so every token would need a script + decision, a table to keep current, and its own measurement. One length errs for Chinese + towards a lower score, not an invented mirror. At the measured ratio (0.570 against 0.649), the + known stale mirror's 0.388 would read about 0.34 in Chinese, still above 0.25. +- **Dictionary word segmentation** (`Intl.Segmenter`, ICU's word break). Its boundaries come from + the ICU version and dictionaries of the Node build, which differ between full and small ICU and + move between Node releases. The same corpus would then grade differently on two machines. This + module hashes with seeded FNV-1a precisely so that a corpus grades the same everywhere. + Character tokens depend only on Unicode's script data. +- **Character n-grams for every script.** This is the language-agnostic basis ADR-0163 named. It + moves every page's shingles, so the threshold and its pinned measurement (0.3879 and 0.0275 over + the repository's captures) would no longer describe anything. The word basis already works for + scripts that space their words. +- **Hangul by character.** Korean writes spaces between words, and its word shingles already + judge it: the Korean vim tutor gives 4224. Changing it would move a working basis to fix + nothing. +- **A second window length chosen per page** by its majority script. Then a mirror that gained an + English section could change basis and score. A weighted window gives every position one rule. +- **Keep counting the floor in shingles.** A page with no word spaces would be judgeable on about + 54 characters (18 words). The floor exists to stop exactly that: a verdict drawn from shared + boilerplate. + +## Consequences + +- A page in Chinese, Japanese, Thai, Lao, Khmer or Burmese is fingerprinted, and a republished + copy of it is one document. The issue's example page is one 76-character sentence repeated forty + times. It now gives 72 shingles, which weigh 24 words, so it stays unjudgeable. The same + sentence repeated in English is unjudgeable too: repetition is not text. +- A Chinese or Japanese page that ADR-0163 judged on clauses is now judged on characters, and its + score moves. No capture in this repository holds such a page. +- No real Lao, Khmer or Burmese text was available on this host. Those scripts are included + because they share Thai's shape: no word spaces, and an alphabet whose marks sit on a + consonant. The Thai measurement stands for them. This is a KNOWN-UNKNOWN, and its trigger is a + corpus in one of them. +- `\p{scx=...}` follows the Unicode version of the running Node. A character first assigned in a + later Unicode version counts as an ordinary letter on an older Node. That affects only + characters no measured page used. diff --git a/docs/adr/README.md b/docs/adr/README.md index cbd5edb4..05e208eb 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -183,6 +183,7 @@ directory and are treated as ADR-equivalent. | [0173](0173-a-topic-request-waits-for-an-approved-topics-folder.md) | **A topic request waits for an approved topics folder.** With no folder approved, auto-collect reports a topic request `waiting` with the reason - no result, no commit - and runs it unchanged on the first cycle after the operator sets the folder, as it already does while paused or over today's cap (#323). Amends ADR-0148 decision 4's last sentence; content and target-safety refusals are unchanged. Rejected: keeping the final refusal, refusing then re-queuing on a settings change, retrying every refusal. | | [0174](0174-a-page-the-kit-refused-for-what-it-is-is-not-fetched-again.md) | **A page the kit refused for what it is is not fetched again within the refresh window** (extends ADR-0100). An 11 MB page, a PDF and a 403 were scraped and paid for on every run; `recentlyGone` now also remembers the size cap and the binary refusal as `refused` on any route, a standing 4xx (not 405/407/408/429, never a rate limit) for the route that got it, and reads a 404/410 in the transports' wording too. 5xx, timeouts, rate limits and transport errors are still retried; `--force` retries anything. Rejected: remembering every failure, leaving the 4xx out, remembering 408/429 or a proxy's 405/407, a structured ledger field, keying all or none of the memory on the transport. | | [0175](0175-a-quote-marker-the-gate-cannot-read-blocks.md) | **A quote marker the gate cannot read blocks** (`citations/quote-unreadable`, fail): a `[quote` with no closing bracket, a space before the colon or no colon was not a marker at all, so an invented sentence in that shape passed where the same sentence written `[quote: ...]` failed; `measure` counts it as a quote not found, not as an unanchored row (#319). Not attempts: inside a read marker, a code span, a URL, or Markdown link text. 0 hits over the repository's 41 projects. Rejected: a tolerant parser that guesses the passage, warn-only, counting it as unanchored. | +| [0176](0176-a-shingle-in-a-script-without-word-spaces-is-fifteen-characters.md) | **A shingle in a script without word spaces is fifteen characters** (supersedes in part ADR-0163, refines ADR-0036). In Chinese, Japanese, Thai, Lao, Khmer and Burmese a token is one character with its marks, and a shingle is the shortest run weighing five words, a character weighing a third (`CHARS_PER_WORD = 3`), so a Japanese page and its republished copy are one document, not two witnesses (#333); `MIN_SHINGLES` is counted in words of text. Measured on Chromium's, vim's and Python's own Chinese, Japanese and Thai text: the plain character 5-gram put two different Thai texts at 0.18 and two translations of one UI at 0.24, fifteen characters at 0.02 and 0.06, with two versions of one text at 0.57 to 0.68 (English 0.65). Every other script shingles as before; no verdict moves over the repository's 417 captures. Rejected: plain character 5-grams, a length per script, dictionary segmentation, character n-grams for every script, Hangul by character, a basis chosen per page, a floor counted in shingles. | | [0180](0180-config-ill-typed-covers-every-typed-config-key.md) | **`config-ill-typed` covers every typed config key.** A `transport`, `searchTransport`, `serpapiKey`, `searxngUrl`, `browserPath`, `projectSkillDir`, `maxAgeDays`, `editGate.settingsPath` or `skillRoots` that is present and not of its type keeps its default and is named, with what it should hold; a `skillRoots` list that loses an entry is named too; an `editGate` that is not an object is `editGate.mode` ill-typed, resolved to `hard-block` (#310). Extends ADR-0121, which named the posture keys only; rejected: posture keys only, a second finding name, an unreadable config on a wrong type, `ask` for a non-object `editGate`. | | [0182](0182-the-gates-judge-a-path-by-its-nearest-gated-project.md) | **The gates judge a path by the project it belongs to in its repository** (#309). git runs the commit hook at the top level, so a project below it was never judged; at an ungated top level each staged path now goes to its nearest project below - a folder whose tracked `research/DISCOVERY.md` new-project rendered, or that holds `research/raw/`, so data folders and the vendored template are not projects and a clone that never received raw/ still is - judged from the index and named in the verdict. The edit gate judges a named file the same way, asks a gated top level alone, and keeps the raw/ and GATE_OFF refusals for every project above the file; doctor says where the commit gate runs. Rejected: judging by the cwd, any one of the four markers below the top, DISCOVERY.md or kit.json alone, raw/ on disk as the only second signal, disk-only markers, an unbounded walk up, judging nested projects under a gated root (deferred), a doctor warning alone. | diff --git a/research-kit/README.md b/research-kit/README.md index b33598d4..3327ba1f 100644 --- a/research-kit/README.md +++ b/research-kit/README.md @@ -568,7 +568,7 @@ node research-kit/bin/selftest.mjs # all of it node research-kit/bin/selftest.mjs gate hook # just these files ``` -2057 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means +2063 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means the assertions settled (ADR-0021), and each test is raced against a watchdog (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s). diff --git a/research-kit/lib/similarity.mjs b/research-kit/lib/similarity.mjs index b8911e54..0b283b31 100644 --- a/research-kit/lib/similarity.mjs +++ b/research-kit/lib/similarity.mjs @@ -16,9 +16,30 @@ // does not try. A `mirror` finding says "these are one document" - what that is worth is // still the reviewer's call. -/** Word shingles. Punctuation, case and URLs are noise for this question. */ +/** + * The scripts that put no space between their words: Chinese, Japanese, Thai, Lao, Khmer, + * Burmese (#333, ADR-0176). In them one CHARACTER, with the combining marks written on it, is + * a token. Script_Extensions, not Script, so the signs these scripts share (the long-vowel + * mark, the iteration mark) count with them. Hangul is NOT here: Korean puts spaces between its + * words, and its word shingles were already judged. + */ +const UNSPACED_SCRIPT = String.raw`[\p{scx=Han}\p{scx=Hiragana}\p{scx=Katakana}\p{scx=Bopomofo}\p{scx=Thai}\p{scx=Lao}\p{scx=Khmer}\p{scx=Myanmar}]`; +const UNSPACED_CHARACTER = new RegExp(String.raw`${UNSPACED_SCRIPT}\p{M}*`, 'gu'); +const STARTS_UNSPACED = new RegExp(`^${UNSPACED_SCRIPT}`, 'u'); + +/** What one token weighs in a shingle, in characters of an unspaced script. */ +function weightOf(token) { + return STARTS_UNSPACED.test(token) ? 1 : CHARS_PER_WORD; +} + +/** + * Shingles: from each token, the shortest run that weighs SHINGLE_WORDS words. A token is a + * word, or one character of a script without word spaces, which weighs 1/CHARS_PER_WORD of a + * word - so a shingle is five words, fifteen such characters, or a mix. Punctuation, case and + * URLs are noise for this question. + */ export function shingles(text, n = SHINGLE_WORDS) { - const words = String(text ?? '') + const tokens = String(text ?? '') .toLowerCase() // Canonically equivalent text is the SAME text: `é` and `e` + U+0301 are one word spelled // once or twice, and composing first is what keeps the combining marks below from making @@ -36,15 +57,42 @@ export function shingles(text, n = SHINGLE_WORDS) { // `toLowerCase` the old class kept exactly a-z, 0-9 and whitespace, which is what these // keep, so no ASCII page's sketch moves (test/similarity.test.mjs pins that). .replace(/[^\p{L}\p{N}\p{M}\s]+/gu, ' ') + // A CHARACTER, where the script puts no space between words (#333, ADR-0176). Without this + // a Japanese or Chinese page was one "word" per clause, or one per page with no + // punctuation, so it shingled to too little, `sketch` returned null, and a republished copy + // of it was an independent witness again. Text in no listed script has no match, so every + // other page tokenises exactly as before. + .replace(UNSPACED_CHARACTER, ' $& ') .split(/\s+/) .filter(Boolean); + const weights = tokens.map(weightOf); + const need = n * CHARS_PER_WORD; const out = new Set(); - for (let i = 0; i + n <= words.length; i += 1) out.add(words.slice(i, i + n).join(' ')); + for (let i = 0; i < tokens.length; i += 1) { + let j = i; + let weight = 0; + while (j < tokens.length && weight < need) weight += weights[j++]; + // Every later start has less text after it, so none of them reaches a full shingle either. + if (weight < need) break; + out.add(tokens.slice(i, j).join(' ')); + } return out; } export const SHINGLE_WORDS = 5; +/** + * How many characters of a script without word spaces weigh one word: a shingle is five + * words or FIFTEEN such characters. Measured, not borrowed (ADR-0176). With one character per + * word - the plain character 5-gram - two different halves of one Thai UI catalog scored + * 0.177 and of a Japanese one 0.109, against 0.015 for the same split of the English catalog + * on words: common short sequences, not shared text, put a Thai pair within reach of + * MIRROR_THRESHOLD. At three, the Thai, Japanese and Chinese splits score 0.021, 0.010 and + * 0.003, and two versions of one catalog (Chrome 154 against an older Electron's Chromium) + * score 0.68, 0.65 and 0.57 against 0.65 in English - the scale the threshold was measured on. + */ +export const CHARS_PER_WORD = 3; + /** How many hashes the sketch keeps. 128 bounds memory and costs ~9% standard error. */ export const SKETCH_SIZE = 128; @@ -52,9 +100,21 @@ export const SKETCH_SIZE = 128; * A document too short to fingerprint honestly. Below this, shingle overlap is dominated * by whatever boilerplate the two pages happen to share, and a `mirror` finding would be * an artefact of the sample size rather than a fact about the documents. + * + * Counted in WORDS of text, in every script (ADR-0176): a shingle starts at each token, and + * a page without word spaces has a token per character, so it reached forty shingles on a + * third of the text an English page needs. A shingle that starts on such a character counts + * 1/CHARS_PER_WORD, and the floor asks the same amount of text of every page. */ export const MIN_SHINGLES = 40; +/** Whether a page's shingles hold MIN_SHINGLES words' worth of text. */ +function enoughText(set) { + let weight = 0; + for (const s of set) weight += weightOf(s); + return weight >= MIN_SHINGLES * CHARS_PER_WORD; +} + function hash32(str, seed) { // FNV-1a, seeded. Deterministic across runs and platforms, which matters because a // sketch decides a gate finding: the same corpus must grade the same way everywhere. @@ -73,7 +133,7 @@ function hash32(str, seed) { */ export function sketch(text) { const set = shingles(text); - if (set.size < MIN_SHINGLES) return null; + if (!enoughText(set)) return null; const mins = new Uint32Array(SKETCH_SIZE).fill(0xffffffff); for (const s of set) { for (let k = 0; k < SKETCH_SIZE; k += 1) { @@ -107,6 +167,12 @@ export function similarity(a, b) { * and the hard case is the STALE mirror, because a current mirror scores ~0.99. Same-host * re-collections of one URL reach 1.0000, which is why this check only speaks about pairs * that `corroboration` was about to call independent. + * + * Re-run 2026-10-19 for the character basis (ADR-0176), over 417 captures and 86,736 pairs: + * 391 shingle exactly as before, the 26 that hold a few characters of a script without word + * spaces move, and no pair changes its verdict. In real Chinese, Japanese and Thai text the + * threshold keeps its scale: two versions of one UI catalog score 0.57 to 0.68 (English 0.65), + * and two different texts at most 0.07. */ export const MIRROR_THRESHOLD = 0.25; diff --git a/research-kit/test/checks.test.mjs b/research-kit/test/checks.test.mjs index 7a04ebae..bc0e0bd5 100644 --- a/research-kit/test/checks.test.mjs +++ b/research-kit/test/checks.test.mjs @@ -679,6 +679,22 @@ test('corroboration: a stale mirror on a second host is one document, not two wi assert.match(findings[0].detail, /across 2 hosts but they are the same document/); }); +test('corroboration: a Japanese page on two hosts is one document, not two witnesses (#333)', () => { + // Thirty sentences ended by 。 alone: the word rule read thirty tokens, too few to judge, so + // the pair graded `independent` - the inversion, for a script without word spaces (ADR-0176). + const WHO = ['無料プラン', '有料プラン', '管理者', '利用者', '開発者', '請求担当']; + const WHAT = ['一分あたりの要求数', '月間の上限', '保存期間', '同時接続数', '応答の形式']; + const DO = ['確認できます', '変更できません', '設定画面に表示されます', '翌月に戻ります', '通知されます']; + const page = Array.from({ length: 30 }, (_, i) => `${WHO[i % 6]}では${WHAT[i % 5]}を${DO[(i * 2) % 5]}`).join('。'); + const dir = withRealPages([ + { url: 'https://vendor.example.jp/docs/limits', markdown: page }, + { url: 'https://mirror.example.net/vendor/limits', markdown: page }, + ]); + const findings = runCheck('corroboration', snapshot(dir)); + assert.equal(findings[0].rule, 'mirror', findings[0].detail); + assert.match(findings[0].detail, /across 2 hosts but they are the same document/); +}); + test('corroboration: a mirror does not outrank two genuinely different pages', () => { // The inversion stated as an ordering. A mirror must never grade better than the // honest `one-voice` that two real pages from one vendor earn. diff --git a/research-kit/test/similarity.test.mjs b/research-kit/test/similarity.test.mjs index 2c8f4b0e..65919099 100644 --- a/research-kit/test/similarity.test.mjs +++ b/research-kit/test/similarity.test.mjs @@ -16,7 +16,7 @@ import { test, describe, assert, fs, path } from './harness.mjs'; import { parseCapture } from '../lib/corpus.mjs'; import { sketch, similarity, shingles, documentGroups, isMirror, closestPair, - MIRROR_THRESHOLD, MIN_SHINGLES, SHINGLE_WORDS, + MIRROR_THRESHOLD, MIN_SHINGLES, SHINGLE_WORDS, CHARS_PER_WORD, } from '../lib/similarity.mjs'; describe('similarity'); @@ -179,6 +179,64 @@ test('canonically equivalent text is one word, composed or decomposed', () => { assert.equal(similarity(sketch(composed), sketch(decomposed)), 1, 'é and e + U+0301 are the same word'); }); +// #333, ADR-0176. Chinese, Japanese and Thai put no space between words, so the rule above read a +// page in them as one "word" per clause - or one per page, with no punctuation - and it +// shingled to too little: `sketch` returned null and a republished copy was an independent +// witness. A character is now a token, and fifteen of them weigh what five words do. +const PHRASES_ZH = ['免费计划每分钟可发送请求', '请求次数超过限制时报错', '账户管理员可以修改设置', '页面上方显示当前的用量', '超过上限后接口返回错误', '每个错误代码都有说明', '用量在每月一日零点重置', '服务接口的版本号说明', '数据在服务器保存九十天', '同时连接数量不能超过十', '响应格式默认是纯文本', '计费周期结束时发送账单']; +/** A page of these phrases in one of many orders, with no punctuation and no space anywhere. */ +const zhPage = (step) => Array.from({ length: 48 }, (_, i) => PHRASES_ZH[(i * step + Math.floor(i / PHRASES_ZH.length)) % PHRASES_ZH.length]).join(''); + +test('a page in a script without word spaces is fingerprinted, and two copies of it are one document', () => { + const page = zhPage(5); + assert.ok(!/[\s\p{P}A-Za-z0-9]/u.test(page), 'no space, punctuation or Latin character may appear, or the word rule would have had something to split on'); + const a = sketch(page); + assert.notEqual(a, null, 'a Chinese page with no punctuation must be judgeable at all'); + assert.equal(isMirror(a, sketch(page.replace('管理员', '负责人'))), true, 'two copies of one page are one document, not two witnesses'); +}); + +test('a Japanese page whose sentences end only in 。 is judged, and so is a Thai page', () => { + // Thirty sentences: under the word rule, thirty tokens and 26 shingles - under MIN_SHINGLES. + const WHO = ['無料プラン', '有料プラン', '管理者', '利用者', '開発者', '請求担当']; + const WHAT = ['一分あたりの要求数', '月間の上限', '保存期間', '同時接続数', '応答の形式']; + const DO = ['確認できます', '変更できません', '設定画面に表示されます', '翌月に戻ります', '通知されます']; + const ja = Array.from({ length: 30 }, (_, i) => `${WHO[i % 6]}では${WHAT[i % 5]}を${DO[(i * 2) % 5]}`).join('。'); + assert.equal(isMirror(sketch(ja), sketch(ja.replace('管理者', '担当者'))), true); + // Thai marks (the vowel and tone signs written on a consonant) stay on their consonant. + const TH = ['การ', 'ตั้งค่า', 'ความ', 'ปลอดภัย', 'บัญชี', 'ผู้ใช้', 'ข้อมูล', 'ส่วนตัว', 'หน้า', 'แสดง', 'จำนวน', 'คำขอ', 'ต่อ', 'นาที', 'เกิน', 'ขีดจำกัด', 'รายเดือน', 'บริการ']; + const th = Array.from({ length: 120 }, (_, i) => TH[(i * 5 + Math.floor(i / TH.length)) % TH.length]).join(''); + const thTokens = [...shingles(th)].flatMap((s) => s.split(' ')); + assert.ok(thTokens.includes('ตั้'), 'a consonant and the marks on it are one token'); + assert.ok(!thTokens.some((t) => /^\p{M}/u.test(t)), 'no mark stands alone as a token'); + assert.equal(isMirror(sketch(th), sketch(th.replace('บัญชี', 'ระบบ'))), true); +}); + +test('two different pages built from the same phrases are not a mirror: why a shingle is fifteen characters, not five', () => { + // A UI catalog or a templated page repeats its phrases; a character 5-gram is shorter than + // most of them, so it matches what the pages share in VOCABULARY, not in text. The pair below + // scores above MIRROR_THRESHOLD on plain character 5-grams. Measured on real catalogs: two + // halves of Chrome's Thai strings, 0.177 on 5-grams, 0.021 on fifteen (ADR-0176). + const a = zhPage(5); + const b = zhPage(7); + const grams = (t) => new Set(Array.from({ length: [...t].length - 4 }, (_, i) => [...t].slice(i, i + 5).join(''))); + const [x, y] = [grams(a), grams(b)]; + const shared = [...x].filter((g) => y.has(g)).length; + assert.ok(shared / (x.size + y.size - shared) >= MIRROR_THRESHOLD, 'the pair must be one that character 5-grams would call a mirror, or this test proves nothing'); + assert.equal(CHARS_PER_WORD, 3); + assert.equal(isMirror(sketch(a), sketch(b)), false); +}); + +test('MIN_SHINGLES asks the same amount of text of a page in every script', () => { + // A token per character gives a page without word spaces three shingles for every word's + // worth of text, so it would pass a count of forty on a third of the text English needs. + let han = ''; + for (let i = 0; han.length < 200; i += 1) han += String.fromCodePoint(0x4e00 + ((i * 7919) % 20000)); + const short = han.slice(0, 130); + assert.ok(shingles(short).size > MIN_SHINGLES, 'the short page has more shingles than the floor counts'); + assert.equal(sketch(short), null, 'and still too little text: 116 shingles at a third of a word each'); + assert.notEqual(sketch(han), null, '186 shingles weigh 62 words, over the floor'); +}); + // The guard for the two fixes above. Every page that is plain ASCII must shingle EXACTLY as the // ASCII-only rule did, because MIRROR_THRESHOLD and the pinned Jaccard in this file were // measured on that: after `toLowerCase`, `[^a-z0-9\s]` kept exactly a-z, 0-9 and whitespace, @@ -195,3 +253,29 @@ test('ASCII text shingles exactly as the ASCII-only rule did', () => { for (let i = 0; i + SHINGLE_WORDS <= oldRule.length; i += 1) expected.add(oldRule.slice(i, i + SHINGLE_WORDS).join(' ')); assert.deepEqual([...shingles(sample)].sort(), [...expected].sort()); }); + +// The guard for ADR-0176: a page in a script that separates its words - and Korean is one - +// shingles exactly as ADR-0163's rule had it, so making a character a token in Chinese, +// Japanese and Thai moves no other page. Passes before and after that change, by design. +test('a page in a script with word spaces shingles exactly as the word rule did', () => { + const samples = [ + 'Тариф включает сто запросов в месяц, а лимит сбрасывается первого числа.', + 'Το δωρεάν πρόγραμμα επιτρέπει δέκα αιτήματα ανά λεπτό σε κάθε λογαριασμό.', + 'تسمح الخطة المجانية بعشرة طلبات في الدقيقة لكل حساب مسجل لدى الخدمة.', + 'התוכנית החינמית מאפשרת עשר בקשות בדקה לכל חשבון רשום בשירות הזה.', + 'मुफ़्त योजना हर मिनट दस अनुरोध की अनुमति देती है और महीने में सीमा रीसेट होती है।', + '무료 요금제는 계정마다 분당 열 번의 요청을 허용하며 한도는 매월 초기화됩니다.', + 'Le forfait gratuit numéro 1 autorise des requêtes limitées chaque mois, déjà réinitialisées.', + ]; + for (const sample of samples) { + const words = sample.toLowerCase().normalize('NFC') + .replace(/https?:\/\/\S+/g, ' ') + .replace(/[^\p{L}\p{N}\p{M}\s]+/gu, ' ') + .split(/\s+/) + .filter(Boolean); + const expected = new Set(); + for (let i = 0; i + SHINGLE_WORDS <= words.length; i += 1) expected.add(words.slice(i, i + SHINGLE_WORDS).join(' ')); + assert.ok(expected.size > 0, `the sample must shingle at all: ${sample}`); + assert.deepEqual([...shingles(sample)].sort(), [...expected].sort(), sample); + } +});