What each module owns, the seams between them, and which ADR governs each decision.
This is a map of the code, not of the research: research/MAP.md is the topic map —
a different artifact with a different job, judged by a different check.
Rule this map lives by (ADR-0007, generalised by ADR-0008): a commit touching this
project's declared code paths — for this repo, research-kit/lib, research-kit/bin,
research-kit/hooks, research-kit/githooks, declared in research/kit.json — carries
this file's update in the same commit. The commit gate refuses such commits without
it. What the rule proves is deliberately modest: the gate checks this file was staged
with the code, not that it is current — a whitespace edit satisfies it, which makes a
stale map a deliberate act rather than mere forgetting. A prompt, not a proof.
This file describes what exists now. Historical rationale — the defects that motivated
a design, the benchmarks behind a threshold, the alternatives that were rejected — lives in
adr/, in the dated reports beside it, and in
architecture-history/, and is linked from here. That
boundary was drawn on 2026-09-20, when this file had grown to 531 lines and a reader asking
"where does this behaviour live?" had to walk through several hundred lines of narrative
about defects that were already fixed. Nothing was deleted; it was moved and linked.
git pre-commit (sh) ─┐ runtime edit-time hook ─┐
▼ ▼
bin/gate.mjs ──── lib/gate.mjs (evaluate) ─── lib/preflight.mjs
(CLI adapter) │ ADR-0007 │ the verdict
│ ▼
│ lib/checks.mjs
│ 12 named checks, pure
│ │
▼ ▼
lib/provenance.mjs ◄──────────── lib/corpus.mjs
hash-chained ledger, migrations one corpus snapshot
▲
bin/decompose.mjs ── selectTransport() ── lib/decompose.mjs
(phase 0 CLI) lib/transport.mjs (gathers, seeds MAP.md)
bin/research.mjs ─── selectTransport() ── lib/research-run.mjs
(phase 1 CLI) lib/transport.mjs (plan, queries, budget, cache)
Both coordinators use the selected adapters through search-session.mjs and
collect.mjs. The invocation and provenance boundaries are shown below.
lib/machine.mjs ──── everything outside the project (config, install state, git
(ADR-0002, 0010, hooksPath, posture, machine role collector|builder) — the only
0020) module that reaches past root. Reads the machine config in
THREE states: absent / readable / UNREADABLE
lib/handoff.mjs ──── the arrival question on a builder: ledger present, every cited
(ADR-0011, 0020) capture on disk, chain verifies — a remedy per cause, no repair path
▲ bin/handoff.mjs (the builder's first command)
▲ lib/doctor.mjs (handoff-* findings, fail on role=builder)
lib/scaffold.mjs ─── project shape: LAYOUT, the four GATE_MARKERS, templates
lib/brief.mjs ────── research/BRIEF.md: the brief's shape (sections, the
judged two, the scaffold/draft/authored state) and the
reader lib/audit.mjs consumes — headings written from it
lib/doctor.mjs ───── diagnostics: aggregates machine + project + gate + chain
lib/installer.mjs ── installs the hooks, records/restores prior state
bin/collect-remote.mjs applies the existing value checker to --repository, --topic,
--client-ref, --prior and --prefer, as well as its workflow and budget flags.
An empty equals/separate value or bare optional value flag is refused by name with
exit 3 before output-directory creation, token lookup or dispatch (#287). Absent
required flags keep their existing refusal, including the repository-only pickup path.
The CLI chooses providers before invoking its coordinator. transport.mjs returns the fetch adapter, search adapter or merged adapters, selection reasons, and selection readiness metadata. Automatic fetch selection may perform a memoized Firecrawl CLI --status probe when the executable is present. Selection does not search or fetch a page. The coordinators decide which queries and pages to attempt; the shared search session and collection writer carry out those decisions.
flowchart TD
R[bin/research.mjs] -->|selectTransport| T[lib/transport.mjs]
D[bin/decompose.mjs] -->|selectTransport| T
T -->|adapters and selection metadata returned to CLI| R
T -->|adapters and selection metadata returned to CLI| D
R -->|runResearch with adapters and optional exhaustion fallback| RR[lib/research-run.mjs]
D -->|decompose with adapters| DD[lib/decompose.mjs]
RR -->|creates and consults per-run exhaustion policy| C[lib/credits.mjs]
RR -->|creates session with policy| S[lib/search-session.mjs]
DD -->|creates session without credits policy| S
S -->|synchronous search calls without a credits policy| SP[Selected search adapters]
S -->|askLive and live during research| C
C -->|supplied patient search function on live adapter| SP
RR -->|selected targets and live fetch adapter| CO[lib/collect.mjs: collectOne]
DD -->|selected material and fetch adapter| CO
CO -->|runScrape seam| FP[Selected fetch adapter]
CO -->|capture and source writes| RAW[research/raw captures and source siblings]
CO -->|appendFetch and locked row writes| P[lib/provenance.mjs and lib/corpus.mjs]
P --> LEDGER[research/raw/.fetches.jsonl]
P --> ROWS[research/EVIDENCE.md and SOURCES.md]
| Responsibility | Owner and boundary |
|---|---|
| Fetch/search choice and readiness information | transport.mjs: explicit flag, environment, machine config, then automatic selection. Fetch and search are separate choices; a search-only adapter is excluded from the fetch registry. |
| Shared search calls, rank interleaving, attribution and meters | search-session.mjs: synchronous provider calls, result-shape normalization, rate-limit patience, one bounded single-provider degradation, and one failure row per failed answer surfaced to the session. The exhaustion policy records its transition separately. |
| Paying-adapter identity and exhaustion transition | credits.mjs: one fresh policy per research run. It recognizes exhaustion only when the failed provider is the exact selected fetch-adapter object (provider === adapter), substitutes that object after a supported switch, and records one exhaustion transition. |
| Local exhaustion-fallback choice | bin/research.mjs: supplies a fallback only when explicitly enabled and the selected adapter exposes creditsExhausted. It prefers an available browser, otherwise http-keyless. |
| Query and target sequencing | research-run.mjs or decompose.mjs: relevance/preference rules, deduplication, budgets, cache decisions, failure sinks and output appropriate to its phase. |
| Durable page provenance | collect.mjs with provenance.mjs and corpus.mjs: cache recheck under the writer lock, capture/source bytes, hashes, ledger append and evidence/source rows. |
| Remote collection request and return | dispatch.mjs and mcp.mjs: workflow dispatch, run identity/status, artifact download, unwrapping and validation. These are separate from local adapter selection and search. |
The search arrows describe conditional invocation. During research, the session calls the supplied policy's askLive and live methods. The policy invokes the coordinator-supplied patient search callback, which uses searchPatiently to call the selected provider synchronously. Without a credits policy, as in decomposition, the session invokes that patient seam directly. The selection arrows above return adapters and metadata to the CLI; they do not represent page or search invocation. The research coordinator also consults the policy for its live fetch adapter, stop/switch decisions and final state.
There are three distinct ways a provider can change:
| Change | Trigger and scope | Spending and limits |
|---|---|---|
| Selection before a run | Automatic fetch selection uses a present authenticated or anonymous Firecrawl CLI, otherwise http-keyless. Automatic search uses the fetch adapter when it can search; a fetch-only browser or anonymous CLI uses keyless search. A configured SerpAPI key selects a rank merge with a search-capable partner. |
No page is fetched by selection. Its reasons and labels describe the selected route, including anonymous mode. Explicit names are not silently replaced by this automatic ladder. |
| Degradation for one query | With one search provider, a failed ask can make one additional ask through the current fetch adapter when it is a different provider and can search. This happens without --fallback. |
The replacement ask can spend on the fetch provider's search meter. Its failure and degradation are recorded. A fetch-only provider cannot rescue a search. Merged provider calls use a sequential loop and rank interleaving; they do not use this same single-provider degradation branch. |
| Credit-exhaustion transition during research | The selected paying adapter recognizes exhaustion. Local default stops. --fallback permits the chosen supported switch once; --no-fallback wins if both flags are present. |
Remaining work uses the chosen fallback in place of the exhausted adapter. A fetch-only browser can fetch remaining pages but cannot answer searches. A fallback's own error is not a second exhaustion transition. Unattempted pages/searches on a stopped run are counted separately from failed or spent work. |
runResearch constructs a new credits policy and search session for each invocation. It uses two rate-limit retries by default for search; collectOne separately uses two page-fetch rate-limit retries by default. Decomposition creates a new search session with zero search rate-limit retries and no credits policy. Its optional page fetches still use collectOne's default retries. Decomposition exposes no exhaustion-fallback flag. These are different coordination policies, although both use the same search and collection seams.
Readiness has two layers. selectSearch preserves an explicit provider's missing-key or missing-instance reason on chosen.search.notReady, allowing a preview or doctor to describe the selection. Both spending CLIs refuse that selection before searching. searchSession.assertReady() has a narrower library contract: it inspects notReady only on the supplied searchAdapter and searchAdapters objects. It does not read the selection object's metadata, inspect every fetch/fallback capability, or inspect an implicit fetch adapter when no search adapter was supplied. searchPatiently separately returns a named failure for a provider that cannot search, throws, or returns a Promise; search adapters are synchronous.
Research follows this sequence:
- For a collection invocation, the CLI checks role policy for a spending run and validates the plan, selects fetch/search adapters, then rejects
chosen.search.notReadyfor a spending run before callingrunResearch. The dry-run path does not take that CLI refusal branch. The coordinator checks CLI-major compatibility only when the fetch adapter name equalsfirecrawl.name(firecrawl-cli); the automatically labelledfirecrawl-cli-anonymousadapter is outside that guard. It then checks session readiness and ledger appendability before search/page calls, including previews. Selection/account probes and the named CLI compatibility probe can happen before those search/page checks. - The coordinator queues explicit plan URLs first, deduplicated by
urlKey. It filters queries by--onlyand skips queries whose cited unknowns are all CLOSED unless--forceis used. Its seen set holds pages already queued in this run; each search result is checked with the existing cache decision before ranking and selection, so fresh cache hits are excluded while stale or forced captures remain eligible. - Each remaining query goes through the session. A merged session asks each planned provider sequentially, normalizes result rows, interleaves by rank, and records every provider that found an exact result URL. The coordinator filters web URLs, relevance or the operator's
prefer, ranks candidates, deduplicates by page identity and appliesperQuery. Search-found pages start as secondary/context sources; a ranking preference does not declare primary authority. - The page budget is
min(plan.maxScrapes, depth allowance). A fresh cache hit or a recent remembered 404/410 costs no new page slot. Preview attempts obey the same page cap while spending nothing. Failed non-exhaustion fetches consume a page attempt; a recognized exhaustion refusal stops or retries the same page on the chosen fallback without counting that refusal as paid work. Search counters and adapter credit estimates remain separate from page attempts; they are not a vendor billing balance. - For each actual page,
collectOnerechecks the cache under the project writer lock. A success writes the capture and any available source sibling, hashes the bytes, appends a fetch-ledger entry, and updates source/evidence rows. The ledger records the actual fetch transport, declared completeness/omissions, source hash/omission, vendor cache metadata where provided, anddiscoveredByfor the ranking provider(s). An explicit plan URL has no ranking attribution. An error/oversized response records afailentry without inventing a capture; an unchanged forced capture can retain its existing evidence row while recording the new fetch. - The coordinator returns page outcomes, discovered results, independent search meters, degradation and exhaustion state. Its usage row is written for actual page spend or reported search usage, including a search-only run. The CLI prints that result; it does not recover authoritative provider accounting from an arbitrary diagnostic string.
Decomposition builds queries from the project topic, filters and deduplicates returned web pages, and interleaves compound-topic material by query. It orders optional scrape targets using likely-owner and relevance rules, uses maxScrapes as a page-attempt cap, and lets cache hits reach further down the candidates. Skipped likely-owner pages remain named for review. It drafts research/MAP.md with the checklist unjudged, candidate material and any captured page outlines; those outlines are read from disk without another fetch. Its --force permits redrafting judged MAP rows; this coordinator does not pass force into its page-cache decisions as research does.
The failure sink is the coordinator's. The session counts and reports the answer returned by creditsPolicy.askLive: that policy can intercept recognized exhaustion of the selected paying adapter and return a fallback answer instead. It records the exhaustion transition as op: credits-exhausted; if the returned fallback answer succeeds, the intercepted refusal does not also become a session op: search failure. Research immediately serializes search failures surfaced to the session into research/raw/.failures.jsonl as op: search; empty and off-topic answers get their own events there, and its credits policy appends the exhaustion event to the same log. Decomposition retains the session's failure objects and renders them in MAP and its search summary. In a merged query, the session delivers a failed ask to the sink immediately, then mutates that retained object's covered field when another provider answers. A retained object can therefore show covered later, while a row serialized at the moment of failure cannot; the research sink intentionally does not serialize that later field. This preserves chronology rather than promising identical snapshots from both sink types. Each run starts with new session meters, failure collections and exhaustion state.
Ledger appendability checks parsing, the finished tail and hash-chain links before spending; the append repeats that structural check under the lock. It does not certify existing capture bodies. verifyLedger, handoff and preflight perform the separate byte/hash and referenced-capture verification. A chain that can accept another entry is not by itself a validated corpus or an approved brief.
Remote dispatch is a separate flow: a CLI/MCP caller invokes dispatch.mjs to start collect.yml, receives its run id, follows that run and downloads/unwraps its corpus artifact for the existing artifact validator. The workflow performs collection on its runner and explicitly passes --fallback; that unattended policy does not change the local default. MCP collect returns the run id promptly, and fetch_corpus reads completion and validation. A downloaded corpus still carries buildAuthorized and review state; successful dispatch or download alone does not authorize building. Existing received-project review/repackaging compatibility is governed by its own ADRs, rather than by the provider selection layer.
Release validation is a separate path — it shares the canonical primitives and nothing else, and it never touches the corpus. It is also optional: decompose, research, preflight, the gate, the brief and handoff import none of it, so the research protocol runs the same with or without it. It came from an older tree under ADR-0029's porting rules, and whether it stays is an open decision (2026-09-30):
bin/researcher-release.mjs ──── validate | conform | fi-validate
│ usage error -> exit 2, always
▼
lib/release-validator.mjs ─── the facade: all 21 exports, unchanged
│ R28-R32 semantics, ledger anchoring, reduction
├── lib/release/json.mjs duplicate-key-safe parsing
├── lib/release/canonical.mjs canonical JSON, hashing, record shape
├── lib/release/schema.mjs the JSON Schema subset
└── lib/release/paths.mjs containment
│
▼
schemas/ · ledgers · pointers · evidence manifests · FI sidecars
read-only; CI diffs conformance/ and schemas/ after every run
Node Python
bin/ledger-conformance.mjs ─┬─ bin/ledger_conformance.py ─┐
bin/fi-sidecar-conformance.mjs ─┼─ bin/fi_sidecar_conformance.py ─┤ agree
bin/property-vector-conformance.mjs ─┴─ bin/property_vector_conformance.py ─┤ PER
bin/conformance_common.py ─┘ VECTOR
one canonicaliser, one float policy
Never by report hash: reportSha256 covers implementation.runtime, so it differs
between languages by design.
| Module | Owns | Governed by |
|---|---|---|
core.mjs |
Generic primitives (fs, paths, dates, slugs) and the artifact constants — file paths and table headers. Knows no semantics. makeSlug(text, fallback, limit) slices then trims, so a cut landing on a separator never leaves one dangling. kitCommand(script, args) (2026-09-27) spells every command the kit prints - remedies, next steps, the gate's fix line - as node <full path of the running kit>/bin/<script>, so it runs from any folder. They said node research-kit/bin/..., which exists only at the repository root: copied from a project folder, a fix crashed with MODULE_NOT_FOUND. Text written into a project's files (the brief, the timeline) travels, so it is spelled by documentCommand (ADR-0051): "$HOME/.agents/research-kit/..." when the running kit is the standard install, its real path anywhere else - unless the project recorded its own spelling: projectKitPath(root) reads kitPath from research/kit.json, which the scaffold writes from --kit, and documentCommand(..., { root }) prefers it (2026-09-28, ADR-0084; a brief drafted from a checkout named that checkout's path in a project scaffolded with $HOME); homeCommand is that standard-install spelling, which README-FIRST uses unconditionally because a package is always read elsewhere. They said node research-kit/bin/... too, which from the project folder they are read in named nothing. spellCommand(file, args) is the spelling itself: a path with a space is double-quoted as it is (ADR-0050) - it was JSON.stringify, which doubled every backslash of a Windows path. CLI entrypoints refuse unknown flags; most use refuseUnknownFlags, while collect-remote and disclosure validate their arguments directly; install, install-hooks, new-project, timeline, decompose, evidence-context and selftest lacked it until 2026-09-27, when install-hooks --unistall would have installed the gates and new-project --topc scaffolded an "Untitled topic". checkFlagValues(flags, specs) (2026-09-27) refuses a value flag given no value, or one outside its choices or not a whole number: parseFlags reads a bare flag as true, and each CLI fell back to a default in silence (research --depth thorough ran on quick's budget, install-hooks --mode block saved a mode read as "ask", new-project --topic scaffolded "Untitled topic"). preflight --only, accepted and never read, is now an unknown flag. parseFlags ends the options at a bare --, as POSIX has it (2026-09-28): it read -- as a flag named "" and every argument after it as a flag, so a CLI refused -- as an unknown option and no positional could start with a dash. parseJson(text) (2026-09-27) is JSON.parse for a file a person may have saved: it drops a leading byte-order mark, which Windows Notepad writes and JSON.parse refuses. readJson, the plan and kit.json readers, research.mjs, the machine config and the Claude settings reader all use it. urlKey(url) (moved from research-run, which re-exports it) is one page's identity across spellings, here so the checks need not load the collector. install.mjs spells its next steps against the copy it just installed, not the download it ran from; new-project.mjs uses kitCommand, and over an existing project says so, naming its topic and a --topic it did not apply (2026-09-27): it reported only "wrote 0, kept 13". realInside(root, abs) (2026-09-28, ADR-0076) resolves every symlink before asking whether a path is inside the project; isInside compares spellings only. homeDir() (2026-09-28) is the home directory, absolute: HOME when it is absolute, else the passwd entry - os.homedir() returns an EMPTY HOME as it is, which made the machine paths relative to the cwd; on Windows with USERPROFILE empty it THROWS instead (seen on windows-latest CI), and machine.mjs called it at import, so every command crashed - the throw falls back to the passwd entry too. writeText (2026-09-28, ADR-0079) writes a scratch dotfile beside the target and renames it over, keeping the mode and writing through a symlink: writeFileSync emptied the target first, so a write cut short destroyed the file it replaced. writeBytes(p, data) (2026-09-28) is that same write for BYTES, and writeText now delegates to it: the audit bundle and the artifact package are Buffers, and they still went through writeFileSync on the target, so a re-create that ran out of quota destroyed the good package it replaced - 2,679,013 bytes became 524,288, measured - which is the one file this kit hands to somebody else. tolerateClosedStdout(stream = process.stdout) (2026-09-28) drops the writes, after the first, on a stdout whose reader has gone: Node turns the next write to a closed pipe into an unhandled 'error' event, so `selftest.mjs |
head -1 died mid-run with a raw stack and exit 1 over a green suite. Any code that is not EPIPE is still rethrown. **canonicalJson** renders iteratively (2026-09-28, Arena break test 11): the recursive form overflowed the stack a few thousand levels deep, so one deeply nested ledger line made handoff.mjs- the builder's first command - print a V8 stack trace and exit 1, which means "the corpus did not arrive". The rendering is byte-identical to the recursive one, corners included, pinned by a test that keeps the old form as its oracle. A cyclic value is refused by name (aTypeError): the recursive form failed fast on one, and a loop would walk it forever. **isRegularFile(p)** was added on 2026-09-28 (Arena break test 12) and is described at its own row below. **siteOf(url)** (2026-09-28) is the registrable domain a URL belongs to, with a short list of shared suffixes (co.uk, github.io, ...) under which each label is a different owner: corroborationandmeasure.mjsjudge independence by it, because by hostnamedocs.firecrawl.devbesidewww.firecrawl.dev was two witnesses - one company describing itself. **operatorPath(root, typed)** (2026-09-28) is a path the operator typed: absolute as it is, anything else inside the project - resolvejoined an absolute--plan /tmp/x/plan.jsononto the root and reported "does not exist";research --plan, readPlanandprior --file use it. **writeFailure(err)** (2026-09-28) turns a refused write (EACCES, EROFS, ENOSPC, EISDIR, ...) into one sentence naming the file and the reason, or null for anything else, which the caller rethrows. **isRegularFile(p)** (2026-09-28, Arena break test 12) is the file-type check every read in the kit now makes: a regular file, or a symlink to one. readTextandsha256Filerefuse anything else WITHOUT opening it - a fifo blocks inopen()until some other process writes to it, and a character device such asdev/zeroallocates until libstdc++ kills the run - so one such file underresearch/raw/hung or crashed every entrypoint, the commit gate included, with no diagnostic at all.statSyncfollows symlinks on purpose: a link to a real file still has an end.documentCommand spells a kit that sits inside the project root relative to it (node research-kit/bin/...), so the kit's own repository regenerates its tracked files byte for byte on any checkout (2026-09-29). **tempBase()** (break-test PR #145, 2026-09-29): a relative TMPDIRis resolved against the process's start cwd and published back throughTMPDIR/TEMP/TMP, so every child agrees; a relative one had made the commit gate's scratch fail and the gate fail open. tolerateClosedStdout names a refused stdout (ENOSPC, EROFS, ...) and exits 2 - not a verdict - where it had thrown a raw stack and exited 1. **tempFreeSpace()** (break-test, 2026-09-29) is the free room on that temp folder, in words, for selftest's ENOSPC-dominated red summary; it lived in bin/selftest.mjs, which never imported node:os, so the one run that needed the diagnosis died with ReferenceError: os is not definedinstead. A static test now refuses any kit module naming a builtin it does not import. WhenstatfsSync itself refuses (ENOSYS), the fallback sentence still names the remedy - point TMPDIR (TEMP/TMPon Windows) at a folder with room (2026-09-30, Arena break test): it had said only "check the free space".checkFlagValuestakes amaxas well as amin(2026-09-29).makeSlugcaps its fallback at the same limit (2026-09-29, Arena break test): it returned the fallback whole, past the length the caller asked for; latent, since every caller's fallback is shorter than its limit.writeFailurealso names ENOENT frommkdir(2026-09-30): a recursive mkdir through a dangling~/.agentslink crashed install with a stack trace. ENOENT from any other call stays unnamed, because a missing copy source would be blamed on the destination.audit, decompose, research, install-hooksandprioralso route a refused write throughwriteFailure(2026-09-30): the first four crashed with a stack trace and exit 1, andprior printed the raw syscall text. Their test runs each case offline - search and fetch point at a closed loopback port. **projectFile(root, rel)** (2026-09-30, ADR-0109) is the one answer to "may this recorded path be read": { abs, problem }, problem outside(by path, or by real path when it exists),missingornot-file, else null. verifyLedgerandexportWarcuse it;corpus, the producer and shared artifact-stateapproval derivation, andevidence-contextnow apply it to their recorded-path reads (#351). Capture reads also enforce the capture-folder boundary by real path; ADR-0109 records the completed trigger. The atomic write's scratch carries at most 100 bytes of the target's name (scratchStem, cut on a character boundary), so a name up to the filesystem's 255-byte limit writes; ENAMETOOLONG is a named refusal (2026-09-30). flushed(stream)/exitAfterFlush(code): a command that prints a report waits for stdout and stderr to drain before it exits, because process.exit() discards what is still queued for a pipe on POSIX - preflight, evidence-context, audit --show, doctor, handoff and selftest end through it (2026-09-30). writeBytesrenames throughrenameRetrying: on Windows a rename that fails with EPERM, EBUSY or EACCES - a file an antivirus or the indexer holds for a moment - is tried again up to 5 times (20 ms doubling); elsewhere, and for any other code, it fails at once (2026-10-01). KIT_VERSION(0.9.5) is the kit's version, stated only here (ADR-0116).compareTextis re-exported fromrelease/order.mjs: identifiers sort by code unit, never by the machine's locale (2026-10-01). tolerateClosedStdoutanswers the callback of every write it drops, a tick later (2026-10-01, break-test PR #186):exitAfterFlushlearns that stdout has drained by writing an empty chunk and waiting for its callback, and the earlier replacement write never called it, soselftest.mjs |
audit.mjs |
The audit family: immutable single-file snapshots under research/audits/ (one full-corpus, one per COVERED/GAP subtopic), their versioning (nextVersion, fingerprint-driven, never overwritten), the manifest reader (listVersions → { known, topics } with the one ordering, resolveVersion → one version's path or null), and the bundle (zipAudit → the latest main audit plus every subtopic audit of that version, in one .zip named after them). bin/audit.mjs prints what those return and does no ordering of its own, so there is exactly one place the ordering is decided. The bundle reads the manifest's latest pointers and never a directory listing, renders nothing, bumps nothing, and refuses rather than guesses: no audits / several topics unnamed / a manifest naming a file that is not on disk (ADR-0019). Derived artifact: reads the corpus, never writes back to it. Generated names are budgeted: TOPIC_SLUG (60) and SUBTOPIC_SLUG (28), because Windows MAX_PATH is 260 and the project root counts against it — this repository produced 114-character relative paths under a 157-character root, and git add refused until core.longpaths was set. The files already written are left as they are: they read, and renaming them would break both the manifest that names them and the history that holds them. Containment resolves both sides the same way: the file and the boundary are each realpath'd before comparison, so a project reached through a symlink still bundles its own files, and the plain-path comparison is kept alongside it to catch a manifest naming ../.. when the target does not exist (history). bin/audit.mjs refuses --version without --show and --topic without --zip (2026-09-27): each was ignored, and audit --version v9 rendered a new audit without mentioning v9. writeAudit chooses the version, writes the immutable files and updates the manifest under withLock: unlocked, two audits at once could interleave (2026-09-28). resolveVersion reads both manifest shapes the kit has written (2026-09-28): the 2026-09-14 writer stored a version's file as a basename in research/audits and its subtopics under the topic, the current one stores main and a flat subtopics on the version. Read-only, it had refused every 0.1 audit - this repository's own - naming undefined as the missing file; a version recording no file now says so. The committed manifest is left as written, because audits are immutable. readAuditFile(root, file) (2026-09-28) is the one read of a file the manifest names, checked inside research/audits/ by name and by realpath before the read; zipAudit and audit --show both use it - --show read the path as given, and printed a file outside the folder that --zip refused. bin/audit.mjs refuses a folder that is not a research project (2026-09-30), exit 2, in the words brief, timeline and prior use: there it quoted a preflight verdict of three blocking findings for a folder where gate and preflight say "not gated". When the map's topic line has changed, bin/audit.mjs says the run started a new chain and names the one already held (2026-09-30), and its printed zip command carries --topic <slug> whenever more than one chain exists. It had forked in silence, and the bare --zip it recommended then refused. --list orders slugs with compareText (2026-10-01): localeCompare read the machine's locale, and under th_TH two slugs differing only by a dash compared equal, so two machines listed one corpus in two orders. Full and subtopic audits preserve the whole-corpus warning reasons from their supplied evaluation (ADR-0143). Version selection includes the rendered gate result in the fingerprint: a changed freshness/policy warning earns a new immutable snapshot, and old renderings earn a warning-aware version on the next render. Existing files are not overwritten. Only a consecutive identical latest observation reuses its version (ADR-0144); returning to an older fingerprint earns the highest retained version plus one, keeping the default bundle current through warning cycles. |
ADR-0143, ADR-0144 |
fi-validator.mjs |
Read-only FI (field-integrity) bundle validator: evidence-manifest and sign-off-sidecar schema conformance, the FI_IDS set and FI_STATUS_PRECEDENCE reduction, and readFiWorkbookProjection — which inflates a workbook entry with zlib.inflateRawSync to read a projection out of it without unpacking anything to disk. Depends on release-validator.mjs for canonical hashing and schema running; nothing else. Writes nothing. |
ADR-0029 |
ledger-conformance.mjs |
The offline, cross-language qualification-ledger conformance runner: five vector kinds (canonical-json, canonical-float, self-excluding-hash, chain-hash, ed25519), strict packet loading, and a deterministic report. Reads only; the sole fs call is readFileSync. loadLedgerVectors pins packetVersion, profile, the signature profile (Ed25519 / base64url-no-padding), a non-empty vector list and unique vector ids — a packet that drifts is refused rather than partly run. Two details carry more weight than their size: base64urlBytes rejects a signature that does not survive a canonical base64url round-trip, so an alternative encoding of the same bytes is not quietly accepted; and the packet's own SHA-256 is attached with enumerable: false, so it can be reported without entering the canonical JSON that is hashed. reportSha256 covers implementation.runtime, i.e. process.version — so it is a within-runtime determinism check, and cross-language agreement is asserted per-vector, never by comparing report hashes. |
ADR-0029 |
fi-sidecar-conformance.mjs |
The second half of the conformance foundation: classifies FI sign-off sidecars and evidence manifests as valid, malformed or tampered, against a vector packet, with a Node and a Python runner that must agree. Read-only — no writes, network, spawn or eval in any of the three files. One of its four tests exists to check the FIXTURE rather than the code: that the vector packet stays synthetic and carries no completed evidence, so a conformance suite cannot quietly become a place real sign-offs are stored. Patch traversal and removal accept every canonical nonnegative array index, including indices after zero; a synthetic two-entry packet verifies Node/Python agreement (#337). | ADR-0029 |
property-replay.mjs |
Captures a failing property seed once, as a self-describing replay envelope, and replays it deterministically afterwards. The only ported module that writes, and it writes exactly one way: fs.writeFileSync(file, …, { flag: 'wx' }) — O_EXCL, so write-once is the kernel's decision rather than a check that can race. Three layers guard it: an existsSync fast path, the exclusive flag, and an EEXIST catch that returns the existing envelope with created: false. A second failure on the same seed therefore cannot overwrite the first, which is the point — the first failure is the evidence. Identity binds family+seed+iteration three ways at once: the filename, the caseId, and inputSha256; validateRegression re-derives the caseId and refuses a mismatch. regressionSha256 self-excludes before hashing, so tampering with any field is caught. Everything after capture is read-only. Pointer order ties break by compareText (2026-10-01). |
ADR-0029 |
property-vector-conformance.mjs |
Exports the property suite's canonical-hash and graph findings as a static vector packet, and re-verifies them — in Node and in Python, against the same file. Read-only in all three files. Its value is that it turns property tests, which generate their own inputs and could drift with the generator, into fixed vectors a second language can check: canonical hashing invariant under object insertion order and sensitive to authored array order, predecessor closure across generated chains, and descendant invalidation that is transitive, sorted, duplicate-free and permutation-invariant. Two of its five tests guard the fixture rather than the code — that the exported vectors stay synthetic, and that the exported graph and hash values stay bound to the property seeds that produced them. | ADR-0029 |
r29-workbook-linkage-validator.mjs |
Cross-reference validator for the R29 reviewer-workbook linkage register: 24 task ids, six expected reference kinds (task/gold/source/calibration/warm/lock), and a status reduction that takes the worst of PASS < INCOMPLETE < FAIL < REOPEN. Zero imports — it is 58 lines of pure functions over { register, catalog, pointer } passed in as arguments, so it touches no filesystem, no network and no clock, and the caller owns every input. That is also why it needed no sealed-record fixture: its tests synthesise a register into a temp directory rather than shipping one. A revoked lock pointer reduces to REOPEN rather than FAIL, which is the distinction the rank order exists for — a withdrawn approval is not the same as a broken link. |
ADR-0029 |
config.mjs |
A compatibility shim, nine lines, re-exporting machine.mjs's config surface so an older import path keeps working. It adds nothing and decides nothing. Listed because it was the one module in lib/ this map did not mention — found by counting files against rows during the 2026-09-20 reconciliation, which is the only way that kind of omission surfaces. |
ADR-0002 |
archive.mjs |
One container format: buildZip(entries) → bytes, writeZip(file, entries) → { file, bytes, entries } (written whole or not at all, through writeBytes, since 2026-09-28), crc32, entryName. ZIP with deflate (store when deflating would only grow the payload), with UTF-8 name encoding declared by bit 11 in both local and central headers, preserving Unicode manifest paths when ordinary ZIP tools extract them (#345); written because the kit has no dependencies and cannot shell out to a zip binary that may not exist. Knows the container and nothing else — what goes inside a bundle is lib/audit.mjs's. Refuses entry names that are not plain relative paths (an archive must not unpack outside its folder) and an archive with no entries. See ADR-0019. Changes no name it is handed (2026-09-28): entryName refuses a backslash - it translated one into a separator, which stored a\b.md as a/b.md while the package manifest still declared the original. |
ADR-0019 |
artifact-zip.mjs |
Reading a ZIP somebody else wrote — the opposite job from archive.mjs, and a separate file on purpose: a writer that also parses hostile input grows a trust boundary in the middle of itself. openZip(bytes) → { entries, names, problems, has, read }, plus checkEntryName, readCentralDirectory, inspectEntries, readEntry and the exported ZIP_LIMITS (entry count, total and per-entry uncompressed bytes, ratio + floor, name length) so tests drive the refusal paths with small fixtures instead of real bombs. The order of operations is the security property: every structural refusal — traversal, absolute, drive letter, backslash, NUL, duplicate, case collision, file/directory-prefix collision, symlink/special entry, encryption, unknown method, counts, sizes, ratio — is decided from the central directory before one byte is inflated. The directory is attacker-controlled too, so it is never the only defence: inflateRawSync runs with maxOutputLength, making the ceiling the runtime's to enforce rather than a number compared against a field the attacker wrote. Names are judged as raw text, never path.normalize'd first — normalising would let a/../../b become ../b and then be judged, when the honest answer is that .. has no business in a research artifact. Touches no filesystem: entries come back as buffers. ZIP64 is refused rather than half-read. |
ADR-0032 |
artifact-validator.mjs |
Ledger verification follows actual archive capture entries and parsed evidence rows, so relabelling roles or declaring zero captures cannot disable provenance checking; an empty failed-collection scaffold still passes (#358). Is this package valid, and does it authorize a build — two questions, kept apart. validateArtifact({ file, expectedClientRef }) → { status, buildAuthorized, packageId, clientRef, workflowRunId, state, errors, warnings }, where status: 'PASS' means internally consistent and says nothing about permission. Offline and read-only by construction: no transport imported, process.env never read, nothing from the package executed, no shell, and the only write is a fs.mkdtemp copy of project/ removed in a finally — needed because the chain and handoff checks are the kit's existing code (readCorpus, verifyHandoff) and reuse beats a second implementation of provenance verification nobody tested. Stable error codes (ZIP-*, MANIFEST-*, FILE-*, LEDGER-*, CAPTURE-*, AUTHORIZATION-INCONSISTENT, SECRET-DETECTED, FORMAT-UNSUPPORTED, CLIENT-REF-MISMATCH) and never a secret value in an error — the report names the pattern and the line. authorizationProblems() is the one implementation of the build rule; the schema encodes it a second time and when the schema catches it first the rule is still reported in its own words. An unsupported format major is INCOMPLETE, not invalid. A crash is BLOCKED, never PASS. Its report carries reviewedBy (review.by, or undeclared for a 1.0.0 package) and judges nothing by it (ADR-0074). Report metadata is surfaced only after the manifest passes schema validation; malformed state and identity fields are null, and the CLI escapes control characters in every text diagnostic so hostile strings cannot forge lines. Reads format majors 1 and 2 (2026-09-30, ADR-0107): a 1.x HUMAN_REVIEW_REQUIRED is reported as REVIEW_REQUIRED, and eraProblems refuses a value from the other major by name - a 2.x package with the old state or review.by: "human", or a 1.x one with the new state. A binary body (NUL in its first 8000 bytes) is not matched against the credential patterns, as a body over 512 KiB is not (ADR-0128, 2026-10-02): docs/ travels whole, and one vendored DLL carrying the PEM marker it parses would have failed a package with no secret in it. Missing, ill-typed, malformed and below-oldest format versions are MANIFEST-SCHEMA/FAIL with regeneration as the remedy; only a numeric format major newer than this reader is FORMAT-UNSUPPORTED/INCOMPLETE (#361). Claimed approvals are re-derived from checked bytes with the portable unpromoted pluralist, 180-day baseline; warnings name that baseline and format-1 legacy compatibility. |
ADR-0032, ADR-0160 |
artifact.mjs |
Raw-cell review reads and explicit ledger reads use projectFile under ADR-0109; Raw reads stay inside the capture folder, and a non-file ledger is refused before reading (#351). Packaging a project without changing it; shared authorization derivation lives in artifact-state.mjs. createArtifact → { bytes, name, manifest, entries, derived }; writeArtifact writes then re-validates with the consumer's own validator before reporting success; its bytes go through writeBytes, so re-creating a package at a name that already holds one cannot trade a good package for a truncated one (2026-09-28). There is no --build-authorized and no option that reaches an authorization field: deriveState(root) runs the real gate with env defaulting to {} so a packaging shell's override cannot become a property of a package that outlives it. mapClassified and briefReviewed have honest machine answers; findingsReviewState answers the third negatively — a Finding byte-identical to what firstFinding re-extracts from the capture was demonstrably never rewritten, which detects the unreviewed case without grading the reviewed one (ADR-0013's line, uncrossed). It repeats the collector's own call exactly - the page body after the front-matter, falling back to the page's title or url from that front-matter - since 2026-09-27: run over the whole file, the extractor read the front-matter as the page, and a URL-only corpus nobody had touched (one JSON capture, run 36287211468) was packaged findingsReviewed: true. packageName never carries the topic (an artifact listing is world-readable on a public repository) and falls back to run-<workflowRunId>; checkClientRef refuses rather than sanitises. Inventory sorted by ZIP path, manifest and digest appended last, digest over the exact manifest bytes. README-FIRST in a package that is not yet approved prints the preflight and brief commands and one create line carrying the package's own identity (repackageArgs, 2026-09-27): the manifest is written once, and the package gave its reviewer no way to get one that says APPROVED_BRIEF. Commands are spelled by homeCommand (ADR-0050). createArtifact checks the identity against the manifest schema's own source rules before deriving or writing anything (sourceProblem, 2026-09-27): create --commit abc wrote a package, then reported it did not validate, and left it on disk. review.by (format 1.1.0, ADR-0074) is the reviewer's declaration read from BRIEF.md by reviewedBy: agent or undeclared (human was retired in format 2.0.0, ADR-0107). Not an approval condition. README-FIRST no longer says "a human reviewed it"; it names the declared reviewer, or says none was declared. collectProjectFiles and the explicit ledger add refuse (OUTSIDE_PROJECT) any path whose real location is outside the project: it followed links and packed outside files, a private key included (ADR-0076). The same walk asks the consumer's own checkEntryName about every name it is about to store and refuses the run before a byte is written (UNPACKAGEABLE_NAME, 2026-09-28): entryName translates a backslash into a path separator, so a file named a\b.md travelled as a/b.md - a directory invented out of a name - while the manifest kept the original, and only the read-back caught the contradiction, after the package had been written. The same refusal covers two files differing only in case (2026-09-28), with the reader's lowercase comparison: the reader's ZIP-CASE-COLLISION fired only after the write. Format 2.0.0 (2026-09-30, ADR-0107): a collected package waiting for review says REVIEW_REQUIRED, and README-FIRST gives the review to the agent alone. collectProjectFiles refuses (UNPACKAGEABLE_FILE) what packaging cannot read to an end - a FIFO, socket or device, and a folder link back into its own ancestry (judged by real path, so a link to a sibling folder inside the project still packages) - before a byte is read (2026-10-01). A reviewed brief is complete and current (2026-10-03, output-reliability audit G1): deriveState approves only when every judged section is present and answered (judgedSection) and a stamped brief's inputs hash matches the corpus (draftStamp, briefInputsHash); briefState alone read a missing section as answered, so a drafted brief with no review sections derived APPROVED_BRIEF on the repository's own corpus, and a brief whose evidence changed after the draft stayed authorized beside hygiene's brief-stale warning. A brief with no stamp predates ADR-0055 - the root's own and ten decision briefs - and keeps its approval: its currency cannot be checked, hygiene/brief-unstamped says so (ADR-0138), a redraft with --force stamps it. README-FIRST in every state and the collection summary preserve the producer evaluation warnings through renderGateWarnings (ADR-0143). Authorization fields and machine package formats are unchanged. bin/artifact.mjs refuses a value flag given no value by name, exit 3, before anything is read or written (2026-10-11, #280): create had asked only whether a required flag was present, so a bare --ref went into the manifest as "true", a bare --run-id as 1 and a bare --output wrote a package named true that re-validated PASS; a bare validate --file tried to open a file called true. Caller references use checkClientRef before token lookup, output-folder creation or dispatch (#375, 2026-10-13); CR/LF are refused by the shared checker, and diagnostics are redacted. The artifact CLI awaits exitAfterFlush for its report, help and refusal endings, preserving queued stdout/stderr and the existing exit status on asynchronous POSIX pipes (#359). |
ADR-0032, ADR-0031, ADR-0052, ADR-0143 |
artifact-state.mjs |
The one approval derivation shared by producer and validator: real gate and handoff, classified map, reviewed findings, authored/current brief; consumers rederive claimed approval under the explicit portable baseline and exact format-1 pre-0069 input hash (ADR-0160). Finding review resolves a blank Raw cell through the shared capture resolver and refuses unresolved captures; every capture read retains project and raw-folder path containment (#351, #357). | ADR-0032, ADR-0160 |
mcp.mjs |
The collector as a Model Context Protocol server, and the first module in this repository built from an APPROVED BRIEF rather than from a plan - every design choice cites an E-row in docs/decisions/2026-09-21-agent-interface/. handle(message, deps) is a pure function of a JSON-RPC message plus injected transport, so the whole surface is tested offline. stdio, not Streamable HTTP: semantics are identical on both, so the difference is operational, and remote costs the entire OAuth 2.1 chain. Two tools, not one - collect returns a run id immediately, fetch_corpus returns a link - because a server cannot initiate a request, so the alternative is one call held open for minutes and an agent that loses a paid run when its process dies. A resource_link, never embedded bytes: the spec defines embedding but states no maximum size, so a link is the defensible option. No tool takes a token and a test asserts no parameter is credential-shaped. PROTOCOL_VERSION is pinned and a mismatch answers -32022 with data.supported, the same discipline as pinning X-GitHub-Api-Version. createStdioLoop frames synchronously inside each data event - overlapping async drains had mutated the buffer concurrently, which a test caught - and answers on two chains: tools/call one at a time in order, everything else on its own chain, so ping and tools/list are answered while a call is in flight (#347, 2026-10-17); they had waited behind it on one shared chain. DUAL-ERA: SUPPORTED_VERSIONS is ['2026-07-28', '2025-11-25', '2025-06-18'] (2025-06-18 since 2026-09-27, ADR-0049: a client on the official SDK up to 1.24.0 asks for it and refuses 2025-11-25, and nothing 2025-11-25 changed touches this server; 2025-03-26 is not served, because the results carry structuredContent and resource_link) and initialize is implemented beside server/discover, because the specification is ahead of every shipped client - the official SDK at v1.30.0 declares 2025-11-25 as its latest and answered the modern-only first version with -32601: unknown method initialize. initialize echoes a serveable requested version and falls back to legacy rather than to this server's preference, because a client throws on a version it does not support. Arguments that fail a tool's schema are a tool result (isError: true, the reason as text) since 2026-09-27 - SEP-1303 in 2025-11-25 says input validation errors should be Tool Execution Errors, so the model can correct itself; they were JSON-RPC -32602, which a client raises as an exception. An unknown tool stays -32602. collect's topic carries minLength: 1, and validateArgs enforces minLength after trimming (2026-09-27): an empty topic had passed and reached GitHub, which answered HTTP 422. fetch_corpus checks out_dir with usableOutDir before it reads the run (2026-09-27), so an unusable folder is an OUT_DIR tool error, not UNKNOWN. fetch_corpus returns reviewedBy, and its text says review is for an agent or a person (ADR-0074). createStdioLoop bounds a line at MAX_LINE (4 MiB, 2026-09-28): a line past it is refused with -32600 as soon as it passes the cap and dropped through its newline, and the next message is served - also while a call is in flight, since the check runs in the data event rather than behind the call (#346, 2026-10-17) - the buffer had grown without limit until a newline came, so a client that sent none could grow the server until it died. No request is lost at end of input (break-test PR #145, 2026-09-29): exitWhenSettled waits for every call already read before exiting (bounded by END_GRACE_MS), and a final request without a trailing newline is served on end - both had been dropped with exit 0. SERVER_INFO.version is KIT_VERSION (ADR-0116); it had announced a fixed 1.0.0. The approved fetch response directs the builder to README-FIRST and the brief for warnings and known unknowns (ADR-0143). It forwards no new machine warning field and derives no new authorization. validateArgs checks allowed keys by own membership and enforces the declared arrays of strings: malformed query/URL values and inherited Object property names become schema tool errors before credential lookup or dispatch. Omitted or empty arrays, and valid string arrays, keep their existing normalization. The shared queriesInput refusal prevents CR/LF inside individual queries or URLs from dispatching extra runner work (#376, 2026-10-12). Rejected query/URL input passes through the existing credential redactor before it becomes tool-result text (#378, 2026-10-12), including caller credentials absent from the server environment. Caller references use checkClientRef before token lookup, output-folder creation or dispatch (#375, 2026-10-13); CR/LF are refused by the shared checker, and diagnostics are redacted. Output folders resolve to absolute paths before access checks; the MCP caller passes that same path to fetchCorpus so the write and resource link identify the same file (#372, 2026-10-13). Non-PASS corpus downloads return only a redacted diagnostic with validator codes and remedies, never a corpus link or review instructions; buildAuthorized stays false (#371, 2026-10-13). |
ADR-0034, ADR-0049, ADR-0143, ADR-0172 |
disclosure.mjs |
What a public run actually exposes, measured rather than asserted. Written because the opposite claim was documented first: the collector's header said dispatch inputs are visible to anyone with read access, and checking it found the topic in none of the readable responses. probeRun sends unauthenticated requests - there is no token parameter, and a probe that quietly authenticated would answer a different question while looking like this one. summarise separates exposedNames (true on any public repo, and the reason the topic is kept out of run and artifact names) from exposedContent and exposedSubject. needleChecked exists so a report can never let "false" read as "no": without a topic to search for, the rendering says not checked. An unreachable endpoint is a refusal to conclude, never a clean result - enforced since 2026-09-27: until then that sentence was true of the findings and false of the verdict, and with no network every probe was unreachable and the command exited 0, "only the shape is public". summarise now reports unmeasured (probes with status 0) and conclusive; exitCode(report) keeps a measured exposure (2, 1) whatever else failed, and otherwise needs every probe answered for 0, else 3; the rendering says UNREACHABLE and "not tried" instead of "refused", shows not measured for a false it could not establish, and closes NOT MEASURED instead of the shape paragraph. An unreachable probe names the cause of the failed fetch through fetchFailure (2026-09-30), not the bare fetch failed. probeRun takes the artifact and job ids from the listings its probes already read (2026-09-30); it had fetched both again inside one try/catch, so a transient failure on the second request left the artifact download and the log untried. |
ADR-0035 |
dispatch.mjs |
Asking the collector to run, from outside GitHub - the seam an agent or a desktop app uses. dispatchCollection (pins X-GitHub-Api-Version: 2026-03-10 on the REQUEST, not merely in a comment), waitForRun, listArtifacts, downloadArtifact, unwrapArtifact, plus tokenFromEnv and redact. No token may be passed as an argument - tokenFromEnv is the only way in, because a credential on a command line reaches the shell history, ps, and any log that echoes its own command; a test asserts the source contains no token flag. redact is PATTERN-based rather than "remove the token we hold", so it also catches a credential the process was never given - one echoed back by a server, a proxy, a redirect. A 204 is a named NO_RUN_ID failure, never an empty success: "the run started and I cannot tell which one" is the failure a caller is least able to diagnose from a status code. Both routes to a run id are requested, not just the pinned version: the X-GitHub-Api-Version: 2026-03-10 header AND return_run_details: true, the optional parameter GitHub added 2026-02-19 that returns the run id on 2022-11-28 as well. Measured before it was trusted - four calls on 2026-09-21 established that the old version honours the parameter and the new version accepts rather than rejects it, so sending both is safe and a server that strips or downgrades the header still answers with a run id. That retired ADR-0031's original fallback, which was to poll /actions/runs and correlate by created_at - a race the design never actually needed. NO_RUN_ID now means BOTH routes were ignored, which is a far narrower diagnosis. waiting is reported as a status, not a stall - it means a protection rule is holding the job. unwrapArtifact opens GitHub's outer ZIP through the same distrustful reader, and passes a hostile or ambiguous container through untouched for the validator to refuse rather than guessing at packaging depth. Every request goes through an injectable fetch, so the whole module is tested offline. Pages by URL (2026-09-27): queriesInput builds the one queries input from search queries and pages to fetch, one per line, and refuses a URL that is not http(s) before anything is dispatched. collect.yml fetches a line that is an http(s) URL instead of searching it, and with only URLs it searches nothing. collect-remote --url and the MCP collect tool's urls share this function, so all three agree. Until then the collector could only search, and a page the caller already knew had to be collected locally. usableOutDir checks the folder before anything is spent (2026-09-27): it creates the folder and tests write access, and an unusable one is a named OUT_DIR failure with a remedy. fetchCorpus calls it before it lists artifacts, the MCP fetch_corpus tool before it reads the run, and collect-remote before it dispatches. The folder had been checked only after the download, as a raw ENOTDIR the MCP server could report only as UNKNOWN, and for collect-remote that came after a paid run and up to half an hour of waiting. fetchCorpus writes the package it downloaded through writeBytes (2026-09-28, ADR-0079): fs.writeFileSync emptied a package already at that name when the write failed. bin/collect-remote.mjs checks --max-pages (1..25), --depth, --runner, --search-transport and --timeout against the ranges and choices collect.yml enforces, before it asks for a token (2026-09-29, Arena break test): --max-pages 8o was dispatched as typed, and the paid run failed on the runner. A dispatch that never left names the cause of the failed fetch through fetchFailure (2026-09-30), still passed through redact; so does .github/scripts/check-action-pins.mjs for a pin it could not check. Both had printed the bare fetch failed. getRun and listArtifacts read their body through readJsonBody (2026-09-30): a 200 whose body is not JSON (a maintenance page, a proxy) is DispatchError BAD_BODY with the status and a remedy, where it had thrown a raw SyntaxError that collect-remote could only call UNKNOWN. getRun, listArtifacts and downloadArtifact reach GitHub through callApi (2026-09-30), so a request that never left is DispatchError NETWORK with its cause (fetchFailure, redacted) and a remedy, as a dispatch already was; they had thrown the raw fetch failed, which collect-remote could only call UNKNOWN. fetchCorpus treats names from the API as data: a wrapper entry is flattened to its file name, an artifact name holding a separator, :, .. or a control character is refused (ARTIFACT_NAME) and nothing is written (2026-10-01). unwrapArtifact hands on a wrapper whose inner package cannot be decompressed unchanged, for the validator to judge, instead of throwing ZIP-READ (2026-10-01). waitForRun retries a failed poll that may pass on its own - no connection, a 429 or 5xx, a body that is not JSON - up to MAX_POLL_RETRIES (5) in a row, reset by a good poll and never past the deadline; 401, 403 and 404 stay fatal (2026-10-01). Every failed poll had ended a watch of up to 30 minutes with RUN_FAILED while the run went on. getRun gives a 401, 403 or 404 its own remedy (a missing run, a refused token), and the timeout remedy names --run-id (2026-10-01). splitRepository admits only two nonempty [A-Za-z0-9._-]+ components other than . and .., so dispatch, run lookup, artifact listing and download refuse route-changing delimiters and control characters before their first request (#379, 2026-10-12). queriesInput refuses CR/LF in either list before trimming and serialization (#376, 2026-10-12), retaining existing empty-input and one-line normalization. A declared item cannot become additional workflow lines. dispatchCollection also uses readJsonBody: a successful non-JSON response is BAD_BODY with the proxy/maintenance-page remedy; valid JSON missing the run ID retains NO_RUN_ID (#377, 2026-10-12). Output folders resolve to absolute paths before access checks; the MCP caller passes that same path to fetchCorpus so the write and resource link identify the same file (#372, 2026-10-13). Default corpus downloads use a fresh private mkdtemp folder per call; existing or dangling symbolic links at the package name are refused as OUT_DIR before the write, and MCP describes the private default (#373). Dispatch and run reads classify a rate-limited 403 from headers or body as RATE_LIMITED, retaining Retry-After seconds and the UTC reset time; permission-only 403 remedies remain unchanged (#374). |
ADR-0033, ADR-0032 |
brief.mjs |
The phase-1 → phase-2 handoff: renders research/BRIEF.md from the corpus, and owns the brief's shape — BRIEF_SECTIONS (six sections, the two that are judged marked), the markers that distinguish still the scaffold from the drafter wrote this (briefState → template | legacy | draft | authored), and the section reader (briefSection / judgedSection) that lib/audit.mjs consumes instead of regexes of its own. The renderer writes headings from that definition, so the writer and its readers cannot drift; the scaffold ships BRIEF_FILE_MARKER, which the substitution pass leaves alone — the alternative, matching the template's prose, made a sentence of English load-bearing. The verified table has one row per claim and source, naming the unknowns it closes (2026-09-26): two unknowns leading with the same E-row used to print its finding twice, word for word - three committed briefs carried such duplicates. Every E-row an unknown cites gets a row, not only the first (2026-09-29): a MoonAliza closure resting on E-03 and E-06 showed E-03 alone, and the row it most depended on never reached the builder. With no KNOWN-UNKNOWN declared, the unknowns section reports that status and names any CLOSED unknown citing no P row: the earlier primary-source assurance was removed in 2026-09-27 and the unconditional closure assurance in ADR-0143. Its commands are spelled by documentCommand, with the project's recorded kitPath first (ADR-0084). --force over a draft, authored or legacy brief keeps it as research/BRIEF.md.bak-<date> (numbered on a second force that day) and says so (2026-09-27): it restored the TODOs and kept nothing of the answers it replaced. Every draft is stamped (2026-09-27, ADR-0055) with a hash of its own text, a hash of what it was drafted from (briefInputsHash: topic, intent, unknowns, evidence rows, and since ADR-0069 the map rows and the ledger and capture counts), and the verdict it was drafted under (gate=), which bin/preflight.mjs compares with its own verdict and notes when they differ. draftStamp reads the stamp back, wherever it stands (the last one, if several), and text after it is an edit like text before it (2026-10-03, ADR-0138): anchored to the end of the file, a line appended after the stamp - a reviewer's note, the Reviewed by: agent line - had made the brief read as unstamped, so a stale brief re-approved the build and brief-stale fell silent. An unedited draft is redrafted without --force and without a backup, and any edit keeps the refusal. A draft written while the gate failed had kept saying "Gate: FAIL" after PASS. A known unknown's day-one step drops a leading "Day one:" before it is labelled (2026-09-27), so a cell written that way is not rendered "Day-one verification: Day one: ...". reviewedBy(text) (ADR-0074) reads the brief's `Reviewed by: agent |
humanline; the drafted brief carries a placeholder that reads asundeclared. A line naming both roles, or two lines that disagree, also read as undeclared; a line inside a code block is not a declaration, and an indented one is. A known unknown's Evidence cell that states what is known and then labels its day-one step mid-sentence renders as "Known so far" plus one "Day-one verification" line, never the label twice (2026-09-29). A cell the author wrote as the brief prints it, starting "Known so far:", keeps that label once too. reviewedByreads onlyagent(2026-09-30, ADR-0107): aReviewed by: humanline reads asundeclared, and the drafted placeholder names only the agent. **PASS reports configured checks, not complete closure or factual correctness** (ADR-0143). The preamble preserves the evaluated warning reasons through renderGateWarnings`; the known-unknown section reports declared status without claiming all unknowns are closed, including on FAIL. The contradictions placeholder asks for corroboration rather than asserting agreement. On FAIL or without an evaluation, the verified section explicitly labels its CLOSED rows as declarations whose verification has not succeeded or has not been evaluated. Six sections are retained. |
corpus.mjs |
Capture and Raw-path checks use projectFile; capture reads stay inside the capture folder by real path, without reading an internal private-file link (#351, ADR-0109 trigger). readCaptures(root, { known, sketches }) reads only the captures a caller does not already hold and sketches them only when asked (2026-10-02, break-test pass 3 F6): the collector's per-fetch refresh had re-read, parsed and MinHash-sketched every capture on disk inside the lock - 41% of a 2000-page run - and rememberCapture had pushed every entry again, n(n+1)/2 copies after n fetches; it now remembers a file once. Sketches serve one gate check (corroboration) and are the gate's to compute. readCaptures indexes every capture by exact URL (byUrl, with the asked-for URL of a redirected fetch as an alias) and by urlKey (byKey, 2026-10-02): cacheDecision asks the exact spelling first and the key second, so a plan that spells a page differently from its capture - www., a trailing slash, a fragment, http for https - is a cache hit and spends nothing; rememberCapture keeps both indexes for a page collected mid-run. Also builds two per-capture indexes alongside the entries, both because readCaptures is already holding every body: sketches (MinHash, for corroboration's mirror grouping) and renderFailures (countRenderFailures, RENDER_FAILURE_MARKERS — how many times a page printed a notice saying its own content failed to load). Neither rides inside captureEntry, which is serialised into artifact manifests. The research corpus: one reader (readCorpus, readCaptures, parseTable, parseCapture, readOverrides) and one writer family (appendRow, upsertRow, appendJsonLine, writeRaw lives in collect), plus the cache decision over the capture index (cacheDecision) — the one freshness predicate the collector, the run, and phase 0 consume. Table machinery: splitRow/escapeCell/tableRow/repairRowArity. Malformation is reported, never absorbed. The capture index has one constructor (captureEntry, used by readCaptures and by the collector when it has just written a page) and one writer (rememberCapture), so the index a run keeps in memory holds the same shape as the index read from disk — callers store the entry they are handed and shape nothing. The format's joins are here too: traceOf (evidence row → the fetch that produced it and the capture behind it), captureOf (row → capture, via the by-file index the reader builds), and claimOf (unknown → the claim it rests on, first cited row, falling back to the unknown's own text). Five consumers re-derived those joins, including the subtle "an empty Raw cell matches on the URL alone" clause; now a format change lands in one place. A research/kit.json that does not parse is a kit-unparsed problem, blocking in corpus-shape like plan-unparsed (2026-09-27): the gate read it as absent and fell back to its default code paths, silently unguarding the ones the operator declared. A row with more cells than its header folds the extras into the table's prose column (Finding, Unknown, Why it matters, Title), not the last one, so the structured columns to its right stay in place, and the table-arity warning says a ` |
inside a cell splits it and to write|(2026-09-27): folded intoRaw, a Finding quoting a |
transport.mjs |
Adapter selection, and the one reader of the operator's transport choice — now for two sides (ADR-0027). The fetch ladder is unchanged: --transport, RESEARCH_KIT_TRANSPORT, the config's transport, then auto-detection (probeFirecrawl — authenticated or anonymous CLI wins, otherwise keyless). The search ladder mirrors it: --search-transport, RESEARCH_KIT_SEARCH_TRANSPORT, the config's searchTransport, then one auto-detect rule — a SerpAPI key wins, otherwise the search side IS the fetch provider, so a machine with no key behaves exactly as before. selectTransport returns { name, adapter, why } still meaning the FETCH side, with search added alongside; that is what let six call sites stay unedited. Exposes TRANSPORTS/TRANSPORT_NAMES (whole-shape adapters), SEARCH_PROVIDERS/SEARCH_PROVIDER_NAMES (a separate map — a search-only module in the first would make ADAPTER_SHAPE a claim it no longer keeps), FETCH_SHAPE, SEARCH_SHAPE, satisfies, isSearchOnly, selectSearch. isSearchOnly reads a module's declared CONTRACTS rather than duck-typing, because a search-only module exports scrape on purpose — to refuse by name. unusedKeyNote(chosen, env) (2026-09-27) is the line research.mjs and decompose.mjs print under the transport line when FIRECRAWL_API_KEY is set but auto-detection fell back to keyless because the CLI is not on PATH: the key is unused, and the line gives the install command. Until then only doctor said so. An explicit --transport http-keyless gets no note. browser is the third transport (2026-09-28, ADR-0088), chosen only by name; SEARCH_PROVIDERS holds only the transports that search, and selectSearch gives a fetch-only transport the keyless search, saying so. With a SerpAPI key the merged search pairs SerpAPI with the fetch side only if it searches; a fetch-only transport (the browser) hands its share to http-keyless (2026-09-29, found by the first live browser run). SEARCH_PROVIDERS includes searxng (2026-09-30, ADR-0104), selected only explicitly; selectSearch reports it notReady without an instance URL, as it does SerpAPI without a key. selectSearch hands the searches of a Firecrawl CLI with no key to the keyless route, as it does a fetch-only transport's: Firecrawl still scrapes for that CLI but refuses its searches ("Alexandria requires a Firecrawl API key", CLI 1.24.6, 2026-10-01). canSearch sits beside isSearchOnly because both are a transport's own capability, asked of any provider: whether it can search at all — the browser fetches only (ADR-0088), and a Firecrawl CLI with no key has a search function and no search. SEARCH_KEY (2026-10-07, ADR-0147) is the metered search provider's credential for a surface that must not name the vendor - its env and config names, read, state (set, and from where; never the key) and redact - consumed by the desktop panel. Registry lookups accept only own members, so inherited names such as constructor, toString and proto receive the existing unknown-name refusal at every precedence level (#349, 2026-10-13). |
ADR-0005, ADR-0027, ADR-0088, ADR-0147 |
serpapi.mjs |
A search-only provider: the first module here that satisfies one contract instead of the whole adapter shape, declared as CONTRACTS = ['search']. Produces no capture, so nothing it returns reaches the fetch ledger and transport-provenance is untouched. Sends exactly engine, q, api_key — no result-count parameter, because none is documented and a response costs the same whatever it carries, so limit is applied client-side; no_cache and async are never sent, which keeps the vendor's free 1-hour cache. The key is read from SERPAPI_API_KEY or the config's serpapiKey, never from a repository, and travels to the child on stdin rather than argv so it never enters the process table. The request URL — the one string that contains the key — is built in the child and never returned; redact() scrubs it from vendor error text on the way back, and command() renders it pre-redacted. A query that contains the key is refused before transmission, because cmd is written into the hash-chained ledger (a committed file) and because sending it would store the credential as a search term on the vendor's systems for 31 days; redacting the rendering alone would have fixed the smaller half. Async fetch behind a synchronous shape via the same self-exec rendezvous http-transport.mjs uses. status() answers from configuration alone and declines to report a remaining allowance, because no endpoint this project has captured reports one. requestUrl(query, key, endpoint) is exported so the exact parameter set is testable rather than buried in the child — the guarantee worth pinning is negative, and negative guarantees rot silently. The endpoint is overridable and guarded: allowedEndpoint permits the vendor's own host over https, or loopback, and nothing else. That exists so the whole path — parent, spawnSync, child, real fetch, real socket — can be driven against a local stand-in instead of being exercised once by hand and called verified; an overridable endpoint on a request carrying an API key is otherwise an exfiltration route, so both halves check it, the parent before spawning and the child before fetching, because the child reads its job off a pipe and does not get to trust it. The one thing no stand-in can prove — that the two meters really are separate — is covered by test/live-collect.test.mjs, an opt-in test (RESEARCH_KIT_LIVE=1 plus a key) that drives bin/research.mjs through a real collecting run in a temp project and asserts the Firecrawl spend against the pages fetched. Opt-in because running it on every suite would cost every developer credits forever; existing because "it costs money" is a reason to gate a test, not to skip it. It also says what the meter's numbers mean (METER_NOTES, 2026-09-26): the 50/hour and 250/month searchUsage reports are the documented Free Plan (E-25, E-27), while SerpAPI's own Account API reported 250/hour for this account (U-9); and SerpAPI's month is a billing cycle, not the calendar month the kit counts (U-10). research.mjs --status prints the selected search adapter's notes, so the vendor stays named only here (NFR-3) and a machine without a key hears nothing about it (FR-5) The key it searches with is the key that selected it: search() reads the machine config when none is passed, as selectSearch does. It defaulted to null until 2026-09-26, so a key held only in ~/.agents/research-kit.config.json selected SerpAPI and then failed every search "no SerpAPI key" - a merged run degraded silently to the other provider on every query It reads the vendor's own meter (account(), ADR-0040): serpapi.com/account.json, free and uncounted (E-26), through the same child-process path, endpoint lock and redaction as search(), copying only whitelisted fields (the payload carries the email, the account id and the key). research.mjs --status asks the selected search adapter for it and prints a vendor meter line; no key, no call. status() stays offline. Its child uses a configured proxy (2026-09-27, ADR-0047): runJob starts it with fetchEnv(env) - it had been started without, after the keyless child was fixed, so behind a proxy every search went around it and failed "fetch failed"; search() and account() hand their env to the child. A child past its timeout is reported as "SerpAPI did not answer within Ns", not spawnSync ... ETIMEDOUT (2026-09-27, seen on a real run) The child runs with maxBuffer: CHILD_OUTPUT_LIMIT and names an overflow (2026-09-28, ADR-0082): under the 1 MiB default a large answer was lost to ENOBUFS. normalizeSearch drops a result row that is not an object and reads a field of the wrong type as empty (2026-09-28): one null row threw and discarded every valid result beside it. The child reads the answer through boundedText (2026-09-28, ADR-0082): with response.text() a 17 MiB answer was accepted, since the output limit bounds only what reaches the parent. |
ADR-0027, ADR-0040, ADR-0047 |
searxng.mjs |
The second search-only provider (2026-09-30), with the contract serpapi.mjs defines: GET <instance>/search?q=...&format=json against a SearXNG instance the operator runs, named by SEARXNG_URL or searxngUrl. No key and no meter. JSON is off in a default instance, so an HTTP 403 is named as "enable json in search.formats", never "no results"; a row with no string url is dropped (the docs type it str | None). Chosen only explicitly. An instance behind basic auth (http://user:pass@host, 2026-09-30) has its credentials taken out of the URL - fetch refuses one that carries them - and sent as an Authorization header (basicAuth); every message that names the instance shows it through shownInstance, password as ***. Every search had failed, and the error printed the password. shownInstance redacts a password from any URL that parses, whatever its scheme, and by pattern from one that does not (2026-09-30): a mistyped htp://user:pass@host - exactly when the instance is shown in an error - had printed the password. basicAuth sends a literal % as typed instead of throwing URIError. A password holding an unencoded / ? # or @ - which no URL parser accepts - is redacted by pattern up to the last @ (2026-09-30). requestUrl builds the endpoint on the pathname, so a query or fragment on the instance URL no longer drops its sub-path (2026-09-30). |
ADR-0104 |
witness.mjs |
The opt-in Wayback witness (2026-09-30): lookup(url, { timestamp }) asks archive.org/wayback/available for the snapshot closest to the capture time, as a child job through the proxy-aware fetch, and returns { witnessed, snapshot, timestamp, status } or { witnessed: false, reason }; it never throws. parseAvailability treats every field as optional, because the API page is from 2013 and says it changes. Lookup only: it never requests a save. Its child reports a failed lookup through fetchFailure like the other children (2026-09-30), and waybackTimestamp takes a date string and returns empty for an invalid date instead of throwing. |
ADR-0106 |
http-transport.mjs |
The keyless adapter, and the second one - which is what makes the transport seam real rather than hypothetical. Pure Node ESM, no dependencies: built-in fetch driven synchronously through a child-process rendezvous (it re-execs itself with a JSON job on stdin), a word-density main-content extractor, HTML→markdown, DuckDuckGo-lite search, same-domain link mapping. Exposes the same seven-function shape as the Firecrawl adapter (pinned by test), never reads an API key, never charges a credit, stamps transport: 'http-keyless', and grades its own completeness (full only ≥1500 chars of markdown, else partial — an honest self-assessment the capture-completeness check then judges). A fetch past its timeout says "no answer within Ns", not spawnSync ... ETIMEDOUT (2026-09-27). It honours a configured proxy (2026-09-26): fetchEnv (in runtime.mjs since ADR-0047, shared with serpapi.mjs) adds NODE_USE_ENV_PROXY=1 to the fetch child's environment when HTTPS_PROXY/HTTP_PROXY is set, because Node's built-in fetch ignores those variables without it; an operator's own NODE_USE_ENV_PROXY, even 0, is left alone. Until then the fetch went around the proxy it was given: in the container where this was found, GitHub answered with its unauthenticated rate-limit 403, because the bypass lost the proxy's authentication (corrected 2026-09-27; the first version said "proxy-only egress"). The flag needs Node 22.21+ or 24+, and doctor warns on older (ADR-0046). It grades by content type (2026-09-26): the fetch child reports Content-Type, and bodyKind routes HTML (or no type) through extraction and the length bar as before; JSON, XML, CSV, plain text and Markdown are kept verbatim and graded full unless empty, because nothing was extracted or dropped; a binary type (PDF, images) is never full, and the reason names the type. A complete 748-character GitHub API response had been graded partial by the HTML bar. A page's own <main>/<article> is believed (2026-09-27, declaredContent): the largest one with at least 100 words is the content, found on its own because the div-block search is lazy and non-overlapping, so a nested in wrapper divs was never a candidate. On a GitHub Docs page that kept 143 characters of a 17,385-character article; now 5,921. Outside declared content, link-heavy blocks (breadcrumbs, sidebars) are navigation, not dropped content; prose outside it is still reported, and the reason says "outside the page's main content". search reports DuckDuckGo's bot check (anomaly modal, no result links) as a failed search with its reason, not as zero results (2026-09-27, ADR-0066); the user agent stays honest. decodeEntities reads a numeric reference to 0, a surrogate or past U+10FFFF as U+FFFD, as HTML does (2026-09-28): String.fromCodePoint threw on the last, and one malformed entity crashed the scrape. search and map decode an href before reading it as a URL, and map drops a fragment from the page it names (2026-09-28): taken raw, ?a=1&b=2 became a parameter named "amp;b", and map's pattern refused any href holding a #, dropping & and every link to a section of another page. The child reads a body through boundedText (runtime.mjs), refusing one past MAX_PAGE_BYTES (16 MiB) whether declared or streamed, where response.text() held any body whole (2026-09-28, ADR-0082). A run of sections is the content (2026-09-28): when the densest block holds under a quarter of a page's words and its <section>s hold at least half, mainContent takes the sections together (basis: 'sections'), and link-heavy navigation around them is not counted as lost - nodejs.org/api/fs.html, 314 sections and no <main>, went from 1,095 kept characters to 253,654. A binary response (PDF, image, archive) is refused by name and kept as no capture (2026-09-30, ADR-0105): read as text it was unrecoverable and could hold no quote, and one 10-page PDF had put 2.3 MB of it into a corpus. The refusal names --transport firecrawl-cli, which converts documents. Tag pairs are found by tagPairs (and replacePairs, stripTags) in linear time, remembering the next > and the next closing tag as the scan moves, where the lazy pair regexes were quadratic on unclosed tags - 9.4 s for 391 KB of <div>; the result links of a keyless search page are resultLinks. test/fixtures/regex-extract.mjs keeps the regex extractor as the oracle the linear one is compared against (2026-09-30). Redirects are followed one hop at a time, and a hop into an internal address (loopback, private, link-local, localhost; internalTarget) is refused unless the URL asked for was internal itself; RESEARCH_KIT_ALLOW_INTERNAL_REDIRECTS=1 lifts it (ADR-0110, 2026-09-30). The fetch child wraps dns.lookup with guardedLookup, so every connection its fetch opens is judged where it is opened: an internal answer is refused (EINTERNAL), unless the URL asked for was internal or the host is the proxy (ADR-0114, DNS rebinding). isInternal is exported for the browser guard (ADR-0118). A refusal names the server's own reason beside the status (serverReason, 2026-10-02): a JSON message, an HTML title or the first line of a text body, one line and bounded (REASON_MAX), so HTTP 403 - the server said: "..." is what the ledger and the terminal record; a github.com 403 had been recorded as HTTP 403 alone while its body named the refuser - this sandbox's proxy, not GitHub (docs/decisions/2026-10-02-github-plain-fetch-refusal). scrape, search and map all go through it. Entities are decoded once, at the end of the conversion (2026-10-03, output-reliability audit G3): the block passes - headings, list items, table cells - convert their bodies with inline(body, { decode: false }), because the page-level pass strips every tag that is left and a < decoded earlier was read there as the start of one; a list item holding x < 5 and y > 2 had come out as x 2, a cell and a heading the same, and a link label inside an item lost everything from its < to the item's end, while a paragraph survived. A link's label is never decoded on its own either, wherever the link stands: the G3 fix had left the page-level pass decoding it inside the link pass, so a paragraph's <a>limit < 10 and burst > 2</a> still came out [limit 2](/x) and &lt; was decoded twice (found 2026-10-03, review of G3); the label's emptiness is judged on its decoded text, so a -only link is dropped in a list item as in a paragraph; and decodeEntities decodes every reference kind in one pass, so &lt; is the text <. The regex oracle in test/fixtures/regex-extract.mjs carries the same rules, so the 3,000-page equivalence test still proves the scanner changed cost and nothing else. The keyless child reads its page through boundedBody and puts the decoder's decodeFallback sentence on its JSON line; the parent grades such a capture partial with the sentence as the reason (withDecodeFallback, verbatim), because a body read in a charset the decoder did not know is not known to be the page (2026-10-03, audit G7). A charset declared only in a <meta> tag is still not read, but such a page's bytes are not valid UTF-8 and the decoder names that, so the capture is graded partial rather than silently wrong (review of G7, 2026-10-03). Under an explicit refusal a name asked for is judged before any fetch (ADR-0155, 2026-10-11): with allowInternalRedirects: false the fetch child judges a NAME asked for with internalTarget - localhost, or a name resolving inward - and refuses it against the URL asked for, before fetching; the guarded lookup still judges it at connection time. A literal address the caller spelled out is the caller's, and the default (undefined) still treats an internal URL the operator asked for as theirs (ADR-0110). |
ADR-0005 |
browser-transport.mjs |
The browser transport (2026-09-28, ADR-0088): renders a page with a local Chromium, --headless --disable-gpu --virtual-time-budget --dump-dom <url> as an argv array, and grades the DOM with the keyless extractor (mainContent, htmlToMarkdown, gradeCompleteness). findBrowser (config browserPath, RESEARCH_KIT_BROWSER, usual paths), browserArgs (--proxy-server from the proxy environment, --no-sandbox only as root, never a TLS-weakening flag), chromeErrorOf (Chromium's own error page, which --dump-dom prints with exit 0). Fetch only: CONTRACTS = ['fetch'], map is the keyless map, and transport.mjs searches keyless for it. A signal other than the timeout's is reported as killed by <SIGNAL> with Chromium's own cause - the last FATAL stderr line, else the last ERROR line, else the last line - never as the timeout (2026-09-29). On Linux it prefers Chrome stable (/opt/google/chrome/chrome), the binary Ubuntu 23.10+'s AppArmor lets sandbox itself, and a "No usable sandbox" crash names the remedies that keep the sandbox - never --no-sandbox for non-root (ADR-0092). Its recorded command is browser <url> (2026-09-30): it had used the keyless formatter whole, so every browser capture's ledger entry read http-keyless browser <url>. scrape renders through renderGuarded, which runs the guard child (browser-guard.mjs) under spawnSync and reads back the DOM, Chromium's exit and the guard's refusals; a page that landed on a refusal is reported by name and not captured (ADR-0118, superseding the ADR-0115 pre-check). The refusal named is the one the dump holds - the guard's refusal page carries its host and reason - and only a dump that is Chromium's own error page falls back to the last recorded (2026-10-02): a dead third-party host refused after the navigation, or on a no-network host the browser's own www.google.com, had been named as what the page reached for. browserArgs never names a proxy: the child adds the guard's, and switches Chromium's background traffic off - which Chromium 141 ignores for its own services, so the guard drops those by name (ADR-0124). A timeout is judged by what Chromium printed, not by its exit (2026-10-01, CI, three times in one afternoon on Windows and Ubuntu): a whole dump after the timeout is the render, because Chromium prints the DOM once when its own budget says the page is done and a hang at exit does not unrender it; a dump holding the guard's refusal is the refusal; no dump at all is the timeout, and the message now carries the last thing Chromium said on stderr, which the timeout branch used to drop. Chromium is given its own deadline, --timeout, ten seconds under the transport's kill timeout and never below five (2026-10-01): with the virtual-time budget alone, one resource that never answers froze the budget - virtual time advances only when the page is idle - so headless never dumped, printed nothing, and was killed with nothing to show, which is what CI's four hangs that afternoon looked like and what a page with a never-answering image reproduces here. At the deadline Chromium dumps the DOM it has and exits, so the page is captured as it stands. A capture is conservatively graded partial when the guard still has pending loads at process completion or process elapsed minus startup reaches the configured deadline threshold. cutAtDeadline reports the configured loading deadline and the guard's request state at process completion (2026-10-08), rather than claiming an observed DOM-dump time or assigning the historical wait to the browser: neither a late process end nor the final pending snapshot proves those facts. The same note rides a refusal's message. A timeout's message says what the launch cost - "took N s to make its first request" or "never made a request in N s" (ADR-0119). The origin's HTTP status comes from Chromium's own net log (2026-10-03, ADR-0137, output-reliability audit G5): --dump-dom reports no status, so every browser capture carried statusCode: '' and the collector's >= 400 rule, which reads a number, never fired - an origin's 403 page was graded full. documentStatusFromNetLog(log, requested) reads the log the guard child had Chromium write (--log-net-log): event and source names are looked up through the log's own constants, never by number; the document is the URL_REQUEST source whose first URL_REQUEST_START_JOB matches the requested URL (compared in canonical form without the fragment, a main frame request winning); each later start on that source is a redirect, and the status is the HTTP/x NNN line of the last HTTP_TRANSACTION_READ_RESPONSE_HEADERS in the final job. scrape then fails a non-2xx answer the way the keyless transport does (HTTP N - the server said: "<title>"), refuses a final URL that is internal by spelling and not the requested host:port (the guard's own exemption), carries statusCode as a number and the post-redirect URL, and grades a render whose log gave no status partial with omitted saying the origin's status was not observed (rule 2 of the ADR) - never guessed from the page's text. A JavaScript navigation is a new source, not a redirect, and is not followed. The guard child's EINTERNAL - a name asked for refused before any launch under an explicit refusal - is reported as that refusal, not as a browser that could not start (ADR-0155, 2026-10-11). scrape and status given no config read the machine's with loadConfig(env) (2026-10-11): the collector calls runScrape(url) with no options, and the empty default meant browserPath was never consulted. |
ADR-0088, ADR-0092 |
browser-guard.mjs |
The guard proxy Chromium renders through, and the child that runs both (ADR-0118). startGuard listens on a loopback port and judges every plain request and CONNECT with judge: the keyless rule (isInternal), a name by the addresses it resolves to, the connection pinned to the address judged; only the host:port asked for is exempt, allowInternal is the operator's own decision, and a configured proxy is the upstream (absolute URLs, CONNECT passed through, Proxy-Authorization from the URL). A refusal is a 403 carrying REFUSAL_MARKER and is recorded. renderThroughGuard runs the browser with --proxy-server for the guard and --proxy-bypass-list=<-loopback>, its own process group on POSIX (killed whole on a timeout), the DOM capped at MAX_PAGE_BYTES, the exit reported after a short grace even when a forked helper holds stdout open. Run as a process it takes one job on stdin and prints one JSON line. The guard keeps a record of every request it carried - seen, each with its outcome (a status, refused, tunnel, an error, or unanswered once the guard closes) - and pending() lists the ones still unanswered, with how long they have waited; the child reports requests, pending and elapsedMs beside the dump, and seen as a timeline from the launch - each request's offset, duration and outcome - because a Windows run then stalled 28 s with nothing pending at the guard, so where the time went became the question (2026-10-01). The child keeps the first 30 lines of the browser's stderr as well as the last 60, with the omission counted between them, and stamps the launch (startedAt): a hung Windows launch's tail showed profile creation 43 s after the launch, so the wait was in the head the tail could not show (2026-10-01). The render timeout starts at the browser's first request through the guard, and the launch is allowed as long again (ADR-0119): the child arms the kill at launch and re-arms it on the first request, and reports startupMs: three Windows CI runs in a row ran the first launch to its deadline and nothing said on what, and the guard is the one place every load passes through. The browser's own service traffic is dropped (2026-10-02, ADR-0124): BROWSER_SERVICE_HOSTS names Chromium's service hosts - clients*.google.com, *.clients.google.com, mtalk.google.com, accounts.google.com, www.google.com, the update, safe-browsing, variations and autofill endpoints on googleapis.com, *.gvt1.com, dl.google.com - and isBrowserService answers for one; a request there is answered 403 with DROP_MARKER, logged dropped, and is neither judged, carried nor counted as a refusal, the page asked for exempt by host and port. Rendering a plain loopback page, Chromium 141 had made nine such connections per render, carried as external, and on a no-network host the last of them - www.google.com, unresolvable - became the verdict of the LIVE test about an internal address. A request before the page's own is the browser's (ADR-0125): with pageFirst, which renderThroughGuard passes, every request is dropped until the host and port asked for has been requested - CI's Chrome stable preconnected to www.gstatic.com 256 ms into a render, before the page, and that host is one real pages use too; the order settles what a list cannot. The CONNECT socket gets its error listener before any answer is written (2026-10-02): a browser resets a CONNECT it has given up on with the 403's body unread, and an ECONNRESET on the handed-over socket with no listener was an uncaught exception that killed the guard child mid-render - an empty report, read by the transport as "the browser exited null", on three CI legs at random points of the render. The child has Chromium write its net log (2026-10-03, ADR-0137): renderThroughGuard makes a private folder (rk-netlog-), adds --log-net-log=<dir>/net-log.json before --dump-dom, reads the file after the render (a log over MAX_PAGE_BYTES is dropped as empty) and deletes the folder on every way out; a folder that cannot be made renders without the flag, which the transport grades as status not observed. The child also reports allowInternal, the internal-address decision it made by resolving the requested URL, so the transport judges the final URL by the same rule. The log travels in full through the child's JSON stdout and is never written to the corpus. A name asked for is judged by what it resolves to, exempt or not (ADR-0155, 2026-10-11): judge refuses a name resolving inward, or localhost, unless the caller allowed this machine's network, whether or not it is the host and port asked for; the exemption covers only the literal-address judgment. Under allowInternalRedirects: false the child judges a name asked for before launching and answers EINTERNAL with no render. |
ADR-0118, ADR-0124, ADR-0125 |
collect.mjs |
One URL's journey: collectOne (cache, force, fail entries, evidence/source rows) and writeRaw (the capture writer, which hands back the corpus's own index entry). Consumes the runScrape seam; stamps transport and completeness from what the adapter declared — invents neither. The Finding cell a row is born with is lib/finding.mjs's: the collector imports firstFinding and does not re-export it, so the extractor has one address. A fetch reports why it happened in words (whyFetched: first capture, refreshed with the age, or --force), not cacheDecision's code - verbatim, a fresh fetch logged as "collected - not-collected" (found in a live-collection log, 2026-09-26). writeRaw never overwrites a capture with different bytes (2026-09-27, ADR-0058): a same-day re-collection of an unchanged page reuses the file, and a changed page is written as <name>.r2.md (then .r3...), which sorts after the original so readCaptures still takes it as the day's latest. The earlier reading had been replaced, and its ledger hash broken. An error page is a failed fetch (2026-09-28): a result whose statusCode is 400 or more is recorded as a fail ledger entry and writes no capture and no EVIDENCE row - Firecrawl answered ok with Google's "Error 503" page and the kit had filed it as evidence; the keyless transport already refused non-2xx answers. recentlyGone (ADR-0100) answers gone without fetching when the ledger's last entry for the URL is a 404 or 410 inside the refresh window. writeRaw refuses (OUTSIDE_PROJECT, a named write refusal in core) a link at the capture's generated name - dangling or not - and a capture folder whose real path leaves the project, before it reads or writes; a project living under a link is judged by real path and still writes (2026-10-01). A front-matter value with a line break is written on one line, so an adapter title or error cannot add a field; the body is written verbatim (2026-10-01). A page whose text exceeds CAPTURE_MAX_BYTES is a failed fetch, named with its size and the limit, before anything is written: the reader refuses such a file as capture-too-large and the gate blocks on it, so a 40 MB answer had become a capture, a ledger entry and a row the corpus could not hold (2026-10-02, break-test pass 3). refreshCaptures, the re-read inside the lock before every fetch, asks readCaptures only for files the snapshot does not hold and for no sketches (2026-10-02): per-capture cost at 400 rows went from 74 ms to 13 ms. A forced re-fetch that comes back byte-identical reuses the capture (writeRaw never overwrites) and lets the row that cites it stand - the fetch is in the ledger, the reason says the page is unchanged, so E-01 stands; it had appended a second row that superseded the first, so every unknown citing unchanged content had to move (break-test pass 3, 2026-10-02). A changed page still goes to .r2 with a row of its own. writeRaw writes the body with LF line endings only, whatever the transport received (2026-10-02): the scaffold's .gitattributes makes git store a capture as LF, and a keyless capture written with the CR bytes copyright.gov served hashed one way on the collector and checked out another way everywhere else, failing handoff on a capture nobody had touched. The one such capture already committed (a licence decision project, E-18) is pinned byte for byte with a -text attribute in that project. refreshCaptures reads the new captures with the ledger's rank (2026-10-03): one ordering rule with the reopen, so a capture another collector wrote since the snapshot is current only if its fetch came later. The collector remembers its own fetch as latest, and refreshCaptures decides every known capture the ledger names again with the rank, since a reused file's position moves while its bytes do not (2026-10-03). |
ADR-0016 |
finding.mjs |
The auto-extracted Finding cell: firstFinding(markdown, fallback) — page markdown in, one line of prose out — over the heuristics behind it (boilerplate/tracking/nav/campaign regex families, the sentence splitter, the table reader, the tier-hero block skip, the graded score, the 220-character cap). Two arguments over ~460 lines: a deep module that was buried in the middle of another one until ADR-0016. Pure by construction — no filesystem, no corpus, no adapter — the one module a caller can exercise with a string. findingWithContext is the real entry point and returns { finding, excerpt, heading, signals, score }: the same selection, plus where on the page it came from, the surrounding paragraph bounded to EXCERPT_CHARS, and which signals fired. firstFinding is a thin wrapper over it, so the sentence the collector writes and the excerpt a reviewer reads cannot be chosen by different code. Ranking is one declared table, RULES — 21 named rules, each with a weight and the reason for it, and explainScore returns the rules that fired alongside the total. Weights used to be two separate runs of value += n, one in score() and one in the selector, so the priority order could not be stated without executing the code and the number a reviewer saw could differ from the number that chose the sentence; a test now asserts no weight is applied outside RULES, and that the reported signals sum to the score. Heading context is an ancestry stack, not the last heading seen, so a claim under #### Capacity still knows it sits inside ## Pricing. The policy is evidence-first: a measured quantity or a metering rule (how usage is counted, which often carries no digit) is strongest; where a sentence sits counts as much as what it says, so a terms-bearing heading and an FAQ answer both score while blog and careers copy is pushed down; and things that only look concrete — one row of a pricing comparison matrix, a number inside a sign-up prompt — are penalised enough to lose to plain prose. The comparison-row rule's tier test is three linear scans (tierThenPeriodThenDigit, per line, since . never crossed a newline) instead of `\b(plan |
tier)\b.*\b(month |
evidence-context.mjs |
Raw cells pass projectFile for both project and capture-folder boundaries before any content is read; private project files, traversal and links leaving research/raw are refused, and directories are named as non-captures (#351, ADR-0109 trigger). One unknown, everything it rests on, in one read-only answer: the unknown's wording and why it blocks, every evidence row it cites, each row's source title / URL / type / retrieval date, the Finding, a bounded excerpt of the capture chosen for overlap with that Finding (and, in a block longer than the bound, windowed where the most distinct tokens of the Finding fall together - 2026-09-27: a JSON capture is one block, and the excerpt was its first 420 characters whatever the Finding quoted), rows that supersede it, and other rows for the same URLs. It decides nothing: it writes no status, offers no verdict, and reports a gap rather than filling it — a test asserts the rendering contains no conclusion and says plainly that the judgement is the reader's, and structural tests assert it imports no verdict machinery and contains no write call at all - the architectural property, which a phrase-based check cannot give. The recorded status is printed LAST and labelled as a recording, because a verdict shown before the evidence anchors the reader. Where the extractor would now pick a different sentence than the row records, that is shown beside it, labelled a suggestion. Reads no environment variable and spawns nothing, so it cannot use a credential. A Raw cell outside the project is reported ("is outside the project, so it is not read") and never read: it printed the outside file into an agent's context (2026-09-28, ADR-0076). bin/evidence-context.mjs --all over a contract with no unknowns yet answers "no unknowns" (or [] with --json) and exits 0 (2026-09-29, break-test): it exited 2 asking for the --all it was given. |
— |
credits.mjs |
What a run does when the paying provider runs dry — ADR-0086's one switch to the fallback, ADR-0129's stop-and-report. The one owner of a run's exhaustion state: fellBack / stopped, the uncollected and skippedSearches accounting, and the single credits-exhausted row the failure log keeps per run. The coordinator sequences and asks it which provider serves each call (exhausted, live, onExhaustion, askLive) and reads the state back for the summary; before the extraction the state lived as closures inside runResearch and stopped.uncollected was incremented from three loop sites. |
ADR-0086, ADR-0129 |
search-session.mjs |
How a query is put to the search providers (ADR-0135), for both coordinators - runResearch and decompose each carried a copy, and read side by side on 2026-10-03 the copies had drifted in six places (a failed first ask's usage counted in one, a plain failure in searchFailures in the other, searchTransport spelled serpapi by one and serpapi+firecrawl-cli by the other under a merge, the notReady refusal in one only, three log lines in one only, covered in the other only). searchSession({ adapter, searchAdapter, searchAdapters, limit, maxRateLimitRetries, sleep, log, record, credits, askOne }) returns name (the paying provider, what the usage row's searchTransport says), meter (the dry run's label, every provider under a merge), providers, assertReady() (a searchAdapter reporting notReady throws SEARCH_PROVIDER_NOT_READY before any ask), ask(text) and meters(). ask is the loop: every provider under a merge, interleaved by mergeByRank and attributed, else the one provider with one bounded degrade to the fetch provider when it canSearch and is not the provider that failed; it answers { ok, query, results, provider, providers, merged, searchId, degraded } or { ok: false, ..., error }. One accounting rule: what an adapter reports is counted on its meter for every ask, succeeded or failed, and every failed ask is one failure on its provider - a query that failed on both providers is two. The failure row is the session's, the sink the coordinator's: record gets { query, error, provider }, with degraded: true, fellBackTo on a degrade and covered: true when another provider of a merge answered; it is handed over at once (so a failure log keeps its chronology beside a credits-exhausted row) and gains covered afterwards. Provider-level lines (waiting, search failed on, degrading to, searched, search found N, search failed, search failed on every provider) are the session's; what a coordinator makes of an answer stays its own. searchPatiently and mergeByRank live here, re-exported by research-run.mjs. With a credits policy the policy's askLive/live decide which provider serves a call (ADR-0133); askOne is the one patient call a coordinator shares with that policy. A session built with no provider at all answers the failed search searchPatiently already names, charged to this transport, rather than throwing on the missing name (2026-10-03, probing after the module landed; only a library caller can reach it). resultRows keeps the shape every reader of a row assumes - an array, each row an object - on both paths (2026-10-03, probing): a provider's null row had ended both coordinators on row.url, and results that were not an array had ended research's noteOutcome on .some; whether a row's URL is usable stays the coordinator's question. assertReady reads notReady off every provider the session may ask, the merge's partners included, not the search side alone (2026-10-03, probing). A provider that throws is a failed search that says so (searchPatiently, 2026-10-03, probing): every adapter returns its failures by contract, so a throw is an adapter's bug the kit cannot prevent; re-thrown it ended both coordinators, and the searches paid for before it never reached the usage row, written at the end of a run - the meter under-recorded real spend. Named <provider> threw: <message>, not retried, degraded and recorded like any failure. An adapter that returns a Promise - an async search() - is a failed search too, saying adapters are synchronous (review of that guard, the same day): it had reached the ok test as a failure with no error text, and its rejection, awaited by nobody, ended the process after all; the Promise is given a catch so it cannot. A thrown value whose own rendering throws (a Symbol, a poisoned message getter) is named a value that cannot be rendered as text (thrownText). |
ADR-0027, ADR-0133, ADR-0135 |
research-run.mjs |
Many URLs, one run: plan/query/url fan-out, discovery, candidate selection, budget tiers (DEPTH_SCRAPES), cache policy. The coordinator; collectOne does the writing, and what a run does when its credits run out is credits.mjs's, asked through one policy object whose state is read back here. A query whose why names unknowns that are all CLOSED in the contract is not searched (2026-10-02, ADR-0127): a search has no cache and was paid on every run of a finished project; the skip is said on the terminal and in discovered[].note, the dry run previews it, and --force runs it. The plan's own URLs are compared with each other by urlKey (2026-10-02, break-test pass 3): a page written down in two spellings - www., a trailing slash, a fragment - is queued once, and each other spelling is a skipped result naming the spelling that was fetched; until then only search results were compared with the plan, and such a plan paid for the page once per spelling. Since ADR-0027 it drives two providers: searchAdapter defaults to the fetch adapter, so an un-updated caller is unaffected, and one bounded fallback covers a search-provider outage — reported, never absorbed, because it moves spend back onto the fetch budget. searchUsage(root) counts the search meter from the usage log per hour and per month; it reports and does not block, because the count is this machine's only and cannot tell a free cached repeat from a billed search, so a guard built on it would refuse work on a number wrong in the blocking direction. The funnel is limit then perQuery, and it only works when they differ: limit is how many results the search is asked for, perQuery how many survive selectCandidates' ranked slice. The collector workflow set both to max_pages, so the slice kept everything the search returned and rankCandidate - the preferred-domain bonus, the docs/terms bonuses, the blog and forum penalty - could not affect a dispatched collection at all. Measured on job-0922h: 8 results, 8 kept, 7 off topic. Ranking had been inert since perQuery was raised to max_pages to fix a breadth bug, a change that widened the intake and destroyed the selection in one line. Widening limit costs nothing, because a search is billed per search and not per result. Failures are counted per provider (searchFailuresOn) as well as in total, because the total mixes meters: searchSummaryLine prints searches 0 on serpapi with "N attempts failed" beside it when the reported provider was asked and failed - on 2026-09-22 the bare zero read as "not used" after a SerpAPI timeout the merge had survived. A clean run prints the line it always did. prefer can name a path on a shared host (2026-09-26, ADR-0044): parsePreference reads an entry as host plus optional path; a bare host covers the host and its subdomains as before, and a host with a path covers that path and below, on exactly that host, at a segment boundary. prefer: github.com had given every GitHub page the owner's bonus, and collect run 36279879229 captured a stranger's run page with it. The input descriptions in collect.yml, collect-remote --help and the MCP tool show the path form, pinned by test. A prefer value may be a list or a string (preferList, 2026-09-27), at the top level and per query: a string is split on commas and whitespace, as collect.yml splits its input. It had been dropped at the top level and spread into single characters per query - silently, both times. One page, one fetch (2026-09-27, urlKey): "already have this page" compares identities - host case, www., scheme, a trailing slash and the fragment dropped; path case and query kept - in the seen set, in selectCandidates (which also keeps one spelling per list) and in decompose's candidate list. Plan URLs enter the seen set before any search runs, and the single-provider path now records its picks as the merged path did. Collect run 36283114657 had paid twice for .../ui-and-api and .../ui-and-api/. A page the budget leaves out is said, not only recorded (2026-09-27): each is logged skipped ... budget exhausted, the result carries overBudget, and research.mjs ends its summary with left N over the budget; --status states the budget the run will use, min(maxScrapes, the depth's cap), where it quoted the depth's cap alone. A plan naming five pages at maxScrapes 2 had printed two lines and nothing about the other three. The result also carries maxScrapes, and the summary's next line says which cap bound the run and what lifts it (2026-09-28): the depth's (naming the --depth that allows more), the plan's maxScrapes, or both at once. A scaffolded plan says depth quick with maxScrapes 10, and a MoonAliza run stopped at 4 without saying the 10 needs a deeper tier; the template's notes now state the tiers too. A dry run names each query it would search and the provider(s) whose meter it would spend - it had previewed nothing for a plan made of queries. bin/research.mjs refuses a plan with no queries and no urls before choosing a transport - exit 2, naming the plan and the shape of a query and a url (2026-09-27): the scaffold's plan is empty, and new-project's step 3 ran on it, printed collected 0 and exited 0. Before that it refuses a plan file that is missing or does not parse, naming which (the parse error used to read as "no queries and no urls"), and a --only that matches no query. Then planProblems(plan) (research-run) names every value it cannot use - a depth outside DEPTHS, a non-integer count, a query without q, a URL that is not http(s) - and research refuses them all at once (2026-09-27): readPlan defaulted each in silence, so a query written {"query": ...} ran nothing and exited 0. Search credits an adapter estimates are summed as searchCreditsEstimate into the usage row and the run's summary line, and searchUsage reports creditsEstimate for --status, labelled as an estimate (2026-09-27, ADR-0065). Vendor-neutral: it sums what adapters report. Every search logs its outcome (found N / search found nothing), and an empty one is recorded as op: search-empty in .failures.jsonl (2026-09-27, ADR-0066). One noteOutcome reports empty and off-topic searches for both the single-provider and the merged path; the merged path had reported neither. A result must carry the query before it costs a scrape (2026-09-27, ADR-0067): selectCandidates takes the query and keeps a result only if matchesQuery finds min(2, half) of its distinctive terms (queryTerms; ADR-0068 replaced ADR-0067's "half", which kept 2 of 38 real search results) in the title, snippet or URL, or isPreferred puts it on a prefer domain. A search whose results all miss scrapes nothing, logs no result matched, and records op: search-off-topic. A possessive counts as its word (2026-09-29): Ollama's had become the term ollama s, which no page carries, so a product's own name stopped counting toward the floor. Searches are counted per paying provider (searchesOn, 2026-09-28): a merged query was logged as two searches under the SerpAPI name and charged to its free-plan meter. The usage row carries searchesOn, searchUsage counts only the named provider's share of a row that has it (older rows as before), and the summary line names each meter. A run survives exhausted credits (2026-09-28, ADR-0086): given a fallbackAdapter, the first search or fetch whose failure the adapter's creditsExhausted recognises switches every later call that would have gone to it - merged searches and the degraded path included - to the fallback, once, and re-fetches the refused page; the refusal stays a fail ledger entry but not a failed page, and the switch is logged, recorded as op: credits-exhausted, and returned as fellBack. A page that is gone stays gone (2026-09-29, ADR-0100): a URL whose last ledger entry is a 404 or 410 inside the refresh window is set aside before the budget is counted and logged gone; --force retries. bin/research.mjs passes http-keyless when the chosen adapter can say what exhaustion looks like, announces it before the run, reports fell back in the summary, and takes --no-fallback. A dry run counts the searches it would make (wouldSearch), and bin/research.mjs prints the most the plan could spend - page fetches up to the budget, the searches, and on Firecrawl the credit rate - as one at most line (2026-09-29): it listed each search and page but never the total. bin/research.mjs and bin/decompose.mjs refuse, exit 2, when the selection reports the search provider notReady and the run would spend (2026-09-30): research passed the provider module rather than the selection, so the searchAdapter?.notReady check here never fired, and decompose had no check. An explicit provider with no key or instance failed each search, and decompose degraded to the fetch provider's search, which on Firecrawl spends credits. The opt-in witness (2026-09-30, ADR-0106): with witness, witnessCapture asks witness.mjs (or the injected witnessLookup) once per newly collected page - never for a cached page, a failed fetch or a dry run - and appends { at, url, raw, ...record } to research/witnesses.jsonl (PATHS.witnesses), outside raw/ and the ledger; a throwing lookup is recorded as witnessed: false and the capture stands. bin/research.mjs passes --witness. rankCandidate matches the first two words of a reason as plain substrings (mentionsWhy, 2026-09-30); it had compiled them into a regular expression, so a(b threw a SyntaxError and a blank reason built ` |
, which matched every URL and gave every candidate the bonus. bin/research.mjs refuses, exit 2, a plan with a duplicate object key (duplicateKey, 2026-09-30) before it plans or spends: JSON.parsekeeps the last value, so the url the operator pasted was not the one collected, and the gate that names the duplicate runs only after collection. canSearch(intransport.mjs, beside isSearchOnly): a fetch-only transport is never asked to search - searchPatientlynames it a failed search, a failed search degrades only to a provider that can search, and an out-of-credits switch hands searches only to a fallback that can (2026-10-01). isWebUrl: selectCandidatesqueues only an http(s) search result with a host - a vendor'sjavascript:orfile:row never reaches a fetch adapter (2026-10-01).canSearchis false for a Firecrawl CLI with no key, so no fallback asks it to search (2026-10-01). **Refuses a ledger it cannot record into before the first fetch** (ADR-0122):assertAppendable(root)runs beside the vendor-CLI and search-provider refusals, dry runs included, so a broken chain costs nothing - the F12 refusal at the record had let the first fetch be paid for. Without afallbackAdapter(the default on a local run, ADR-0129, 2026-10-02) exhaustion STOPS the run: the first refusal is recognised either way now, every later search is not run and every later page is reporteduncollected(never failed, never spent),stoppedcarries{ reason, provider, during, uncollected, skippedSearches }, and the failure log's credits-exhaustedrow saysaction: "stopped". bin/research.mjsthen prints the two ways forward and exits 2;--fallbackopts into the ADR-0086 switch, whichcollect.ymlandlive-collection.ymlpass because nobody is there to decide.canSearchmoved totransport.mjsbesideisSearchOnly(2026-10-03, architecture pass): a transport's own capability, asked of any provider, had lived in the coordinator;decompose.mjsand one test import it from its new home. **The asking of the search providers moved tosearch-session.mjs** (2026-10-03, ADR-0135): runResearchbuilds one session beside its credits policy, refuses throughsession.assertReady(), asks session.ask(text)per query, maps the session's failure rows to the failure log'sop: "search"rows, and readssession.meters()into the usage row and the result; what it does with an answer -noteOutcome, selectCandidates, the target row (from: 'search'under a merge,from: 'query: ... (via ...)'alone) - stays here. Two counts changed with the move, deliberately: a failed first ask's reported usage is counted whether or not the degrade answers, and a plain failure counts insearchFailuresas it already did insearchFailuresOn. Search-found pages are filtered through the existing candidate-specific cacheDecisionbefore ranking and theperQueryslice: fresh hits remain free, stale or forced captures remain eligible, andseen` holds only targets queued in this run so plan/search aliases and later queries cannot queue the same page twice (2026-10-08). |
firecrawl.mjs |
All vendor knowledge: the CLI transport (exec seam, which carries an argv array and spawns with shell: false — no URL or query is ever concatenated into a shell string), the Windows .cmd-shim resolution (resolveProgramPath/resolveInvocation: full path off PATH+PATHEXT, a real .exe spawned directly, a .cmd/.bat run through cmd.exe as argv and only after every argument passes cmdSafeArg — a denylist of cmd.exe metacharacters (CMD_UNSAFE_CHARS), refused and never re-quoted — a denylist, so an ordinary query with a space passes while an interpreter character does not. unsafeCmdArgs(argv) is the plural reporting form and the single owner of "which arguments would be refused": resolveInvocation calls it rather than filtering inline, so the refusal and the message naming the offenders cannot drift apart. Measured against the allowlist the ADR-0029 source tree still uses, the two agree on every character that can break out of cmd quoting — " % & ` |
< > ^ ( )`` `` tab newline — and differ on ' `{` `}` `` and any non-ASCII character, which an allowlist of ASCII punctuation refuses by construction. That is why `é` is a permitted argument here and a hostile one there, and why that tree's hostile-argv corpus could not be ported), payload normalisation (`normalizeSearch`/`normalizeScrape`/`normalizeMap` — `normalizeSearch` reads the CLI's real `{success, data: {web: […]}, creditsUsed}` object, pinned at `test/fixtures/firecrawl-search-1.23.3.json`), status parsing (`--status`, the flag; `firecrawl status` is not a command), `map`/`search`/`scrape` adapters. `command(args)` survives as a display-only rendering (dry-run output, the ledger's `cmd` annotation) that nothing executes. Stamps `transport: 'firecrawl-cli'` — narrowed to `firecrawl-cli-anonymous` by `transport.mjs` when the CLI has no credential — and grades completeness at the 1500-character bar rather than asserting `full`. Tested CLI: 1.25.3 (from 1.25.2, 2026-10-03: the release is one fix - `--objective` optional for alexandria feedback, firecrawl/cli#301 - with nothing on scrape, search or `--status`, and its output, captured for three credits, reads unchanged, `test/fixtures/-1.25.3.`; from 1.24.6 on 2026-10-02, `docs/decisions/2026-10-02-firecrawl-cli-1-25`: its `--status`, search and scrape output captured for three credits and read unchanged by the parsers, `test/fixtures/-1.25.2.` - the scrape receipt goes to stderr, `data.tools` is populated and reaches no result; before that from 1.23.3, 2026-09-26): 1.23.3's bundled axios 1.15.2 fails behind an HTTPS proxy that requires CONNECT tunnels (405 on every call); 1.24.6 bundles axios 1.18.0, and its status, search and scrape output were read by these parsers before the move - search's new fields are additive `search` reports `searchesUsed: 1` and a `creditsEstimate` of 2 per 10 results, rounded up, at least 2 (2026-09-27, ADR-0065, E-03/E-14): Firecrawl searches had cost credits the kit never recorded. The CLI runs with `maxBuffer: CHILD_OUTPUT_LIMIT` and names an overflow (2026-09-28, ADR-0082): under the 1 MiB default a large answer was lost to ENOBUFS. `creditsExhausted(text)` (2026-09-28, ADR-0086) recognises the API's 402 text, "Insufficient credits ...", which is all the CLI passes on - the status never reaches the kit. `normalizeSearch` and `normalizeMap` drop a row that is not an object (2026-09-29, Arena break test): one null row in the CLI's stdout threw on `row.url` and took every valid result with it - the shape `serpapi.normalizeSearch` was hardened against on 2026-09-28. A field of the wrong type - url, title, description, body, status - reads as empty in normalizeScrape and normalizeSearch, as in serpapi and searxng; one hostile shape is fed to all three parsers (2026-09-30). `rateLimitWaitMs` caps a vendor-stated wait at `MAX_RATE_LIMIT_WAIT_MS` (60 s): the number is the vendor's text, and it is slept with the corpus lock held (2026-10-01). The payload is the first complete JSON value in stdout (`parsePayload`: each bracket tried as a start, the value ending where its brackets balance outside strings, at most 64 starts), and `scrape` refuses an exit of 0 that printed nothing, no readable JSON, or a page with no text - each a named failure, never a capture (2026-10-02, break-test pass 3: a banner with a bracket, a receipt line after the JSON, a cut payload and an empty stdout had each become a "collected" capture of 0 characters). A scrape is a live fetch (2026-10-03, ADR-0139, gap audit rank 2): the argv carries `--max-age 0`, because the vendor serves a cached copy up to two days old by default and a capture stamped with today's date could be yesterday's page - reproduced live, and the kit's own 1.25.3 fixture is a `cacheState: hit`; `normalizeScrape` keeps `metadata.cacheState` and `cachedAt`, which `writeRaw` writes into the front matter and `appendFetch` into the ledger entry when the vendor sends them, so a cached answer, if one still arrives, says so. |
bin/mcp-server.mjs |
The stdio entrypoint an MCP client launches as a subprocess. No flags: a stdio server's configuration is its environment. Calls honourEnvProxy() before touching either stream, because where only the startup flag exists the server starts again on the same pipes (ADR-0047). stdout carries messages and nothing else - a stray console.log corrupts the stream and surfaces as a malformed response to whatever request was in flight - so every diagnostic goes to stderr, including the one line reporting whether a credential was found. It calls tolerateClosedStdout() before its first write (2026-09-30): a client that disconnects mid-response made the kernel's EPIPE an unhandled error event, a raw stack and exit 1 that read as a server defect. A real pipe test (` |
head -c 100) and a source-order pin cover it. **Startup refusal does not echo argv values** (2026-10-09): it reports the count and environment-variable remedy on stderr, before runtime/proxy/token work. The direct check includes positional arguments; --help` is handled first, and a bare invocation still starts normally. |
bin/measure.mjs |
The judge-free citation report for the current project (ADR-0089); --json for machines. A report, never a gate; exit 2 outside a research project. |
ADR-0089 |
bin/export-warc.mjs |
The corpus as one WARC 1.1 .warc.gz for archive tools (ADR-0090); --out (default research-corpus.warc.gz), written whole with writeBytes after readWarc re-reads it; a refused write is named by writeFailure, exit 2 (2026-09-29). A copy, never a gate; exit 2 outside a research project. |
ADR-0090 |
bin/disclosure.mjs |
Probes a run and reports what a stranger can read. Calls honourEnvProxy() before its first request (ADR-0047). Exit codes ARE the finding, so it can gate something later: 0 shape only, 1 content readable, 2 subject leaked, 3 could not measure - which includes any probe that never reached GitHub, unless an exposure was measured anyway (exitCode in the lib). An unknown flag matching token/auth/key/secret is refused with the reason - authenticating this probe would answer a different question. |
ADR-0035 |
bin/collect-remote.mjs |
The one command an agent or a desktop app needs: dispatch the collector on GitHub, learn which run it started, wait, download, unwrap, and validate with the same code a human would run. Calls honourEnvProxy() before its first request, so a proxy-only network can dispatch at all (ADR-0047). Prints the run id before it waits, so a caller that dies mid-wait can still find its run - and a caller that never sees that line knows the dispatch itself did not land. No --token flag, and an unknown flag matching token/auth/key/secret/pat is refused with the reason rather than the spelling. Exit codes separate the four things a caller does differently about: 0 valid (still not authorized to build), 1 invalid package, 2 run failed or incomplete, 3 could not start, 4 dispatched and still running with the run id on stdout. An --out it cannot write is refused (exit 3, OUT_DIR) before the token is read or a run dispatched (2026-09-27); --no-wait downloads nothing and skips the check. Each failed poll the watch survives is printed with its number (2026-10-01). --run-id <n> picks up a run already dispatched - after --no-wait, a timeout (exit 4) or a lost connection - waits, downloads and judges it, and refuses --topic beside it (2026-10-01). The remedies and exit 4 had pointed at that flag while it did not exist, so the only way back to a run was to pay for another. A failure that names no way out now says how to pick the run up. A bare --out, --workflow or --ref is refused as --<name> needs a value (exit 3) beside the five checked values (2026-10-11, #270): the parser reads a bare flag as true, so --out had become the folder true, created before the token was asked for, and --workflow or --ref would have dispatched against the literal name "true". The shared query/URL validator refuses CR/LF inside an individual item before output-folder checks, token lookup or dispatch (#376, 2026-10-12). Rejected URL diagnostics pass through redact before stderr (#378, 2026-10-12), including credentials supplied by the caller rather than this process. Caller references use checkClientRef before token lookup, output-folder creation or dispatch (#375, 2026-10-13); CR/LF are refused by the shared checker, and diagnostics are redacted. |
ADR-0033 |
bin/researcher-release.mjs |
The release CLI: validate, conform, fi-validate. A usage error returns 2 from every subcommand, and a status maps to its own code — PASS 0, FAIL/REOPEN 1, INCOMPLETE 2, BLOCKED 3. 4 is the unrecognised-status fallback and never a usage error, so a caller can distinguish "did not run" from "ran and produced something unknown". --help prints the usage to stdout and exits 0 (2026-09-27), as do path-authority, ledger-conformance, fi-sidecar-conformance and artifact. All five had exited 2. --report is written through writeText (2026-09-28, ADR-0079), whole or not at all. fi-validate --report names a report path the environment refuses through writeFailure and exits 2 (2026-09-30); it had crashed with a stack and exit 1, its code for FAIL. test/hardening.test.mjs now requires every bin/ command that imports a writer from lib/ to call writeFailure, or to name the handler that makes it unnecessary. |
— |
bin/panel.mjs |
Starts the desktop panel (lib/panel.mjs) and opens the browser at its address (explorer.exe on Windows, open, xdg-open; --no-open prints it only). Flags: --project <folder>, --port <n>, --no-open. Stops on Ctrl+C. A Windows desktop shortcut is a target the operator writes (README), not an installed launcher (ADR-0050). |
ADR-0147 |
release-validator.mjs |
Read-only R28–R32 release evidence validation: envelope/predecessor/role/visibility/promotion checks, ledger anchoring, and deterministic PASS/INCOMPLETE/FAIL/REOPEN reduction. It never rewrites records, pointers, registries, or role rosters. Also the facade over the four primitives below: all 21 of its original exports are re-exported unchanged, so the eighteen files importing from it were not touched. 1324 → ~1010 lines. Record ids and file lists sort by compareText (2026-10-01), the order the Python runner uses. validateRelease hands an explicit genesisPath and checkpointsDir to the ledger check, holds them to the ledger's below-the-root rule, and refuses either without a ledger (2026-10-11): until then both were dropped unread. |
R28–R32 validator contract |
release/json.mjs |
Duplicate-key-safe JSON parsing. JSON.parse keeps the last duplicate key silently, which for a hashed record is the difference between one payload and a different payload with the same digest; it also refuses a UTF-8 BOM, which JSON.parse accepts and hashing does not. MAX_JSON_DEPTH (256, 2026-09-28): deeper nesting is refused by name as the reader descends; it recursed, so the release validators reported a deep document as a stack overflow, and Node accepted depths Python could not canonicalise. |
ADR-0083 |
release/canonical.mjs |
Canonical JSON, SHA-256, same (canonical equality), and the flat-vs-nested record shape (metaOf/payloadOf/nestedRecord) plus the hashes built on it. The most-reused primitive in the kit, and the one place duplication has already cost this repository — three Python runners each carried their own copy of this algorithm, two drifted, and one document hashed to two different digests. canonicalJson refuses an unpaired UTF-16 surrogate in a key or a value (2026-09-28), as the canonical-serialization spec's rule 3 says and the Python runners always did: it escaped one as \ud800 and hashed it, so one vector PASSED in Node and was refused in Python. validateRelease reports it as ENV-PAYLOAD, where it had reported a hash mismatch. |
— |
release/schema.mjs |
The JSON Schema subset the checked-in schemas actually use. The release contract deliberately has no runtime dependency on Ajv or a package manager, so this implements that vocabulary and nothing more — a keyword left silently unimplemented accepts everything, which is the failure mode that matters in a validator. | — |
release/paths.mjs |
Containment (inside, safePath). Security-sensitive: a registry names its own record paths and can arrive from another machine with a corpus, so without this a manifest could point anywhere. Checked before the read and again against realpath. |
— |
release/order.mjs |
compareText(a, b): the one order for identifiers - slugs, file names, ids, ISO dates - by UTF-16 code unit. Total, the same on every machine, what canonicalJson hashes keys by and what the Python conformance runners sort by (key=utf-16-be). In the ported layer so the validators may use it (ADR-0029); core.mjs re-exports it. localeCompare with no locale read the machine's: under da_DK Firecrawl before firecrawl, under sv_SE zebra before ålder, under th_TH a-b equal to ab (break-test PR #185, 2026-10-01). |
— |
path-authority-validator.mjs |
Read-only Git-origin/path-authority snapshot validator: strict envelope schema conformance, self-excluding canonical envelope hashes, portable root/realpath containment, isolated-environment coverage, required Git-origin proof, artifact-diff classification, and deterministic status reduction. It never writes snapshots, transcripts, profiles, Git config, or release state. Files sort by compareText (2026-10-01). |
Git-origin/path-authority snapshot validator contract |
decompose.mjs |
Phase 0 coordination: recipe parsing, docs-host discovery, material gathering (scrape-budget accounting via the corpus's cacheDecision — a cache hit is not an attempt), draft MAP writing. Contains no judgment — it seeds the checklist and gathers material; it never writes a status. A failed search reaches the map and the terminal (2026-09-26): MAP.md lists each failure (query, provider, one scrubbed line of the vendor's error, and any fallback) and tells three empty maps apart - a dry run, a quiet topic, an outage - and searchSummary gives the CLI its count. Until then the failures were kept in the result (F20) and shown nowhere: after every search failed, the map told the operator to stop dry-running. The error is scrubbed because MAP.md is committed and vendor error text has echoed request URLs with keys in them. Its searches are metered (2026-09-27): phase 0 runs four searches and now writes a usage row like research-run's (command: 'decompose', searchesUsed summed from what each provider reports), so research.mjs --status counts them. Until then two runs made 8 SerpAPI searches that the vendor's meter showed and the local one - labelled "this box", though it is one project on this machine - did not. A dry run still seeds the map (its documented purpose) and now logs each search a real run would make and the provider whose meter it would spend, as research's dry run does (2026-09-27): it had previewed none of the four searches. The CLI's closing steps name research/plan.json. resolveTopic decides what is mapped (2026-09-27, ADR-0056): plan.json's topic when --topic is omitted, and a refusal naming both when --topic differs (spacing and case aside). A project still called UNTITLED_TOPIC accepts any --topic. The CLI had rewritten the map's topic to whatever it was given, and the brief took its title from the map. loadRecipe names the recipes that exist when one is not found (2026-09-27): --recipe api had answered with only the folder it looked in. Its usage row carries searchCreditsEstimate too (ADR-0065), counted once per search including a fallback search (a duplicated line had counted fallbacks twice until 2026-09-27). With --max-scrapes, a candidate is scraped only if it carries the query that found it (matchesQuery, ADR-0068); every candidate still appears in the map. Under a merged search selection it asks every provider each query and interleaves by rank (mergeByRank, as research does; 2026-09-28): the CLI passed only the first while printing that both were searched, so phase 0 scraped what SerpAPI alone had ranked. The usage row carries searchesOn; a provider that missed a query another answered is listed in the map as answered by the other provider and is not counted lost. A compound topic is searched part by part (2026-09-28, ADR-0085): topicQueries reads subject: part, part; part as one search per part (a one- or two-word part carries the subject, and so does a part that refers back with a pronoun, ADR-0098; at most MAX_TOPIC_PARTS, 6) instead of the whole topic four times, and interleaveByQuery shares the candidate list one per part. Searched whole, MoonAliza's three-question topic mapped forum threads and a coffee-scale blog, and in search order its third question got none of the map's 20 candidates. A plain topic keeps its four searches. Outlines (2026-09-29, ADR-0091): after gathering, outlineOf(body) reads the ##/### headings of every material page with a capture on disk (at most MAX_OUTLINES pages, 12 headings each; code blocks, links, zero-width marks, repeats and OUTLINE_FURNITURE dropped) into a ## Outlines seen in the material section - STORM's table-of-contents step without its model, unjudged, no fetch. Furniture also covers GitHub's file-view chrome and Discourse per-post bylines (OUTLINE_FURNITURE_PATTERNS, 2026-09-29). The scrape budget follows scrapeOrder: likely-owner hosts first, by docsHosts score, then the rest (ADR-0094). Owners are counted by ownerOf (ADR-0096): on a shared host - github.com, gitlab.com, huggingface.co, medium.com and the like - the owner is the account (the segment after orgs/ or users/; the host's own sections such as topics stay the host's), so five strangers' repositories no longer pool into one "likely owner" that outranks the vendor's docs. A page on the owners tying for the top score (at least MIN_NAMED_OWNER_SCORE, 3) that the relevance floor skips is listed in the map under "Skipped on the likely owner", with plan.json urls as the way in (ADR-0097): the owner's terse "OpenAI compatibility" page had been skipped with only a log line to show for it. The floor itself still holds for it - trusting the owner past it re-admitted RR-9's postgresql.org home page. A compound-topic part that opens with "the" carries the subject, as a pronoun part does (2026-09-30, ADR-0103): "the bot limiter", searched alone, filled a SearXNG map with audio mastering limiters. resolveTopic refuses a plan the corpus reader flagged for a duplicate object key (2026-09-30), in the reader's words: JSON.parse keeps the last topic, so every search was built from the second while the operator read the first. Its searches go through searchPatiently (one call, no retries), and a failed search degrades only to a fetch provider that can search - the browser cannot (2026-10-01). Only an http(s) search result with a host (isWebUrl) is taken into the material, so only such pages are listed or scraped (2026-10-01). A damaged ledger is refused before any provider is asked (2026-10-03, outside audit): assertAppendable(root) at the top of decompose, as runResearch has had since ADR-0122 - phase 0 spends too, and it made four searches and one fetch before collectOne threw. Its searches go through the search session (2026-10-03, ADR-0135), the same one runResearch asks: decompose builds it with no rate-limit patience and a sink that keeps each failure row for the map, refuses a notReady provider through session.assertReady() before the ledger check (until then only the CLI refused, so a library caller spent), asks session.ask(query) per query and keeps every http(s) result as material, and reads session.meters() into its usage row. With the move it gained the three lines runResearch already logged (searched, search found N, search failed on every provider), its usage row's searchTransport names the paying provider under a merge as research's does (it had joined every provider with +), its searchFailures counts every failed ask (a covered miss and a degraded first ask included) and its degraded counts only the degrades that answered - research's counts, under the ADR's one rule. A bare --limit or --max-scrapes is refused (2026-10-11, #269): both go through checkFlagValues (int, min: 0) at the top of the CLI with the other value flags, so a flag with no value fails as --<name> needs a value instead of being read as true - which the hand-written Number(raw) check it replaces turned into 1, so --dry-run --max-scrapes reported "scrape up to 1" and a real run paid for one page. Recipe loading refuses front matter with no dimensions and blank dimension labels, naming the file and physical line. The CLI presents INVALID_RECIPE as usage failure (exit 2) before map writes (#356). |
ADR-0085, ADR-0091, ADR-0094, ADR-0096, ADR-0097, ADR-0098, ADR-0135 |
dimensions.mjs |
The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (coverageOfUniversals). Recipes add on top of it; nothing replaces it. |
— |
provenance.mjs |
The tamper-evident fetch ledger: appendFetch, verifyLedger, rebuildLedger (the one metadata-migration path — guarded, per-entry migrations records, chained op: 'migration' boundary), and the overrides log writer. Also the collector's exclusive section (withLock), acquired with O_EXCL (flag: 'wx') so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only after EEXIST proves someone holds the file, and a fresh lock that names no pid — the window O_EXCL leaves between create and write — is waited for, never broken. Reentrant in-process, so appendFetch can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. verifyLedger also classifies a body hash that does not recompute (isLineEndingRewrite, hashText): folding CRLF back to LF either restores the recorded hash or it does not, so body-unmodified problems carry kind: 'line-endings' verifyLedgerreports a ledger-named capture it cannot read asraw-unreadableinstead of throwing out of preflight (ADR-0077). It refuses a capture that is not a REGULAR FILE before it opens it, for the reasonreadTextdoes (2026-09-28, Arena break test 12): a fifo named by the ledger hung every command that verifies a corpus, and a ledger symlinked todev/zeroended them withstd::bad_alloc` rather than a verdict. A lock write that fails (a full disk) closes its descriptor and removes the empty lock it just created before the error reaches the caller; it leaked both (2026-09-28). |
'modified'and the consumer can name the right cause instead of guessing (ADR-0020). A source entry withoutsourceSha256is abody-unmodified/unhashed` problem, never skipped (2026-10-04, break-test pass 4 F4): the no-hash skip was written for the first ledgers' captures, and a source never had that history, so a rechained entry naming an unverifiable sibling is refused. |
quotes.mjs |
Quote anchors (2026-09-28, ADR-0087): quoteAnchors(text) reads the [quote: ...] markers of a Finding cell, split into fragments on an ellipsis - a marker may hold balanced brackets one level deep, a code subscript or a Markdown link (2026-09-29: it ended at the first ], and the stub it kept was what got checked); normalizeForMatch is the comparison-only form (NFKC, Markdown link text, quotes, dashes, emphasis, escapes, whitespace, case); anchorFound(fragments, body) requires at least one nonempty normalized fragment and matches every nonempty fragment in order, exact after normalization, with no fuzzy fallback; MIN_QUOTE_WORDS is 3, and MIN_QUOTE_CHARS is 40 (2026-10-02, ADR-0126): a quote is short only under both, measured by quoteWords and quoteChars across the fragments - two rows of the kit's own decision project quoted a 130-character dependency map, one whitespace-separated "word", and were warned as anchoring almost nothing. Pure; checks.mjs reads the capture and reports. quoteAnchors takes one pair of quotation marks around the whole passage off before matching (unwrap): [quote: "the sentence"] was quote-not-found for a sentence the capture held word for word (2026-10-01). normalizeForMatch also drops Unicode's format characters (\p{Cf}: zero-width space, soft hyphen, word joiner, the joiners and directional marks) after NFKC, on both sides: ## [\u200B](...#bundled-skills) Bundled skills in this repository's own corpus made [quote: ## Bundled skills] unfound, and 32 of its 188 captures hold such characters (break-test 2026-10-01). An anchor that normalizes entirely to empty cannot count as found: raw formatting characters or three formatting-only words used to clear the short-quote floor while every search needle was skipped, falsely passing citations and inflating the measurement count (2026-10-08). Formatting-only fragments alongside real text remain ignored; nonempty fragments still require exact ordered matches. |
ADR-0087 |
measure.mjs |
How a corpus's citations hold up, without a judge (2026-09-28, ADR-0089): measureCorpus(root) - link works (capture on disk, in the ledger, unmodified, not an error page), quote-anchored rows and quotes found (quotes.mjs), full captures, closed unknowns on a P row and on two or more hosts, known unknowns, ledger verifies; factCheck is named and never computed. renderMeasure for people; bin/measure.mjs prints either, --json for machines, refuses outside a project, and never gates. |
ADR-0089 |
warc.mjs |
The corpus as WARC 1.1 (2026-09-28, ADR-0090): exportWarc(root) - a warcinfo record, then per ledger scrape entry whose capture is on disk and unchanged a resource record (the text after the front matter, text/markdown) and a metadata record (WARC-Concurrent-To; transport, completeness, the ledger's hash and seq); WARC-Block-Digest: sha256:, record IDs derived from the ledger entry (uuidFrom, RFC 9562 v8), each record its own gzip member; an edited capture is left out and named in skipped. readWarc splits a stream back into records and throws on bad framing. Validated with warcio on every corpus. exportWarc reads a ledger-named capture only if it is inside the project (isInside and realInside, ADR-0076) and a regular file (2026-09-30); anything else is skipped with its reason. A ledger naming ../ or a link out of research/raw had packed that outside file into the archive, and a FIFO would have hung the export. exportWarc reads the path through projectFile (2026-09-30, ADR-0109). The record is parsed from the bytes whose hash passed, never a second read of the file (2026-09-30). |
ADR-0090 |
prior.mjs |
The registered prior: what was expected BEFORE the evidence, and the proof that it was. registerPrior writes research/PRIOR.md and chains an op: 'prior' entry carrying its sha256, so two things stop being editable at once - the text, through body-unmodified, and its POSITION before the first scrape, through the chain itself. Three dishonest paths, each measured rather than argued: revising the prediction once the answer is known fails the chain; registering late is refused by the tool; and chaining one late BY HAND with every hash correct is caught by provenance/prior-precedes-collection, which exists because appendFetch is a public export and the gate does not get to trust that the CLI was used. Nothing grades the prediction (ADR-0013) - not its specificity, not its reasonableness, and above all not whether it turned out right, since a prior that was wrong has done its job and a gate that preferred correct ones would teach the opposite habit. The single content rule is a length floor, stated as a floor. Absence is SILENT at the gate: a prior is optional, and a warning on every project that skipped one is a warning nobody reads. bin/prior.mjs registers and reads it back; bin/research.mjs prints a one-line nudge before a spending run, because after the first page lands there is no second chance. It also reaches the REMOTE path, and that half is what makes the rest useful: the collector workflow scaffolds its own project on the runner with an empty ledger, so a prior registered on the operator machine cannot reach the corpus that comes back - and every corpus collected since collect-remote shipped came from that route. The prediction therefore travels as a dispatch input and is registered on the runner between the map and the collection, through env and an argv array like the topic (ADR-0020), with the topic's measured disclosure profile (ADR-0035). A test pins that the registration step precedes collect, because if it ever drifts below it, bin/prior.mjs refuses and the feature stops working silently on a workflow CI never runs. |
ADR-0013, ADR-0020, ADR-0039 |
checks.mjs |
The ordered contract-check registry: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, evidence-supersession, collection-attempts, hygiene, corpus-shape, corroboration — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. partial-render applies only to a capture behind an EVIDENCE row some unknown cites (ADR-0099): on a context-only row its instruction had no cited text to confirm. A check that instructs the reviewer owns the accuracy of the instruction, not just of the verdict: capture-completeness/partial-render names the EVIDENCE row and the [render-reviewed: ...] syntax because it used to name a renderReview: front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking provenance/body-unmodified failure instead of a closed warning. The wording is pinned by test for that reason. The same duty covers what a PASS claims: gate-integrity used to pass with "no override in effect" when what it measured was that no override was RECORDED - and git commit --no-verify, the escape hatch this kit documents most prominently, records nothing. It now says what it counted and names the bypass it cannot see. Found by sweeping all thirteen pass messages for claims wider than their measurement; it was the only one. supersededRows(corpus) is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": evidence-supersession refuses an unknown still resting on a replaced row, transport-provenance goes quiet about a replaced row nothing relies on (but not about one still cited), and hygiene stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. collection-attempts counts an attempt through the rows a KNOWN-UNKNOWN cites, not only a URL typed into it (2026-09-26): an E-row is a capture of its URL, so citing one is the stronger proof of having reached for it. Until then a KNOWN-UNKNOWN citing the page it tried was warned as never attempted - found by the 2026-09-26-actions-sept-changes project's own gate, and no test exercised its verdict (only the registry-order test named it). citations/url-mismatch (2026-09-27) fails a row whose URL is not, by urlKey, the page its capture was fetched from: a row could name another page and pass, and the URL cell is what a reader is shown as the source. provenance/ledger-unparsed names its repair (2026-09-27): a torn last line gets doctor --fix-arity, which drops exactly that; a broken line inside the chain is restored from git. It gave only the JSON error. A row is never compared with itself (2026-09-27): supersededRows skips a same-ID pair and hygiene/duplicate-url leaves the same ID to duplicate-id; a row pasted twice failed with "E-01 has been superseded by E-01". hygiene/future-date fails a Retrieved date more than a day ahead (2026-09-27): 2030-01-01 had passed, hiding the row's age from every freshness check. hygiene/duplicate-id fails rather than warns (2026-09-27, ADR-0054): two rows under one ID leave every citation, COVERED cell and brief reference to it ambiguous, because lookups keep the last row. A contract with two U-1 rows had passed. corroboration weighs each unknown ID once, so it no longer prints the same single-source warning twice. hygiene/brief-stale warns when a stamped brief's inputs hash no longer matches the corpus (2026-09-27, ADR-0055) and names the command that redrafts it; hygiene/brief-unstamped warns on a drafted brief with no stamp (2026-10-03, ADR-0138) - its currency cannot be checked and it keeps its approval, so the gate says so and names the redraft that stamps it; a template or a hand-written legacy brief gets no such warning. hygiene/duplicate-url treats two same-day rows for one URL as a refresh when the ledger records a fetch for each (2026-09-27, ADR-0060); a pasted row, with one fetch behind two rows, is still a duplicate. discovery-contract/build-intent tells a missing heading from an empty one, from the corpus's intentHeading (2026-09-27): a renamed heading read as "is empty". citations checks quote anchors (2026-09-28, ADR-0087): each [quote: ...] in a Finding must occur in the row's capture (quote-not-found, blocking) and carry at least three words or 40 characters (quote-too-short, warn; the character floor is ADR-0126, 2026-10-02); the matching lives in quotes.mjs. citations/raw-error-status warns when a row cites an HTTP 4xx/5xx capture from before that refusal, unless the row names the status itself (this repository's E-15 keeps a 404 as the record of a failed guess). unknown-closure/secondary-only (2026-09-28) warns when a CLOSED unknown cites no P row and at least one S row: the registry promised "a primary row" and only L was checked, so three corpora closed everything on S rows with no warning (found by the first measurement). Retrieval dates compare by compareText (2026-10-01). runChecks appends each check's findings one at a time (2026-10-01, break-test PR #188): push(...check.run()) passed every finding as an argument, and past the engine's limit - 60,000 to 130,000 findings from one check, by Node line - the verdict died with a raw RangeError; preflight, doctor, brief and audit crashed, and the commit gate answered internal error. Reachable from about 66,000 evidence rows. The other spread sites (priorOrder, doctor's gateHealth) are bounded by the contract's unknown count and left as they are. hygiene fails a malformed ID (2026-10-03, audit G2): each malformed-id problem the corpus reader records becomes a fail beside duplicate-id, for the same reason - the ID is how every check finds a row; corpus-shape skips the kind so one typo is not reported twice. A table-split problem fails the same way (2026-10-03, gap audit rank 1): the row no check reads is named with its file and line, and corpus-shape does not repeat it. Freshness is judged by the capture's own date (2026-10-03, audit G8): unknown-closure ages a cited row by freshnessDate - the capture's front-matter retrieved, written by the collector at the fetch and under the ledger's hash, with the table's cell as the fallback for a row whose capture carries no parseable date - and names the judging date when the two differ; hygiene warns date-mismatch when a row's cell and its capture name different calendar days (retrievedDay), with the remedy of writing the capture's date in the row. Editing E-19's cell alone had made its stale warning disappear while the capture on disk still said 2026-09-20. The capture judged is the one the row NAMES (ownCapture, review of G8 the same day): captureOf answers a blank Raw cell with the URL's latest capture, and judged by that a row with no Raw cell borrowed a later fetch's date, lost its stale warning and drew a date-mismatch telling the operator to write the later date in; a row naming no capture is judged by its cell and has nothing to reconcile against. A future date in the capture fails future-date as one in the cell does, naming the capture, and the stale hint's "fresher capture of the same URL" is found by capture date (freshnessDay), not by the cells. Same-day supersession follows the ledger (2026-10-03, audit G4): supersededRows orders two rows of one URL by the day freshness judges them by (freshnessDay - the capture's date; ordered by the table cell, one edit to an older row's date had failed the gate on the genuinely newest row and told the operator to cite the older one; review of G4, the same day) and then by the ledger position of each row's cited capture (ledgerRank, the rule newer applies for the collector); a capture no fetch recorded ranks below every fetched one, and only among those does the table's order still break the tie. With A fetched, then B, then A again, the collector had held A current while the check said B superseded it, and swapping two evidence rows swapped the verdict. uncited-capture says "was collected" only of a file a ledger scrape entry names; a file under research/raw no fetch produced is named as such (2026-10-04, break-test pass 4 F5a). discovery-contract/contract-missing states that the contract is missing or could not be read (2026-10-08): the reader uses the same null fallback for absence and refused reads, so that finding cannot establish absence. Both states keep the existing failing rule; no filesystem-state API or new check is introduced. Evidence URL comparisons use the adopted page identity (2026-10-09, audit rank 14): supersession groups, stale-citation fresher-row hints and duplicate-URL hygiene compare through urlKey (ADR-0136), as the collector and cache already do. Capture dates, blank-Raw table-date fallback, same-day ledger ordering, the same-ID exclusion and the two-recorded-fetch refresh rule retain their existing roles. Captured URL spellings remain visible; query strings and path case remain distinct. |
ADR-0004, ADR-0026, ADR-0087, ADR-0136 |
similarity.mjs |
Near-duplicate detection between two captured pages, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash sketch; similarity estimates Jaccard, isMirror compares it to MIRROR_THRESHOLD, and documentGroups single-link-clusters rows into distinct DOCUMENTS. Exists because corroboration counted HOSTS and therefore inverted on a mirror: a republished copy on an unrelated host scored independent - the registry's best grade - while carrying one witness plus a staleness freshness cannot see, since that check grades when a page was FETCHED. MIRROR_THRESHOLD = 0.25 is chosen from measurement, not taste: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (nodejs.org x lira.epac.to, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. sketch returns null below MIN_SHINGLES, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). |
ADR-0036 |
preflight.mjs |
The verdict and the verdict context. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: verdictContext(root) reads the corpus, the gate state (readGateState), and the operator's evidencePolicy together — same snapshot, same verdict, by construction. |
ADR-0004 |
gate.mjs |
Gate policy: isGated (the four markers), evaluate (commit/edit), the GATE_OFF override recording, what phase 1 may commit (ADR-0048: research/, the root SCAFFOLDING - .gitattributes, .gitignore, AGENTS.md, START_HERE.md - and docs/ARCHITECTURE.md while isEmptyArchitectureMap says its staged bytes, read by readStagedText, are still the template's; until 2026-09-27 only research/, so a new project's first git add -A was refused and its .gitattributes could not travel), and the architecture-map rule (architectureMapBreach over loadGateConfig — guarded paths are project configuration, research/kit.json, defaults src/lib/bin/scripts/app). Contains no repo-specific path. Staged paths remain data, with checked acquisition (ADR-0154): the default commit CLI lazily calls readStagedPaths after the ungated and GATE_OFF returns. One actual git diff --cached --name-only --no-renames -z supplies paths only on successful complete output; the existing 64 MiB capture rejects nonzero exit and buffer failure, and the reader rejects malformed NUL data. Successful zero bytes means []; unreadable scope means null and raises the existing architecture-map breach even after green preflight, ordinary exit 1 under either posture. Explicit arrays/null and the existing stdin/repeated --staged options retain their meaning; edits never acquire a list. One deliberate exception to "no CLI parsing" remains for explicitly supplied stdin data (stagedPathsFromStdin / splitPathList), because argv was O(n²) to build and bounded besides. It returns null for "stdin could not be read" and [] for "read, and empty", preserving unknown scope rather than assuming an empty set. No list read claims an atomic snapshot with later index materialization. stdinIsReadable() asks isatty(0) rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; /dev/null is a character device but reads as immediate EOF). The architecture-map block's fix says to update the map, then git add it (2026-09-27): a bare git add docs/ARCHITECTURE.md stages nothing while the map is unchanged, so the printed fix left the commit blocked. bin/gate.mjs no longer closes that block with "Phase 1 is not done until preflight prints PASS" - a map block happens only when the gate passes. isPhaseOneEdit(root, file) answers the same question for the edit hook, about one file. The empty-index verdict — a gated project whose research/ is untracked, every fresh project's first commit — removes the scratch dir materializeIndex made before it learned the index was empty (2026-09-28): it returned before scratch was assigned, so the finally that removes it never ran, and every such verdict leaked one temp directory. Integrity before scope (ADR-0093): a failing verdict whose findings include an INTEGRITY_RULES provenance rule blocks the commit even when it is confined to research/ - the ADR-0048 allowance covers completeness findings only. stdinIsReadable(stdin = process.stdin) takes the stream to ask about, so the hook's terminal case is testable (2026-10-01). The suite rule (ADR-0120, 2026-10-01): suiteBreach(root, stagedPaths, { run }) runs after the map rule, in the commit gate only, and only where isKitCheckout finds the kit's four markers and a staged path is under research-kit/ (or the list is unknown); runSuiteHere spawns the checkout's bin/selftest.mjs with RESEARCH_KIT_RESULT_FILE pointed at a scratch file and RESEARCH_KIT_ALLOW_UNSUP=1 (unsupported is not red, ADR-0108), git's hook environment stripped (GIT_INDEX_FILE, GIT_DIR, GIT_PREFIX and kin - the same REPOSITORY_LOCATION list as the edit hook, else every scratch repository's git in the suite would act on the index being committed), and reads the file, never stdout: failures block naming the counts, no result blocks as a crashed runner, a run past its budget is stopped and blocks as unfinished, the fix naming RESEARCH_KIT_GATE_TIMEOUT before the override, and a green run whose runner exited non-zero (the README count check) blocks too. suiteOwed(root, stagedPaths) is the predicate alone; suiteBreach's announce (evaluate's announceSuite) is called right before the run, so bin/gate.mjs says the suite is running only when it is - not when the map rule or preflight refused the commit first. evaluate takes runSuite so the verdict is testable without a suite. The suite's budget follows the gate's watchdog (ADR-0142, 2026-10-05): suiteBudgetMs(env) is SUITE_TIMEOUT_MS (65 min, twice the slowest honest run measured, 1825 s on the operator's Windows PC) unless RESEARCH_KIT_GATE_TIMEOUT is set to a positive duration gateTimeoutSeconds reads (seconds, an optional fraction, an optional s/m/h/d, at most nine digits before the point - the grammar the hook's gate_timeout_readable checks too; 0 and anything outside it leave the default), and then that watchdog less GATE_REST_S (120 s, the hook's default watchdog for a gate that runs no suite), never more than half a short one, at least 1 ms and at most 2^31 - 1 - so the gate stops the suite and blocks before the hook's watchdog fires, whenever its own work fits what is left. runSuiteHere stops the runner with SIGKILL: the harness's SIGTERM listener kept a synchronously blocked runner alive past its budget on POSIX; a stopped run's scratch is left in the temp folder. The budget had been 20 minutes, which that PC's green suite met on every kit commit, and the variable had reached only the watchdog: set below the budget, the watchdog killed the gate first and a fail-open machine allowed a commit whose suite never reported. The fix lines no longer say --no-verify is recorded (2026-10-03, outside audit): git skips this hook, so nothing can record it; checks.mjs had said so since ADR-0035 while the gate's own fix text, the README, AGENTS.md and CONTEXT.md said the opposite. Red suite diagnostics (2026-10-08): suiteBreach forwards up to five failed identifiers, each limited to 160 characters with controls normalized, in its existing detail string; omitted or clipped data is named, and legacy missing labels are explicit. Counts and verdict keys are unchanged. runSuiteHere still removes its scratch result; no result retention system is added. |
ADR-0002, ADR-0007, ADR-0008, ADR-0020, ADR-0093, ADR-0120, ADR-0142, ADR-0154 |
doctor.mjs |
Read-only diagnostics: machine + project + gate + chain health in one report. Consumes the verdict context (verdictContext / readGateState) for the chain-verified corpus and the gate-state triple; adds one machine read of its own (the git-resolved hooksPathEffective) for report. Role-aware (ADR-0010): the Firecrawl findings are informational on a builder, and the handoff findings (ADR-0011) are blockers there — the collection entrypoints' refusal is what makes that split honest. The handoff findings come from the owner, not from a second composition: doctor passes the corpus it already read to verifyHandoff(root, { corpus }), so the arrival question has one implementation and one set of pins. One deliberate write: the repository-local hooksPath override log (the third override is silent by nature — the bypassed hook cannot report it). gateHealth and runDoctor accept explicit Git/config/settings paths and a child environment so diagnostics can run against disposable machine fixtures without consulting host state. commitGateState asks WHOSE pre-commit is at core.hooksPath - current / foreign / unusable - because executable is not the same as ours. It used to ask only whether some executable hook was there, the same existence-for-correspondence mistake the deploy check made; settingsState has asked the question of the EDIT gate since a registration here named our hook inside a different checkout, and core.hooksPath is a single machine-wide setting that husky, lefthook and pre-commit all rewrite. Pure and separate from gateHealth, which reads real git config - a rule testable only by fishing a finding out of an array on a host-dependent path is a rule nobody tests (ADR-0004), which is why the older test for it asserts the severity is one of three. Ownership is decided before executability, because the first version asked about the mode first and so reported a non-executable FOREIGN hook as unusable with chmod +x beside it - advice to repair somebody else's gate while leaving this one just as absent. That ordering bug was invisible on Windows, where hookExecutability returns ok for any file that exists, and surfaced on Linux in CI. On a collector, handoff findings are info in the collector's words ("does not exist yet - nothing has been collected here; a builder handed this corpus would be refused"), since 2026-09-26: they were graded pass with the builder's wording, so a fresh project printed "pass ... the corpus arrived without its ledger". On a builder they remain blockers, worded as before. It reports the Node line and, when a proxy variable is set, whether this Node sends the kit's requests through it A collector without the Firecrawl CLI is told the exact install command, npm install -g firecrawl-cli@<tested> from cliInstallSpec() (2026-09-27; it said "install the Firecrawl CLI" and nothing anywhere said how). (proxy: keyless pages, search requests and remote collection; a warning naming the version to move to - or, for a value Node cannot parse, the spelling it needs - since the Firecrawl CLI has its own proxy handling) - measured on 22.20.0, where the fetch went around the proxy (ADR-0046, ADR-0047). A collector that explicitly chose a keyless or browser transport passes the Firecrawl checks; nothing chosen still fails (ADR-0095). The Firecrawl fixes (firecrawl-cli, firecrawl-auth) name the no-account route beside the install or login - "transport": "http-keyless" in the machine config, the setting that makes them pass (ADR-0095) - where the CLI fix had offered only a per-run flag and the login fix nothing (2026-09-29). editGateState reads an entry whose hooks is not a list, or a hook that is null, as naming nothing (2026-09-30); [null] had thrown a TypeError. scanForSecrets catches a folder it cannot read where it reads it, and skips a folder it has already walked by real path (2026-09-30): one try/catch around the whole walk had ended the scan at the first unreadable folder, leaving every file after it unscanned and reporting nothing found. scanForSecrets records a cap stop separately from the scanned count (2026-10-08, bounded-scan audit repair): it keeps the 2000-text-file limit and reports incomplete coverage only when that cap actually prevents further traversal; ending at the final file's boundary is not itself incomplete. runDoctor reports a capped scan as a secret-scan warning, beside any unchanged critical secret hits, instead of reporting a capped no-hit scan as passing. It names no unobserved remaining-file count and makes no whole-repository clearance claim. bin/doctor.mjs reports KIT_VERSION, as kitVersion in --json and on its last line (ADR-0116). Reports each ill-typed posture key as a config-ill-typed warning naming the key, the values the kit reads, and the state it resolved to (ADR-0121). The firecrawl-cli finding judges the installed version with the run's own rule (cliCompatibility): an unsupported major is a fail with the remedy on a collector that needs the CLI and information elsewhere, a newer minor passes and says what it was tested against - doctor had printed pass 2.0.0 for a CLI the run then refused (2026-10-02, break-test pass 3). scanForSecrets skips a binary file and counts it (skippedBinary, named in the coverage line) instead of matching credential patterns against an executable's string table (ADR-0128, 2026-10-02): run on a Windows home folder it reported fourteen committed keys, every one the PEM marker OpenSSH and libssh2 parse, or random bytes spelling sk-. handOnState (2026-10-03, ADR-0134; since 2026-10-04 defined in installer.mjs and re-exported here, ADR-0141, so the install judges what it records by the same rule): what the hooks hand on to - none, missing, kit, other - and gateHealth warns on kit and missing; the maintainer's machine handed every commit on to the previous implementation's gate, which crashed and failed open while doctor said READY. handOnState resolves a relative folder from the repository's top level (base), where git runs its hooks from; doctor had resolved .custom-hooks against its own working directory and from a subdirectory called a working hand-on missing (fourth audit round, 2026-10-03). scanForSecrets skips research/raw at ANY depth (2026-10-04): the skip matched the relative path research/raw, so only the root's captures were outside the scan and a nested decision project's were read as the project's own files - CI tripped google-api-key on a Google page's source sibling (ADR-0140), Google's own public keys; a capture is page content, and the folder exists wherever a nested project does (ADR-0030). The shared unusable-proxy check also rejects proxy URLs carrying a non-root path, query or fragment, so those configurations warn instead of promising proxy use (#380, 2026-10-12). machineHealth uses the shared envProxyChoice with injectable execArgv (#381, 2026-10-12). An explicit disabled choice warns that requests go direct and names how to enable the proxy, rather than promising proxy use from Node version support alone; the raw variable value is never echoed. |
ADR-0010, ADR-0011, ADR-0095, ADR-0134 |
panel.mjs |
The desktop panel (ADR-0147, 2026-10-07): createPanel({ project, env, kitRoot, kitHome }) serves research-kit/panel/ (one page, a script, a stylesheet) and a small JSON API on 127.0.0.1 only, a thin shell over commands that already exist. It owns no verdict: doctor, handoff, preflight and install run as children with the fixed argument lists in PANEL_COMMANDS, in the chosen project folder, and their output is shown with the search key redacted. No command from the page collects - research.mjs and decompose.mjs are absent from PANEL_COMMANDS; the one way pages are fetched through the panel is auto-collect (lib/auto-collect.mjs, ADR-0148), for a builder's validated request, on a collector, within the operator's caps. Its routes are GET /api/requests (the queue and the meter), POST /api/autocollect (the settings), /api/requests/check and /api/requests/resume. It writes two things: the search provider's key into the machine config through transport.mjs's SEARCH_KEY (so the vendor's name stays in the adapter and the registry, NFR-3), refused when the config resolves inside a git repository (Rule 6), or with HTTP 409 and a role-repair remedy when an unreadable config has no usable snapshot (#364); and which project it shows, a full path to an existing folder, never searched for. The topic and BRIEF.md are read, never written (ADR-0056). builderInstructions is the hand-off text: the collector's push, ledger included, then the builder's own steps in $HOME spelling (ADR-0050) - declare the role, handoff, preflight, inspect an authored complete brief and its current inputs, never collect. Its readiness checklist requires classified map rows and rewritten Findings; a file-presence observation grants no build authorization. Commit-report placeholders require actual observed results. A collected Git-request result waits for the collector to record review/closure before dependent building. Every request passes four checks: the bound address, a Host header naming it (DNS rebinding), an Origin, when sent, that is its own, and for /api/ the per-launch token, initialized through the launch URL fragment and required in x-panel-token. The launcher prints that capability URL; terminal-log and browser-history privacy are not guaranteed by this module. POSTs must be JSON, so a plain cross-site form is refused. Update is refused when the panel runs from the installed copy, which has nothing newer to install. Command runs, the project switch and auto-collect cycles share one lock (running, handed to createAutoCollect as lock): none runs beside another, so no preflight or update reads the corpus or the kit tree while a cycle writes it. Request-body overflow discards retained chunks and stops parsing further input. The handler sends the existing 413 JSON refusal with Connection: close, allowing the response to flush before the socket closes; the exact byte limit remains admitted (#324). |
ADR-0147, ADR-0148 |
requests.mjs |
A builder's fact request as a file (ADR-0148, 2026-10-07): research/requests/<id>.json, the fact-request skill made machine-readable. requestProblems(request, { perRequestPages }) validates it as untrusted input: the fixed fields (fact, blocks, urls, queries, prefer, unknown, maxPages, topic), bounded sizes, and urlProblem refuses a non-http(s) URL, one with a user name or password, and one naming this machine or an internal address (isInternal from http-transport.mjs, ADR-0110). A request over the per-request cap is refused, not trimmed. listRequests reads the folder, skipping the collector's own <id>.result.json and <id>.plan.json and ignoring a name that is not an id, and a request whose file, .plan.json or .result.json is a symbolic link (a builder's link would make the collector read or overwrite a file of its own machine; writeText follows links). applyRequest and writeResult refuse to write through a link as well. applyRequest writes the contract row (a new OPEN U-n, or an existing non-CLOSED row; it refuses requests naming a closed unknown) and the plan.json entries, and the plan for that run alone, with maxScrapes set to the request's pages. contractUnknownStatus reads the status of a contract row for continuation checks. writeResult records the outcome beside the request. Terminal and partial results are trusted only when recordTrustedResult has recorded hashes of both request and result in the machine-local research-kit.autocollect.json; isTrustedResult checks those hashes so a builder cannot submit a result as if the collector wrote it. The auto-collect settings (off/auto, pages a request, pages a day, interval, the folder for new topics) and the daily meter live in research-kit.autocollect.json beside the machine config (readAutoCollect, saveAutoCollect, recordSpend, setPaused); no key is in them. |
ADR-0148 |
auto-collect.mjs |
The collector listening (ADR-0148, 2026-10-07): createAutoCollect({ project, env, kitRoot, exec, lock }) returns cycle, status, schedule, resume, stop. A cycle resolves the project folder to its final path (fs.realpathSync.native) before comparing it with git's top level, so a Windows 8.3 short name or a symlinked folder cannot turn a path git stages into one outside the repository. status() uses that same canonical project path so its trust lookup matches collector records across alternate Windows path spellings. A cycle refuses unless collectionPolicy says the machine may collect, and skips while another run holds the shared lock (the panel's; its own when none is given). Otherwise it runs git pull --ff-only; a failed pull is surfaced and blocks the cycle before requests are read or processed. On success, it goes through each request. Results received over git are ignored unless their request and result hashes match collector-local trust records, including in status(); trusted terminal results retry pending commit/push delivery without collecting again, and trusted partial results carry continuation state and resume only while their contract unknown is not CLOSED. An invalid one gets a refused result, committed and pushed. A valid one, within the per-request and daily caps and with research/ clean, is applied (requests.mjs), then its target-specific research/requests/<id>.plan.json runs, followed by preflight.mjs, whose verdict is recorded, not acted on. It then stages research/ and the ledger by name (git add -f), commits, and pushes when there is an upstream. Accounting travels on the fixed Node research child's private IPC descriptor (ADR-0149), never its human output: exactly one bounded, typed report with measured pages within the admitted allowance and the exhaustion provider. Missing, invalid, duplicate or over-allowance reporting reserves the remaining admitted allowance against the daily meter, labels actual spend unknown, records a terminal failed result and pauses. Earlier measured continuation attempts stay counted once. A valid received report survives failed exit or later interruption; interruption also pauses. Diagnostic output keeps a bounded redacted tail with a truncation notice, and failure detail is bounded separately. The legacy spentPages/creditsStopped exports remain diagnostic compatibility helpers. It never passes --fallback: received exhaustion accounting makes the loop commit the partial corpus and continuation state, pause until the operator resumes, then reuse the saved target, unknown and remaining page allowance (ADR-0129). A topic request gets new-project.mjs and decompose.mjs --max-scrapes 0 in a dated folder under the folder the operator approved, never over an existing project (ADR-0056). It is refused when no folder is approved. Diagnostic details, refusal reasons and preflight verdicts are scrubbed with the existing redactor before result files or commit reports are written (2026-10-08). Status also scrubs inherited trusted result details, delivery diagnostics and pause text without rewriting those files or changing their trust hashes; the configured redactor defines which secret is recognized. Every child gets a fixed argument list; exec is injectable for tests. The panel labels its unchanged numeric meter as pages used or reserved. Allowance regression coverage (2026-10-09): the four-cycle credit exhaustion and resume fixture runs real synthetic Node scripts against persisted request, plan, result and meter files, with a finite injected Git executor. Native round-trip, partial-delivery and topic-resume fixtures retain real Git and corpus-transfer assertions. The test watchdog remains 60 seconds. Complete Git answers (ADR-0151): the bounded child byte buffer reports a private truncated flag. Repository and Git metadata paths, upstream, ahead count, commit subject and changed paths, dirty state, and pending delivery status/diff all refuse truncated output before it can authorize work; a malformed ahead count or missing quiet-diff exit also blocks delivery. A newly written partial result is trusted before fingerprinting, so refused inspection stages nothing and cannot make a restart collect it again. Exit-only operations retain bounded diagnostic logs. Admission survives collector termination (ADR-0152): reserveCollection saves the newly admitted allowance and an owned in-flight pause atomically before the fixed research child starts; a failed meter write prevents spawn. settleCollection refunds unused allowance once on a valid receipt, preserves earlier spend and distinct pauses, and retains the full charge for unknown/spawn-failed attempts. Whole collector termination leaves its charge/pause across restart or project switch. Continuation reserves only its remaining newly admitted pages. Day-aware settlement never refunds yesterday against today; cross-day completion charging is conservative, and an over-cap completion remains paused. The existing file writer is atomic, while the mutex remains per collector instance; racing independent instances and power-loss durability are not certified by this correction. Successful result.next says captures were delivered: the builder reads/judges, the collector records Git Findings, closures, MAP and the current authored BRIEF; another fetch is only for a missing external fact. ADR-0052 local received-package review remains available, and derived approval stays role-agnostic (2026-10-08, ADR-0150). Git command failures select the first fatal: or error: cause before applying the existing diagnostic bound; output without a cause marker and kit run summaries retain the final-line fallback (#326). |
ADR-0148, ADR-0149, ADR-0150, ADR-0151, ADR-0152 |
machine.mjs |
Config saves preserve declarations (#364): saveConfig patches a raw JSON object or its usable last-good snapshot, retaining unpatched role and posture typos for diagnostics rather than serializing resolved defaults. A broken config without a usable snapshot refuses an unrelated save with CONFIG-UNREADABLE; an explicit valid role declaration can repair it. The existing claudeGate migration remains, and retired keys are dropped. The config carries searxngUrl (2026-09-30, ADR-0104): a URL, not a secret. shape() keeps only the non-empty strings of skillRoots: every other key it reads is type-checked, and this one was checked as an array and passed its contents to path.join, so {"skillRoots":[42]} killed doctor with a raw ERR_INVALID_ARG_TYPE stack (break-test 2026-10-01). A dropped entry is silent; naming it would be a new finding, which the freeze does not allow. editGate.settingsPath is taken only as a non-empty string, else the anchor (2026-10-02, arena #198): it was the one key shape() passed through raw, and a numeric value reached path.dirname in installEditGate and killed install-hooks with a raw stack - the sibling of the skillRoots check, found a day later because the first fix was never swept to it. An ill-typed posture key resolves to its restrictive state (ADR-0121, 2026-10-02): shape() resolves a failOpen, evidencePolicy or editGate.mode that is present and not of its type to false, strict and hard-block through one postureKey helper, lists it in illTyped, and readMachineConfig/posture carry illTypedKeys beside retiredKeys on every path, the snapshot one included. Absent keys keep their defaults. Until then a typo fell to the lax default, the only place a typo relaxed a gate in silence. browserPath is read and kept (2026-10-11): it was missing from shape(), so the key the browser transport's errors name was never read, and every saveConfig - install-hooks --role among them - erased it. Only non-array JSON objects may create or restore a last-good snapshot (#368, 2026-10-13). A parsed scalar/array leaves an existing good snapshot intact; an unusable backup falls back to fail-closed with role unknown. |
Everything outside the project: machine config (fail-open posture, editGate mode and settings path, evidencePolicy, role), read in three states — absent (nothing configured; the fail-open default is the answer), readable, and unreadable (the file exists and does not parse), which holds the last config that parsed from a snapshot beside it and, with nothing to hold, fails closed. readMachineConfig() returns the settings and state/source/error, because the settings alone cannot say whether fail-closed was configured or is the fallback this machine adopted (ADR-0020). Install state, git core.hooksPath (scope-aware readers that expand Git include and includeIf directives, including scoped reads, so installation preserves and restores the included hooks folder; #369), and the runtime anchors — RUNTIME_ANCHORS, the one place a runtime's paths are named (settings file, personal skill roots, project-relative skill dir), each a default that runtimePaths() / skillLocations() let the machine config override (ADR-0012); Git readers/writers accept an explicit child environment, preserving production defaults while allowing R1's disposable machine/Git fixture to isolate global/system origins; reads a retired config key's value into the current shape rather than Its six home-based paths (settings, skills, agentsHome, kitHome, the config, the install state) use homeDir(), so HOME="" no longer makes them relative (2026-09-28). dropping a setting the operator had tightened, and reports a retired ENVIRONMENT variable (CLAUDE_SETTINGS_PATH) without ever honouring it. Owns the machine role and the collection policy that follows from it (machineRole, collectionPolicy, collectionRefusal) — the single answer to "may this machine collect?". The only module that reaches past the project root. localHooksPathOverride asks git from cwd (2026-10-03, outside audit): git finds the repository by walking up, so a subdirectory - a nested decision project, the folder install-hooks is run from - is governed by the same local core.hooksPath as the root; the probe looked for .git beside cwd and said null below it, and doctor's gateHealth goes through the same probe now. previousHooksPath(opts) reads the folder the install replaced (2026-10-03), the reader beside setPreviousHooksPath, with --type=path as hand-on.sh reads it - the raw value made doctor call a working ~/custom-hooks missing (second audit, 2026-10-03). repoTopLevel(opts) is git rev-parse --show-toplevel from a cwd, or null outside a repository (2026-10-03). skillRoots(env) is the personal skill roots - the config's skillRoots, else the anchor's - which skillLocations() and the skill set's deploy both read (2026-10-06, ADR-0146). Install-state reads require an object and shape existing path, path-array and skill-hash fields before consumers use them; doctor names an existing unreadable or pathless record in its deploy warning (#365). Snapshot resolution keeps source naming the broken config and carries snapshotSource separately; doctor and collection refusal distinguish held settings/role from fallback-closed instead of claiming the snapshot failed to parse (#366). |
handoff.mjs |
The arrival question: verifyHandoff(root) reports a ledger that is absent or empty, an EVIDENCE.md that is absent or holds no evidence table (handoff-evidence-missing, handoff-evidence-unparsed; the report carries rows, and the CLI and doctor print the count - 2026-10-02: with no rows there were no citations, so "every cited capture on disk" held vacuously and handoff said OK over a corpus whose table a sync tool had dropped), every capture an evidence row names that is not on disk, and a chain that does not verify — each named. Read-only by design: the machine that asks cannot collect the missing bytes. The report also carries lineEndings, the captures whose body hash fails only because git rewrote CRLF into them on checkout, and handoffRemedy(report) picks the remedy from the cause rather than printing one text for every way a corpus can look broken (ADR-0020): HANDOFF_REMEDY when something did not travel (the collector must push research/raw/, dotfiles included), lineEndingRemedy when everything travelled and this machine rewrote it (pin with .gitattributes, re-check out, touch no collector and spend no credits), both when a report holds both. Consumed by bin/handoff.mjs and doctor (blocker on role=builder, silent on a collector; doctor passes the snapshot it already read, so the question is composed once for both). The report carries nothingCollected (no ledger entry, no evidence row, no capture) since 2026-09-27, so a caller can tell a project never collected from a corpus that failed to travel. The finding names are unchanged, so the doctor and the artifact validator read them as before. The report also carries brief: { state, todo } (briefReview, 2026-09-27, ADR-0061): the CLI's "handoff OK" is followed by a note when the brief is a draft (naming the judged sections still TODO) or has not been drafted. It is never a finding, and the exit stays 0. The line-ending remedy is judged by running it (2026-09-27, ADR-0062): per affected capture it prints git rm --cached (the index entry only) then git checkout HEAD --, which rewrites the file as LF through the pin, and it prints the pin (PIN_LINES, to add in an editor) only when .gitattributes lacks it. Every printed command is a plain git command with no comment, so cmd and PowerShell run it as written (ADR-0070). --renormalize had fixed only the index, so the capture handoff reads stayed CRLF. A third cause (2026-09-28): a ledger present and empty while captures are on disk lost its entries where the captures are - a one-line ledger whose torn tail doctor --fix-arity dropped - so ledgerLost gets ledgerLostRemedy(): restore it from git, or re-collect with --force. It was given the push remedy, which on the collector pushes the same empty ledger. A ledger that is merely short is the same cause: a cited capture on disk that no entry records is handoff-capture-unledgered (2026-09-28) and gets the same remedy - handoff had said OK and exited 0 while preflight failed it as fetch-entry-exists. CORPUS_ADD (2026-09-28) is how the kit prints putting a corpus into git - git add research/, then git add -f research/raw/.fetches.jsonl - shared by the push remedy and the commit gate's fix: both said git add -f on the whole folder, which also committed the machine-local .usage.jsonl, .failures.jsonl, .diagnostics.jsonl and .fetches.lock. A fourth cause (2026-10-04, break-test pass 4, F5b): a capture whose body hash fails and NOT by line endings (body-unmodified/modified) was changed after its fetch, and the report carries it as altered; it no longer counts as "did not travel", and alteredRemedy(files, { isRepo }) prints, per file, git status --porcelain -- to tell a local change from a committed one, then git checkout HEAD -- followed by handoff again (no folder-wide checkout, which would discard uncommitted work beside it), and for the committed case git log --oneline -- to find the commit that held the fetched bytes and restore from it, here or on the collector; without git metadata it prints no command. Judged by running it (2026-10-04, external review): a dirty status says only that this checkout changed the file after the commit - a capture committed altered and edited again is dirty too - so the remedy no longer claims the committed copy is the fetched one; and --force is no longer offered, because a new fetch writes a new capture beside this one, or refuses to overwrite a differing source sibling, while verifyLedger still checks the entry that names this file. The push remedy it used to get sends the same bytes again when the change was committed, and sends an operator to another machine when the committed copy here was one checkout away. safePathspec(file) (2026-10-04, two external reviews) decides whether a capture path is printed inside a git command at all: letters, digits, ., _, - and /, the alphabet the collector's own names come from. Anything else is named but never printed as a command, because no one spelling is read literally by cmd, PowerShell and sh alike and by git: a space unquoted is two pathspecs, $copy inside double quotes is expanded to nothing by PowerShell, and [12] is a git glob whatever the shell did, so git checkout HEAD -- restored a neighbour and discarded its uncommitted work. For those the remedy prints commands that type no name at all: the operator lists them, one per line, in handoff-names.txt (NAMES_FILE) and runs git --literal-pathspecs checkout HEAD --pathspec-from-file=handoff-names.txt (the line-ending remedy adds the git rm --cached step the same way) - "quote it for your shell and prefix :(literal)" was not enough, because cmd expands %NAME% inside double quotes and PowerShell $name (third external review, 2026-10-04, verified by running in cmd); --pathspec-from-file hands git the bytes of the file with no shell in between, and --literal-pathspecs switches pattern syntax off (both git 2.25+). Such a name reaches the ledger rarely but not only by hand: captureName embeds the URL's host as it is, so an IPv6 literal or an adapter-supplied host outside DNS's alphabet produces one (third review). The classification runs over EVERY affected file (classify), not the five the text names: an unsafe sixth file got no note and "handle the remaining files the same way, each by name" - an instruction to type the glob that restores a neighbour (third external review, 2026-10-04). insideRepository(root) (2026-10-04, two external reviews) decides whether the runnable remedies are printed, and it asks git: git -C <root> rev-parse --is-inside-work-tree. It was root/.git alone, so a nested decision project (ADR-0030), which has none of its own, was told it had "no git metadata"; then a walk up for a .git entry, which walked past a bare repository the project sat inside and printed a checkout git refused with "this operation must be run in a work tree". The commands are git's, so git decides whether they run; without git on PATH the walk is the fallback: it walks the PHYSICAL path (realpath first - a project reached through a junction into a repository had read as outside it, and one linked out of a repository as inside; third review) and reads GIT_CEILING_DIRECTORIES as git does: absolute entries only, each through realpath until an empty entry, after which the rest are taken as written. Git's pathspecs are relative to the cwd, so the printed commands run unchanged from the project folder. The tests that want "no git metadata" from a scratch fixture set GIT_CEILING_DIRECTORIES above it and go through handoffRemedy, so a detection that always says "repository" fails them. repositoryState(root) is the full answer, { inside, refused }: when git answered with something other than yes, "not a git repository" or "must be run in a work tree" - fatal: detected dubious ownership - refused carries its first line and both remedies say git refused to read the repository and to fix that first, where they said "no git metadata, re-copy the corpus" (third review, 2026-10-04); no command is printed either way. Unparsed ledger lines carry their physical line and tail position in ledgerUnparsed; handoff chooses the existing tail-repair or git-restore remedy. Until parsing is restored, secondary sequence/hash/link failures and unledgered captures do not establish travel failure or entry loss; independently missing captures still get the travel remedy (#296). |
ADR-0011, ADR-0020 |
bundle.mjs |
The archive this project arrived as, and how far the tree has moved from it. BUNDLE_INDEX.md lists 80 SHA-256 digests taken at the 2026-09-17 handover; until ADR-0028 nothing had ever read them, which is how ADR-0022 could treat them as a constraint for three days without noticing the constraint was already violated. verifyBundle(root) sorts every entry into unchanged / drifted / unexpected / missing, where "drifted" means a file in EXPECTED_TO_DRIFT — the corpus, the map, the ADR index, the manifest itself, (from 2026-09-20) CONTEXT.md and the root README.md, which are documentation ABOUT the code and go stale the moment it moves, and (from 2026-09-21) AGENTS.md, which is the file the agent protocol is written in — caught drifting by this check when ADR-0030 gave decision research a committed home and the rule that sends an agent there had to be where agents read their rules — each of which moves because one of this project's own rules requires it to. The list is deliberately narrow: exempting a planning document would leave the report unable to say anything, and a test asserts none is exempted. Hashes bytes, not decoded text. Reports; never fails a build — a changed file is a fact about history, and the only thing that makes it a problem is nobody knowing. bin/bundle.mjs --json prints verifyBundle's result as JSON (2026-09-28): the flag was accepted and never read, so a caller parsing it got the markdown report. START_HERE.md joined EXPECTED_TO_DRIFT on 2026-09-29: the archived copy held the author's machine paths, and the portable one is the template rendered with $HOME. |
ADR-0028 |
git-config.mjs, config.mjs |
Compatibility shims → machine.mjs. |
ADR-0002 |
runtime.mjs |
The host prerequisites a command actually needs: checkNode (22+), checkGit, checkPython (3.11+ since 2026-09-28, ADR-0078 - one number with the READMEs and CI's pin, reading either stream because Python 2 printed its version to stderr, trying python3 first and continuing past an old interpreter rather than stopping at it — python is an alias for 2.7 on the most common Linux and older-macOS layout, and stopping there reported "no usable Python" with a good 3.12 one name away; an interpreter whose --version prints no version is passed over like an old one, and a refusal quotes what it printed - it was accepted as "proceeding" until 2026-09-28, and failed later in the conformance tests; it names the interpreter it chose as exe (2026-09-28), and the test harness's findPython runs that one - the harness had its own probe, python first with no version check, so an old alias failed every cross-language test), MAX_PAGE_BYTES, boundedText, CHILD_OUTPUT_LIMIT and outputOverflow (2026-09-28, ADR-0082) bound every transport child - boundedText what a child downloads, the other two what it hands back: they ran under spawnSync's 1 MiB default, so a page just over it failed with ENOBUFS - and a Firecrawl scrape that large was paid for and lost - and an overflow is now one sentence, its partial output discarded; and requireRuntime({ node, git, python }) which exits 3 — distinct from a product failure — naming the missing piece and how to install it. Deliberately per-command rather than one gate that demands everything: Python is used by the cross-language conformance runners and nothing else, and a command that only reads the corpus has no business refusing to run because a language it never calls is absent. The Node line, judged (2026-09-27, ADR-0046): nodeLine passes a supported line, warns on an odd line below 27 (never LTS, end-of-life after six months) and fails below the floor; nodeHonoursEnvProxy says whether fetch can use a configured proxy - 22.21+ on the 22 line, 24.0+ after, never 23 (docs/decisions/2026-09-27-node-support). A configured proxy, for every request the kit makes (ADR-0047): PROXY_VARIABLES / proxyVariable; fetchEnv for a child that fetches; and for a command that fetches in its own process, envProxyPlan / honourEnvProxy - set through http.setGlobalProxyFromEnv() on 24.14+/25.4+, reexec with the flag on 22.21+/24.0-24.13 (the same standard streams, stop signals forwarded, the child's exit code, and --disable-warning=UNDICI-EHPA - Node 22's "EnvHttpProxyAgent is experimental" on every run, and only that warning), unsupported with one stderr line naming the version to move to, none when no proxy is set or the operator chose at startup (the variable even at 0, --use-env-proxy in argv or NODE_OPTIONS), invalid when a value Node reads (lowercase wins) is not an http(s) URL - unusableProxy; with the flag on such a value crashes every fetch inside Node, so it is left off, and the message shows proxySpelling without ever echoing the value. fallbackCost(name) (2026-09-30) words what falling back onto the fetch provider's search costs: free on http-keyless and browser, which have no meter, and "this spends fetch credits" for any other, because a provider the kit does not know may be metered. The line had said "spends fetch credits" on the keyless transport. fetchFailure(err) (2026-09-30) is the keyless and SerpAPI children's error text: Node's fetch failed plus the reason on err.cause (getaddrinfo ENOTFOUND host, ECONNREFUSED, redirect count exceeded), with any URL in the cause cut to its host so a key in a query string never rides along. Every network failure had read fetch failed. fetchFailure also reads the first inner error of an AggregateError cause that carries no message or code of its own (2026-09-30), as a dual-stack connection failure can. fetchFailure cuts every URL to its host in the failure's own message as well as in its cause (hostOnly, both return paths): Node throws a credential-bearing or unparseable URL with no cause and the whole URL in the message, and one signed URL in a plan was written nine times into the ledger as that entry's error (break-test 2026-10-01). A body is read in the charset the server declared (2026-10-03, output-reliability audit G7): charsetOf(contentType) reads the label, decoderFor(contentType, bytes, who) picks the decoder - a byte-order mark first, then the declared label through TextDecoder, a label it does not know falling back to UTF-8 with a sentence in fallback - and boundedBody returns { text, charset, fallback }; boundedText is its text alone, for the JSON answers of SerpAPI, SearXNG and the Wayback Machine, which RFC 8259 makes UTF-8. Every body had been decoded as UTF-8 whatever the header said, so a windows-1252 page's euro sign captured as U+FFFD and the hash proved a page the server never sent. A legacy label over bytes that are valid multi-byte UTF-8 - the common server misconfiguration - is read as UTF-8, because such bytes are almost never intended as Latin-1 and the always-UTF-8 reading had got those pages right by accident. The review of that fix (2026-10-03) added three readings and took one back: bytes that are not valid UTF-8 under a UTF-8 label, or none - the page that names its charset only in a <meta> tag - are a named fallback instead of silent U+FFFD; a UTF-16 label over a body with no NUL byte (hasNul), which UTF-16 text of a page cannot avoid, is not believed and the body is read as UTF-8, where it had decoded to CJK garbage end to end; an unknown label over pure ASCII is an exact reading, not a fallback; and boundedText reads as UTF-8 whatever the label says, because the JSON callers had always had that reading and a parse decides what it means - boundedBytes is the shared bounded read beneath both. charsetOf drops a stray quote (charset="utf-8). nodeDecodesWindows1252 is the version rule behind nodeLine's second fail (2026-10-11, ADR-0159): Node 22.13.0 through 22.22.0 read windows-1252 as Latin-1 (nodejs/node#60888), so the declared-charset reading above silently lost 0x80-0x9F on those builds; doctor names the defect and the first good build, 22.22.1. usableProxyUrl mirrors Undici origin parsing: pathname must be / and search/hash must be empty (#380, 2026-10-12). A non-origin proxy URL follows the existing invalid plan, preserving direct requests and the named diagnostic rather than crashing during proxy initialization. envProxyChoice distinguishes an unspecified startup choice from enabled/disabled, with the final explicit flag taking precedence over the variable and command-line flags over NODE_OPTIONS (#381, 2026-10-12). proxyChosen uses that reader to retain the existing no-override rule; only an exact variable value of 1 enables the proxy. |
ADR-0046, ADR-0047, ADR-0159 |
scaffold.mjs |
Project shape: LAYOUT (12 entries), the four GATE_MARKERS (pinned by ADR-0001 and its test), template rendering, placeholder validation, hook executability. .gitattributes is a LAYOUT entry but not a gate marker: a project without it still gates, it just breaks on a Windows builder (ADR-0020). UNTITLED_TOPIC names the placeholder topic, so decompose can tell a project without a topic from one that has one (ADR-0056). A folder created while scaffolding a brand-new project is not counted as repaired (2026-09-27): only a folder missing from a project that already existed is a repair. MERGED_FILES (.gitignore, .gitattributes) are merged into an operator's existing copy (2026-09-27, ADR-0064): missingKitLines finds the kit's rule lines it lacks (whitespace collapsed) and they are appended once under a marked comment, so .env stays out and the ledger stays in. validateProject warns kit-rules-missing and agents-md-foreign (an AGENTS.md without the kit's heading, kept but named). A created folder counts as repaired only when research/ already existed (2026-09-27): the operator's own .gitignore had made a fresh folder read as a project. |
ADR-0001, ADR-0020 |
installer.mjs |
Installs the git hook (records/restores prior hooksPath) and the edit-time gate (backs up, repairs known corruption, refuses unfamiliar states). The hook was renamed (A10.4): the installer recognises a registration left at a retired name, replaces it rather than adding a second, and reports the repair — retiredRepairNote. And whether the deployment is still the kit you have (deployedDrift, driftNote). deploy is a full mirror, so drift is exact rather than heuristic: every source file should be present at the destination with the same bytes, and nothing else should be there. Doctor measures the drift against the home the install state recorded (the one the hooks point at; it had measured KIT_HOME and named the recorded one), and run from the deployed kit itself - node ~/.agents/research-kit/bin/doctor.mjs, the copy an agent runs - it says so as info, with the command that does measure, instead of pass ... matches this tree: a tree compared with itself always matches, and a line appended to the deployed lib/core.mjs had passed that way (break-test pass 3, 2026-10-02). This exists because doctor's deploy check read the install STATE and stopped - "kit deployed ", pass, READY - and recorded is not current. Measured on this machine 2026-09-22, that passed over a deployment missing 119 files and carrying 38 stale ones, including a skill describing a protocol without the step shipped that morning. An agent invoking the skill runs the DEPLOYED copy, so every fix since the last install existed only in the repository while the machine reported ready. The skill is reported separately from the kit because it drifts independently and is the file that tells an agent what the protocol IS. Reported, never repaired: install.mjs acts, and a health check that quietly rewrote what it measured would erase the evidence of the problem it found - pinned by a test that asserts the missing file is still missing afterwards. removeEditGate drops the PreToolUse list and hooks object its entry leaves empty, and removes a settings file with nothing left (2026-09-27): --uninstall on a machine with no settings file left {"hooks": {"PreToolUse": []}} behind. removeCommitGate restores the recorded hooksPath only while the current value is still the kit's own folder (2026-09-27, ADR-0063), and otherwise leaves it and names it: an uninstall had unset a machine's /opt/myhooks, which the kit never set. installEditGate refuses, as the commit gate already did, when hooks/edit-gate.mjs is not deployed (2026-09-27): it had registered a command naming a missing file, so every Edit would run a hook that crashed. deployedDrift asks isDirectory, not exists, of the kit home and each skill location, and listTree guards its readdirSync/statSync (break-test, 2026-09-29): RESEARCH_KIT_HOME naming a regular file sent readdirSync at it, and doctor died with a raw ENOTDIR stack; a file there now reads as not deployed. listTree walks each real folder once (2026-09-29): a folder link back to an ancestor re-listed the tree on every lap until ELOOP, about 40 phantom "extra" files per real one. deploy({ dryRun }) lists what the deploy would actually prune - retired files that exist, plus the mirror's extras (2026-09-29, Arena break test): it listed every retired name whether or not it was there, so a machine never installed read "would prune" three files, and it never named a stale file the mirror would remove. readSettings also refuses a hooks.PreToolUse node of a type the walkers do not handle (knownHookShape, 2026-09-30): [null], a number, a string, or an entry whose hooks is not a list of objects reads as unfamiliar, and the file is not rewritten. It had reached the walkers and thrown a raw TypeError out of install-hooks.mjs. An entry with no hooks key is still the known shape. knownHookShape also requires hooks itself, each entry and each hook to be plain objects (2026-09-30): "hooks": "none", null, a list or a number had passed as the known shape, and the install spread the string into the file. Project binding is checked before deployment writes (2026-10-08, ADR-0006): deploy({ into }) refuses a missing path, a regular file, or a directory with none of .git, AGENTS.md, or research/; .git may be a file or directory so worktrees remain eligible, AGENTS.md must be a file, and research/ must be a directory. The same refusal precedes the mirror, personal/project skill copies and install-state update on the real run and the preview; install.mjs --into reports it with exit 2. The guard had been absent, so a typo could deploy the home copy, prune its old files and bind skills into an unrelated folder. Config readers retain their separate last-good snapshot policy. deploy refuses, touching nothing, a RESEARCH_KIT_HOME that holds files and is neither a kit (lib/core.mjs, bin/gate.mjs and bin/preflight.mjs, all three: one alone took a project with a lib/core.mjs for a kit) nor the recorded deployment, --dry-run included; install exits 2 (ADR-0111, 2026-10-01). The mirror had deleted a folder's own files and reported them as retired. The refusal reads only the first few files it names (2026-10-01): counting all of them took 1.6 s for /usr. installCommitGate records the hooks folder it replaced as research-kit.previousHooksPath in git's global config, never the kit's own, and removeCommitGate clears it (ADR-0113). A dry run answers what the real run answers (2026-10-01, arena #196): installCommitGate checks that githooks/pre-commit is deployed before it previews, and installEditGate's check no longer reads !dryRun && !exists(hook) - on a machine with nothing deployed --dry-run promised both gates and exited 0 while the real run refused both and exited 1. Both previews carry ok: true, one result shape for every caller, and bin/install-hooks.mjs tests ok before dryRun: the first cut reordered the CLI without that key and a healthy deployed machine printed commit gate: undefined, which a CLI test over a deployed kit now pins. A repository-local core.hooksPath is reported at install time (2026-10-03, ADR-0131): installCommitGate takes cwd, probes it with localHooksPathOverride, and returns displaced: { cwd, local } on the real run and the dry run alike; bin/install-hooks.mjs says it on stderr with the remedy and exits 0, and the install goes through - the gate is machine state. The installer had read and written only the global path and said "installed" from inside a husky repository where the gate would never run (outside review, 2026-10-02, three rounds). A dry run writes nothing on any path (2026-10-03, outside audit): removeCommitGate, removeEditGate and uninstall take dryRun and answer before any write, and bin/install-hooks.mjs reads --dry-run FIRST - it was read after the role, the posture and the uninstall had run, so --dry-run --role builder --fail-closed saved both and --dry-run --uninstall unset the installed commit gate. Both tree walks skip __pycache__ (2026-10-04, break-test pass 4 F1): CPython's cache from the conformance runners was deployed with the kit and then counted as drift on every machine that had run the suite. The skill set (2026-10-06, ADR-0146): SKILL_SET_DIR (skills/) and shippedSkills(from) - every directory there holding a SKILL.md, sorted; deploy mirrors each to <root>/<name> in every skillRoots() root and, with into, beside the bound research-first skill (boundSet); deployedDrift measures each against every root where research-first is deployed, so a root installed before the set existed is drift, reported, never repaired. A same-named skill that is not the kit's is left as it is (2026-10-06, review): a set skill is written only where its folder is absent or its SKILL.md is, byte for byte, the one the kit last deployed there (skillSetHashes in the install state, kept across deploys) or the one it would deploy now - the content decides, never a recorded path, which outlives the kit's copy; any other is returned as skillConflicts (the dry run included), printed by install.mjs, and skipped by drift. The first cut overwrote a user's own brainstorming/SKILL.md and left its other files mixed with the kit's. A set skill the kit owns is mirrored (copyTree with mirror) and its extras count as drift (orphaned), unlike research-first's (PR review, 2026-10-06): the deploy had copied over the folder, so a reference a later kit dropped stayed and could still steer the skill, and doctor called it current. handOnState lives here since 2026-10-04 (ADR-0141; doctor re-exports it): what the hooks hand on to - none, missing, kit, other - and installCommitGate records no folder it classifies as kit (handsOnTo), on a first install or a re-install from the state file. doctor's one-line remedy for that hand-on (ADR-0134) had lasted one session on the operator's machine: the installer kept the old kit's githooks/ in the install state and the next install-hooks run wrote it back into git config, and the old gate blocked commits again on its outdated checks. The state file and git config agree once an install has run; a missing folder stays recorded, for the uninstall restore. Read-only deployed files remain replaceable (2026-10-08, break-test pass 7 correction): copyTree restores write permission on an existing regular-file destination before copying, independently of the incoming source's mode, and explicitly applies the source's ordinary permission bits with owner-write after copying. This covers repeated read-only-source deployment and replacement of earlier read-only output from a normal writable source. Windows chmod changes write permission only; the copied file's read-only attribute is cleared, while POSIX ordinary execute bits are preserved and githooks keeps its final HOOK_MODE (E-04/E-05 in docs/decisions/2026-10-07-break-test-external-facts). Source-mode propagation was observed in the original Linux probe rather than promised by the copy API; directory, ownership and ACL failures remain separate limits. |
ADR-0012, ADR-0131, ADR-0141 |
render.mjs |
Terminal tables and Markdown findings, with no policy or I/O. renderTable sizes each column with a fold, not a spread (2026-09-28, Arena break test 12): Math.max(header, ...rows.map(...)) passes one ARGUMENT per row, and V8 exhausts the stack somewhere past 100,000 of them - measured, 50,000 rows rendered and 200,000 threw RangeError: Maximum call stack size exceeded, which is a raw stack trace where a report belongs. Nothing in the kit produces a report that large today; a renderer that dies on a large input is still a renderer with an undocumented ceiling. renderGateWarnings(verdict) presents an existing evaluation in Markdown for the brief, audits and package reading guides (ADR-0143). It preserves every warning reason, remedy and supplied row/unknown target, labels whole-corpus evaluation time, and distinguishes unavailable observations from an evaluated zero. Presentation only: it performs no I/O or policy judgment. |
ADR-0143 |
timeline.mjs |
TIMELINE.md generation and the diagnostics-log appender the gate CLI records through. Its regenerate command is spelled by documentCommand, with the project's recorded kitPath first (ADR-0084). The When column writes a day as a date and a real time as YYYY-MM-DD HH:MM UTC (2026-09-27): it had mixed evidence dates with the ledger's day-precision T00:00:00.000Z stamps. A migration row gives its recorded reason or note and a prior row its file, not url -> raw ("undefined -> undefined", fixed 2026-09-29). Events sort by compareText on their timestamp (2026-10-01). Timeline details fold whitespace before escaping table pipes, keeping multiline fetch errors in a single Markdown row; the root timeline is regenerated from its intact corpus (#354, 2026-10-13). |
ADR-0051 |
Not a module, but part of the map: examples/release-evidence/ (added 2026-09-20) is
six synthetic release packages plus build.mjs (regenerates them, keeping their hashes
correct) and run-example.mjs (validates what is on disk, and never calls build.mjs).
It exists because researcher-release validate checks a package — files that refer to
each other — and the schemas describe each file's shape while saying nothing about how
they connect, so the only way in was to assemble something plausible and learn the
relationships from error codes. One of the six passes on purpose: a record the registry
does not list is never validated, so PASS means "everything declared is intact", not
"nothing unexpected is here". test/release-examples.test.mjs asserts the packages still
produce their documented verdicts, that the committed files match what build.mjs
generates, and that the README's table names every package that exists and none that does
not.
research.mjs (→ research-run; auto-collect alone opens its private IPC channel, over which it sends the run's spent/exhaustion report and waits for send callback/disconnect before exit, while actual parent receipt remains authoritative, ADR-0149; refused on a builder by role; names the kit's own refusals with their message and exit 2 - NAMED_RUN_REFUSALS - instead of re-throwing them as a stack trace and exit 1, which a torn ledger tail did until 2026-10-01), researcher-release.mjs (→
read-only R28–R32 validation and schema conformance), handoff.mjs (→
verifyHandoff, the builder's first command; on a collector whose report says nothingCollected it says nothing has been collected yet and names research.mjs, rather than "something did not travel", ADR-0057), preflight.mjs (→ the verdict; its summary
line names the setting that decided the verdict, printing [--strict, over evidencePolicy=pluralist] rather than the declared policy alone — --strict promotes every
warning while evidencePolicy=strict promotes only the three POLICY_CHECKS, and labelling
a flag-driven failure with the policy name told a reader the policy had failed them;
ADR-0153: after existing argument and check-name validation, a cwd with none of the four
gate markers returns the same ungated scope in text and JSON. JSON keeps the existing
keys and types but reports pass:false, counts 0/1/0, one warn/gate/not-gated notice
and exit 0; this unjudged scope notice is not promoted by strict settings and grants no
build approval. The shared verdict library and all gated formatting stay unchanged),
gate.mjs
(→ evaluate; --staged-stdin reads the piped path list; --posture answers 0 allow /
1 fail-closed / 2 unreadable-and-closed, so no shell has to parse JSON), doctor.mjs (--fix-arity), install.mjs (deploy + --into per-project
binding, role-aware next steps; collector keyless instructions name the selected machine config rather than a per-run research flag, builder instructions require complete/current review, and an unresolved role prints diagnostics before role-specific advice (2026-10-08, ADR-0150); tolerateClosedStdout before its first write, so a reader that leaves early does not turn a finished deploy into exit 1, 2026-09-30), install-hooks.mjs (--fail-closed, --role; a CONFIG-UNREADABLE save refusal is named with exit 2 before the requested config patch is written, #364),
new-project.mjs (its first next step omits --topic once the project has one, ADR-0056), decompose.mjs (phase 0 CLI, refused on a builder by role; --limit requires at least 1 at the CLI boundary, matching the library's positive search-result limit, while --max-scrapes still allows 0, #320),
timeline.mjs, audit.mjs (lists and resolves versions through the manifest's
reader), brief.mjs (drafts the handoff with all evaluated warning reasons, remedies and
row/unknown targets, labeled as whole-corpus observations; it reports the brief's state; it,
timeline.mjs and prior.mjs refuse with exit 2, writing nothing, in a folder holding none of the gate
markers - they wrote research/ into any folder until 2026-09-28, prior a PRIOR.md and a ledger; timeline, brief, new-project,
doctor --fix-arity and install name a write the system refused - could not write <file>: <CODE> (why), exit 2, via writeFailure - where they printed a stack trace, 2026-09-28),
path-authority.mjs (offline Git-origin/path-authority snapshot validation and
schema conformance; refuses a missing value for --file, --schema or --root by
name before reading any input, as a usage error with exit 2),
researcher-release.mjs (ported 2026-09-20, ADR-0029 — three subcommands over the two
validators: validate for R28–R32 release evidence, conform for schema-only checking,
fi-validate for an FI bundle. It duplicates no validation: every check comes from
release-validator.mjs or fi-validator.mjs through their public exports. Reads no
environment variable at all, so it cannot depend on a credential or a config. Status
maps to exit code — PASS 0, FAIL/REOPEN 1, INCOMPLETE 2, BLOCKED 3, unknown 4 —
and a usage error exits 2 from every subcommand, printing the usage block to stderr;
its argument parser names a value flag without a value before resolving any path,
including when the next token is another flag. validate refuses blank, non-finite or
negative --snapshot-max-age-ms values as usage errors before judging evidence (#343).
The validateRelease and validateSnapshotEvidence library boundaries report an invalid
age allowance as INPUT-SNAPSHOT-MAX-AGE / INCOMPLETE; SNAPSHOT-STALE is reserved for
an actual snapshot older than a valid allowance;
test/support-policy.test.mjs pins that by running the CLI. It writes exactly one thing,
only when asked: fi-validate --report <file>),
selftest.mjs (every requested group must match a test file before any test starts; an unmatched group is named and refused with exit 2, including beside a valid group; #332. CLI regressions isolate child report paths from the parent suite; the whole suite, queued by the harness — which awaits every test,
so ok means the assertions settled; ADR-0021. It runs from the repository root whatever
the caller's cwd, resolving a relative RESEARCH_KIT_RESULT_FILE against the caller's
folder first; started anywhere else, it crashed at import before any test ran - ADR-0071); RESEARCH_KIT_ALLOW_UNSUP=1 lets a LOCAL run whose only blockers are UNSUP tests exit 0, ending "NOT a full pass" and writing unsupportedWaived into the result file, and is ignored when CI is set (2026-09-30, ADR-0108); a block whose only blockers are unsupported tests ends "No test failed ... this is not a red suite", naming the opt-in, where it said "A red suite stops work" - the protocol's stop-the-line sentence, which stopped a break-test at its baseline on a host with no Chromium (2026-10-02); below the Node floor it refuses before any test runs, exit 2, naming the floor and the fix - on Node 20 it ran and reported 28 red tests each saying "this kit needs 22 or newer" (2026-10-02; RESEARCH_KIT_TEST_NODE_VERSION is the harness's seam); the result file names the failing tests (failed: [labels]) beside their count, so two runs compare as sets - a count cannot show one test starting to fail while another starts to pass (break-test pass 5, 2026-10-04); a RESEARCH_KIT_RESULT_FILE that cannot be written is named on stderr and a green run exits 2 for it, where it exited 0 and left CI's summary reading "crashed before reporting" beside a green step (2026-10-02); it gives the run a scratch home and removes every RESEARCH_KIT_* variable but the ones that describe the host or the run (the result file, the unsupported opt-in, the browser, the gate timeout, the live opt-ins, the harness's RESEARCH_KIT_TEST_* seams), and the vendor keys, before the first test (2026-10-02, ADR-0123) - the tests that spawn kit commands read the operator's machine config, deployed kit and CLI login, so a strict posture config turned 18 tests red and this container's http-keyless config hid two cli tests red on any host whose Firecrawl CLI is installed and not logged in; a test file that throws while it is imported is reported as a named FAIL by importTestFiles in the harness, and the rest of the suite still runs and counts - one such file used to abort the whole run with no count); it tries the temp folder once before the run and, if it cannot be used, names it and the TMPDIR fix at the top and under a red summary - an unwritable one gave 541 EACCES failures and no cause; a closed stdout no longer kills the run - | head -1 printed an unhandled EPIPE and exited 1 while the suite was still going, so a green run read as broken (2026-09-28); outside a repository checkout - a deployed kit - it refuses with exit 2 and points at doctor, where 80 tests had failed on a healthy install, ADR-0080; a temp folder that does not exist yet is created by its probe and by the harness's tempDir through createTempFolder, which creates each missing ancestor with its own non-recursive mkdir and records only a mkdir that succeeded, each at the front of the list so it stays deepest first at every moment (a sort after the loop was skipped by a mkdir that threw part-way, and an ancestor already recorded was left behind: 2026-10-04, second review) - recording from the existence check before it left a window in which another process created the same chain and this run removed that process's empty folder (2026-10-04, external review) - and removes them with the scratch at exit by a non-recursive rmdir that refuses anything non-empty - the run had left the created chain behind, empty, /nonexistent/dir/xyz for that TMPDIR and an empty folder inside the checkout for a relative one: break-test pass 4, F3, 2026-10-04).
githooks/pre-commit— POSIX sh commit-gate wrapper. Its posture reader reads afailOpenthat is present and not the unquoted wordtrueas fail-closed (fail_open_in, ADR-0121) - it looked for the wordfalse, so a quoted"false"read as fail-open - and drops a leading byte-order mark before its structural check, agreeing withparseJson(a pinned posture row per BOM state). With HOME unset or empty it takes the home directory from node (os.homedir(), else the passwd entry), aslib/machine.mjs'shomeDir()does: underset -uit stopped on$HOMEand refused every commit with a shell error (ADR-0072). Tracked as mode 100755. It dispatches one default commit CLI invocation, whose checked Node reader lists the staged set with--no-renames, so agit mvout of product code names both paths and cannot pass as a change confined to research/ (2026-09-30). The earlier successful Git probe preceded an unchecked second pipeline; if that actual read failed, sh kept the gate's success status and the verdict received an empty list. ADR-0154 removes both shell reads: the actual Node read must finish successfully and completely, and unknown scope cannot clear the map rule. Once the gate allows (or fails open) it hands on to the repository's own.git/hooks/pre-commit, and every other hook name git knows has a pass-on file beside it that does the same (ADR-0112): git runs hooks from one directory, and the machine-widecore.hooksPathhad silenced every repository's own. The hand-on lives ingithooks/hand-on.sh, sourced by every hook, and runs what git would have run without the kit: the global hooks folder the install replaced, recorded asresearch-kit.previousHooksPathin git's global config, else the repository's own (ADR-0113). In the kit's own checkout - the same three markerslib/gate.mjsreads, plustest/- it widens the watchdog unlessRESEARCH_KIT_GATE_TIMEOUTis set, because the suite rule (ADR-0120) runs the suite inside the gate and the 120 s default would kill it and report an internal error on every kit-code commit (2026-10-01). The widened watchdog is 4020 s,lib/gate.mjs'sSUITE_TIMEOUT_MSplusGATE_REST_S, pinned to their sum by test, so the gate decides an overrunning suite before the watchdog can (ADR-0142, 2026-10-05; it was 1500 s against a 1200 s budget, so the node side always fired first). An operator'sRESEARCH_KIT_GATE_TIMEOUTsets the watchdog and, less 120 s (or half of one under 240 s), the suite's budget; the hook checks it withgate_timeout_readable, in the grammarlib/gate.mjsreads, and names and ignores a value outside it - timeout(1) takes a sign, an exponent or hex that the gate did not, and for+40the watchdog fired first (review of ADR-0142, 2026-10-05). A test holds the two readers to one table. that is part of its contract, not a detail: git skips a hook it cannot execute, and says nothing. It was tracked 100644 until 2026-09-20, so every Linux and macOS clone of this repository had a commit gate that silently never ran — invisible on Windows, wherecore.filemodeis off, and found in ten seconds by the first CI run on Ubuntu.hooks/edit-gate.mjsis deliberately not executable: it is invoked asnode hooks/edit-gate.mjs, never directly. See ADR-0001 andhookExecutability. Fail-open when node or the kit is missing, with the one self-contained posture reader pinned tolib/machine.mjsby test (ADR-0002). The hook uses one fixed Node invocation, and the gate reads the NUL-delimited staged paths as checked Git output (ADR-0154), never accumulating paths into argv. Explicit--staged-stdincallers remain supported. The oldset -- "$@" --staged "$p"loop re-copied the whole growing list on every iteration, so the hook was O(n²) in staged files — 225 seconds of silence on 30,000 paths, which reads as git being broken. It is also watchdogged (timeout/gtimeoutwhere coreutils exists,RESEARCH_KIT_GATE_TIMEOUT, default 120s), and a watchdog kill (124) is an internal error — named on stderr, then decided by the same fail-open / fail-closed posture as any other broken gate, never a hang and never a silent pass (ADR-0020).hooks/edit-gate.mjs— the edit-time adapter;permissionDecisionsemantics per E-05. The wire keys (hookEventName: 'PreToolUse',hookSpecificOutput) are the runtime's protocol, not the kit's vocabulary (ADR-0012). A call whose every target (tool_input.file_path/notebook_path, resolved against the payload'scwd) is phase-1 work - underresearch/, or the project'sSCAFFOLDING- is allowed unjudged, bylib/gate.mjs'sisPhaseOneEdit(2026-09-27). The hook never looked at the file, so while phase 1 was open it interrupted every edit toresearch/MAP.mdandDISCOVERY.md: the work phase 1 is made of. The map is not exempt, because an edit gives it content. A cwd with no gate markers is judged by its repository's top level, the root the commit gate uses, walked up to in the cwd's own spelling (ADR-0053): fromsrc/the hook answered "not a gated project" to every edit. A target insideresearch/raw/is denied (2026-09-27, ADR-0059), before the phase-1 allowance and whatever the verdict: captures and the ledger are written only byresearch.mjs. OnlyeditGate.mode=offandresearch/GATE_OFFlet such an edit through. Creatingresearch/GATE_OFFis also denied before the phase-1 allowance (2026-10-10 audit), on a passing or failing verdict, with the same deliberate off-switches. The exact target and descendants are protected: a Write into a descendant creates the directory that the override existence check accepts. Sibling names remain ordinary research. This protects the matched edit tools; shell writes and operator-controlled hooks remain outside that boundary. It had been allowed as phase-1 work, and a forged ledger line with a correct hash passes preflight. The subfolder discovery (git rev-parse --show-toplevel) runs with git's repository-location variables removed (2026-09-28): aGIT_DIR/GIT_WORK_TREEcarried by the editor process made git name another repository, and code edited fromsrc/was allowed as "not a gated project". A payload that is valid JSON but not an object, or whosecwd/project_diris not a string, reads as no payload and the process cwd (2026-09-29, Arena break test): either had crashed the hook with a raw stack and exit 1 on every edit. AsksstdinIsReadable()before reading the payload and allows, with the reason, when stdin is a terminal:readFileSync(0)on a TTY waited forever for an end of file (break-test PR #185, 2026-10-01).schemas/— JSON Schema contracts consumed byrelease-validator.mjs; schemas are read-only inputs and are never generated or rewritten by the validator.skills/<name>/SKILL.md— the skill set (ADR-0146): one directory per skill, deployed as siblings of research-first in every skill root and--intoa project.skill-routerroutes by role and gate state, and its## Skill table(held bytest/skill-set.test.mjs) never sends a builder to a skill that collects. Ten directories come from the recorded SkillsMDs baseline (Apache-2.0,REUSE.toml), with renames and marked auto-build adaptations listed in NOTICE and ADR-0146; the rest are the kit's own. Guidance only (ADR-0006): no skill passes a gate, and owner consent/prose duties remain review responsibilities. Router/build handoffs require complete/current review, preserve authored unstamped and local received-package compatibility, and keep Git review records with the collector (ADR-0150).skill/SKILL.md,template/,recipes/— the protocol as text. Recipes carry machine-readable frontmatter dimensions (specializations ofdimensions.mjs, never replacements).START_HERE.md(kit root andtemplate/) — the operator's twin ofAGENTS.md: where things are, the three lines of starting research, the four-command resume block, one line on credits. Table-first, no protocol vocabulary, scaffolded into every project and deployed with the kit (ADR-0018). The two copies differ only in how the kit path is spelled ($HOME/.agents/research-kitvs{{KIT}}). Every command in them, and in every template file and the skill, double-quotes its path, and a project that travels spells the kit$HOME/.agents/research-kit(ADR-0050): that reaches node whole in every PowerShell, bash and zsh, with a space in the home folder, where the bare~it replaced reached node literally in Windows PowerShell 5.1, 7.4 and 7.5. START_HERE says what cmd needs (%USERPROFILE%). Not aLAYOUTentry: no rule needs it, so a project without it reports nothing.docs/ARCHITECTURE.md(this file) andresearch/kit.json— the map of the code and the declaration of what code it guards. Both travel with the scaffold (ADR-0008); both are LAYOUT entries, neither is a gate marker.
release-validator.mjs exposes the read-only verifyQualificationLedger seam. It
checks canonical payload/record/chain hashes, duplicate event IDs, signed Ed25519
promotion pointers, promotion-commit target/closure links, and cached-head consistency.
Ed25519 verification consumes the raw 32-byte SHA-256 digest and requires canonical,
unpadded base64url signature text. ledger-conformance.mjs, its Node CLI, and the
standard-library Python runner execute one shared synthetic packet for those hash domains
and signature bytes; no runner holds private key material.
Qlog segments are scanned without mutation; unresolved prepare/intent records, torn
tails, missing commits, invalid signatures, or stale heads are INCOMPLETE and cannot
qualify a package. researcher-release validate enables it with --ledger, --head,
--keys, and --genesis-hash.
The same module exposes pure computeDescendantInvalidation, which authenticates
synthetic pointer identities, rejects cycles/missing roots before emitting output,
walks target ancestry with duplicate convergence, and returns package/generation/
record-ordered revocation projections for rollback planning.
Added 2026-09-20. The Python runners exist to answer the same question a Node runner answers, about the same packet, and to disagree loudly when the two implementations differ. That only works if they canonicalise identically — and until this change they did not.
Each of the three runners carried its own copy of the canonicaliser. Two were hand-written
byte-for-byte alike; the third reached for json.dumps(sort_keys=True), which is a
different algorithm. The divergence was real and provable:
{"a": 1.0} Node -> {"a":1} sha 015abd7f... Python -> {"a":1.0} sha c29a44ab...
{"a": -0.0} Node -> {"a":0} sha 45b619e9... Python -> {"a":-0.0} sha 952b7dc4...
Cross-language agreement held only because no vector in any packet contained a float. A suite that compares 28 chosen inputs does not establish that two functions agree; it establishes that they agree on 28 inputs. The tests were green and the claim was false.
bin/conformance_common.py is now the single implementation — standard library only, like
the runners it serves. It holds canonicalisation, ECMAScript number formatting, JSON string
escaping, duplicate-key-rejecting parsing, the shared packet preflight, and report
rendering. Each runner keeps a thin binding and nothing else.
A packet nested deeper than MAX_JSON_DEPTH (256) is refused before json.loads runs, by a flat scan (ADR-0083, 2026-09-28): the Node reader enforces the same number, and without it Node accepted depths at which this canonicaliser hit the recursion limit, so the two runners disagreed.
An integer literal too large for a double is refused at parse as non-finite number (parse_js_integer, 2026-09-28), as the Node reader refuses it: canonicalising one through float() raised OverflowError, and int() past 4300 digits ValueError - tracebacks where Node reported.
A packet nested too deep to parse is refused like any unparseable one (2026-09-28): the
parse sites catch RecursionError, which is not a JSONDecodeError, and each runner had
printed a traceback where its Node twin reported FAIL (test/conformance-hostile.test.mjs).
A packet opening with a UTF-8 BOM is refused by name in all six runners (2026-09-29):
property_vector_conformance.py reads its own packet rather than through read_packet, so it
lacked the BOM floor and relayed CPython's "Unexpected UTF-8 BOM (decode using utf-8-sig)".
Two packets that PARSE crashed the Python runners while canonicalising (2026-09-28): an
unpaired-surrogate object key raised UnicodeEncodeError in the key sort before json_string
could refuse it (the sort now encodes with surrogatepass), and a packet a few hundred levels
deep raised RecursionError in canonical_json, past the parse sites. The ledger and property
runners' main now catch ConformanceError and RecursionError; the FI runner already caught
both as ValueError and RecursionError.
Two decisions in it are load-bearing:
js_numberimplements ECMAScriptNumber::toStringdirectly rather than post-processing Python'srepr. Both languages produce shortest-round-trip digits, so the digits were never the problem; everything around them was, and each difference is a different SHA-256:1.0/1,-0.0/0,1e-07/1e-7,1e+17/100000000000000000,1e-05/0.00001. Post-processingreprwould have caught the first four and missed the fifth, because the two languages switch to exponential notation at different thresholds — not at different formats.- An integer outside +/-253 is rendered the way
JSON.parsein JavaScript would render it, not the way Python's arbitrary-precisionintprints it** (2026-09-28, Arena break test 12).9007199254740993IS9007199254740992after a JavaScript parse, and123456789012345678901234567890is1.2345678901234568e+29;str()on a Pythonintprinted the digits the author wrote, so the two languages hashed DIFFERENT BYTES for one input. Found by differential testing 3,926 generated structures - 107 disagreed, all of them this case - and no shipped vector carried such an integer, which is why the chosen inputs looked like agreement: the same way the float divergence above hid behind vectors that happened to contain no float. Integers INSIDE the range go throughstr()unchanged, so the fix cannot rewrite a small one. float_policyis a required, named argument, because the repository has two deliberate policies that were previously indistinguishable from a bug.rejectrefuses a float outright: the ledger and property vectors carry hashes and chain positions, where a float is meaningless and its presence means the packet is wrong, so refusing says more than canonicalising would.normalizeformats it exactly as JavaScript would, for FI vectors describing documents that may legitimately carry one. An unrecognised policy raises rather than defaulting.
ConformanceError subclasses ValueError deliberately. Every runner's domain logic
already catches ValueError around a single document, because a malformed document is a
vector result — the FI packet carries one with a duplicate fiId whose entire purpose
is to be rejected and recorded as FAIL. A bare Exception escapes those handlers and
turns an expected failure into a crash.
What the regression vectors can and cannot pin. qualification-ledger-vectors.json
gained thirteen canonical-float vectors (QL-F-01…QL-F-13) covering both exponent
thresholds from both sides, the smallest denormal, the largest finite double, a repeating
fraction, nested floats, and a float beside integers in one array. Each declares
floatPolicy explicitly, and only normalize is expressible — which is itself worth
recording: in JavaScript 1.0 is 1, so a Node runner parsing the packet has no
float left to reject by the time it sees the value. A reject vector would pass on the
Node side for a reason unrelated to the policy it claims to test. The reject policy is
therefore a Python-side property, tested directly in
test/canonical-float-policy.test.mjs, which hands real Python floats to the module
rather than routing them through JSON. That file also compares the two canonicalisers as
functions over inputs nobody put in a packet, and asserts structurally that no runner
has grown a fourth private copy.
- Transport seam —
runScrape/search/scrape/command/mapare injected adapters (ADR-0005), chosen bylib/transport.mjs. Two fetch adapters exist — the Firecrawl CLI (lib/firecrawl.mjs) and the keyless HTTP one (lib/http-transport.mjs, same seven-function shape, pinned by test) — which is what makes the seam real rather than hypothetical. Since ADR-0027 the seam has two sides: a fetch provider and a search provider, resolved independently, withlib/serpapi.mjssatisfying the search contract alone. Every collection path is testable with no key, no credits, no network; the ledger'stransportfield records which adapter actually fetched, andtransport-provenancejudges that. A search records its provider separately — in the usage log, the failure log, and the ledger's optionaldiscoveredByon any entry whose URL a ranking chose — because who chose a page and who fetched it stopped being the same answer once there were two providers. A URL a person wrote intoplan.jsoncarries nodiscoveredByat all, so its absence means "nobody ranked this". A search itself never becomes a ledger entry. Acmdstring is an annotation and proves nothing. Inside the Firecrawl adapter there is a second, narrower seam:exec(argv, opts). It carries an argv array, never a command string (ADR-0020) — the string form was the shell-injection vulnerability, because a URL and a query are attacker-controlled data in this kit's threat model (aplan.jsontravels through git; a query is pasted back from a web result).spawnandplatformare injectable on that seam too, so both invocation routes — the shell-free POSIX one and the validated Windows cmd.exe one — are pinned by test on any host, and one test spawns a real stub CLI and then checks that the file an injected$(touch …)would have created does not exist. - Corpus snapshot seam — checks are pure functions over
readCorpus()output plus options (ADR-0004). New rule = new check = new fixture test; the verdict never changes shape. The joins a consumer needs are the snapshot owner's too (traceOf,captureOf,claimOf), so "how a row points at its evidence" changes in one module and is proved by one fixture. - Verdict seam — one
runPreflight, three consumers, and one owner of the verdict's inputs:verdictContext(root)inlib/preflight.mjsreads the corpus, the gate state, and the operator'sevidencePolicytogether. A caller that injects a corpus gets its gate state from the snapshot's own root (recorded by the reader) — never the process cwd — so the same snapshot cannot yield a different verdict. Severity policy (pluralist warns / strict fails) is the operator's machine config; the agent does not pick it. - Machine seam — one module reaches outside the project (ADR-0002); the pre-commit
hook's inline posture reader is the single deliberate duplication, pinned to the module by
test. The pin covers every state, not just the two the readers used to have
(ADR-0020): each row of the agreement table now sets both files — the config and the
last-known-good snapshot — so no row inherits state from the row above it, and each
asserts the message that names which branch decided, because three of them exit 1 for
three different reasons an operator has to tell apart to repair the right file. The two
readers share one exit-code vocabulary: 0 allow, 1 a config that parses says
fail-closed, 2 unreadable and resolved to blocking. The code carries the resolved
posture, not the state — an unreadable config whose held snapshot said fail-open is 0,
warned loudly — because two readers of one posture that disagree about what a code
means is the drift the pin exists to catch.
The machine's role (
collector | builder) is read here too (ADR-0010): the collection CLIs askcollectionPolicy()before they touch an adapter, and doctor asks it before it judges a missing credential. - Handoff seam — the corpus crosses machines through git, so the arrival question has
one owner (
lib/handoff.mjs, ADR-0011) with two consumers and no repair path: doctor (a blocker onrole=builder, silent on a collector, whoseraw-dangling/ledger-missing/ledger-chainfindings already own the same states) andbin/handoff.mjs(exit 1, every missing path named). One owner also means one remedy per cause (ADR-0020): the diagnosis lives inlib/provenance.mjs, which owns the hash comparison and can prove a rewrite by folding; the report carries the verdict aslineEndings; and both consumers printhandoffRemedy(report)instead of a constant, so neither can drift into telling an operator to re-push a corpus that already arrived. The per-entrydetailstays one line for the same reason the remedy is stated once — a paragraph per capture buries the capture names. - Write seam — captures and rows go through one writer family; the ledger is append-only
with hash-chained entries; migrations are the one sanctioned rewrite and must leave traces
(R9) that the provenance check cross-verifies. A write hands back what it wrote:
writeRawandcollectOnereturn the capture's index entry, built bycaptureEntry, so a caller indexes a fact instead of reconstructing one. The chain'sseq/prevare a read-modify-write, so every writer ofresearch/raw/.fetches.jsonlholds oneO_EXCLlock while it does it (ADR-0020) — an unlocked append is not a smaller version of the same operation, it is the operation with the guarantee removed. Two collectors arriving at once (an agent run and an operator run, cron plus interactive) is a routine situation, and the corruption they used to produce was unrecoverable:repairLedgerTailno-ops on a chain broken in the middle andrebuildLedgerrefuses it, so the corpus was dead and the credits were spent twice. - Phase-0 seam — decompose gathers, dimensions seed, the agent fills,
subtopic-coveragejudges. The tool never writes a status; the check refuses omission but accepts dismissal.
What holds now. Why it holds — the defects, benchmarks and rejected alternatives behind
each line — is in the ADR named beside it and in
architecture-history/.
| Invariant | Where it is decided |
|---|---|
| Search and fetch providers resolve independently; one seam, two sides | ADR-0027 |
| The fetch ledger is append-only and hash-chained; the chain is evidence, not truth | ADR-0011 |
No data reaches a shell: argv arrays, shell: false; the one Windows .cmd route refuses rather than re-quotes |
ADR-0020 |
| A path list is data on a pipe, not arguments | ADR-0020 |
Exclusivity is the kernel's: wx / O_EXCL, never check-then-act |
ADR-0020 |
| A test verdict is awaited, never printed early; each test is raced against a watchdog | ADR-0021 |
| A machine config is read in three states — absent / readable / unreadable — and an unreadable one fails closed | ADR-0020 (refines ADR-0002) |
Line endings are corpus integrity: .gitattributes pins LF in the repo and in template/ |
ADR-0020 |
| A builder machine does not collect; its collection CLIs refuse before reaching an adapter | ADR-0010, ADR-0011 |
| An unsupported prerequisite blocks; it is never a silent skip | ADR-0021 |
Validators are read-only; the only writes are fi-validate --report and property-replay, both explicitly requested |
ADR-0029 |
Validator inputs are not mutated — CI diffs conformance/ and schemas/ after every run |
ADR-0029 |
researcher-release usage errors return exit 2 from every subcommand |
— |
An artifact's authorization is derived, never supplied: artifact.mjs has no --build-authorized, refuses it (and every near spelling) with the reason rather than ignoring it, and deriveState runs the real gate with env defaulting to {} so a packaging shell's override cannot travel |
ADR-0032 |
A package being valid is not a package being authorized: status: PASS and buildAuthorized are separate fields, and the CLI says so in words on a valid collected corpus |
ADR-0032 |
A ZIP's structural faults are refused from the central directory, before inflation; the inflation ceiling is maxOutputLength, not a field the archive declared about itself |
ADR-0032 |
No dispatch input reaches a shell: ${{ inputs.x }} inside a run: block is substitution before the shell parses, so inputs arrive through env: and then an argv array. Enforced across every workflow, not only the newest |
ADR-0033, ADR-0020 |
The collector's spend gate is a protected environment whose existence is proven, because GitHub silently creates an unprotected one when a workflow names a missing environment; a missing credential refuses rather than falling back to a keyless route. The live adapter test is held to the same rule since 2026-09-26 (vars.RESEARCH_KIT_LIVE_ENV in live-collection.yml): its first dispatch named an environment nobody had created, and it failed safely only because the auto-created one was empty |
ADR-0033 |
A job that can read a credential has no cache path between runs: both paid workflows declare top-level cache-mode: none (2026-09-26), and a test refuses a job-level override. They used no cache and logged Cache mode: write, the trusted-trigger default; a denied cache step logs and continues, so it cannot fail a paid run |
ADR-0042 |
A claim's support is weighed, not just checked: corroboration reports whether a closed unknown rests on one reading, several readings of one source, a republished copy of one document across hosts (mirror, via lib/similarity.mjs), or genuinely distinct documents — it never fails a corpus on its own, because single-sourcing is often the right answer. Rows are deduped by id first: citedIds returns every E-## mention, so prose naming one row twice used to produce it twice, and once grouping folded the duplicate it reported a "republished copy" about a row that was merely mentioned twice |
ADR-0036 |
corroboration reports the closest pair it measured on an independent verdict, and does not act on it. The threshold has two measured blind spots, and containment - the obvious repair for the second - ranks a genuinely different pair (0.506) ABOVE a real mirror (0.380) over 2404 pairs, so trying harder would report second sources as copies. The number informs a reviewer; the grade does not move. A [single-witness: ...] note on an unknown that cites two sites draws single-witness-stale, worded as what it is: the check counts sites, not authors, so a note saying both pages share an author stands (2026-09-29). |
ADR-0036 |
A reviewer note is not a citation: stripReviewNotes in lib/corpus.mjs removes [single-witness: ...] and [render-reviewed: ...] before citedIds runs. Those notes are prose written INSIDE the same cell the citations are read from, so explaining why no second witness exists - by pointing at a row that demonstrates it - turned that row into a citation, and the unknown read as corroborated while its own note denied it. It caught its author three times in two days across three projects; the guard was right every time and the mechanism was the problem. The raw cell is untouched, because the check that reads the reason needs it |
ADR-0038 |
A finding a reviewer has considered and overruled is recorded, not repeated forever: [single-witness: …] in an unknown's Evidence cell and [render-reviewed: …] in an evidence row's Finding cell turn a true-but-permanent warning into a pass that prints the reason. A reason under 30 characters is named (single-witness-unreasoned) rather than obeyed, and a note left on an unknown that has since been corroborated is reported (single-witness-stale) so the acknowledgement cannot rot. The render note lives on the ROW because verifyLedger hashes the whole capture file — a capture is evidence and stays byte-immutable, a judgement about it is corpus prose |
ADR-0038 |
The collector lock is exclusive on a local filesystem, and not promised on a network one: Node states "the exclusive flag might not work with network file systems", a condition ADR-0020 never carried. Measured 2026-09-22 - 24 racers, 1 winner, 23 EEXIST on win32 - so the guarantee holds where it was tested and nothing detects where it is standing. Evidence: docs/decisions/2026-09-22-wx-lock/ |
ADR-0020 |
With a SerpAPI key present the search side asks BOTH providers and interleaves by rank (mergeByRank, lib/research-run.mjs). The meters are separate, so each is asked once rather than either more. Round-robin is the point: on 2026-09-22 one provider returned eight US financial-regulation pages for a question about EU deforestation and the other returned seventeen on topic, and concatenation would have spent the whole page budget before reaching the working one. Every merged row carries EVERY provider that returned it (providers, joined into rankedBy), not the first one the loop reached. Crediting the first made agreement look like discovery: a live run had sixteen results and nine distinct, so seven URLs came from both and all seven were attributed to whichever provider was asked first - provenance that reads as a measurement and is really an artefact of loop order. An explicitly pinned provider stays single, because pinning exists to make two runs comparable |
ADR-0027 |
| The SEARCH side is selectable at dispatch: collect.yml takes a search_transport input (auto | serpapi |
A rate limit is an instruction to wait, not a failure: rateLimitWaitMs (lib/firecrawl.mjs) reads the delay the vendor states in its own error text, and collectOne sleeps and retries, bounded at two. Firecrawl free tier allows 10 requests a minute and the collector did not throttle, so a run started inside a busy minute shed six of eight fetches and recorded them as ordinary failures. The vendor number is used rather than an invented backoff, for the reason the kit researched for Retry-After: when a server says when to come back, that is the number. Non-rate-limit errors are NOT retried - a 404 does not become a 200 if you wait. Search waits too (searchPatiently, lib/research-run.mjs): until 2026-09-26 only fetches did, and a measurement put the 11th and 12th Firecrawl searches of one minute into 429 - E-01's 10 /search per minute - which a run logged as failures, so a plan with more than ten queries lost the rest of that minute. Same vendor number, same bound of two, per provider in a merged search; a non-rate-limit search failure still costs one call and still reaches the fallback |
ADR-0005 |
A collection reports how much of the topic own vocabulary reached the captures - topicMatch in lib/research-run.mjs, printed by bin/research.mjs right after collecting and deciding nothing. It exists because a run returned eight captures about US financial regulation for a question about EU deforestation and passed every structural check: no check asks whether the evidence is about the topic. Two thresholds were measured and REJECTED - per-capture 0.5 flags RFC 9728 in the agent-interface corpus, which never says MCP and is an independent witness precisely because it does not; per-corpus at-least-one-strong flags delivery-architecture, which scores identically to the genuine failure because its topic is internal vocabulary and its evidence is GitHub docs. Telling those apart needs meaning (ADR-0013), so the number is surfaced at the moment a reviewer can act on it and the grade does not move |
ADR-0013 |
A run reports what landed separately from what it cost: collected is captures on disk, spent is budget consumed (collected + failed). They were one number under the label collected, so a run that fetched 2 pages and lost 6 to a Firecrawl rate limit reported collected 8 while its artifact carried 2. The free tier is 10 requests per minute and the collector does not throttle, so a paired run started inside the same minute sheds most of its fetches |
ADR-0005 |
A stdio server takes no options, and refuses any it is given (2026-09-28, Arena break test 12). bin/mcp-server.mjs accepted every argument and read none, so --zzz-not-a-flag printed its banner and started serving, exit 0. The entrypoint checks the whole argument vector directly, including positional arguments; refuseUnknownFlags only sees parsed named flags and returns normally for an empty flag map, even with an empty whitelist. A refused invocation reports its argument count and environment guidance without echoing supplied values (2026-10-09). The realistic harms are a client configured with --directory /some/project - the spelling other servers take - collecting somewhere else in silence, and a token passed as --token <secret>, which both puts a credential in the process table and is discarded, so every tool then refuses with "no token". |
|
An entrypoint refuses a flag it does not know rather than ignoring it: refuseUnknownFlags in lib/core.mjs is used after parseFlags by entrypoints such as research.mjs and preflight.mjs; other entrypoints enforce their own argument contracts directly. parseFlags knows no flag names by design, so an entrypoint that skips the check gets silence — the flag is parsed, never read, and the command proceeds to its default. On 2026-09-22 that default was a real collection: research.mjs --totally-made-up-flag spent 26 credits against this repository's corpus, and audit.mjs wrote twelve files. The same survey found a dead --max-scrapes in two workflows, where the page bound people believed it set was really plan.maxScrapes — real in collect.yml, and absent in live-collection.yml, whose max_pages was validated and then ignored |
ADR-0020 |
A capture that says it failed to render is reported, though it arrived whole: completeness grades the transport and cannot see a page built by script that arrives complete and empty. The rule matches sentences a page PRINTS, never a guess at whether its text looks substantial — emptiness itself was measured and found mechanically undetectable, in both directions (ADR-0037) |
ADR-0037 |
A privacy claim is measured, not asserted: bin/disclosure.mjs re-runs the probe, and a report never lets an unchecked thing read as a clean one |
ADR-0035 |
| A credential that resolves in a job declaring no environment is not behind the approval gate, and fails the run: the scope question GitHub will not answer directly is answered by asking it from a context that can see only one scope | ADR-0033 |
The vendor package name lives in lib/firecrawl.mjs (CLI_PACKAGE, cliInstallSpec), never in a workflow: the CLI is firecrawl-cli, and the npm package called firecrawl is the SDK and ships no binary. Both workflows installed the wrong one for weeks, and only a real dispatch could find it |
ADR-0005, ADR-0033 |
| A module exports only what something else imports — a test-only seam counts, a promise with no reader does not. Eleven such exports arrived with the artifact and dispatch work and were narrowed back to module scope on 2026-09-21 | — |
A template token substituted into a .json file is JSON-escaped; markdown gets it verbatim |
ADR-0033 (found by a hostile topic) |
| Supported CI platforms are Linux and Windows; macOS is best-effort and untested | README |
The project is the current working directory; there is no --project |
ADR-0017 |
| Enforcement surfaces stay at two: the git commit gate and the edit-time hook | ADR-0006 |
| The kit names no runtime: anchors are overridable, retired names are kept only to be recognised and repaired | ADR-0012 |
| Two languages answer per vector, never by comparing report hashes | ADR-0029 |
Counted against the filesystem, not estimated:
lib/ modules |
64 (59, plus the 5 in lib/release/) — every one has a row above |
bin/ entrypoints |
33, of which 4 are Python: 3 conformance runners and the module they share |
schemas/ |
12 |
conformance/ vector packets |
3 |
| checks in the registry | 13 |
| supported CI platforms | 2 |
The schemas/ count became 12 on 2026-09-21 when artifact-manifest.schema.json arrived
with the portable artifact format (ADR-0032). It is not one of the ported release schemas,
and the sentence below is about those: of the twelve the port could have brought, eleven
did.
Two artifacts are deliberately absent, which is why the ported counts are 11 and 3
rather than 12 and 4: trap-register.schema.json and dashboard-status-vectors.json exist in
the tree this layer was ported from and are referenced by nothing here — no module,
binary, test or fixture names either. A schema no code validates against drifts silently
until somebody trusts it, so they arrive with the code that needs them or not at all
(ADR-0029 rule 4).
The full-suite test count is reported by selftest.mjs and is deliberately not repeated
here. A manually maintained total went stale twice — the kit README claimed 326 when
there were 565, then 565 when there were 584. selftest.mjs now owns that number and
fails the run if the one claim that remains, in research-kit/README.md, disagrees with
it.
lib/property-replay.mjs owns immutable capture and read-only replay of failing seeded
canonical-hashing, predecessor-closure, and descendant-invalidation cases. A capture self-hashes while excluding
regressionSha256, binds the generated input hash, and is created once with wx; a later
failure for the same family, seed, and iteration cannot replace it. The property suite writes
one only after an assertion failure. bin/property-replay.mjs verifies an envelope, recreates
the exact synthetic input, and reports that one case (or a lexically ordered directory) without
rerunning the suite or mutating inputs. Profile, envelope, or generator-input-hash drift fails
closed, so a changed generator cannot claim to reproduce an older failure.
For --all, the CLI refuses a missing directory as INCOMPLETE (exit 2), naming
the path and --directory remedy. An existing empty directory explicitly reports
that no regressions were captured; directory results in JSON carry their path and
regression count. These checks remain read-only; library listing behavior is unchanged.
schemas/r29-workbook-linkage.schema.json is the strict 24-row R29 workbook-linkage
envelope; lib/r29-workbook-linkage-validator.mjs resolves each declared link only against
an explicit sealed-record catalog and current R29 pointer. It rejects cross-task members,
missing/sealed-state/hash/predecessor mismatches, bad calibration traps, the Live/Frozen warm
scope violation, and a non-ready pointer without reading arbitrary record directories or
mutating evidence.
conformance/property-graph-hash-vectors.json is the reviewable, static export of two
declared property generator cases: one canonical-hash/record-chain value and one R28–R33
graph. It carries expected closure and rejection outcomes for the base graph, a self-cycle,
and a duplicate pointer. lib/property-vector-conformance.mjs and
bin/property-vector-conformance.mjs verify it with the release-validator implementation;
bin/property_vector_conformance.py reproduces canonical hashing and graph traversal using
only the Python standard library. Both CLIs are read-only, reject malformed or duplicate-key
packets, and normalize their vector rows identically for offline comparison.
lib/fi-validator.mjs is the read-only join between the completed FI workbook,
the thirty normalized FI sidecars, and the content-addressed evidence manifest.
It validates the shipped strict schemas, hashes the exact workbook and every manifest-named
regular file under the explicit root, resolves evidence ownership and declared hashes, and
reduces case state with REOPEN > FAIL > BLOCKED > ROLLED-BACK > INCOMPLETE > PASS.
Its deliberately narrow XLSX reader reads ZIP/XML projection cells only; it never invokes an
office application or changes the source package. researcher-release fi-validate exposes the
same seam with explicit paths and writes a report only when --report is supplied.