Merge pull request #69 from StepenkoAnatoli/feat/registered-prior

A prediction is only a prediction if it was chained first

Author: StepenkoAnatoli

.github/workflows/collect.yml

@@ -91,6 +91,10 @@ on:
         default: 'quick'
         type: choice
         options: [probe, quick, normal]
+      prior:
+        description: 'What you EXPECT the evidence to say, and what you know you cannot know yet. Optional, and this is the only moment it can be registered - it is chained into the ledger ahead of the first page and refused afterwards. Nothing grades it; being wrong is the point (ADR-0039). Same disclosure as topic: not in any name, and not in an unauthenticated response.'
+        required: false
+        default: ''
       client_ref:
         description: 'PUBLIC on a public repository - it becomes the run name and the artifact name. Use an opaque id (job-0417), never the subject. Optional: correlation does not need it - pin X-GitHub-Api-Version and read workflow_run_id. Shape [A-Za-z0-9][A-Za-z0-9._-]{0,63}.'
         required: false
@@ -399,6 +403,33 @@ jobs:
             execFileSync(process.execPath, [`${process.env.KIT}/bin/decompose.mjs`, "--topic", process.env.TOPIC, "--dry-run"], { stdio: "inherit" });
           '
 
+      # The prediction, chained BEFORE the first page - which is the only place it can go.
+      #
+      # This step exists because the mechanism was nearly useless without it. A prior
+      # registered on the operator's laptop cannot reach this corpus: the runner scaffolds a
+      # fresh project above, with an empty ledger, so the two corpora have nothing in common.
+      # Every corpus in this repository since the remote collector shipped was collected on
+      # THIS path, so a prior that only worked locally would have been a discipline for a
+      # route nobody takes.
+      #
+      # Like TOPIC, it reaches node through the ENVIRONMENT and then an argv ARRAY, never
+      # through the expression engine and never concatenated into a command line (ADR-0020).
+      # Its disclosure profile is the topic's, measured in ADR-0035: dispatch inputs did not
+      # appear in any unauthenticated response, and it reaches no run name or artifact name.
+      # It is NOT a credential and must never carry one - it is a sentence about what you
+      # expect, published in the artifact that comes back.
+      - name: register the prior, if one was dispatched
+        if: inputs.prior != ''
+        env:
+          PRIOR: ${{ inputs.prior }}
+        run: |
+          set -e
+          cd "$SCRATCH"
+          node -e '
+            const { execFileSync } = require("node:child_process");
+            execFileSync(process.execPath, [`${process.env.KIT}/bin/prior.mjs`, process.env.PRIOR], { stdio: "inherit" });
+          '
+
       - name: collect
         env:
           FIRECRAWL_API_KEY: ${{ secrets.FIRECRAWL_API_KEY }}

docs/ARCHITECTURE.md

@@ -145,6 +145,7 @@ else, and it never touches the corpus:
 | `decompose.mjs` | Phase 0 coordination: recipe parsing, docs-host discovery, material gathering (scrape-budget accounting via the corpus's `cacheDecision` — a cache hit is not an attempt), draft MAP writing. Contains **no judgment** — it seeds the checklist and gathers material; it never writes a status. | — |
 | `dimensions.mjs` | The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (`coverageOfUniversals`). Recipes add on top of it; nothing replaces it. | — |
 | `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). | ADR-0003 (corpus-adjacent), ADR-0020 | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not.
+| `prior.mjs` | **The registered prior: what was expected BEFORE the evidence, and the proof that it was.** `registerPrior` writes `research/PRIOR.md` and chains an `op: 'prior'` entry carrying its sha256, so two things stop being editable at once - the text, through `body-unmodified`, and its POSITION before the first scrape, through the chain itself. Three dishonest paths, each measured rather than argued: revising the prediction once the answer is known fails the chain; registering late is refused by the tool; and chaining one late BY HAND with every hash correct is caught by `provenance/prior-precedes-collection`, which exists because `appendFetch` is a public export and the gate does not get to trust that the CLI was used. **Nothing grades the prediction** (ADR-0013) - not its specificity, not its reasonableness, and above all not whether it turned out right, since a prior that was wrong has done its job and a gate that preferred correct ones would teach the opposite habit. The single content rule is a length floor, stated as a floor. Absence is SILENT at the gate: a prior is optional, and a warning on every project that skipped one is a warning nobody reads. `bin/prior.mjs` registers and reads it back; `bin/research.mjs` prints a one-line nudge before a spending run, because after the first page lands there is no second chance. **It also reaches the REMOTE path, and that half is what makes the rest useful**: the collector workflow scaffolds its own project on the runner with an empty ledger, so a prior registered on the operator machine cannot reach the corpus that comes back - and every corpus collected since `collect-remote` shipped came from that route. The prediction therefore travels as a dispatch input and is registered on the runner between the map and the collection, through `env` and an argv array like the topic (ADR-0020), with the topic's measured disclosure profile (ADR-0035). A test pins that the registration step precedes `collect`, because if it ever drifts below it, `bin/prior.mjs` refuses and the feature stops working silently on a workflow CI never runs. | ADR-0013, ADR-0020, ADR-0039 |
 | `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. | ADR-0004, ADR-0026 |
 | `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 |
 | `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 |
@@ -450,8 +451,8 @@ Counted against the filesystem, not estimated:
 
 | | |
 |---|---|
-| `lib/` modules | **49** (45, plus the 4 extracted into `lib/release/`) — every one has a row above |
-| `bin/` entrypoints | **29**, of which **4 are Python**: 3 conformance runners and the module they share |
+| `lib/` modules | **50** (46, plus the 4 extracted into `lib/release/`) — every one has a row above |
+| `bin/` entrypoints | **30**, of which **4 are Python**: 3 conformance runners and the module they share |
 | `schemas/` | **12** |
 | `conformance/` vector packets | **3** |
 | checks in the registry | **13** |

docs/adr/0039-a-prediction-is-only-a-prediction-if-it-was-chained-first.md

@@ -0,0 +1,106 @@
+# ADR-0039 — A prediction is only a prediction if it was chained first
+
+- **Date:** 2026-09-22
+- **Status:** accepted
+- **Area:** prior, provenance, brief
+- **Depends on:** ADR-0003 (the hash-chained ledger), ADR-0013 (the gate does not grade reasoning)
+- **Evidence:** the three corpora named below, and the measurements in this file
+
+## Context
+
+Three corpora in this repository carry a prediction written before collection, and in each
+case the prediction is worth more than the finding that follows it.
+
+The **EUDR** prior was a year wrong on both dates. That is the only reason its brief can say
+the corpus supplied something the reviewer could not have — without the prior on record, the
+brief would have been eight captures agreeing with each other and no evidence that anyone's
+belief had moved. The **Tavily** prior held. That is the only reason its brief can say so
+without it reading as a boast, because the claim is checkable rather than remembered.
+
+The habit was mine and it was unenforced. It has exactly one failure mode, and it is not
+forgetting.
+
+**It is writing the prediction afterwards, once the answer is known, and remembering it as
+foresight.** Nobody has to be dishonest for that to happen. "I thought that" is a memory,
+and memories reshape themselves around what turned out to be true. An unenforced prior is
+therefore not weak evidence of foresight — it is no evidence at all, because the honest
+version and the reconstructed version are the same file.
+
+## Decision
+
+A prior is **optional**, and when it exists its ORDER is **cryptographic, not procedural**.
+
+1. The prediction lives in `research/PRIOR.md`, outside `research/raw/` so it is never read
+   as a capture.
+2. Registering it appends an `op: 'prior'` entry to the hash-chained fetch ledger, carrying
+   the file's sha256 in `bodySha256` and its path in `raw`.
+3. Two things stop being editable at that moment, and neither needed new machinery:
+   - the **text**, because `verifyLedger` already hashes anything a ledger entry names, so
+     revising the prediction fails `provenance/body-unmodified`;
+   - the **position**, because every later entry's `prev` is the previous entry's
+     `entrySha256`, so moving a prior after a scrape rewrites every hash that follows it.
+4. `provenance/prior-precedes-collection` fails a prior registered after a scrape.
+5. The drafted brief quotes it verbatim, above `## Intent`.
+
+## The three paths, measured
+
+Not reasoned. Each was performed against a real corpus and the result recorded.
+
+| path | what happened |
+|---|---|
+| Edit `PRIOR.md` after registering | `fail provenance/body-unmodified` — *does not match the hash recorded at fetch* |
+| Register after collecting, via `bin/prior.mjs` | refused, exit 2, nothing written |
+| Chain a prior at seq 2 **by hand**, every hash correct | chain verifies, and `fail provenance/prior-precedes-collection` |
+
+The third is why the check exists at all. `appendFetch` is a public export and will happily
+chain a prior after the evidence with a perfectly valid chain — so the gate verifies the
+order itself rather than trusting that the tool was used. Without it, the first two refusals
+would be a fence with a gate left open beside it.
+
+## What is deliberately not checked
+
+**Nothing grades the prediction.** Not whether it was specific, not whether it was
+reasonable, and above all not whether it turned out right.
+
+This is ADR-0013 applied to the one place where violating it would be most tempting and most
+destructive. A gate that preferred correct priors would teach the exact opposite of the
+habit: it would reward hedging, vagueness, and predicting the safe thing. **A prior that was
+wrong has done its job** — the EUDR one is the most valuable artefact in that corpus
+precisely because it was wrong.
+
+The single content rule is a **length floor** of 80 characters, and it is stated as a floor
+rather than a judgement. It stops `"fine"` from being a registered prediction. It cannot
+stop `"fine, and here are ninety more characters of nothing"`, and it is not meant to.
+
+## Why absence is silent
+
+A project with no prior produces **no finding at all** — not a warning, not a passing one.
+
+A warning that fires on every project that skipped an optional step is a warning nobody
+reads, and it would turn the habit into a toll. This repository already learned that from
+`partial-render`, which stood permanently on three captures it actively relied on until the
+review could be recorded and closed.
+
+The nudge lives where it can still be acted on instead: `bin/research.mjs` prints one line
+before a spending run, because after the first page lands there is no second chance.
+
+## Why the brief carries it
+
+A prior left in its own file is a file nobody opens. The brief is the one document phase 2
+is required to read, so the drafter emits the prediction there, above the findings.
+
+That placement also removes the quiet edit. An author who has just learned they were wrong
+does not have to decide whether to mention it — the text is already on the page, and the
+ledger already fixed it. The two cases, confirmed and demolished, look identical to the
+renderer, which is what keeps it from becoming a scoreboard.
+
+## Consequences
+
+- One prior per corpus. A prior you can re-register is a prior you can retry until it is
+  right. Under ADR-0030 each decision gets its own nested project, so each gets its own
+  prediction; a project that opens a genuinely new question after collecting should say so
+  in the brief, where it is labelled as hindsight.
+- The corpora that already exist have no registered prior and never will — the mechanism
+  cannot be applied retroactively, which is the whole point. Their priors stay where they
+  are, in prose, and should be read as the weaker thing they are.
+- `usageSummary.ledgerEntries` counts the prior, because it is one.

research-kit/README.md

@@ -60,6 +60,7 @@ The project is the **current working directory**. The kit takes no project argum
 node research-kit/bin/new-project.mjs . --topic "<topic>"   # the canonical shape
 node research-kit/bin/decompose.mjs --topic "<topic>"       # phase 0: the map, statuses blank
                                                             # then YOU mark COVERED/DISMISSED/GAP
+node research-kit/bin/prior.mjs "<what you expect>"         # optional, and only possible NOW
 node research-kit/bin/research.mjs                          # collect (spends credits)
 node research-kit/bin/preflight.mjs                         # do not build until PASS
 node research-kit/bin/brief.mjs                             # the phase-1 -> phase-2 handoff
@@ -80,6 +81,7 @@ node research-kit/bin/handoff.mjs     # did the corpus arrive whole?
 | `preflight.mjs` | the verdict (`--checks`, `--check <name>`, `--strict`, `--json`) |
 | `gate.mjs` | the verdict for a hook (`--gate commit\|edit`, `--staged-stdin`, `--posture`) |
 | `research.mjs` | collect (`--depth`, `--refresh-days`, `--force`, `--dry-run`, `--status`, `--transport`) |
+| `prior.mjs` | register what you expect **before** collecting, chained so it cannot be moved or rewritten afterwards; with no argument, prints the one on record |
 | `decompose.mjs` | phase 0 (`--topic`, `--recipe`, `--recipes`, `--max-scrapes`, `--dry-run`) |
 | `handoff.mjs` | the arrival question, with the remedy picked from the cause |
 | `brief.mjs` | draft the handoff (`--state`, `--force`) |
@@ -395,7 +397,7 @@ node research-kit/bin/selftest.mjs            # all of it
 node research-kit/bin/selftest.mjs gate hook  # just these files
 ```
 
-874 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
+888 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
 the assertions settled (ADR-0021), and each test is raced against a watchdog
 (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).