Merge pull request #18 from StepenkoAnatoli/fix/live-predicate-is-integrity-not-sufficiency

Run the live predicate for real, and fix what it could never have passed

Author: StepenkoAnatoli

.github/workflows/live-collection.yml

@@ -18,10 +18,21 @@ name: live-collection
 #   1. the vendor CLI is a major this adapter supports
 #   2. the collection command exited 0
 #   3. a ledger was written, and it parses
-#   4. handoff exits 0        - the corpus arrived whole
-#   5. preflight exits 0      - the corpus passes the same checks a synthetic one does
+#   4. handoff exits 0                  - the corpus arrived whole
+#   5. the INTEGRITY checks pass        - provenance, corpus-shape, citations,
+#                                         transport-provenance, hygiene
 #   6. the repository is unchanged
 #
+# NOT the whole preflight verdict, and this was found by running the predicate for real
+# rather than by reading it. `preflight` answers "is the RESEARCH sufficient to build",
+# which is a human-authored question: a scratch project has no unknowns in its contract,
+# so it fails `discovery-contract/no-unknowns` and exits 1 no matter how perfectly
+# collection worked. Requiring it would have made this workflow fail on every run, always,
+# for a reason having nothing to do with the transport it exists to test.
+#
+# The checks above are the ones collection can actually answer: the chain verifies, every
+# cited capture was fetched and is on disk, the corpus parses, rows carry URLs and claims.
+#
 # Diagnostics are still captured when a step fails, but they are captured in a way that
 # preserves the exit status rather than discarding it.
 #
@@ -191,7 +202,7 @@ jobs:
             console.log(`${parsed.length} ledger entries, ${fetched.length} successful`);
           ' "$ledger"
 
-      - name: the corpus arrived whole, and passes its own checks
+      - name: the corpus arrived whole, and passes its integrity checks
         run: |
           set -e
           cd "$RUNNER_TEMP/live-project"
@@ -205,15 +216,19 @@ jobs:
             echo "::error::handoff failed: the collected corpus did not arrive whole"
             exit 1
           fi
-          if ! node "$kit/bin/preflight.mjs" --json > preflight.json; then
-            echo "::error::preflight failed on the collected corpus"
-            cat preflight.json || true
-            exit 1
-          fi
-          node -e '
-            const r = JSON.parse(require("fs").readFileSync("preflight.json", "utf8"));
-            console.log(`preflight: ${r.status ?? r.verdict}`);
-          '
+          # The INTEGRITY checks, one at a time, each required.
+          #
+          # Not the whole verdict: preflight also judges whether the research is
+          # sufficient to build, and a scratch project has no unknowns in its contract -
+          # so it fails and exits 1 however well collection worked. Verified by running
+          # it: `discovery-contract/no-unknowns` blocks a freshly scaffolded project.
+          for check in provenance corpus-shape citations transport-provenance hygiene; do
+            if ! node "$kit/bin/preflight.mjs" --check "$check"; then
+              echo "::error::integrity check '$check' failed on the collected corpus"
+              exit 1
+            fi
+          done
+          echo "all integrity checks passed on the collected corpus"
 
       - name: the repository is unchanged
         run: |

docs/live-predicate-verification-2026-09-21.md

@@ -0,0 +1,86 @@
+# Live predicate, verified over a real network — 2026-09-21
+
+`live-collection` was written to verify a real collection and had never been run. This
+records what happened when its success predicate was executed for real, keyless, against
+live sites — and the defect that found.
+
+**Status: local keyless live predicate verified.** Not "live-collection verified". The
+distinction is the point of this file.
+
+## Method
+
+A scratch project outside the repository, `--transport http-keyless`, two direct URLs and
+no search provider, so nothing was spent. Each condition of the workflow's predicate was
+run and its exit code read.
+
+## The defect: the predicate asked a question collection cannot answer
+
+The workflow required **`preflight` to exit 0** on the scratch project it collects into.
+Preflight exits **1** there, and correctly:
+
+```
+fail  discovery-contract/no-unknowns   the contract enumerates no blocking unknowns -
+                                       a contract with no unknowns makes no claim
+FAIL  1 blocking, 4 warning(s), 8 passing
+```
+
+Preflight answers *"is the research sufficient to build?"* — a **human-authored**
+question. A freshly scaffolded project has no unknowns in its contract, so it fails that
+check however perfectly collection worked. **The workflow would have failed on every run,
+always, for a reason having nothing to do with the transport it exists to test.**
+
+Reading the workflow would not have found this. Running it did, on the first attempt, and
+it cost nothing because the keyless route is free.
+
+### The correction
+
+The predicate now requires the checks collection *can* answer — the integrity ones, each
+required individually:
+
+| check | what it asserts |
+|---|---|
+| `provenance` | the chain verifies and every cited capture was actually fetched |
+| `corpus-shape` | the corpus parses and the shape is there |
+| `citations` | every evidence row has a URL, a claim, and a cached page behind it |
+| `transport-provenance` | which adapter fetched each cited capture |
+| `hygiene` | duplicate rows, uncited captures, unparseable dates |
+
+All five pass on a freshly collected scratch corpus. A test now refuses any executable
+line invoking `preflight.mjs` without `--check`.
+
+## What ran, and what it returned
+
+| condition | result |
+|---|---|
+| scaffold a scratch project | ok |
+| `research.mjs --dry-run` | exit 0 |
+| `research.mjs` (2 pages, keyless, real network) | exit 0, 2 collected, 0 failed |
+| ledger exists, parses, records a successful fetch | 2 entries, 2 successful |
+| `handoff.mjs` | exit 0 — "every cited capture on disk, chain verifies" |
+| the five integrity checks | all exit 0 |
+| repository unchanged | no change under `research/` |
+
+Both captures graded `partial`, which the keyless adapter reports honestly rather than
+claiming `full` — itself a small confirmation that the completeness grading survives a
+real fetch.
+
+## What this does and does not establish
+
+**Established, over a real network:**
+
+- the predicate logic is satisfiable by a real corpus, not only by fixtures
+- real network behaviour reaches the collector, and the ledger records it
+- ledger → handoff → integrity-check integration holds outside synthetic tests
+- the "repository unchanged" guard is meaningful: collection wrote only to the scratch path
+
+**Still unproven, and not claimed:**
+
+- the Firecrawl CLI install path in the workflow
+- `cliCompatibility` against a real installed CLI in CI
+- metered credential handling
+- the GitHub environment secret wiring
+- the Actions runner path itself — `workflow_dispatch` has never fired
+
+The gap is now "the CI wiring and the metered route are untested" rather than "the whole
+thing is untested", which is smaller and specific. Closing the rest needs the
+`live-collection` environment and one real dispatch.

research-kit/README.md

@@ -197,7 +197,7 @@ node research-kit/bin/selftest.mjs            # all of it
 node research-kit/bin/selftest.mjs gate hook  # just these files
 ```
 
-644 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
+645 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
 the assertions settled (ADR-0021), and each test is raced against a watchdog
 (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).
 

research-kit/test/architecture-map.test.mjs

@@ -286,3 +286,32 @@ test('the live workflow does not swallow the checks that decide its result', ()
     'the summary prints a URL prefix, which discloses what was being researched');
   assert(/hostname/.test(yaml), 'the summary should reduce URLs to hostnames');
 });
+
+test('the live workflow requires integrity, not research sufficiency', () => {
+  // Found by RUNNING the predicate, not by reading it.
+  //
+  // The workflow originally required `preflight` to exit 0 on the scratch project it
+  // collects into. Preflight answers "is the RESEARCH sufficient to build" - a
+  // human-authored question - so a freshly scaffolded project fails
+  // `discovery-contract/no-unknowns` and exits 1 however perfectly collection worked.
+  // The workflow would have failed on every run, always, for a reason having nothing to
+  // do with the transport it exists to test.
+  //
+  // What collection CAN answer is integrity: the chain verifies, captures are on disk,
+  // the corpus parses. Those are required per-check instead.
+  const file = path.join(REPO, '.github', 'workflows', 'live-collection.yml');
+  if (!fs.existsSync(file)) return;
+  const yaml = fs.readFileSync(file, 'utf8');
+  const executable = yaml.split('\n').filter((line) => !line.trim().startsWith('#'));
+
+  const wholeVerdict = executable.filter((line) => /preflight\.mjs/.test(line) && !/--check/.test(line));
+  assertEqual(wholeVerdict.length, 0,
+    'the workflow requires the whole preflight verdict from a scratch project, which can '
+    + `never satisfy it - it has no unknowns to prove:\n  ${wholeVerdict.join('\n  ')}`);
+
+  assert(/--check "\$check"|--check \$check/.test(yaml),
+    'the workflow should require named integrity checks rather than the overall verdict');
+  for (const check of ['provenance', 'corpus-shape', 'citations']) {
+    assert(yaml.includes(check), `the integrity predicate no longer names ${check}`);
+  }
+});