Merge pull request #64 from StepenkoAnatoli/feat/topic-match-signal

The topic signal is reported, because both thresholds for enforcing it were wrong

Author: StepenkoAnatoli

docs/ARCHITECTURE.md

@@ -427,6 +427,7 @@ each line — is in the ADR named beside it and in
 | The collector lock is exclusive **on a local filesystem, and not promised on a network one**: Node states "the exclusive flag might not work with network file systems", a condition ADR-0020 never carried. Measured 2026-09-22 - 24 racers, 1 winner, 23 EEXIST on win32 - so the guarantee holds where it was tested and nothing detects where it is standing. Evidence: `docs/decisions/2026-09-22-wx-lock/` | ADR-0020 |
 | The SEARCH side is **selectable at dispatch**: collect.yml takes a search_transport input (auto | serpapi | firecrawl-cli) and collect-remote.mjs a matching --search-transport, passed as RESEARCH_KIT_SEARCH_TRANSPORT. ADR-0027 built the ladder and the collector never exposed it, so a dispatched run always took whatever auto-detect picked and two providers could not be compared on one topic through the kit at all. auto becomes the EMPTY STRING, falsy in the ladder, so the default is the previous behaviour exactly | ADR-0027 |
 | A **rate limit is an instruction to wait, not a failure**: `rateLimitWaitMs` (lib/firecrawl.mjs) reads the delay the vendor states in its own error text, and `collectOne` sleeps and retries, bounded at two. Firecrawl free tier allows 10 requests a minute and the collector did not throttle, so a run started inside a busy minute shed six of eight fetches and recorded them as ordinary failures. The vendor number is used rather than an invented backoff, for the reason the kit researched for Retry-After: when a server says when to come back, that is the number. Non-rate-limit errors are NOT retried - a 404 does not become a 200 if you wait | ADR-0005 |
+| A collection reports **how much of the topic own vocabulary reached the captures** - `topicMatch` in lib/research-run.mjs, printed by bin/research.mjs right after collecting and deciding nothing. It exists because a run returned eight captures about US financial regulation for a question about EU deforestation and passed every structural check: no check asks whether the evidence is about the topic. Two thresholds were measured and REJECTED - per-capture 0.5 flags RFC 9728 in the agent-interface corpus, which never says MCP and is an independent witness precisely because it does not; per-corpus at-least-one-strong flags delivery-architecture, which scores identically to the genuine failure because its topic is internal vocabulary and its evidence is GitHub docs. Telling those apart needs meaning (ADR-0013), so the number is surfaced at the moment a reviewer can act on it and the grade does not move | ADR-0013 |
 | A run reports **what landed separately from what it cost**: `collected` is captures on disk, `spent` is budget consumed (collected + failed). They were one number under the label `collected`, so a run that fetched 2 pages and lost 6 to a Firecrawl rate limit reported `collected 8` while its artifact carried 2. The free tier is 10 requests per minute and the collector does not throttle, so a paired run started inside the same minute sheds most of its fetches | ADR-0005 |
 | An entrypoint **refuses a flag it does not know** rather than ignoring it: `refuseUnknownFlags` in `lib/core.mjs`, called immediately after `parseFlags` in all eleven bins. `parseFlags` knows no flag names by design, so an entrypoint that skips the check gets silence — the flag is parsed, never read, and the command proceeds to its default. On 2026-09-22 that default was **a real collection**: `research.mjs --totally-made-up-flag` spent 26 credits against this repository's corpus, and `audit.mjs` wrote twelve files. The same survey found a dead `--max-scrapes` in two workflows, where the page bound people believed it set was really `plan.maxScrapes` — real in `collect.yml`, and **absent** in `live-collection.yml`, whose `max_pages` was validated and then ignored | ADR-0020 |
 | A capture that **says it failed to render** is reported, though it arrived whole: `completeness` grades the transport and cannot see a page built by script that arrives complete and empty. The rule matches sentences a page PRINTS, never a guess at whether its text looks substantial — emptiness itself was measured and found mechanically undetectable, in both directions ([ADR-0037](adr/0037-an-empty-capture-is-not-mechanically-detectable.md)) | [ADR-0037](adr/0037-an-empty-capture-is-not-mechanically-detectable.md) |

research-kit/README.md

@@ -395,7 +395,7 @@ node research-kit/bin/selftest.mjs            # all of it
 node research-kit/bin/selftest.mjs gate hook  # just these files
 ```
 
-861 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
+864 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
 the assertions settled (ADR-0021), and each test is raced against a watchdog
 (`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).
 

research-kit/bin/research.mjs

@@ -7,10 +7,11 @@
 //
 // `--dry-run` and `--status` still work there, because they spend nothing.
 
-import { parseFlags, flagList, refuseUnknownFlags } from '../lib/core.mjs';
+import { parseFlags, flagList, refuseUnknownFlags, resolve, readText } from '../lib/core.mjs';
 import { collectionPolicy, collectionRefusal } from '../lib/machine.mjs';
 import { selectTransport, TRANSPORT_NAMES, SEARCH_PROVIDER_NAMES } from '../lib/transport.mjs';
-import { runResearch, readPlan, usageSummary, DEPTHS, DEPTH_SCRAPES } from '../lib/research-run.mjs';
+import { runResearch, readPlan, usageSummary, topicMatch, DEPTHS, DEPTH_SCRAPES } from '../lib/research-run.mjs';
+import { parseCapture } from '../lib/corpus.mjs';
 import { heading } from '../lib/render.mjs';
 
 const { flags } = parseFlags(process.argv.slice(2));
@@ -115,8 +116,25 @@ depth      ${run.depth} (budget ${run.budget} scrapes)
 collected  ${run.collected}
 cached     ${run.cached}
 failed     ${run.failed}
-spent      ${run.spent} (budget consumed: collected + failed)
-`);
+spent      ${run.spent} (budget consumed: collected + failed)`);
+
+// The topic signal, printed at the one moment it helps: the pages are on disk and nobody
+// has read them yet. It decides nothing - see `topicMatch` for the two thresholds that were
+// measured and rejected - but a low number on a fresh collection is the prompt to check the
+// URLs, which is exactly the step that caught a wholly off-topic corpus by hand.
+//
+// Only what THIS run collected is measured. Judging the whole corpus would blend a bad
+// collection into whatever was already there and hide the thing worth seeing.
+const freshBodies = run.results
+  .filter((r) => r.status === 'collected' && r.entry?.file)
+  .map((r) => parseCapture(readText(resolve(root, r.entry.file)) ?? '').body);
+const match = topicMatch(readPlan(root, typeof flags.plan === 'string' ? flags.plan : '').topic, freshBodies);
+if (match) {
+  process.stdout.write(`topic      ${match.best.toFixed(2)} best of ${match.terms} topic terms, ${match.strong} capture(s) >= 0.50`
+    + (match.strong === 0 ? ' - NOTHING matched strongly; read the URLs before trusting this' : '')
+    + '\n');
+}
+process.stdout.write('\n');
 // The second meter reports separately, and a degradation is never silent: it means the
 // run quietly moved spend back onto the fetch budget.
 if (run.searchTransport !== run.transport) {

research-kit/lib/research-run.mjs

@@ -344,3 +344,62 @@ export function searchUsage(root, { now = new Date(), provider = '' } = {}) {
     caveat: 'counted from this machine\'s own runs; a repeat served from the provider\'s free cache is counted here but not billed',
   };
 }
+
+/**
+ * How much of the topic's own vocabulary appears in what was just collected.
+ *
+ * REPORTED, NEVER ENFORCED, and the measurement that settled that is worth keeping.
+ *
+ * It exists because a collection can return eight captures about the wrong subject and pass
+ * every structural check in the kit. On 2026-09-22 a run on the EU Deforestation Regulation
+ * returned SEC filings, CFPB regulations and California water boards - the search had
+ * matched the word "regulation" - and `preflight` had nothing to say, because no check asks
+ * whether the evidence is about the topic.
+ *
+ * Two thresholds were measured against that failure and every committed corpus here:
+ *
+ *   PER CAPTURE, share >= 0.5: rejected. It flags RFC 9728 in the agent-interface corpus,
+ *   which never says "MCP" - and not saying it is exactly what makes it an independent
+ *   witness. The rule would punish the best evidence in the corpus.
+ *
+ *   PER CORPUS, at least one capture >= 0.5: rejected too, and this is the decisive one.
+ *   `delivery-architecture` scores max 0.25 with zero strong captures - IDENTICAL to the
+ *   failure - because its topic is "How Research-Kit should be delivered to a non-technical
+ *   user" and its evidence is GitHub Actions documentation. Semantically related, lexically
+ *   disjoint. No threshold separates that from a corpus about the wrong subject entirely.
+ *
+ * Telling those apart needs meaning, which ADR-0013 refuses. So the number is printed at the
+ * moment it helps - right after collecting, before anyone has read the pages - and decides
+ * nothing. A reviewer seeing 0.25 on a fresh collection knows to check the URLs; on the EUDR
+ * run that is precisely the step that caught it, performed by hand.
+ */
+const TOPIC_STOPWORDS = new Set(['the', 'a', 'an', 'of', 'for', 'and', 'or', 'to', 'in', 'on',
+  'at', 'by', 'after', 'before', 'current', 'date', 'with', 'from', 'is', 'are', 'as', 'eu',
+  'its', 'it', 'this', 'that', 'how', 'what', 'when', 'does', 'do', 'can', 'not', 'without',
+  'versus', 'vs', 'should', 'be', 'was', 'were', 'has', 'have', 'than', 'they', 'their']);
+
+export function topicTerms(topic) {
+  return [...new Set(String(topic ?? '').toLowerCase()
+    .replace(/[^a-z0-9/.-]+/g, ' ')
+    .split(/\s+/)
+    .filter(Boolean))]
+    .filter((word) => word.length > 2 && !TOPIC_STOPWORDS.has(word));
+}
+
+/**
+ * `{ terms, best, strong }` over the given capture bodies, or `null` when the topic carries
+ * too few distinctive terms to say anything - which is not the same as a low score.
+ */
+export function topicMatch(topic, bodies) {
+  const terms = topicTerms(topic);
+  if (terms.length < 3 || !bodies.length) return null;
+  const shares = bodies.map((body) => {
+    const lower = String(body ?? '').toLowerCase();
+    return terms.filter((term) => lower.includes(term)).length / terms.length;
+  });
+  return {
+    terms: terms.length,
+    best: Math.max(...shares),
+    strong: shares.filter((share) => share >= 0.5).length,
+  };
+}

research-kit/test/collect.test.mjs

@@ -7,6 +7,7 @@ import { readCorpus, parseTable } from '../lib/corpus.mjs';
 import { HEADERS } from '../lib/core.mjs';
 import { collectOne, writeRaw, captureName, bodyHashOf } from '../lib/collect.mjs';
 import { rateLimitWaitMs } from '../lib/firecrawl.mjs';
+import { topicMatch } from '../lib/research-run.mjs';
 import { verifyLedger } from '../lib/provenance.mjs';
 import { runResearch, readPlan, rankCandidate, selectCandidates, DEPTH_SCRAPES, usageSummary } from '../lib/research-run.mjs';
 
@@ -331,3 +332,56 @@ test('collectOne gives up after a bounded number of rate-limit retries', () => {
   assert.equal(outcome.status, 'failed');
   assert.match(outcome.reason, /Rate limit exceeded/);
 });
+
+// ---------------------------------------------------------------- topic match
+
+test('topicMatch separates the off-topic collection from the on-topic one', () => {
+  // The real pair, from the same query on 2026-09-22: SerpApi returned eight pages about US
+  // financial regulation, Firecrawl returned eight about the EU Deforestation Regulation.
+  // The off-topic corpus passed every structural check in this kit.
+  const topic = 'EU Deforestation Regulation 2023/1115 application date large operators SMEs current after delay';
+
+  const offTopic = [
+    'Regulation D exempt offerings for small businesses. The application date for filing...',
+    'Regulation Z truth in lending. Large creditors must apply...',
+    'California water efficiency legislation and conservation regulation...',
+  ];
+  const onTopic = [
+    'The EUDR, Regulation (EU) 2023/1115 on deforestation-free products. Application for large operators and SMEs...',
+    'EU Deforestation Regulation 2023/1115: application postponed. Large operators and SMEs face a delay...',
+  ];
+
+  const off = topicMatch(topic, offTopic);
+  const on = topicMatch(topic, onTopic);
+  assert.equal(off.strong, 0, 'nothing in the off-topic set should match strongly');
+  assert.ok(on.strong > 0, 'the on-topic set must match');
+  assert.ok(on.best > off.best, `on-topic must outscore off-topic: ${on.best} vs ${off.best}`);
+});
+
+test('topicMatch declines to judge a topic with too few distinctive terms', () => {
+  // Refusing is not scoring zero. A two-word topic makes the share 0, 0.5 or 1 and the
+  // number means nothing - reporting it would invite a reader to act on noise.
+  assert.equal(topicMatch('the EU', ['anything at all']), null);
+  assert.equal(topicMatch('', ['x']), null);
+  assert.equal(topicMatch('EU Deforestation Regulation dates', []), null, 'no captures, nothing to measure');
+});
+
+test('topicMatch REPORTS and never decides - the thresholds that were rejected', () => {
+  // Pinned so nobody promotes this into a gate later. Measured 2026-09-22 across every
+  // corpus in this repository:
+  //
+  //   per-capture >= 0.5 flags RFC 9728 in the agent-interface corpus, which never says
+  //   "MCP" - and not saying it is what makes it an independent witness.
+  //
+  //   per-corpus "at least one strong capture" flags delivery-architecture, whose topic is
+  //   "How Research-Kit should be delivered to a non-technical user" and whose evidence is
+  //   GitHub Actions docs. It scores max 0.25 with zero strong captures - IDENTICAL to the
+  //   genuine failure. No threshold separates them.
+  //
+  // So the shape of the return value is the contract: numbers, no verdict.
+  const m = topicMatch('EU Deforestation Regulation 2023/1115 operators', ['deforestation regulation for operators']);
+  assert.deepEqual(Object.keys(m).sort(), ['best', 'strong', 'terms']);
+  assert.equal(typeof m.best, 'number');
+  assert.equal(m.pass, undefined, 'a pass/fail field would make this a gate');
+  assert.equal(m.severity, undefined);
+});