Skip to content

Commit ad9111e

Browse files
Merge pull request #66 from StepenkoAnatoli/feat/merge-search-providers
Ask both search providers and interleave by rank
2 parents 6232c9e + c27b101 commit ad9111e

6 files changed

Lines changed: 157 additions & 4 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -426,6 +426,7 @@ each line — is in the ADR named beside it and in
426426
| A reviewer note is **not a citation**: `stripReviewNotes` in lib/corpus.mjs removes `[single-witness: ...]` and `[render-reviewed: ...]` before `citedIds` runs. Those notes are prose written INSIDE the same cell the citations are read from, so explaining why no second witness exists - by pointing at a row that demonstrates it - turned that row into a citation, and the unknown read as corroborated while its own note denied it. It caught its author three times in two days across three projects; the guard was right every time and the mechanism was the problem. The raw cell is untouched, because the check that reads the reason needs it | ADR-0038 |
427427
| A finding a reviewer has **considered and overruled** is recorded, not repeated forever: `[single-witness: …]` in an unknown's Evidence cell and `[render-reviewed: …]` in an evidence row's Finding cell turn a true-but-permanent warning into a pass **that prints the reason**. A reason under 30 characters is named (`single-witness-unreasoned`) rather than obeyed, and a note left on an unknown that has since been corroborated is reported (`single-witness-stale`) so the acknowledgement cannot rot. The render note lives on the ROW because `verifyLedger` hashes the whole capture file — a capture is evidence and stays byte-immutable, a judgement about it is corpus prose | [ADR-0038](adr/0038-a-judgement-the-gate-can-read.md) |
428428
| The collector lock is exclusive **on a local filesystem, and not promised on a network one**: Node states "the exclusive flag might not work with network file systems", a condition ADR-0020 never carried. Measured 2026-09-22 - 24 racers, 1 winner, 23 EEXIST on win32 - so the guarantee holds where it was tested and nothing detects where it is standing. Evidence: `docs/decisions/2026-09-22-wx-lock/` | ADR-0020 |
429+
| With a SerpAPI key present the search side asks **BOTH providers and interleaves by rank** (`mergeByRank`, lib/research-run.mjs). The meters are separate, so each is asked once rather than either more. Round-robin is the point: on 2026-09-22 one provider returned eight US financial-regulation pages for a question about EU deforestation and the other returned seventeen on topic, and concatenation would have spent the whole page budget before reaching the working one. Every merged row carries the provider that surfaced it, so the ledger keeps a claim about search quality checkable. An explicitly pinned provider stays single, because pinning exists to make two runs comparable | ADR-0027 |
429430
| The SEARCH side is **selectable at dispatch**: collect.yml takes a search_transport input (auto | serpapi | firecrawl-cli) and collect-remote.mjs a matching --search-transport, passed as RESEARCH_KIT_SEARCH_TRANSPORT. ADR-0027 built the ladder and the collector never exposed it, so a dispatched run always took whatever auto-detect picked and two providers could not be compared on one topic through the kit at all. auto becomes the EMPTY STRING, falsy in the ladder, so the default is the previous behaviour exactly | ADR-0027 |
430431
| A **rate limit is an instruction to wait, not a failure**: `rateLimitWaitMs` (lib/firecrawl.mjs) reads the delay the vendor states in its own error text, and `collectOne` sleeps and retries, bounded at two. Firecrawl free tier allows 10 requests a minute and the collector did not throttle, so a run started inside a busy minute shed six of eight fetches and recorded them as ordinary failures. The vendor number is used rather than an invented backoff, for the reason the kit researched for Retry-After: when a server says when to come back, that is the number. Non-rate-limit errors are NOT retried - a 404 does not become a 200 if you wait | ADR-0005 |
431432
| A collection reports **how much of the topic own vocabulary reached the captures** - `topicMatch` in lib/research-run.mjs, printed by bin/research.mjs right after collecting and deciding nothing. It exists because a run returned eight captures about US financial regulation for a question about EU deforestation and passed every structural check: no check asks whether the evidence is about the topic. Two thresholds were measured and REJECTED - per-capture 0.5 flags RFC 9728 in the agent-interface corpus, which never says MCP and is an independent witness precisely because it does not; per-corpus at-least-one-strong flags delivery-architecture, which scores identically to the genuine failure because its topic is internal vocabulary and its evidence is GitHub docs. Telling those apart needs meaning (ADR-0013), so the number is surfaced at the moment a reviewer can act on it and the grade does not move | ADR-0013 |

‎research-kit/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -395,7 +395,7 @@ node research-kit/bin/selftest.mjs # all of it
395395
node research-kit/bin/selftest.mjs gate hook # just these files
396396
```
397397

398-
867 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
398+
872 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
399399
the assertions settled (ADR-0021), and each test is raced against a watchdog
400400
(`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).
401401

‎research-kit/bin/research.mjs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,7 @@ if (!chosen.search.sameAsFetch) {
102102
const run = runResearch(root, {
103103
adapter: chosen.adapter,
104104
searchAdapter: chosen.search.adapter,
105+
searchAdapters: chosen.search.adapters ?? null,
105106
plan: typeof flags.plan === 'string' ? readPlan(root, flags.plan) : null,
106107
depth: typeof flags.depth === 'string' ? flags.depth : '',
107108
refreshDays: flags['refresh-days'] === undefined ? null : Number(flags['refresh-days']),

‎research-kit/lib/research-run.mjs‎

Lines changed: 76 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,39 @@ export function rankCandidate(url, { prefer = [], why = '' } = {}) {
6363
return value;
6464
}
6565

66+
/**
67+
* Interleave several providers' results by RANK, not by concatenation.
68+
*
69+
* WHY RANK AND NOT ORDER. On 2026-09-22 a search for the EU Deforestation Regulation
70+
* returned, from one provider, eight pages about US financial regulation and nothing about
71+
* the subject - while the other returned seventeen, all on topic. Concatenating would have
72+
* let the failing provider spend the entire page budget before the working one was reached.
73+
* Round-robin bounds that: each provider contributes its best result, then its second, and
74+
* a provider that is wholly wrong costs at most its share.
75+
*
76+
* `provider` is carried on every row so the ledger records WHICH search surfaced a URL -
77+
* that is how a claim about search quality stays checkable afterwards instead of being an
78+
* impression.
79+
*
80+
* Duplicates are the interesting case rather than a nuisance: when both providers return the
81+
* same URL the first occurrence wins and keeps its finder, so the count of rows attributed
82+
* to each provider is not inflated by agreement.
83+
*/
84+
export function mergeByRank(lists) {
85+
const merged = [];
86+
const taken = new Set();
87+
const depth = Math.max(0, ...lists.map((l) => l.results.length));
88+
for (let rank = 0; rank < depth; rank += 1) {
89+
for (const list of lists) {
90+
const row = list.results[rank];
91+
if (!row?.url || taken.has(row.url)) continue;
92+
taken.add(row.url);
93+
merged.push({ ...row, provider: list.provider });
94+
}
95+
}
96+
return merged;
97+
}
98+
6699
export function selectCandidates(results, { prefer = [], perQuery = 3, seen = new Set() } = {}) {
67100
return results
68101
.map((row) => ({ ...row, score: rankCandidate(row.url, { prefer }) }))
@@ -83,6 +116,8 @@ export function runResearch(root, {
83116
// The SEARCH side (ADR-0027). Absent means "the fetch adapter", which is what every
84117
// caller did before the split - so an un-updated caller behaves exactly as before.
85118
searchAdapter = null,
119+
// Several providers: asked in parallel and interleaved by rank. One stays one.
120+
searchAdapters = null,
86121
plan = null,
87122
depth = '',
88123
refreshDays = null,
@@ -147,7 +182,8 @@ ${compatibility.remedy}`);
147182
let searchFailures = 0;
148183
let degraded = 0;
149184

150-
const searcher = searchAdapter ?? adapter;
185+
const searchers = (searchAdapters ?? []).filter(Boolean);
186+
const searcher = searchAdapter ?? searchers[0] ?? adapter;
151187
const searchName = searcher?.name ?? adapter?.name ?? '';
152188

153189
const seen = new Set(corpus.captures.entries.map((e) => e.url).filter(Boolean));
@@ -173,6 +209,45 @@ ${compatibility.remedy}`);
173209
discovered.push({ query: text, results: [], note: 'search not run under --dry-run' });
174210
continue;
175211
}
212+
// MORE THAN ONE PROVIDER: ask each, interleave by rank, and attribute every row.
213+
// The meters are genuinely separate (ADR-0027), so this costs one search on each rather
214+
// than more of either. A provider that fails here does not stop the run - the others
215+
// still contribute, and the failure is recorded like any other.
216+
if (searchers.length > 1) {
217+
const lists = [];
218+
for (const one of searchers) {
219+
const r = one.search(text, { limit: settings.limit });
220+
if (Number.isFinite(r.searchesUsed)) searchesUsed += r.searchesUsed;
221+
if (!r.ok) {
222+
searchFailures += 1;
223+
log(` search failed on ${one.name}: ${r.error}`);
224+
appendJsonLine(root, PATHS.failures, {
225+
at: new Date().toISOString(), op: 'search', query: text, provider: one.name, error: r.error,
226+
});
227+
continue;
228+
}
229+
lists.push({ provider: one.name, results: r.results ?? [] });
230+
}
231+
if (!lists.length) {
232+
log(` search failed on every provider: ${text}`);
233+
continue;
234+
}
235+
const merged = mergeByRank(lists);
236+
log(` searched ${lists.map((l) => `${l.provider} ${l.results.length}`).join(', ')} -> ${merged.length} distinct`);
237+
discovered.push({ query: text, results: merged, provider: lists.map((l) => l.provider).join('+'), searchId: null });
238+
for (const candidate of selectCandidates(merged, { prefer, perQuery: settings.perQuery, seen })) {
239+
targets.push({
240+
url: candidate.url,
241+
type: DEFAULT_SOURCE_TYPE,
242+
usedFor: typeof query === 'object' ? (query.why ?? '') : '',
243+
from: 'search',
244+
rankedBy: candidate.provider ?? '',
245+
});
246+
seen.add(candidate.url);
247+
}
248+
continue;
249+
}
250+
176251
let found = searcher.search(text, { limit: settings.limit });
177252
let ranker = searchName;
178253

‎research-kit/lib/transport.mjs‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,11 +177,22 @@ export function selectSearch({ explicit = '', env = process.env, config = null,
177177
}
178178

179179
if (serpapi.readKey({ env, config: settings })) {
180+
// BOTH, not one. The meters are separate, so asking each costs one search apiece rather
181+
// than more of either - and on 2026-09-22 that difference was the whole corpus: a query
182+
// about the EU Deforestation Regulation returned eight US financial-regulation pages
183+
// from SerpAPI and nothing on topic, while the fetch provider's own search returned
184+
// seventeen, all on topic. Either provider alone can fail that badly on a given query;
185+
// `mergeByRank` bounds the damage by letting each contribute its best result first.
186+
//
187+
// `adapter` still names the provider that pays the separate meter, so every existing
188+
// caller and every status line keeps meaning what it meant.
180189
return {
181190
name: serpapi.name,
182191
adapter: serpapi,
183-
why: 'a SerpAPI key is configured - searching on its own meter, leaving the fetch budget for pages',
192+
adapters: [serpapi, side.adapter],
193+
why: `a SerpAPI key is configured - searching on its own meter AND with ${side.name}, merged by rank`,
184194
searchOnly: true,
195+
merged: true,
185196
};
186197
}
187198

‎research-kit/test/transport.test.mjs‎

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ import { test, describe, assert, tempDir, fs, path, KIT_ROOT } from './harness.m
77
import { readText } from '../lib/core.mjs';
88
import * as firecrawl from '../lib/firecrawl.mjs';
99
import * as httpKeyless from '../lib/http-transport.mjs';
10-
import { TRANSPORTS, TRANSPORT_NAMES, selectTransport, probeFirecrawl } from '../lib/transport.mjs';
10+
import { TRANSPORTS, TRANSPORT_NAMES, selectTransport, selectSearch, probeFirecrawl } from '../lib/transport.mjs';
11+
import { mergeByRank } from '../lib/research-run.mjs';
1112

1213
describe('transport');
1314

@@ -406,3 +407,67 @@ test('O-4: an unauthenticated machine is read as unauthenticated, not as a parse
406407
assert.equal(s.authenticated, false);
407408
assert.equal(s.credits, null);
408409
});
410+
411+
// ---------------------------------------------------------------- merged search
412+
413+
test('mergeByRank interleaves, so one failing provider cannot eat the budget', () => {
414+
// The case that motivated this. On 2026-09-22 a query about the EU Deforestation
415+
// Regulation returned eight US financial-regulation pages from one provider and nothing
416+
// on topic, while the other returned seventeen, all on topic. Concatenation would have
417+
// spent the whole page budget on the wrong provider before reaching the right one.
418+
const bad = { provider: 'serpapi', results: [{ url: 'sec' }, { url: 'cfpb' }, { url: 'water' }] };
419+
const good = { provider: 'firecrawl-cli', results: [{ url: 'europa' }, { url: 'ey' }, { url: 'wiki' }] };
420+
421+
const merged = mergeByRank([bad, good]);
422+
assert.deepEqual(merged.map((r) => r.url), ['sec', 'europa', 'cfpb', 'ey', 'water', 'wiki']);
423+
424+
// Whatever the budget, the good provider is represented. At three pages the old
425+
// behaviour would have taken sec/cfpb/water and nothing else.
426+
assert.ok(merged.slice(0, 3).some((r) => r.provider === 'firecrawl-cli'),
427+
'the working provider must reach the first slots');
428+
});
429+
430+
test('mergeByRank attributes every row, and agreement does not double-count', () => {
431+
// The ledger records which search surfaced a URL, which is what keeps a claim about
432+
// search quality checkable rather than an impression.
433+
const a = { provider: 'serpapi', results: [{ url: 'shared' }, { url: 'a2' }] };
434+
const b = { provider: 'firecrawl-cli', results: [{ url: 'shared' }, { url: 'b2' }] };
435+
436+
const merged = mergeByRank([a, b]);
437+
assert.equal(merged.filter((r) => r.url === 'shared').length, 1, 'a shared URL appears once');
438+
assert.equal(merged.find((r) => r.url === 'shared').provider, 'serpapi', 'the first finder keeps it');
439+
assert.ok(merged.every((r) => r.provider), 'every row names its finder');
440+
});
441+
442+
test('mergeByRank handles ragged and empty lists without inventing rows', () => {
443+
const long = { provider: 'x', results: [{ url: '1' }, { url: '2' }, { url: '3' }] };
444+
const short = { provider: 'y', results: [{ url: 'only' }] };
445+
assert.deepEqual(mergeByRank([long, short]).map((r) => r.url), ['1', 'only', '2', '3']);
446+
assert.deepEqual(mergeByRank([{ provider: 'z', results: [] }]), []);
447+
assert.deepEqual(mergeByRank([]), []);
448+
});
449+
450+
test('a configured SerpAPI key selects BOTH providers, not one', () => {
451+
// This is what the operator believed was already happening, and was not: the ladder
452+
// picked a single provider and the second key bought a meter rather than a second look.
453+
const search = selectSearch({
454+
env: { SERPAPI_API_KEY: ['4', '7'].join('') + 'd'.repeat(62) },
455+
probe: () => ({ ok: true, authenticated: true }),
456+
});
457+
assert.equal(search.merged, true);
458+
assert.equal(search.adapters.length, 2, 'both providers must be offered');
459+
assert.equal(search.adapters[0].name, 'serpapi', 'the separately metered one goes first');
460+
assert.ok(search.why.includes('merged by rank'));
461+
});
462+
463+
test('an explicitly pinned provider stays a single provider', () => {
464+
// Pinning exists to make two runs comparable. If pinning still merged, it would not.
465+
const search = selectSearch({
466+
explicit: 'serpapi',
467+
env: { SERPAPI_API_KEY: ['4', '7'].join('') + 'd'.repeat(62) },
468+
probe: () => ({ ok: true, authenticated: true }),
469+
});
470+
assert.equal(search.merged, undefined);
471+
assert.equal(search.adapters, undefined, 'a pinned choice offers no second provider');
472+
assert.equal(search.name, 'serpapi');
473+
});

0 commit comments

Comments
 (0)