max_pages was capped at 3 by perQuery, on every dispatched collection

Found while comparing the two search providers, and it explains the thing
the comparison was set up to investigate.

collect.yml seeds exactly ONE query and left perQuery at the scaffold
default of 3. selectCandidates does `.slice(0, perQuery)`, so the effective
ceiling on a dispatched run is min(maxScrapes, queries x perQuery) = 3 -
whatever max_pages says. The input advertises 1..25 and anything above 3 was
inert.

Observed and unexplained at the time: run 35692202192 was dispatched with
--max-pages 4 and collected exactly 3. So were sea-assets and build-sea,
both asking for more and both collecting 3. Every corpus this repository
collected through the runner saw three candidates.

That reframes the search-quality question it was found by. "9 of 13
candidates were SEO tutorials" came from a LOCAL decompose run, which lists
what search returned; the runner never fetched beyond rank 3. So one poor
result was a third of the corpus, and the genuinely normative source could
sit at rank 4 and never be reachable. For the test query - HTTP 429
Retry-After - the local run surfaced rfc-editor.org and the dispatched runs
could not, because both providers ranked it below third.

perQuery now follows max_pages, and limit rises with it. Spend is unchanged:
maxScrapes still bounds the fetches, and this only widens the list they are
chosen from. live-collection.yml had the same ceiling and gets the same fix.

The provider comparison itself is not answerable yet. Both providers
returned MDN at rank 1 for the spec query and differed only at ranks 2-3,
which is too narrow a window to judge them by. It gets re-run against the
raised ceiling.

855 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: StepenkoAnatoli

.github/workflows/collect.yml

@@ -374,10 +374,18 @@ jobs:
             const plan = JSON.parse(fs.readFileSync("research/plan.json", "utf8"));
             plan.topic = topic;
             plan.maxScrapes = Number(process.env.MAX_PAGES);
+            // perQuery is the REAL ceiling on breadth: selectCandidates slices each
+            // query's results to it, so one seeded query and the scaffold default of 3
+            // capped every dispatched run at three candidates however large max_pages
+            // was. Matching it to max_pages is what makes the advertised 1..25 control
+            // mean what it says. Nothing about SPEND changes - maxScrapes still bounds
+            // the fetches, and this only widens the list they are chosen from.
+            plan.perQuery = Number(process.env.MAX_PAGES);
+            plan.limit = Math.max(Number(process.env.MAX_PAGES), plan.limit ?? 8);
             plan.queries = [{ q: topic, why: "the dispatched topic", prefer: [] }];
             plan.urls = [];
             fs.writeFileSync("research/plan.json", `${JSON.stringify(plan, null, 2)}\n`, "utf8");
-            console.log(`plan written: 1 query, maxScrapes ${plan.maxScrapes}`);
+            console.log(`plan written: 1 query, maxScrapes ${plan.maxScrapes}, perQuery ${plan.perQuery}`);
           '
 
       - name: seed the map, spending nothing

.github/workflows/live-collection.yml

@@ -205,6 +205,7 @@ jobs:
             const p = "research/plan.json";
             const plan = JSON.parse(fs.readFileSync(p, "utf8"));
             plan.maxScrapes = Number(process.env.MAX_PAGES);
+            plan.perQuery = Number(process.env.MAX_PAGES);
             fs.writeFileSync(p, JSON.stringify(plan, null, 2));
             console.log("plan written: maxScrapes " + plan.maxScrapes);
           '