Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ Each release of Research-Kit, newest first. The version is `KIT_VERSION` in

## Unreleased

- The tested Firecrawl CLI is 1.25.3 (from 1.25.2): the release is one fix, `--objective`
optional for alexandria feedback (firecrawl/cli#301), with nothing on scrape, search or
`--status`; its output was captured from the real CLI anyway (three credits) and is read
unchanged by the adapter's parsers (`research-kit/test/fixtures/*-1.25.3.*`). `doctor`, the
install line and the workflows' install spec name 1.25.3.
- Internal: one search session (`lib/search-session.mjs`, ADR-0135) owns how a query is put to
the search providers - the merge, the one degrade to the fetch provider, the rate-limit
patience, the meters, the failure rows and the refuse-before-spend readiness gate - and
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ exists for Claude Code only; every other agent gets the commit gate.
| Node 22+ | everything | Node 22, 24 and 26 are each tested on every commit |
| Git | everything | projects are git repositories; the commit gate is a git hook |
| Python 3.11+ | the test suite only | the cross-language conformance runners; without it the suite blocks rather than passing |
| Firecrawl CLI 1.25.2 and a Firecrawl account | metered collection | `npm install -g firecrawl-cli@1.25.2`, then `firecrawl login`. Free tier: 1,000 credits a month, no card |
| Firecrawl CLI 1.25.3 and a Firecrawl account | metered collection | `npm install -g firecrawl-cli@1.25.3`, then `firecrawl login`. Free tier: 1,000 credits a month, no card |
| Chromium or Chrome | the `browser` transport | free; reads pages built by JavaScript |
| SerpAPI key or a SearXNG instance | searching, optional | a second meter, so search stops competing with fetching |

Expand Down Expand Up @@ -445,7 +445,7 @@ evaluated and deliberately not wired in, because its terms allow retention for t
**Firecrawl setup**

```bash
npm install -g firecrawl-cli@1.25.2
npm install -g firecrawl-cli@1.25.3
firecrawl login
```

Expand Down
2 changes: 1 addition & 1 deletion docs/ARCHITECTURE.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/decisions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ repository. The retention rule is **ADR-0102**:
| [2026-09-29-perspective-discovery](2026-09-29-perspective-discovery/) | STORM-style perspective discovery without a language model | ADR-0091 | active |
| [2026-09-30-searxng-search](2026-09-30-searxng-search/) | SearXNG's JSON Search API as a keyless search transport | ADR-0104; `lib/searxng.mjs` | active |
| [2026-09-30-wayback-witness](2026-09-30-wayback-witness/) | The Wayback Machine as a third-party witness for captured pages | ADR-0106; `lib/witness.mjs` | active |
| [2026-10-02-firecrawl-cli-1-25](2026-10-02-firecrawl-cli-1-25/) | Whether the tested Firecrawl CLI pin should move from 1.24.6 to 1.25.2 - it did, on 2026-10-02, after the brief's first step (a fixture capture of 1.25.2's real output, read unchanged by the parsers; `research-kit/test/fixtures/*-1.25.2.*`) | `lib/firecrawl.mjs` (`TESTED_CLI_VERSION`) | active |
| [2026-10-02-firecrawl-cli-1-25](2026-10-02-firecrawl-cli-1-25/) | Whether the tested Firecrawl CLI pin should move from 1.24.6 to 1.25.2 - it did, on 2026-10-02, after the brief's first step (a fixture capture of 1.25.2's real output, read unchanged by the parsers; `research-kit/test/fixtures/*-1.25.2.*`); the pin moved on to 1.25.3 on 2026-10-03 by the same first step | `lib/firecrawl.mjs` (`TESTED_CLI_VERSION`) | active |
| [2026-10-02-similar-tools-survey](2026-10-02-similar-tools-survey/) | Which similar agent-research and evidence-provenance tools on GitHub and Hugging Face do something this kit should take - none as code; two mechanisms weighed and deferred in its brief (a signature on ledger entries, anchoring the chain head with a third party) | the freeze (ADR-0117) stands; the brief's deferred items name their triggers | active |
| [2026-10-02-github-plain-fetch-refusal](2026-10-02-github-plain-fetch-refusal/) | Why a plain fetch of github.com answered 403 and which free route to name - the block was this sandbox's egress proxy, not GitHub's (its rate-limit 403 carries `x-ratelimit-remaining: 0`, this one did not); `raw.githubusercontent.com` and the browser transport are the measured free routes; the one change is that the keyless failure quotes the server's own reason | `lib/http-transport.mjs` (`serverReason`) | active |
| [2026-10-02-licence-and-captured-pages](2026-10-02-licence-and-captured-pages/) | Which licence the kit should publish under so that people and agents may install and run it, and whether the cached third-party pages under every `research/raw/` may be redistributed - raised by an outside review that found LICENSE (all rights reserved) contradicting the README's install instruction. The brief recommends PolyForm Shield 1.0.0 (use and modify, no competing product) or Apache-2.0 if adoption outweighs control, and for the captures a NOTICE in the archives' words with `research/raw/` excluded from the code licence via REUSE; the owner chose Shield on 2026-10-02 | ADR-0130; `LICENSE`, `NOTICE`, `REUSE.toml` | active |
2 changes: 1 addition & 1 deletion research-kit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,7 +453,7 @@ node research-kit/bin/selftest.mjs # all of it
node research-kit/bin/selftest.mjs gate hook # just these files
```

1512 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
1515 tests, offline, no key and no network. The runner **awaits** every test, so `ok` means
the assertions settled (ADR-0021), and each test is raced against a watchdog
(`RESEARCH_KIT_TEST_TIMEOUT`, default 60s).

Expand Down
9 changes: 8 additions & 1 deletion research-kit/lib/firecrawl.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -455,8 +455,15 @@ export function cliVersion({ execFn = exec, ...opts } = {}) {
* no `url`, so none reaches a result) beside `data.web`; `--status` is the same text. The
* bundle is unchanged (SDK 4.40.0, node >= 22), so the proxy behaviour that forced the
* previous move is not in question.
*
* Moved from 1.25.2 to 1.25.3 on 2026-10-03. The release is one commit, "fix: make --objective
* optional for alexandria feedback" (firecrawl/cli#301), with nothing on scrape, search or
* `--status`; the bundle is unchanged (SDK 4.40.0, axios 1.18.0, node >= 22). Captured anyway,
* the same way (test/fixtures/*-1.25.3.*): the three outputs read unchanged by these parsers -
* the receipt on stderr, `data.tools` populated and reaching no result, `--status` the same
* text with a 1,500-credit cycle behind it.
*/
export const TESTED_CLI_VERSION = '1.25.2';
export const TESTED_CLI_VERSION = '1.25.3';
export const SUPPORTED_CLI_MAJOR = 1;

/**
Expand Down
1 change: 1 addition & 0 deletions research-kit/test/fixtures/firecrawl-scrape-1.25.3.json

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Scrape ID: 01a10321-2ecb-73a9-9ec2-ddcd9c51dcc5
1 change: 1 addition & 0 deletions research-kit/test/fixtures/firecrawl-search-1.25.3.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"success":true,"data":{"tools":[{"provider":"github-com","capability":"repositories/contributors","description":"One page of a repository's contributors ordered by commit count on the default branch, with login, account type and contribution count. Anonymous (email-only) contributors are not listed. GitHub refuses this listing for repositories with very large histories (for example torvalds/linux), which surfaces as an error naming the reason."},{"provider":"github-com","capability":"repositories/issue_counts","description":"Exact issue counts for a repository (pull requests excluded, unlike the repository's open_issues_count): the total for the given state plus one count per requested label, from GitHub's issue search. Makes 1 + labels.length search requests; at most 5 labels per call because GitHub allows 10 keyless searches per minute per IP."},{"provider":"github-com","capability":"repositories/issues","description":"One page of a repository's issues filtered by state and labels (all given labels must match), sorted by created/updated/comments. Pull requests are excluded unless include_pull_requests is true, so a page may hold fewer than per_page records; pull_requests_excluded says how many were dropped. Each record carries title, state, labels, assignees, milestone, comment and reaction counts, body and timestamps."}],"web":[{"url":"https://docs.firecrawl.dev/sdks/cli","title":"CLI - Firecrawl Docs","description":"## Documentation Index\nThe Firecrawl CLI works standalone or with skills that AI coding agents like Codex, Claude Code, Cursor, and OpenCode can discover and use automatically.\n\n## [​](https://docs.firecrawl.dev/sdks/cli#installation) Installation\n```\n# Install globally with npm\nnpm install -g firecrawl-cli\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli#commands) Commands\n### [​](https://docs.firecrawl.dev/sdks/cli#scrape) Scrape\n#### [​](https://docs.firecrawl.dev/sdks/cli#scrape-options) Scrape Options\n```\n# Extract only main content (removes navs, footers)\nfirecrawl https://example.com --only-main-content\n\n# Wait for JavaScript rendering\nfirecrawl https://example.com --wait-for 3000\n\n# Take a screenshot\nfirecrawl https://example.com --screenshot\n\n# Extract structured JSON with a schema\nfirecrawl https://example.com --format json --schema '{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}'\n\n# Run lightweight scrape actions before extraction\nfirecrawl https://example.com --actions '[{\"type\":\"wait\",\"milliseconds\":1000}]'\n\n# Select proxy mode\nfirecrawl https://example.com --proxy basic\n\n# Redact personally identifiable information\nfirecrawl https://example.com --redact-pii\n\n# Include/exclude specific HTML tags\nfirecrawl https://example.com --include-tags article,main\nfirecrawl https://example.com --exclude-tags nav,footer\n\n# Save output to file\nfirecrawl https://example.com -o output.md\n\n# Pretty print JSON output\nfirecrawl https://example.com --format markdown,links --pretty\n\n# Force JSON output even with single format\nfirecrawl https://example.com --json\n\n# Show request timing information\nfirecrawl https://example.com --timing\n```\n\n### [​](https://docs.firecrawl.dev/sdks/cli#crawl) Crawl\n#### [​](https://docs.firecrawl.dev/sdks/cli#crawl-options) Crawl Options\nCLI\n\n```\n# Limit crawl depth and pages\nfirecrawl crawl https://example.com --limit 100 --max-depth 3 --wait\n\n# Include only specific paths\nfirecrawl crawl https://example.com --include-paths /blog,/docs --wait\n\n# Exclude specific paths\nfirecrawl crawl https://example.com --exclude-paths /admin,/login --wait\n\n# Include subdomains\nfirecrawl crawl https://example.com --allow-subdomains --wait\n\n# Crawl entire domain\nfirecrawl crawl https://example.com --crawl-entire-domain --wait\n\n# Rate limiting\nfirecrawl crawl https://example.com --delay 1000 --max-concurrency 2 --wait\n\n# Pass scrape options to each crawled page\nfirecrawl crawl https://example.com --scrape-options '{\"formats\":[\"markdown\"],\"onlyMainContent\":true}'\n\n# Send crawl completion events to a webhook\nfirecrawl crawl https://example.com --webhook '{\"url\":\"https://example.com/webhook\",\"events\":[\"completed\"]}'\n\n# Cancel an active crawl\nfirecrawl crawl <job-id> --cancel\n\n# Custom polling interval and timeout\nfirecrawl crawl https://example.com --wait --poll-interval 10 --timeout 300\n\n# Save results to file\nfirecrawl crawl https://example.com --wait --pretty -o results.json\n```\n\n## [​](https://docs.firecrawl.dev/sdks/cli#examples) Examples\n### [​](https://docs.firecrawl.dev/sdks/cli#full-site-crawl) Full Site Crawl\nCLI\n\n## [​](https://docs.firecrawl.dev/sdks/cli#open-source) Open Source\nThe Firecrawl CLI and all three skill segments are open source on GitHub:\n\n- [`firecrawl/cli`](https://github.com/firecrawl/cli) — the CLI and CLI skills (live web work)","position":1},{"url":"https://github.com/firecrawl/cli","title":"CLI and Agent Skill for Firecrawl - Add scrape, search, and ... - GitHub","description":"# 🔥 Firecrawl CLI\n## Installation\n```\nnpm install -g firecrawl-cli\n```\n\n## Quick Start\n```\nfirecrawl https://example.com\n```\n\n## Global Options\n### Check Status\n```\n 🔥 firecrawl cli v1.20.0\n\n ● Authenticated via stored credentials\n Concurrency: 0/100 jobs (parallel scrape limit)\n Credits: 500,000 / 1,000,000 (50% left this cycle)\n```\n\n## Tips & Tricks\n[Permalink: Tips & Tricks](https://github.com/firecrawl/cli#tips--tricks)\n\n### CI/CD Usage\n[Permalink: CI/CD Usage](https://github.com/firecrawl/cli#cicd-usage)\n\n```\n# Set API key via environment\nexport FIRECRAWL_API_KEY=${{ secrets.FIRECRAWL_API_KEY }}\nfirecrawl crawl https://docs.example.com --wait -o docs.json\n\n# Use self-hosted instance\nexport FIRECRAWL_API_URL=${{ secrets.FIRECRAWL_API_URL }}\nfirecrawl scrape https://example.com -o output.md\n```\n\n## Documentation\nFor more details, visit the [Firecrawl Documentation](https://docs.firecrawl.dev/).\n\n## Languages\n- [TypeScript98.3%](https://github.com/firecrawl/cli/search?l=typescript)\n- [Shell1.1%](https://github.com/firecrawl/cli/search?l=shell)\n- [Other0.6%](https://github.com/firecrawl/cli/search?l=Other)\n\nYou can’t perform that action at this time.","position":2,"category":"github"},{"url":"https://www.firecrawl.dev/blog/introducing-firecrawl-skill-and-cli","title":"Introducing Firecrawl Skill and CLI: The Complete Web Data Toolkit for ...","description":"# Introducing Firecrawl Skill and CLI: The Complete Web Data Toolkit for Agents\n## Meet Firecrawl CLI\n[Firecrawl CLI](https://docs.firecrawl.dev/sdks/cli) is a unified command-line interface designed for agents and developers who need **reliable web data.**\n\n### Context-Efficient Data for Agents\nFirecrawl CLI uses a file-based approach for context management and bash methods for efficient search and retrieval.\n\n### Industry-Leading Reliability\nIt handles JavaScript rendering, dynamic content, authentication flows, and complex page structures that break other scrapers.\n\n## Get Started\nOr use the CLI directly:\n\n```\n$ npm install -g firecrawl-cli\n$ firecrawl auth --api-key fc-YOUR_KEY\n$ firecrawl scrape https://example.com\n```","position":3}]},"id":"01a10321-273e-74bd-821f-5c1339008379","creditsUsed":2}
9 changes: 9 additions & 0 deletions research-kit/test/fixtures/firecrawl-status-1.25.3.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@

🔥 firecrawl cli v1.25.3

● Authenticated via FIRECRAWL_API_KEY
Concurrency: 0/2 jobs (parallel scrape limit)
Credits: 1,175 / 1,500 (78% left this cycle)
.firecrawl: not found - no local cache
.gitignore: missing - add .firecrawl/ to ignore cache

50 changes: 47 additions & 3 deletions research-kit/test/transport.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -656,7 +656,7 @@ test('O-4: the status parser reads the CLI\'s actual output', () => {
});

// Captured from firecrawl-cli v1.24.6 on 2026-09-26, the version TESTED_CLI_VERSION moved to
// that day (and on from, 2026-10-02 - the 1.25.2 fixtures below).
// that day (and on from, 2026-10-02 and 2026-10-03 - the 1.25.2 and 1.25.3 fixtures below).
// Why it moved: 1.23.3 bundles axios 1.15.2, which sends plain proxied requests that an
// HTTPS proxy requiring CONNECT tunnels refuses with 405 - every call failed behind one
// (research/BRIEF.md, 2026-09-26 addendum). 1.24.6 bundles axios 1.18.0.
Expand Down Expand Up @@ -698,10 +698,10 @@ const REAL_SEARCH_1252 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firec
const REAL_SCRAPE_1252 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-scrape-1.25.2.json'));
const REAL_SCRAPE_1252_STDERR = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-scrape-1.25.2.stderr.txt'));

test('O-4: the status parser reads the TESTED version\'s actual output (1.25.2)', () => {
test('O-4: the status parser reads 1.25.2\'s actual output, the previously tested version', () => {
assert.ok(REAL_STATUS_1252, 'the fixture must exist');
const s = firecrawl.parseStatus(REAL_STATUS_1252);
assert.equal(s.version, firecrawl.TESTED_CLI_VERSION, 'the fixture and the tested version must be the same release');
assert.equal(s.version, '1.25.2', 'the fixture is the release its name says');
assert.equal(s.authenticated, true);
assert.equal(s.credits, 154);
assert.equal(s.creditLimit, 1000);
Expand Down Expand Up @@ -732,6 +732,50 @@ test('O-4: 1.25.2 scrape stdout is one JSON value - the receipt is on stderr - a
assert.equal(page.transport, 'firecrawl-cli');
});

// Captured from firecrawl-cli v1.25.3 on 2026-10-03, the same way (a scratch prefix, the key
// per command, three credits, stdout and stderr apart). The release is one fix - `--objective`
// optional for alexandria feedback (firecrawl/cli#301) - with nothing on scrape, search or
// `--status`; the capture says so rather than the note: the same fields come back.
const REAL_STATUS_1253 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-status-1.25.3.txt'));
const REAL_SEARCH_1253 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-search-1.25.3.json'));
const REAL_SCRAPE_1253 = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-scrape-1.25.3.json'));
const REAL_SCRAPE_1253_STDERR = readText(path.join(KIT_ROOT, 'test', 'fixtures', 'firecrawl-scrape-1.25.3.stderr.txt'));

test('O-4: the status parser reads the TESTED version\'s actual output (1.25.3)', () => {
assert.ok(REAL_STATUS_1253, 'the fixture must exist');
const s = firecrawl.parseStatus(REAL_STATUS_1253);
assert.equal(s.version, firecrawl.TESTED_CLI_VERSION, 'the fixture and the tested version must be the same release');
assert.equal(s.authenticated, true);
assert.equal(s.credits, 1175);
assert.equal(s.creditLimit, 1500, 'the plan behind the capture allows 1,500 a cycle; the parser reads the comma');
assert.equal(s.concurrencyLimit, 2);
});

test('O-4: 1.25.3 search output normalises as 1.25.2\'s did - data.tools still reaches no result', () => {
const raw = JSON.parse(REAL_SEARCH_1253);
assert.ok(Array.isArray(raw.data.web) && raw.data.web.length, 'the fixture must carry real results');
assert.ok(Array.isArray(raw.data.tools) && raw.data.tools.length, 'the fixture must carry the populated tools list');
const rows = firecrawl.normalizeSearch(REAL_SEARCH_1253);
assert.equal(rows.length, raw.data.web.length);
assert.deepEqual(rows.map((r) => r.url), raw.data.web.map((r) => r.url));
assert.deepEqual(rows.map((r) => r.position), [1, 2, 3]);
for (const row of rows) assert.match(row.url, /^https?:\/\//);
assert.equal(JSON.stringify(rows).includes('capability'), false, 'a tool entry leaked into the results');
assert.equal(firecrawl.creditsUsed(REAL_SEARCH_1253), 2, 'the vendor says what the search cost');
});

test('O-4: 1.25.3 scrape stdout is one JSON value - the receipt is on stderr - and normalises whole', () => {
assert.doesNotThrow(() => JSON.parse(REAL_SCRAPE_1253), 'stdout holds nothing before or after the payload');
assert.match(REAL_SCRAPE_1253_STDERR, /^Scrape ID: [0-9a-f-]+\s*$/, 'the receipt went to stderr');
const page = firecrawl.normalizeScrape(REAL_SCRAPE_1253, 'https://docs.firecrawl.dev/sdks/cli');
assert.equal(page.url, 'https://docs.firecrawl.dev/sdks/cli');
assert.equal(page.title, 'CLI | Firecrawl');
assert.equal(page.statusCode, 200);
assert.ok(page.markdown.length > 20000, `the whole page text is kept (${page.markdown.length} chars)`);
assert.equal(page.completeness, 'full');
assert.equal(page.transport, 'firecrawl-cli');
});

test('O-4: stripAnsi removes the ESCAPE, not just the bracket sequence', () => {
const ESC = String.fromCharCode(27);
assert.ok(REAL_STATUS.includes(ESC), 'the fixture carries real escapes');
Expand Down
Loading