Skip to content

Commit 657375d

Browse files
Merge pull request #68 from StepenkoAnatoli/fix/topic-signal-limits
Tavily's exclusion holds, and using the kit to prove it found a bug in the kit
2 parents 7d3b3f9 + 021309d commit 657375d

30 files changed

Lines changed: 5131 additions & 6 deletions

‎docs/ARCHITECTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ else, and it never touches the corpus:
145145
| `decompose.mjs` | Phase 0 coordination: recipe parsing, docs-host discovery, material gathering (scrape-budget accounting via the corpus's `cacheDecision` — a cache hit is not an attempt), draft MAP writing. Contains **no judgment** — it seeds the checklist and gathers material; it never writes a status. | — |
146146
| `dimensions.mjs` | The universal, domain-agnostic subtopic checklist and its mechanical presence-matching (`coverageOfUniversals`). Recipes add on top of it; nothing replaces it. | — |
147147
| `provenance.mjs` | The tamper-evident fetch ledger: `appendFetch`, `verifyLedger`, `rebuildLedger` (the one metadata-migration path — guarded, per-entry `migrations` records, chained `op: 'migration'` boundary), and the overrides log writer. Also **the collector's exclusive section** (`withLock`), acquired with `O_EXCL` (`flag: 'wx'`) so the kernel refuses a second creator instead of letting it overwrite; staleness is judged only *after* EEXIST proves someone holds the file, and a fresh lock that names no pid — the window `O_EXCL` leaves between create and write — is waited for, never broken. Reentrant in-process, so `appendFetch` can serialize its own read-modify-write inside a run that already holds the lock, and release removes only the acquisition it made (pid + nonce). Every writer of the chain — append, tail repair, rebuild — goes through that one lock. `verifyLedger` also **classifies** a body hash that does not recompute (`isLineEndingRewrite`, `hashText`): folding CRLF back to LF either restores the recorded hash or it does not, so `body-unmodified` problems carry `kind: 'line-endings' | 'modified'` and the consumer can name the right cause instead of guessing (ADR-0020). | ADR-0003 (corpus-adjacent), ADR-0020 | **Release is ownership-positive (fixed 2026-09-21):** the lock is unlinked only when the nonce POSITIVELY matches. An unreadable lock is left alone — while a collector works its lock can be truncated, recovered as stale by a second collector and replaced by that collector's own, and unlinking then would delete a lock this process does not hold and admit a third writer to the section. Failing to release is recoverable by the staleness path; deleting someone else's is not.
148-
| `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape — twelve, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. | ADR-0004, ADR-0026 |
148+
| `checks.mjs` | The **ordered contract-check registry**: discovery-contract, citations, provenance, transport-provenance, gate-integrity, unknown-closure, subtopic-coverage, capture-completeness, **evidence-supersession**, collection-attempts, hygiene, corpus-shape, **corroboration** — thirteen, in a pinned order. Every check is a pure function of the corpus snapshot + options; none reads the filesystem. A check that **instructs** the reviewer owns the accuracy of the instruction, not just of the verdict: `capture-completeness/partial-render` names the EVIDENCE row and the `[render-reviewed: ...]` syntax because it used to name a `renderReview:` front-matter line, and a reviewer who obeyed it edited a hashed capture and got a blocking `provenance/body-unmodified` failure instead of a closed warning. The wording is pinned by test for that reason. **`supersededRows(corpus)`** is the one shared answer to "which rows have been replaced by a fresher capture of the same URL": `evidence-supersession` refuses an unknown still resting on a replaced row, `transport-provenance` goes quiet about a replaced row nothing relies on (but not about one still cited), and `hygiene` stops calling a refresh a duplicate while still catching two rows of one URL fetched the same day. Those last two are ADR-0026-aware: they warn about the state that ADR requires. | ADR-0004, ADR-0026 |
149149
| `similarity.mjs` | **Near-duplicate detection between two captured pages**, and nothing else. Word shingles (5-word, case- and punctuation-folded, URLs stripped) reduced to a 128-hash MinHash `sketch`; `similarity` estimates Jaccard, `isMirror` compares it to `MIRROR_THRESHOLD`, and `documentGroups` single-link-clusters rows into distinct DOCUMENTS. Exists because `corroboration` counted HOSTS and therefore **inverted** on a mirror: a republished copy on an unrelated host scored `independent` - the registry's best grade - while carrying one witness plus a staleness `freshness` cannot see, since that check grades when a page was FETCHED. `MIRROR_THRESHOLD = 0.25` is **chosen from measurement, not taste**: all 1081 capture pairs in this repository were compared on 2026-09-21, the known stale mirror (`nodejs.org` x `lira.epac.to`, six majors apart) scored 0.3879 and the highest-scoring genuinely different cross-host pair scored 0.0275 - a fourteen-fold gap, so the line is not delicate. `sketch` returns **null** below `MIN_SHINGLES`, and every caller must read null as "cannot tell" rather than "different", which is why two unjudgeable rows stay two documents instead of merging into an invented mirror. Hashing is seeded FNV-1a so a sketch is identical across runs and platforms - a fingerprint that decides a gate finding must grade the same corpus the same way everywhere. Judges no meaning and cannot tell a correct page from a wrong one (ADR-0013). | ADR-0036 |
150150
| `preflight.mjs` | **The verdict** and the **verdict context**. Assembles findings from CHECKS into pass/fail; one function, three callers (commit gate, edit gate, CLI) so they cannot disagree. Owns the verdict's input unit: `verdictContext(root)` reads the corpus, the gate state (`readGateState`), and the operator's `evidencePolicy` together — same snapshot, same verdict, by construction. | ADR-0004 |
151151
| `gate.mjs` | Gate policy: `isGated` (the four markers), `evaluate` (commit/edit), the GATE_OFF override recording, and the architecture-map rule (`architectureMapBreach` over `loadGateConfig` — guarded paths are project configuration, `research/kit.json`, defaults `src/lib/bin/scripts/app`). Contains no repo-specific path. **One deliberate exception to "no CLI parsing"**: the commit gate's staged-path list arrives as *data on a pipe* (`stagedPathsFromStdin` / `splitPathList`), because argv was O(n²) to build and bounded besides. It returns **null for "stdin could not be read" and `[]` for "read, and empty"** — conflating them would hand the verdict an empty staged set, which it allows, i.e. a gate that silently stops gating. `stdinIsReadable()` asks `isatty(0)` rather than inferring from the file type, which gets two cases wrong in opposite directions (a socketpair is not a FIFO; `/dev/null` is a character device but reads as immediate EOF). | ADR-0002, ADR-0007, ADR-0008, ADR-0020 | **An unreadable index BLOCKS (fixed 2026-09-21).** The commit gate judges staged bytes (ADR-0024); when `git checkout-index` fails it used to judge the working tree and note that it had. Noting is not enough — the two differ exactly when it matters, so an environmental git failure silently reinstated the defect ADR-0024 exists to prevent. It now blocks with `judged: 'nothing'` and names `--no-verify` as the recorded override. A project that is not a git repository still judges the tree: there is no index to fail to read. Index materialisation is also **chunked** (250 paths per call) rather than spreading every tracked path into one argv, which on Windows hit the ~32k command-line cap for a large corpus and produced exactly that git failure.
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# The corpus's line endings are integrity, not style.
2+
#
3+
# A capture's body hash is taken over the LF bytes the collector fetched. git's smudge
4+
# filter rewrites every text file on a default Windows checkout (core.autocrlf=true), so
5+
# without this file the whole corpus fails `body-unmodified` on arrival with nothing
6+
# actually wrong with it. This file outranks a reader's core.autocrlf. See ADR-0020.
7+
8+
research/raw/* text eol=lf
9+
*.jsonl text eol=lf
10+
research/*.md text eol=lf
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
node_modules/
2+
.DS_Store
3+
4+
# Credentials never belong in a research project (AGENTS.md Rule 6). Note that
5+
# `firecrawl env` writes the key into .env, which is why the rule says never to run it
6+
# inside a repository.
7+
.env
8+
.env.*
9+
*.pem
10+
*.key
11+
12+
# The Firecrawl CLI saves scrape output into .firecrawl/ in whatever directory it runs
13+
# from. That is a vendor cache, not evidence - the kit's own captures live in
14+
# research/raw/ with a ledger entry behind each one.
15+
.firecrawl/
16+
17+
# The kit's local review logs are not evidence.
18+
research/raw/.usage.jsonl
19+
research/raw/.failures.jsonl
20+
research/raw/.diagnostics.jsonl
21+
research/raw/.fetches.lock
22+
23+
# The chain IS evidence and must travel. The explicit negation is here so a broader
24+
# dotfile rule added later cannot silently exclude it - this repository lost its ledger
25+
# exactly that way once.
26+
!research/raw/.fetches.jsonl
27+
!research/overrides.log
Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Research-first project bootstrap
2+
3+
Agent instructions for this project. Read this before planning, designing, or building
4+
anything here.
5+
6+
## How this project works: two phases, two roles
7+
8+
**Phase 1 - research (this kit).** Close every blocking unknown with fetched primary
9+
sources, then pass the gate. The output is a briefing, not code: `research/DISCOVERY.md`
10+
(the contract), `research/EVIDENCE.md` (claims with cached pages behind them), and
11+
`research/BRIEF.md` (the handoff). Phase 1 never writes product code.
12+
13+
**Phase 2 - build (someone else's job).** A builder - another agent, a different model, or
14+
a human - takes `research/BRIEF.md` plus `research/` and implements. They should not need
15+
to re-research anything: if they do, phase 1 was incomplete, and the fix is to collect the
16+
missing fact rather than to let the builder guess it.
17+
18+
The gate is the handoff point. Before it passes, phase 2 does not start.
19+
20+
## Two machines, two roles
21+
22+
The role is machine config - `role: "collector" | "builder"` in
23+
`~/.agents/research-kit.config.json`, default `collector`; declare it with
24+
`node ~/.agents/research-kit/bin/install-hooks.mjs --role builder`.
25+
26+
| | **collector** (the operator's PC) | **builder** (a sandbox, a CI box, a laptop) |
27+
|---|---|---|
28+
| holds | the Firecrawl key | no key, no Firecrawl egress |
29+
| runs | `decompose.mjs`, `research.mjs` | `handoff.mjs`, `preflight.mjs`, the build |
30+
| a missing key is | a **FAIL** | **informational** |
31+
| must | push `research/raw/` including its dotfiles | **not collect** - the collection CLIs refuse (exit 2) |
32+
33+
**If you are on a builder machine and there is no brief: you do not collect.** A page
34+
fetched by hand is not evidence in this kit, and a builder that "re-collects" a missing
35+
page forges a corpus instead of reporting a gap. Name the missing fact instead.
36+
37+
A builder's first command is:
38+
39+
```
40+
node ~/.agents/research-kit/bin/handoff.mjs
41+
```
42+
43+
The remedy depends on the cause, and the command says which: something that did not
44+
travel is the collector's to push (`git add -f research/raw/`, dotfiles included);
45+
a corpus that travelled whole and was rewritten on checkout here is fixed **here**, with
46+
`.gitattributes`, and costs no credits.
47+
48+
## Starting in the wrong place: ask which project, do not hunt for it
49+
50+
The project is the **current working directory**. The kit takes no project argument and
51+
never will. If the cwd is not the project the operator means - the kit's own repo, a
52+
parent folder, an unrelated checkout - **ask which project**, and wait. Do not search the
53+
filesystem, do not infer it from a name, do not look it up on a remote. Searching a
54+
person's disk to guess his intent reads directories nobody authorised, and the question
55+
it replaces takes one line.
56+
57+
Signals: no `AGENTS.md`, no `research/` directory, or a `research/` corpus plainly about
58+
something else. An existing corpus about another topic is the signal doing its job.
59+
60+
## Resuming after an interruption: resume, do not restart
61+
62+
The gate is a pure function of repository state, so orient from disk before touching
63+
anything. `doctor.mjs` and preflight answer *where this project is*; the last commit
64+
report and `docs/ARCHITECTURE.md` say *what was happening and why*; `git status` answers
65+
what is unfinished, and its uncommitted changes are the interrupted task - partial and
66+
unverified, work in progress, never state to trust. The corpus is already collected and
67+
spent credits are not spent again.
68+
69+
## Rule 1 - research first, then build
70+
71+
The sequence is decompose -> contract -> collect -> gate -> brief.
72+
73+
0. **Decompose** (collector machine): `node ~/.agents/research-kit/bin/decompose.mjs --topic "<topic>"`
74+
drafts `research/MAP.md` seeded with the universal checklist. The tool contains no
75+
judgment. Mark every row COVERED (citing U-## rows), DISMISSED (reason required), or
76+
GAP.
77+
1. **Write the intent** in `research/DISCOVERY.md` under `## Build intent`, and enumerate
78+
the blocking unknowns FROM the map.
79+
2. **Collect**: `node ~/.agents/research-kit/bin/research.mjs --plan research/plan.json`. Prefer the page
80+
that *owns* the fact over any write-up about it.
81+
3. **Rewrite** each auto-extracted `Finding` cell into a real claim, keeping the `Raw`
82+
cell pointing at the cached page.
83+
4. **Gate**: `node ~/.agents/research-kit/bin/preflight.mjs`. **Do not start building until it prints
84+
PASS.** Evidence must be *fetched*, not typed, and by a named transport.
85+
5. **Hand off**: write `research/BRIEF.md`.
86+
87+
There are exactly three overrides - `git commit --no-verify`, a deliberate
88+
`research/GATE_OFF`, and a repository-local `core.hooksPath` - and all three are recorded
89+
in `research/overrides.log`. If you take one, say so in your reply.
90+
91+
## Rule 2 - questions are for intent, never for facts
92+
93+
At most **3** questions, once, up front, and only about things no document can answer. If
94+
a question's answer is in public documentation, it is a research task. **Never answer
95+
"insufficient info" and stop.**
96+
97+
## Rule 3 - known unknowns are allowed, silence is not
98+
99+
Mark a genuinely unreachable fact `KNOWN-UNKNOWN` and write the day-one verification step
100+
in the `Evidence` cell. An unlabeled gap is the failure mode this project exists to
101+
prevent.
102+
103+
## Rule 4 - source quality
104+
105+
`P` primary/official carries the design. `S` secondary is context. `L` lead-only is a
106+
hint, never proof. Never invent a citation. Flag contradictions instead of averaging them.
107+
108+
## Rule 5 - cost discipline
109+
110+
Every scrape spends credits. Plan the queries in `research/plan.json` first, reuse the
111+
cache (`--refresh-days`), and check the budget with `node ~/.agents/research-kit/bin/research.mjs --status`.
112+
113+
## Rule 6 - secrets
114+
115+
The key lives in the CLI config or the environment. Never write it into this repository.
116+
117+
## The standing protocol
118+
119+
Five rules for how work here is committed and recorded. Documentation, not enforcement:
120+
no check judges prose, because a gate that judges prose is a gate that will be wrong.
121+
122+
1. **Architecture map current in the same commit.** A commit touching a declared code
123+
path (`research/kit.json`) stages `docs/ARCHITECTURE.md` with it. Enforced - and a
124+
prompt, not a proof: the gate checks the map was *staged*, not that it is *current*.
125+
2. **Commit report: five named parts** - what changed / why / what it touched / what you
126+
verified / what you got wrong and fixed. The got-wrong line is not optional; if
127+
nothing was gotten wrong it is written as "nothing to report".
128+
3. **An ADR for any choice with a rejected alternative**, dated, with the reason a future
129+
explorer would need to avoid re-suggesting it.
130+
4. **One commit per task - the revert test.** `git revert <sha>` undoes it alone.
131+
5. **A red suite stops work**, reported immediately and alone, **with the cwd recorded**.
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Start here
2+
3+
For me, the operator. Not for the AI.
4+
5+
---
6+
7+
## Where things live
8+
9+
| What | Where |
10+
|---|---|
11+
| The kit itself | `~/.agents/research-kit` - installed once, never moves |
12+
| Research for THIS project | `<project folder>\research\` - working files, leave alone |
13+
| My one pasteable file | `<project folder>\research\audits\` - named by topic + version + date |
14+
15+
Nothing gets copied between folders. Each project keeps its own research, inside itself.
16+
17+
---
18+
19+
## Starting new research
20+
21+
1. **Open the project folder** - the actual project, never the kit's own folder.
22+
2. **Say what I want researched.** The agent reads the skill automatically and starts.
23+
3. **When it's done, ask for the audit** - that's my one file:
24+
```
25+
node ~/.agents/research-kit\bin\audit.mjs
26+
```
27+
Written to `research\audits\` in the project folder. One attachment instead of ten
28+
pastes:
29+
```
30+
node ~/.agents/research-kit\bin\audit.mjs --zip
31+
```
32+
33+
If I'm ever unsure which folder I'm in, check before starting - the agent should ask if
34+
it isn't sure, not go searching my disk.
35+
36+
---
37+
38+
## Picking up where I left off
39+
40+
From inside the project folder:
41+
42+
```
43+
node ~/.agents/research-kit\bin\doctor.mjs
44+
node ~/.agents/research-kit\bin\preflight.mjs
45+
git log --oneline -5
46+
git status
47+
```
48+
49+
Then tell the agent to resume. Nothing committed is ever lost, and credits already spent
50+
are not spent again.
51+
52+
---
53+
54+
## If something looks broken
55+
56+
```
57+
node ~/.agents/research-kit\bin\doctor.mjs
58+
```
59+
60+
It names every problem and prints the exact fix. Run it, fix what it says, run it again.
61+
Stop at READY.
62+
63+
---
64+
65+
## Budget
66+
67+
1,000 Firecrawl credits a month. Resets monthly. Stops cleanly at zero - no surprise
68+
charges. A research cycle is roughly 40 credits.
69+
70+
---
71+
72+
## The rules the AI follows
73+
74+
Live in `AGENTS.md` inside the kit and every project it touches. I don't need to know
75+
them - any AI that opens the repo is bound by them.
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# ARCHITECTURE - the map of the code
2+
3+
What each module owns, the seams between them, and which decision governs each. This is
4+
a map of the **code**, not of the research: `research/MAP.md` is the topic map - a
5+
different artifact with a different job, judged by a different check.
6+
7+
The rule this map lives by: a commit touching a **declared code path** (declared in
8+
`research/kit.json`, defaulting to `src` / `lib` / `bin` / `scripts` / `app`) carries
9+
this file's update in the **same commit**. The commit gate refuses such commits without
10+
it.
11+
12+
What the rule proves is deliberately modest: the gate checks this file was **staged**
13+
with the code, not that it is current - a whitespace edit satisfies it. That makes a
14+
stale map a deliberate act rather than mere forgetting. A prompt, not a proof.
15+
16+
## The one picture
17+
18+
```
19+
_Replace this with the shape of the thing you build: the entry points, the modules
20+
they reach, and the seams between them._
21+
```
22+
23+
## Modules - what each one owns
24+
25+
| Module | Owns | Governed by |
26+
|---|---|---|
27+
| _module_ | _the one concept it is responsible for_ | _ADR or decision_ |
28+
29+
## The seams
30+
31+
_Where modules touch, what is injectable, and why testing can stay offline._
32+
33+
## Current shape, for the record
34+
35+
_Counts and invariants a reader can check against the code: how many checks, how many
36+
adapters, which defaults._

0 commit comments

Comments
 (0)