Skip to content

docs(break-test): pass 8 report - network-off and root probes clean, one Low finding in handoff's subdirectory message - #279

Merged
StepenkoAnatoli merged 2 commits into
mainfrom
arena/9aac6bbb-research-kit
Oct 11, 2026
Merged

StepenkoAnatoli merged 2 commits into
mainfrom
arena/9aac6bbb-research-kit

Conversation

@StepenkoAnatoli

Copy link
Copy Markdown
Owner

Documents-only: the break-test skill (pass 8) run against this repository at 195b2ca, one commit, one new report.

What changed

  • docs/build-report-2026-10-11-break-test.md (new) - the pass-8 report: the probe set, the single finding, what was tried and abandoned, and what was not probed.
  • docs/README.md - the reading map gains its row.

No kit code, no tests, no corpus.

Why

The break-test skill was run on this repository. Its deliverable is the report; the previous pass's report lives beside it (pass 7, 2026-10-07), which this one continues.

What it found

  • Baseline: three full-suite runs at 195b2ca, each 1782 passed, 0 failed, empty failure set, 4 unsupported (the browser-guard/browser-transport LIVE tests; no Chromium on the probe host), exit 1 per ADR-0108. Root gate: preflight PASS, handoff OK, doctor exit 1 with its one by-design blocker.
  • The gaps pass 7 left open are closed. Pass 7 could not isolate the network (unshare -n refused). Here unshare -rn works, so the suite ran with no DNS and no outbound sockets: 1782/0/4, identical to baseline. A root-in-userns control (unshare -r) was also identical. Pass 7's other gap - no dedicated repeated-run flakiness programme - is closed by 70 repeated runs of the seven timing-sensitive files (concurrency, http-linear, proxy, large-response, panel, stdout-flush, browser-guard): zero flakes. Seven full passes in total (three baselines, root, no-network, TZ=Pacific/Kiritimati, two simultaneous suites) with the same 1782/0/4.
  • One finding, not fixed:

    [F1] handoff from inside a project's research/ folder says nothing has been collected and advises collecting there (Low, Possible)
    Repro: cd <project>/research && node "$HOME/.agents/research-kit/bin/handoff.mjs" prints "nothing has been collected in this project … Collect it here", while the parent directory holds a complete corpus and prints OK. Harm is bounded - the suggested command refuses to spend (research.mjs there exits 2, no plan.json) - but the diagnosis is wrong and points at a metered action. Not fixed on purpose: detecting "this cwd is the project's research/ folder" is new behaviour, and the freeze (ADR-0117) makes it the owner's decision. The report carries a recommended minimal change.

  • Everything else probed produced a named diagnostic rather than a crash: no HOME, minimal env -i, no git on PATH, read-only HOME and research/, RESEARCH_KIT_HOME naming a regular file, a taken panel port, a nonexistent --project, a missing capture, a CRLF-smudged corpus, Hebrew + space paths, a 272-character path, unicode TMPDIR, ulimit -n 64, core.autocrlf=true.

Verification

  • Verified: selftest.mjs on the committed head in the real checkout - 1782 passed, 0 failed, failure set empty, 4 unsupported, exit 1, 172.0 s, 87 files, Node 22.22.3.
  • Verified: the four CI extra steps by hand - release example 6/6 PASS; the three Node<->Python conformance pairs agree per vector; timeline.mjs (60 events) and export-warc.mjs (55 records, 27 captures) regenerate; git status --porcelain empty. preflight PASS (0 blocking, 1 hygiene/brief-unstamped warning), handoff OK (33 ledger entries, 27 evidence rows, chain verifies).
  • Verified: the two doc-facing test files - selftest.mjs repo-hygiene support-policy 31 passed / 0 failed - and git diff --check clean.
  • Not run: Windows, macOS, Node 24/26, and the browser LIVE tests (no Chromium on the probe host). CI covers the first three on this PR. The browser LIVE tests stay the report's top open risk.
  • Environment note: the probe host differs from pass 7 in one useful way (unshare -rn permitted) and is otherwise the same kind of container; the report states both, and its "Not probed" section is explicit about the rest.

What I got wrong

Two probe defects of mine, both recorded in the report and not counted as kit findings: the first network-off run was started without ip link set up on loopback (its FAILs were mine; discarded, re-run clean), and one preflight invocation used a kit-relative path from a project folder that does not contain the kit (Node MODULE_NOT_FOUND).

Merge

The report is documentation only and the branch carries no kit change; merging is the operator's call. No gate override of any kind was used.

…, one Low finding

what changed: docs/build-report-2026-10-11-break-test.md is new, and docs/README.md's
reading map gains its row. No kit code changed.
why: the break-test skill was run against this repository at 195b2ca; the report is its
deliverable and records the probe set, the one finding, and what was not probed.
what you verified: selftest.mjs 1782 passed / 0 failed / 4 unsupported, three times in the
real checkout with an empty failure set; preflight PASS; handoff OK; the four CI extra
steps (release example 6/6, three conformance pairs agree per vector, timeline and
export-warc regenerate to the committed bytes, tree clean); the probe set recorded in the
report; git status --porcelain empty before and after.
what you got wrong: two probe defects, both recorded in the report - a network-off suite
run started without `ip link set lo up` (discarded, then re-run clean) and one preflight
invocation from a folder that does not contain the kit.

Co-authored-by: arena-agent <297053741+arena-agent@users.noreply.github.com>

Copy link
Copy Markdown
Owner Author

CI record for 9b7eb1c: all seven required jobs passed on the first push (run 38125774529) - platform (ubuntu-latest) 3m10s, platform (ubuntu-26.04) 2m53s, platform (windows-latest) 8m26s, node (24) 3m22s, node (26) 3m27s, archive tree 3m16s, aggregate suite pass. Note for the record: the Windows leg was the 2026-10-11 flake host in the report's risk 1 (the LIVE browser-guard meta-refresh test, which passed here). The four browser LIVE tests remain the report's top open risk; this run did not exercise them again.

@StepenkoAnatoli
StepenkoAnatoli merged commit 2faaf0e into main Oct 11, 2026
7 checks passed
@StepenkoAnatoli
StepenkoAnatoli deleted the arena/9aac6bbb-research-kit branch October 11, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant