Skip to content

Latest commit

 

History

History
237 lines (163 loc) · 29.2 KB

gates.md

File metadata and controls

237 lines (163 loc) · 29.2 KB
title Gates
description I changed some files. Which command do I run before I push?
icon shield-check
noindex true

If you changed anything at all, run pnpm verify (format:check, lint, typecheck, test:unit; about two minutes). If you also touched the tool surface, install, desktop or Rust, section 1 below names the one extra command your change needs. CI is path-routed too, and gate counts a skipped job as green, so a gate you skip here may also be skipped in CI.

One question this file answers: I changed some files. Which command do I run?

The why behind the gate design (tiers, the merge-gate/release-gate split, what is still unbuilt) lives in gate-plan.md. This file is the routing table.


0. Last verified

Every gate below was executed end to end against main on 2026-08-12 (macOS, M-series, v2.6.0), except the four re-run for the v3.1.0 release review on 2026-09-16 (macOS, M-series, feat/v3.1.0): verify, test:e2e, gate:install and lint:docs. A green row means somebody watched it go green, not that it is supposed to be green. A row nobody has watched since v2.6.0 is a row making a claim about a different codebase.

Rows marked ↻ were re-measured on 2026-09-09 (macOS, M-series, this branch). The numbers they replace were wrong by a lot, and wrong in the way that is hardest to notice: test:unit was recorded here as "5,725 tests / 613 files" and, forty lines further down in the same file, as "4,315 tests". Two copies of one number, neither derived from anything. The battery was 33 in one row and 32 in another, against 38 spec files on disk.

A ↻ row's COUNT was re-derived; a ↻ row's ✅ is still the August sweep unless the row says otherwise. test:unit was actually re-run (turbo run test:unit test:guards --force, then pnpm test:bench, summed from vitest's own per-package output). The battery's spec count was recounted from ORDER and DESKTOP in apps/e2e/run.mjs cross-checked against ls apps/e2e/specs; it was not re-run, so its ✅ and its wall clock are still August's.

Gate Result Wall clock
pnpm build ✅ ~40s
pnpm lint (+ both guard self-tests) ✅ ~15s
pnpm typecheck ✅ ~10s
↻ pnpm test:unit ✅ 9,210 tests / 934 files across 10 packages plus the bench harness ~70s
pnpm format:check ✅ ~10s
pnpm test:integration ✅ 14/14, re-measured 2026-09-11 17s
↻ pnpm test:e2e ✅ 39/39 specs, 334 checks, on 2026-09-11. The battery now has 44 specs. This row was not re-run for 3.8.0 490s (Aug)
pnpm test:e2e:desktop ✅ 3/3 (Electron 22, electron-vite 6, Tauri 18), 2026-10-01; includes native headless visibility and idle durability 58s
node apps/e2e/soak.mjs --self-check ✅ <1s
node apps/e2e/matrix.mjs --self-check ✅ <1s
pnpm matrix:compat --only cursor ✅ 4/4 ~10s
pnpm bench ✅ 10/10 (was failing before this sweep, see below) 279s

pnpm bench was broken and nobody knew. suite-rre.mjs recorded four flows that asserted no observable consequence and then demanded a pass verdict from whole-suite replay, which correctly grades an assertion-free suite unverifiable. The product got more honest about false greens; the benchmark measuring it did not follow, so the whole run aborted at script 9 of 10 and replay-determinism never ran at all. Each flow now carries a success oracle.

That paragraph used to end "nothing in CI runs it, so it can only rot silently", and it is left here because it is the reason the bench job below exists. It is no longer true: ci.yml has a bench job that runs pnpm bench:full then pnpm bench:gate, path-routed on the files that can move token cost. What is still true is the narrower claim in section 4: bench/ as a whole is measurement, and only the replay + observation-cost numbers are gated.

pnpm gate:install was run on 2026-09-11 and passed 10/10 scaffolds, which is a stronger claim than this paragraph used to make. It matters this release because pnpm -r publish selects thirteen packages and two of them gained a prepack during v3; the gate is the only thing that runs those prepacks against a real registry.

The Windows and Rust jobs are still not run here: they are CI-only. They are green on main per the last CI run, which is a weaker claim than every row above and is stated that way on purpose.

Tauri headless mode now hides the native window on macOS 14+. The earlier offscreen workaround could leave the window on a display; the 2026-10-01 run caught a window close followed by a normal app exit during the idle check. Reticle now disables WebKit background throttling before hiding the loaded window. The packaged gate confirms native invisibility, successful concurrent captures, and command response after an idle pause. Older macOS versions retain the offscreen fallback; their runtime behavior was not reverified in this sweep.


Quick product check

Run pnpm test:smoke after pnpm install --frozen-lockfile. Install Chromium once with pnpm exec playwright install chromium (Linux CI uses --with-deps). The command builds the server and its dependencies, starts an isolated fixture and daemon, connects over stdio MCP, and drives a real Chromium session. It needs no scaffold generator, registry, API key, or running app. It checks a true assertion, a click with both DOM and HTTP consequences, a false assertion, and a deliberately broken button. Unknown or inconclusive results fail. A separate counter in the fixture server verifies that exactly one write happened.

Evidence is written to artifacts/smoke/: result.json, calls.json, daemon output, and MCP stderr. The browser phase has a 120-second watchdog; duration is recorded, not used as a performance assertion. This is a quick check of the default MCP surface and zero-install reader. It does not replace framework installation, state/source mapping, desktop, or full E2E coverage. Its runtime is not yet qualified on CI; local verification in the restricted workspace cannot bind loopback ports.

The smoke CI job is included in gate and preserves the first failure without retrying it. The longer E2E battery remains required independently. The npm publish workflow also requires this smoke before publishing, including on a manual dispatch.

1. The routing table

Find the row that matches what you changed. Run its commands. That is the whole rule.

You changed Run Cost
Quick product feedback pnpm test:smoke recorded in artifacts/smoke/result.json; CI timing pending
Anything at all pnpm verify ~2 min
The tool surface, the wire contract (core), or an observer ↑ and pnpm test:e2e +~8 min
reticle init, @reticlehq/vite-plugin, @reticlehq/next, @reticlehq/babel-plugin, anything a user runs before their first session ↑ and pnpm gate:install +~15 min
@reticlehq/electron, adapters/realm/tauri, the IPC observer, desktop capture ↑ and pnpm test:e2e:desktop +~3 min
open-verification, the adjudicator, WebRealm, or anything a verdict is derived from ↑ and pnpm gate:conformance +~3 min
Session selection, the artifact root, or anything that tells git worktrees apart ↑ and pnpm gate:worktrees +~20s
Telemetry, feedback, or anything that emits an event ↑ and read contributing/telemetry/contract.md first. pnpm test:e2e covers it (telemetry-events-test) n/a
adapters/realm/tauri (Rust) ↑ and cd adapters/realm/tauri && cargo fmt --check && cargo clippy --all-targets -- -D warnings +~2 min
Docs, README, comments only pnpm format:check seconds

Why routing exists. The full set is roughly 35 minutes. A gate people resent is a gate people route around, so only the tier that can see your change is worth your time.

CI routes too, so this table is not just about your time. It used to be true that CI ran everything regardless, which made skipping a local gate cost you nothing but a slower red. That is no longer so: the expensive gates only run when a path they care about changed, on the same reasoning as the table above. Machine time spent proving that a documentation edit did not break an Electron app is machine time nobody reads the result of.

Which means a gate skipped locally can also be skipped in CI, if what you changed did not match its paths. The routing is deliberately generous, and the paths are checked (server/src/ci-routing-paths.test.ts fails if a path in the workflow matches no real file, so a rename cannot quietly switch a gate off). But if you are doing something the paths would not predict, run the gate rather than assuming.

CI gate runs when
verify (lint, docs lint, types) and unit-tests (the unit suite), in parallel; format-check is its own job always
macos always, but only a narrow platform-sensitive slice of the tests
windows a pull request that touches TypeScript source, scripts or the lockfile; always in the merge queue, main, nightly and on demand; full build, typecheck, unit suite, and daemon runtime checks
smoke product changes on PRs; always in the queue and on main, nightly, and manual runs
e2e a pull request that touches a package or app the battery boots; always in the merge queue and on main. Split into three parallel shards (E2E_SHARD=k/3); the integration suite and the soak run in shard 1
rust (Linux) and rust-macos changes to the standalone Cargo tree or CI workflow; full runs nightly and on demand
install-packages and install-gate installation paths, CLI setup, build adapters, browser SDK, dependency graph, or gate machinery. A pull request runs three Linux scaffolds and vite-react on Windows. The merge queue, the nightly and a manual run run all thirteen scaffolds on Linux and on Windows. One prepack produces the same tarballs for all cells
the install gate's self-test gate machinery changes, nightly, and manual runs. --with-self-test shares one registry publish across the control and positive phases
desktop-e2e only when desktop code changed
early-gates and break-matrix a change to init/, server/src/command/cli/, server/src/command/setup/, install/ or break/; nightly and on demand. The shipped CLI's first-run refusals on Linux, macOS and Windows; the hostile-environment matrix
launchers a change to install/; nightly and on demand. Both launchers parse, are pure ASCII, and install the published CLI on Linux, macOS and Windows (PowerShell 5.1 and 7)
package-quality always: publint, are-the-types-wrong and the pack-size budgets for every published package
bench in the merge queue and on a push to main when something that could move the numbers changed, nightly, and on demand. It is ~19 minutes, so a pull request runs it only when it changes the tool surface (server/src/surface/ or core/src/wire/tool-names.ts): those changes move the price on every turn, and review needs the number. The queue stops a token regression before it reaches main

The install control blocks Reticle WebSockets in the browser, leaving the daemon alive. Every requested scaffold must fail only its session check and must have attempted the blocked socket. Registry errors, crashes, inconclusive transport, missing results, and an unexpectedly passing control fail qualification. There are no scaffold waivers. The positive phase additionally calls real MCP assertions and a DOM action through the installed SDK; hasCapabilities alone cannot pass it. pnpm gate:install --only vite-react --with-self-test exercises both phases locally.

CI's install-packages job runs the real prepacks once and uploads tarballs plus a manifest. Each cell validates the commit, package inventory, versions, and SHA-256 digests before restoring emitted code and publishing those tarballs to its private Verdaccio. Init still performs a real dependency install. The Windows cells consume the Linux-built packages, as consumers of a Linux-built release do; Windows compilation remains covered by the Windows job. An ordinary local pnpm gate:install still runs prepack itself. INSTALL_GATE_PACKAGES=<directory> explicitly selects a prepared artifact; node scripts/pack-install-gate.mjs <empty-directory> produces one.

This removes repeated package compilation, not framework coverage. Generator downloads still use the upstream scaffold versions declared in the gate, including floating latest references; npm's content cache reduces downloads but does not make those dependency resolutions reproducible. Before claiming a latency or flake-rate improvement, compare completed PR and merge-group runs on GitHub, including first-attempt failures and time waiting for runners. That qualification is pending.

2. Every gate, and what each one can actually see

Each gate exists because the ones above it are blind to something. That blindness is the column that matters.

Gate Command Proves Blind to CI job
Build pnpm build every package compiles and emits anything at runtime verify
Lint pnpm lint style rules, plus the dependency-boundary and lossy-transform guards anything not expressible as a rule verify
Typecheck pnpm typecheck types agree across package boundaries runtime behaviour verify
Unit pnpm test:unit every package's unit tests, no browser anything crossing a package boundary at runtime unit-tests
Repo guards turbo run test:guards the @reticlehq/server tests that scan OTHER trees (docs, apps/, bench/, the workflows, the skills), held apart from the unit suite so the wide cache key is 2 seconds rather than 40 anything inside a package unit-tests (inside pnpm test:unit)
Format pnpm format:check Prettier n/a format-check
JavaScript security audit node scripts/check-js-audit.mjs --self-test && node scripts/check-js-audit.mjs critical dependency advisories fail and high ones are printed as warnings without failing; two exact, unpatched development-tool paths are reviewed exceptions documented in repository settings; changed paths, versions, dependency placement, or patch availability fail vulnerabilities absent from the advisory registry; the reviewed exceptions until patched verify (CI only)
Integration pnpm test:integration real headless Chromium: browser pool, crash isolation, framework adapters, withReticle the MCP surface, the daemon e2e
Web e2e battery pnpm test:e2e 44 specs (ORDER in apps/e2e/run.mjs) against 3 booted servers and a real browser (the tool surface, the daemon lifecycle, transport faults, telemetry, trace shape), plus the soak desktop runtimes; the install e2e
Desktop battery pnpm test:e2e:desktop two real Electron main processes (plain Vite + electron-vite) and a packaged Tauri binary, driven headless web-only paths desktop-e2e
Conformance pnpm gate:conformance drives the published specification's scenarios against this implementation on a real browser, a real Electron shell AND a command-line subject (run-self, run-desktop, run-cli), scoring every verdict through the spec's own adjudicate rather than Reticle's kernel the seven scenarios no fixture can plant, reported ABSENT, plus the four above the claimed profile, reported not asked. All three arms are negatively controlled: gate:conformance:self-test PLANTS a subject answering yes to every claim and requires the gate to refuse it. Both halves are needed and they catch different things -- the coverage floor catches a suite whose plantable scenarios are FALLING, the plant catches a gate that has gone BLIND. The CLI arm briefly had a --self-test that only inspected the run it was handed, which on a healthy tree exits non-zero and would redden CI on a working build; it plants at the client seam now, the way the browser arm does. One scenario still passes under the lie and should: healthy-app-real-claim genuinely expects a yes. The lie is injected ABOVE the realm, so a supervisor that spawned nothing is not caught by this control alone ~3 min; runs in CI as the conformance job, path-routed on the protocol, the suite, the realm and the two fixture apps
Install gate pnpm gate:install scaffolds 13 pristine apps across 2 OSes (3 Linux and 1 Windows cell on a pull request; all 26 in the merge queue, nightly, on main and on demand), publishes this checkout to a local Verdaccio, lets init install itself, boots each app in a real browser, polls for a session that advertised capabilities install complexity; see fixtures.md install-gate
Matrix records pnpm matrix:validate every submitted client-compat record is well-formed whether the client actually works matrix-records
Windows (CI only) that the code runs at all on the majority platform: build, typecheck, the unit suite and the daemon runtime checks the e2e battery windows
macOS (CI only) the POSIX-but-not-Linux surface (paths, spawn, lsof, temp/state dirs) plus a real daemon's lifecycle everything Linux already covers; it deliberately no longer re-runs the unit tier at a 10x billing multiplier macos
Rust cargo fmt/clippy/check adapters/realm/tauri compiles and lints on Linux, macOS, and cross-checks Windows everything JS rust and rust-macos (when the Cargo tree or the CI workflow changed; full runs nightly and on demand)

The single required status check is gate. It passes when every job above either succeeded or was deliberately skipped by path routing, and fails on anything else. Adding a job to ci.yml is half the work; adding it to gate's needs: list is the other half. A job missing from that list runs, reports, and is structurally incapable of blocking a merge.

Guards that self-test

Five checks prove they can still fail before they are trusted. A guard that has never refused anything is not a guard, so each has a negative control that CI runs first:

node scripts/check-boundaries.mjs --self-test        # catches a synthetic bad dependency graph
node scripts/check-lossy-transforms.mjs --self-test  # catches an unclassified lossy export
node apps/e2e/soak.mjs --self-check                  # the soak gate still reddens on an unanswered call
node apps/e2e/matrix.mjs --self-check                # the validator still refuses a malformed record
pnpm gate:install:self-test                          # mis-wires every scaffold; the gate MUST go red

The install gate also compares init's plan with apps/e2e/install-baseline.json, because a step that changes its mark or leaves the plan passes every other check. If you change the plan on purpose, run pnpm gate:install --update-baseline and commit the diff. Add --only <id> to record one scaffold again; the write merges, so the other scaffolds stay.


3. Gates that are not automatic

These are real and they work; they are not on the PR path, so they only run when somebody runs them.

Command What it is When
pnpm gate:soak:record a short recorded soak (20 rounds, 2 s idle). It re-records bench/soak-history.jsonl and bench/TOOL-PROFILE.md. For the half-hour release soak, run node apps/e2e/soak.mjs --rounds 60 --idle-ms 30000 --record before a release
pnpm matrix:compat drives each MCP client's config exactly the way that client would; writes a machine record for docs/matrix/ before a release, per client
pnpm knip unused files, exports, and dependencies when the repo feels heavy
node scripts/verify.mjs in the sibling reticle-fixtures repo installs this commit into each real third-party app in its fixtures.json (install complexity, not regressions) see fixtures.md

4. bench/ is not a gate

bench/ is measurement and research. Most of it blocks nothing and is allowed to bit-rot in a way a gate is not. But "nothing runs in CI" is no longer true, and was left standing here for a release after it stopped being: ci.yml has a bench job, gated on changes.bench, that runs pnpm bench:full and then pnpm bench:gate. It is the only thing in CI that measures TOKEN COST, which is how a release once shipped a measurable token regression with every other gate green. Read bench/README.md before touching it: it says which scripts are live and which are one-off studies kept as evidence for a published claim.

The one exception worth knowing: pnpm bench + pnpm bench:gate is a working regression gate for the replay numbers, and it is run by hand before a release.


5. When a gate fails and you think it is the gate's fault

Sometimes it is. The specific failures worth recognising:

  • EADDRINUSE / "died during boot". A previous run left something on :8787, :4310, or :3100. run-ci.sh stops its own fixture processes on exit. An occupied port is refused; identify its owner before stopping anything.
  • Killing port 4400 with lsof -ti tcp:4400 | xargs kill -9. This SIGKILLs the reticle mcp proxy too, because the proxy holds a client socket on the bridge port. Do not kill by port. The web and desktop batteries now default to 14400 (RETICLE_PORT overrides it), and cleanup requires recorded process ownership. Framework integration uses 15400 (RETICLE_INTEGRATION_PORT); conformance uses 15401 (CONFORMANCE_BRIDGE_PORT).
  • A timing assertion. If a test asserts Date.now() - t < N, that is a bug in the test, not a flake to re-run. Assert the bound (output size, a truncation flag), or use a generous per-test timeout. See harness-rules.md.
  • An INCONCLUSIVE verdict. The harness is telling you the transport did not stay up, so it is claiming nothing about the product. That is the harness working, not the product failing.

The four rules every tier obeys, and the incident behind each, are in apps/e2e/harness-rules.md.

Reorganising a directory

A directory with a long flat listing is a symptom, not a defect. What says whether it is badly organised is the number of directory pairs that reach for each other -- those cannot be read, moved or tested apart -- and server/src/directory-reach.test.ts already tracks it.

So a grouping is an improvement only if that number stays flat. Check before you move:

node scripts/safe-to-group.mjs server/src/surface/tools harness-explore harness-hunt harness-gate

It answers SAFE or UNSAFE by the same rule the test uses: a group is unsafe exactly when some directory it reaches out to also reaches back into it. It is a prediction; the test decides.

Two rules learned the expensive way:

  • Move only files that import no sibling. A group containing something that imports back into its old home creates a mutual pair with its own parent.

  • Six registries key on a source PATH, and a move breaks them silently. They are correct and useful individually; together they are a class, and the class was discovered one move at a time. Grep for the file you are moving before you move it. In the order they were found:

    Registry What it holds
    dispatch-attribution.test.ts files allowed to dispatch an ACT
    library-path-boundary.test.ts declared crossings into the install-time surface
    on-disk-versions.test.ts every versioned on-disk format
    orphan-modules.test.ts modules with no production importer, and why
    config-search-depth.test.ts the config walkers, opened by path
    presenter-dead-exports.test.ts retired artwork that must stay out of the bundle

    A general guard was attempted and abandoned: a path in a string cannot be told apart from a fixture filename or an output path without guessing, and the noisy version of this check is worse than none: it would be switched off inside a week and take real coverage with it.

  • Being a leaf among siblings is not enough. A file can import no sibling and still close a cycle once its directory has a name -- tool-kit.ts imports no sibling and reaches flows, and flows reaches back. Invisible while both sat in one directory.

Why the conformance gate is not inside a battery

pnpm gate:conformance runs on its own rather than as a spec in pnpm test:e2e, and the reason is worth stating so nobody "tidies" it in:

  • The web battery boots the demo API with REFLECT_MS=6000. The conformance runner boots its own on the same port. Folded into the battery it would find the battery's backend already answering, score every scenario against a deliberately-slowed API, and still print a number. A verdict suite silently measuring a different system is the exact failure it exists to catch.
  • It gates REGRESSION, not the score. earned is none and stays none until the fixture grows: half the scenarios have no plant on any subject here, and ABSENT is never a pass. --gate fails only when a scenario that could be planted was driven and answered wrongly. That number is zero today, so the gate is clearable; gating on earned would be a red board nobody can clear.

It runs in CI as the conformance job, path-routed on the protocol, the suite, the realm, the two functions in core that build a protocol subject, and the two fixture apps the scenarios are planted into. That is a narrower question than "could the desktop battery break", so it has its own filter rather than borrowing one. It is in the gate aggregate, so it can actually stop a merge.

It is a separate job rather than a step inside e2e for two reasons. That job boots the demo API with REFLECT_MS=6000, so a conformance run beside it would score every scenario against a deliberately-slowed backend while still printing a number. Both also want port 8787. It stays cheap despite driving Electron, because the desktop half needs a display and not Rust: it drives apps/electron-smoke, not the packaged Tauri binary that makes desktop-e2e the slowest job in the file.

Run it by hand too when you touch the protocol, the adjudicator, or anything a verdict is derived from; the path filter is a routing decision, not a definition of what can break a verdict.

Path routing

CI path routing lives in scripts/ci/changes.mjs, and its output is written to the changes job's step summary. Each changed file is mapped to its nearest workspace package in scripts/ci/tier-map.json, which can add only e2e and windows; every other tier comes from the regexes in scripts/ci/tier-patterns.json. A path that matches no tier and is not docs-only, or a package missing from the map, turns every expensive job on. Any file under .github/workflows/ or scripts/ci/ does the same.

Flake quarantine

.github/flaky.json is the only place a tolerated gap may be named. Each entry needs an issue and an expires date. The flaky-quarantine job checks the file with scripts/ci/flaky-expiry.mjs. An entry with no issue or no expiry always fails it. An expired one fails it on the nightly, a manual run on main, and any pull request that edits flaky.json; anywhere else it is a warning, so a date alone never reddens an unrelated change. Nothing waits on that job, so an expired entry never stops the nightly from running the jobs it is about. gate still requires it.

An entry means one of two things, depending on what it names:

  • e2e or integration: a tolerated flake. The job runs as usual. Without an entry, a failure fails on the first attempt. With one, a failure retries once and prints a warning, and every attempt's logs are uploaded, including a later green one. No entry of this kind exists today.
  • bench or an install cell (install-gate/<os>/<scaffold>): a job that does not run in the merge queue yet. It runs after merge, nightly and on a manual run, with no retry. The parity guard (post-merge-parity.test.ts) requires an entry for every job or install cell that runs after merge but not in the queue. No entry of this kind exists today. A job that needs one joins the merge queue, and leaves this file, after three consecutive green nightlies on main.

A red nightly

nightly-watch runs after gate on the scheduled run, and on a manual run on main. When anything failed it opens one Nightly CI is red issue labelled nightly-red, or comments on the open one. Forks skip the schedule; a dispatch still runs everything anywhere.