Skip to content

Commit 830fc18

Browse files
elliotBraemelliotBraem
committed
tests
1 parent f90face commit 830fc18

7 files changed

Lines changed: 265 additions & 35 deletions

File tree

‎.github/workflows/README.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,12 @@ Workflows:
99
- `CI` — lint, audit, typecheck, framework tests, and the vital-plane suites (pull requests only; slow regression suites are `workflow_dispatch`-only for triage)
1010
- `Deploy` — production deployment via `bos deploy` (build → bundle upload → FastKV publish → GHCR image push → Railway; triggered by push to `main`)
1111
- `Staging` — the same deployment against the staging environment (`--env staging`, triggered by push to the `staging` branch)
12-
- `Release` — changeset versioning and npm publish (triggered by push to `main`, canonical changesets flow); when a publish is imminent, the full regression matrix gates it first
13-
- `Full regression` — reusable matrix (`regression-full.yml`): unit + dev + start + backcompat + framework regression, serially, on one runner. Consumed by the Release gate and the nightly
14-
- `Backcompat nightly` — nightly main-tip proof: the registry sentinel plus the full regression matrix via `regression-full.yml`
12+
- `Release` — changeset versioning and npm publish (triggered by push to `main`, canonical changesets flow); when a publish is imminent, the regression gate (`tier: release` — unit + start + backcompat) gates it first
13+
- `Full regression` — reusable matrix (`regression-full.yml`): unit + dev + start + backcompat + framework regression as PARALLEL planes (wall = slowest plane), tiered by input — `tier: release` = unit + start + backcompat (the publish gate), `tier: nightly` = the full matrix (the nightly canaries the dev + framework planes)
14+
- `Backcompat nightly` — nightly main-tip proof: the registry sentinel plus the full regression matrix (`tier: nightly`) via `regression-full.yml`
1515
- `consumer-ci` / `consumer-deploy` / `consumer-staging` — reusable workflows for generated child repos (see [Consumer workflows](#consumer-workflows))
1616

17-
Design: the test tiers run where a decision is made (docs/agents/testing.md) — T0 local lefthook hooks, T1 vital-plane CI on pull requests, T2 the full regression matrix before a release and nightly on the main tip. There is no `push: main` CI trigger and no merge queue. `Deploy` and `Release` trigger directly on push and gate themselves: they re-verify the commit is still the main tip (stale runs skip; the run for the new tip deploys/releases instead), and Deploy additionally verifies the commit landed as a pull request (PR provenance — a raw direct push has no associated PR and fails the gate; `workflow_dispatch` with the `force` input is the deliberate escape hatch).
17+
Design: the test tiers run where a decision is made (docs/agents/testing.md) — T0 local lefthook hooks, T1 vital-plane CI on pull requests, T2 the regression gate (unit + start + backcompat, `tier: release`) before a release and the full matrix nightly on the main tip (`tier: nightly` — parallel planes). There is no `push: main` CI trigger and no merge queue. `Deploy` and `Release` trigger directly on push and gate themselves: they re-verify the commit is still the main tip (stale runs skip; the run for the new tip deploys/releases instead), and Deploy additionally verifies the commit landed as a pull request (PR provenance — a raw direct push has no associated PR and fails the gate; `workflow_dispatch` with the `force` input is the deliberate escape hatch).
1818

1919
The changesets version PR never runs the matrix: `detect-changes` fast-paths `changeset-release/*` PRs to an all-skipped run, so the required `CI gate` check goes green immediately (`package.json`-only diffs would otherwise trip every `detect-changes` arm). That plane is validated by release.yml instead — its regression gate runs the full matrix before any actual publish. The version PR merges via the manual merge button.
2020

@@ -73,9 +73,10 @@ The changesets version PR never runs the matrix: `detect-changes` fast-paths `ch
7373
(a maintainer merge button click)
7474
4. Version Packages PR merges → no changesets remain
7575
↓
76-
Full regression gate
77-
(regression-full.yml — the matrix
78-
must be green before npm sees bytes)
76+
Regression gate (tier: release)
77+
(regression-full.yml — unit +
78+
start + backcompat must be green
79+
before npm sees bytes)
7980
↓
8081
pnpm run release (scripts/publish-release-packages.ts)
8182
↓

‎.github/workflows/backcompat-nightly.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,3 +53,5 @@ jobs:
5353
full-regression:
5454
name: Full regression (main tip — unit + dev + start + backcompat + framework)
5555
uses: ./.github/workflows/regression-full.yml
56+
with:
57+
tier: nightly

‎.github/workflows/regression-full.yml‎

Lines changed: 235 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,53 @@
1-
# Root-only full-regression matrix — never synced to children (same class as
2-
# release.yml). The T2 tier of docs/agents/testing.md: consumed by the
1+
# Root-only full-regression matrix -- never synced to children (same class
2+
# as release.yml). The T2 tier of docs/agents/testing.md: consumed by the
33
# release gate (release.yml: every npm publish is preceded by a green run)
44
# and the nightly main-tip check (backcompat-nightly.yml). Per-PR CI does
5-
# not call this — the tiering contract keeps PRs on the vital plane and
5+
# not call this -- the tiering contract keeps PRs on the vital plane and
66
# pays the matrix exactly where a release is decided.
7+
#
8+
# Two tiers of the same matrix, selected by the 'tier' input, run as
9+
# PARALLEL planes (wall = the slowest plane, not their sum):
10+
#
11+
# nightly -- the full matrix: unit + dev + start + backcompat + framework.
12+
# release -- the publish gate: unit + start + backcompat only. The dev
13+
# and framework planes pin the 'bos dev' machinery (source-mode dev
14+
# server, process death) -- dev-plane concerns app-code changes cannot
15+
# break, whose user journeys the start plane's full suite already
16+
# covers. The gate proves the production artifact (start) and the
17+
# published-chain contract (backcompat); the nightly canaries the dev
18+
# plane. ci.yml keeps workflow_dispatch triage jobs for every plane.
19+
#
20+
# Planes share one setup shape (install -> build template -> playwright)
21+
# and differ in what they additionally pay: 'start' builds the docker
22+
# fixture in-job (no prebuilt image on called runs -- the regression-image
23+
# job is a ci.yml dispatch-run optimization), 'framework' pays neither
24+
# playwright nor the template build (its stacks resolve every-plugin from
25+
# TS source, ADR 0018). Skipped planes (release tier) never fail the
26+
# called workflow.
27+
728
name: Full regression
829

930
on:
1031
workflow_call:
32+
inputs:
33+
tier:
34+
description: "release = unit + start + backcompat / nightly = full matrix incl. dev + framework"
35+
required: false
36+
type: string
37+
default: "nightly"
1138

1239
permissions:
1340
contents: read
1441

1542
jobs:
16-
regression:
17-
name: Full regression (unit + dev + start + backcompat + framework)
43+
start:
44+
name: Start plane (http + browser, full suite)
1845
runs-on: ubuntu-24.04
19-
timeout-minutes: 90
46+
timeout-minutes: 45
2047
env:
2148
CI: true
2249
CORS_ORIGIN: http://localhost:4100
2350
BETTER_AUTH_SECRET: regression-test-secret-do-not-use-in-production
24-
BOS_TEARDOWN_ADOPTION: "1"
25-
BOS_FRAMEWORK_STACK_SLOTS: "2"
26-
BOS_FRAMEWORK_READY_TIMEOUT_MINUTES: "4"
2751
steps:
2852
- name: Checkout repository
2953
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 (Node 24)
@@ -63,29 +87,219 @@ jobs:
6387
run: pnpm run test:regression:browser:install
6488

6589
- name: Free disk space
66-
# The start suites' in-job `docker build --target regression` fallback
67-
# needs headroom for the container's fresh pnpm store, the full
68-
# node_modules COPY into the regression stage, and the export layers.
90+
# The start suites' in-job docker build needs headroom for the
91+
# container's fresh pnpm store, the full node_modules COPY into the
92+
# regression stage, and the export layers.
6993
run: |
7094
sudo rm -rf /usr/local/lib/android /usr/local/.ghcup /opt/hostedtoolcache/CodeProfiler /usr/local/share/boost || true
7195
sudo docker image prune -af || true
7296
df -h /
7397
74-
- name: Run the full regression matrix
75-
run: |
76-
pnpm run test:regression:unit
77-
pnpm run test:regression:dev
78-
pnpm run test:regression:start
79-
pnpm run test:regression:backcompat
80-
pnpm run test:regression:framework
98+
- name: Run the start plane (http:start + browser start:ssr + start:csr)
99+
run: pnpm run test:regression:start
81100

82101
- name: Upload regression artifacts on failure or cancel
83102
if: ${{ !success() }}
84103
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 (Node 24)
85104
with:
86-
name: regression-full-artifacts
105+
name: regression-start-artifacts
87106
path: |
88107
.bos/logs/**
89-
tests/regression/fixtures/death-contract/.bos/logs/**
90108
playwright-report/**
91109
test-results/**
110+
111+
unit-backcompat:
112+
name: Unit + backcompat plane
113+
runs-on: ubuntu-24.04
114+
timeout-minutes: 30
115+
env:
116+
CI: true
117+
CORS_ORIGIN: http://localhost:4100
118+
BETTER_AUTH_SECRET: regression-test-secret-do-not-use-in-production
119+
steps:
120+
- name: Checkout repository
121+
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 (Node 24)
122+
123+
- name: Setup pnpm
124+
uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
125+
126+
- name: Setup Node
127+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
128+
with:
129+
node-version: "24"
130+
131+
- name: Setup Go
132+
uses: actions/setup-go@7a3fe6cf4cb3a834922a1244abfce67bcef6a0c5 # v6.2.0 (Node 24)
133+
with:
134+
go-version: "1.22"
135+
136+
- name: Get pnpm store dir
137+
id: pnpm-cache
138+
run: echo "STORE_PATH=$(pnpm store path)" >> "$GITHUB_OUTPUT"
139+
140+
- name: Cache pnpm store
141+
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 (Node 24)
142+
with:
143+
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
144+
key: ${{ runner.os }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
145+
restore-keys: |
146+
${{ runner.os }}-pnpm-
147+
148+
- name: Install dependencies
149+
run: pnpm install --frozen-lockfile --ignore-scripts
150+
151+
- name: Build every-plugin dist
152+
run: pnpm run build template
153+
154+
- name: Run regression lib unit tests
155+
# First in the plane: no browser needed, so it fails before the
156+
# Playwright install below pays its cost.
157+
run: pnpm run test:regression:unit
158+
159+
- name: Install Playwright chromium
160+
run: pnpm run test:regression:browser:install
161+
162+
- name: Run the backcompat plane (http + browser)
163+
run: pnpm run test:regression:backcompat
164+
165+
- name: Upload regression artifacts on failure or cancel
166+
if: ${{ !success() }}
167+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 (Node 24)
168+
with:
169+
name: regression-backcompat-artifacts
170+
path: |
171+
.bos/logs/**
172+
playwright-report/**
173+
test-results/**
174+
175+
dev:
176+
name: Dev plane (dev server)
177+
# Dev-plane canary, not a gate: pins the 'bos dev' machinery
178+
# (source-mode dev server, --ssr boot) that app-code changes cannot
179+
# break -- see the workflow header.
180+
if: inputs.tier == 'nightly'
181+
runs-on: ubuntu-24.04
182+
timeout-minutes: 30
183+
env:
184+
CI: true
185+
CORS_ORIGIN: http://localhost:4100
186+
BETTER_AUTH_SECRET: regression-test-secret-do-not-use-in-production
187+
steps:
188+
- name: Checkout repository
189+
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 (Node 24)
190+
191+
- name: Setup pnpm
192+
uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
193+
194+
- name: Setup Node
195+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
196+
with:
197+
node-version: "24"
198+
199+
- name: Setup Go
200+
uses: actions/setup-go@7a3fe6cf4cb3a834922a1244abfce67bcef6a0c5 # v6.2.0 (Node 24)
201+
with:
202+
go-version: "1.22"
203+
204+
- name: Get pnpm store dir
205+
id: pnpm-cache
206+
run: echo "STORE_PATH=$(pnpm store path)" >> "$GITHUB_OUTPUT"
207+
208+
- name: Cache pnpm store
209+
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 (Node 24)
210+
with:
211+
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
212+
key: ${{ runner.os }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
213+
restore-keys: |
214+
${{ runner.os }}-pnpm-
215+
216+
- name: Install dependencies
217+
run: pnpm install --frozen-lockfile --ignore-scripts
218+
219+
- name: Build every-plugin dist
220+
run: pnpm run build template
221+
222+
- name: Install Playwright chromium
223+
run: pnpm run test:regression:browser:install
224+
225+
- name: Run the dev plane (http:dev + browser dev:ssr + dev:csr)
226+
run: pnpm run test:regression:dev
227+
228+
- name: Upload regression artifacts on failure or cancel
229+
if: ${{ !success() }}
230+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 (Node 24)
231+
with:
232+
name: regression-dev-artifacts
233+
path: |
234+
.bos/logs/**
235+
playwright-report/**
236+
test-results/**
237+
238+
framework:
239+
name: Framework plane (process death)
240+
# Dev-plane canary, not a gate: pins the CLI's lifecycle machinery
241+
# (teardown, orphan adoption, port cleanup) under deliberate process
242+
# death. Two stack slots: every test but one boots the confined
243+
# death-contract fixture (~100MB of fake node services); the single
244+
# full-stack boot (sigterm) adds ~2GB -- see ci.yml's regression-framework
245+
# job for the resource-history rationale.
246+
if: inputs.tier == 'nightly'
247+
runs-on: ubuntu-24.04
248+
timeout-minutes: 20
249+
env:
250+
CI: true
251+
BETTER_AUTH_SECRET: regression-test-secret-do-not-use-in-production
252+
BOS_TEARDOWN_ADOPTION: "1"
253+
BOS_FRAMEWORK_STACK_SLOTS: "2"
254+
BOS_FRAMEWORK_READY_TIMEOUT_MINUTES: "4"
255+
steps:
256+
- name: Checkout repository
257+
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 (Node 24)
258+
259+
- name: Setup pnpm
260+
uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
261+
262+
- name: Setup Node
263+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
264+
with:
265+
node-version: "24"
266+
267+
- name: Setup Go
268+
uses: actions/setup-go@7a3fe6cf4cb3a834922a1244abfce67bcef6a0c5 # v6.2.0 (Node 24)
269+
with:
270+
go-version: "1.22"
271+
272+
- name: Get pnpm store dir
273+
id: pnpm-cache
274+
run: echo "STORE_PATH=$(pnpm store path)" >> "$GITHUB_OUTPUT"
275+
276+
- name: Cache pnpm store
277+
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5 (Node 24)
278+
with:
279+
path: ${{ steps.pnpm-cache.outputs.STORE_PATH }}
280+
key: ${{ runner.os }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
281+
restore-keys: |
282+
${{ runner.os }}-pnpm-
283+
284+
- name: Install dependencies
285+
run: pnpm install --frozen-lockfile --ignore-scripts
286+
287+
- name: Build every-plugin dist
288+
run: pnpm run build template
289+
290+
# No "Build every-plugin dist" step here: the suite boots only 'bos
291+
# dev', which resolves every-plugin from TS source through tsx
292+
# (ADR 0018) -- a fresh checkout boots the same. The setup body above
293+
# already skips both the Playwright install and the template build.
294+
295+
- name: Run framework regression suite
296+
run: pnpm run test:regression:framework
297+
298+
- name: Upload regression artifacts on failure or cancel
299+
if: ${{ !success() }}
300+
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 (Node 24)
301+
with:
302+
name: regression-framework-artifacts
303+
path: |
304+
.bos/logs/**
305+
tests/regression/fixtures/death-contract/.bos/logs/**

‎.github/workflows/release.yml‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -112,14 +112,19 @@ jobs:
112112
done
113113
echo "rc=$rc" >> "$GITHUB_OUTPUT"
114114
115-
# The release gate: the full regression matrix must be green before npm
116-
# sees anything. Dispatched as a manual retry (workflow_dispatch), the rc
117-
# detection runs the same way — a deliberate re-publish re-proves the tree.
115+
# The release gate: the regression matrix must be green before npm sees
116+
# anything. Tier 'release' = unit + start + backcompat — the production
117+
# artifact and the published-chain contract. The dev and framework planes
118+
# are nightly canaries (see regression-full.yml's header). Dispatched as a
119+
# manual retry (workflow_dispatch), the rc detection runs the same way — a
120+
# deliberate re-publish re-proves the tree.
118121
regression:
119-
name: Release gate — full regression
122+
name: Release gate — regression (unit + start + backcompat)
120123
needs: [gate]
121124
if: needs.gate.outputs.release == 'true' && needs.gate.outputs.rc == 'true'
122125
uses: ./.github/workflows/regression-full.yml
126+
with:
127+
tier: release
123128

124129
release:
125130
name: Process Changesets

0 commit comments

Comments
 (0)