Skip to content

native-e2e.md

Latest commit

 

History

History
470 lines (397 loc) · 27.9 KB

native-e2e.md

File metadata and controls

470 lines (397 loc) · 27.9 KB
Raw

Native e2e runner

Runbook and evidence log for the native WebDriver e2e runner (e2e-native/, make test-e2e-native).

The native runner drives the real Maru application — the real WKWebView DOM, a real PTY, and IME input — against the real Rust backend. It is macOS-only and lives in e2e-native/, separate from the Playwright suite in e2e/. Nothing it adds reaches a shipped build: every runner-only affordance sits behind the default-off native-e2e cargo feature and the VITE_NATIVE_E2E build flag, and scripts/check-native-e2e-isolation.mjs proves against the produced artifacts (the JS bundle inside make verify, the built binary inside make release-checks) that neither leaks into a production build.

How to run

make test-e2e-native

The target builds the frontend with VITE_NATIVE_E2E=1 (pnpm build:frontend:native-e2e), forces the maru crate to re-embed that frontend (cargo does not track dist/ as a build input, so the target touches src-tauri/build.rs between the two builds), builds the app with --features native-e2e, then runs WebdriverIO (e2e-native/wdio.conf.ts) against it. Each spec receives a fresh temporary fixture workspace and points the app at it through MARU_NATIVE_E2E_HOME / MARU_NATIVE_E2E_CONFIG_DIR; the developer's real workspace is never opened (D-09), and the app's outbound providers stay unconfigured for the whole run (D-11).

Fixture lifecycle and terminal readiness

The launcher seeds the initial root before the service launches its app. Each later spec is seeded before service restart, and the prior app must exit before its root is removed. Process selection checks exact argv, launcher parent, physical cwd and creation time; ambiguity preserves the fixture and fails the run. An intentional reloadSession inside a spec retains that spec's root.

On macOS 14+, the default-off native-e2e build derives a WKWebView data-store identifier from the canonical fixture config directory. A spec cannot inherit another spec's terminal tabs or the installed app's default store. The actual Mocha root hook checks a fixture-owned storage marker at startup; the restored terminal native spec checks that the same marker survives its app restart. Home/config isolation alone does not isolate WKWebView localStorage.

Fresh startup uses the same 20s profile deadline to require the actual menu dispatcher, completed boot/settings hydration, writable settings, the exact owned workspace paths and the seeded Welcome catalog entry. The build-gated reader returns fixed status fields only. A retained-profile check inside a restart still checks its marker without waiting for boot or rewriting metadata.

Actual Mocha hooks propagate setup errors. WDIO config beforeTest catches and logs rejected hooks, so it cannot enforce fixture readiness. Between tests only the authored Welcome leaf is restored atomically, and identical bytes cause no write; live .maru and registry directories are never recursively reset. Launcher/cleanup failures also make the final runner exit nonzero.

Terminal helpers wait for topology-created sessions and their real prompts before an explicit Shell launch, including restored split panes. They bind only one new physically visible, active and focused session and retain that identity through prompt/focus assertions. Failure diagnostics include pane/session IDs, rectangles, visibility and activeElement, without input or terminal contents. The restoration case requires one visible split body, one left/right pane with distinct IDs and nonempty real prompts, and the original left session identity. Split failures also report body visibility, prompt-readiness booleans and only the fixture's terminal layout flags; they do not replay the menu command. Readiness conditions replace session-set quiet-period guesses; existing timeout and text/paint assertions remain intact. Issue #388 requires five consecutive hosted native runs at the accepted fix, not retry-to-green closure.

Menu tests wait for a registered dispatcher and require a receipt from their single command invocation; bridge-object presence alone is insufficient. A failed document-mode assertion reports active rail modes and list state, while keeping its original deadline. The startup pick guard stays pending throughout boot, even when the picked mode matches stored settings, so an in-flight Today launch cannot override that explicit choice.

The settings owner also retains local mode/layout intent against older whole snapshots. Save origins and baselines bind to the scheduled value, including coalesced saves; acknowledgement uses actual merged readback. Read-start revisions preserve newer choices even after acknowledgement. Choices made before writes become possible are persisted through the existing lifecycle.

The dist/ the run leaves behind is deliberately not shippable — it carries the debug bridge. Re-run pnpm build:frontend before inspecting a production artifact; the ship-isolation guard inside make verify fails the build on a runner bundle.

To run a single spec file, use pnpm exec wdio run e2e-native/wdio.conf.ts --spec e2e-native/specs/pty.spec.ts. Passing -- --spec … through pnpm test:e2e:native does not work — pnpm forwards a literal -- and wdio silently runs the whole suite.

Which target runs which suite: make test-e2e is the Playwright suite — Chromium against the Vite dev server with mocked IPC. make test-e2e-native is WebdriverIO against the real backend in the real app. They share no runner and no premise, so a green result from one says nothing about the other.

Phase 08 responsiveness harness (plans 08-27/08-28)

Plan 08-27 added a feature-gated saturation/source-race harness on this runner (PERF-01/PERF-02); plan 08-28 closed the phase with production-isolation proofs. What exists and where it lives:

  • Feature-gated Rust harness (src-tauri/src/native_e2e.rs, behind cfg(feature = "native-e2e")): installs a deterministic two-worker Tokio runtime (TOKIO_WORKER_THREADS=2, exported in e2e-native/wdio.conf.ts onPrepare before the launcher spawns the app), serves a same-runtime async probe (native_e2e_async_probe), and drives allowlisted load control (native_e2e_load_control) that holds a 2500ms pre-work window inside the real blocking closures of git_status, scan_vault and skills_sync_source. Both commands are registered in the feature-gated generate_handler entries and allowlisted in docs/performance/phase08-native-allowlist.json; neither exists in a default-feature build.
  • Fixed fixtures, disposable roots: every fixture lives under a per-run mkdtemp root reached only through MARU_NATIVE_E2E_HOME; the change/revert seam refuses any source whose canonical path escapes that root. No live workspace, credential or public remote is touched (bare remotes are local bare repos). The spec's after reseeds checkouts and restores the registry from the seeded backup; it never deletes directories, so the app-side filesystem watchers survive.
  • Calibration / negative control / fixed run: the spec first calibrates idle probe/parse p95, freezes thresholds from those samples, then proves the blockingAsync negative control violates the frozen probe bound (loaded p95 4789ms vs the 27ms bound — the stall is the measurement), and finally proves the isolated saturation run meets the bounds with four overlapping real operations on two workers across three rounds (probe p95 2ms, parse p95 7ms, max stall 9ms, zero missing samples). Measured numbers land in artifacts/native-responsiveness/{calibration,thresholds,negative-control,fixed-run}.json (untracked, local only).
  • Ship isolation (scripts/check-native-e2e-isolation.mjs, D-09/D-10): the bundle half (runs inside pnpm build:frontend, so every make verify) scans dist/assets/*.js for the bridge namespace __MARU_NATIVE_E2E__ and both native-only command names; the binary half (--binary <path>, wired into make release-checks before the debug-artifact prune) scans the produced executable for the WebDriver plugin crate names, the MARU_NATIVE_RESPONSIVENESS_HOOK marker and both command symbols. A feature-on artifact fails both halves observably — verified 2026-09-06: the feature binary was rejected with exit 1 reporting all five markers, and the native-e2e dist was rejected with exit 1 reporting all three bundle markers.

Exact commands to reproduce the normal-artifact proof from a tree that last built the runner:

pnpm build:frontend                                    # normal dist, scans itself
touch src-tauri/build.rs                               # force re-embed of dist
cd src-tauri && cargo build && cd ..                   # default-feature binary
node scripts/check-native-e2e-isolation.mjs            # dist + manifest half
node scripts/check-native-e2e-isolation.mjs --binary src-tauri/target/debug/maru

Both scans must print their PASS line. If the last build was make test-e2e-native, expect the first scan to fail — that is the guard working; the rebuild commands above restore a shippable artifact.

The phase-level evidence closure gate (node scripts/check-command-isolation.mjs --all --expected-count 365, also a make verify step via check-command-isolation) is hermetic — it reads sources and recorded evidence only, and never builds or launches the app.

CI placement

Verdict: ci-viable (per-condition evidence in ## Spike log below). Consequences:

  • Selected pull requests: CI calls native-e2e.yml through workflow_call for the native/IPC/window paths listed in its header or the native-e2e label. The reusable suite runs on macos-26, and its result is a dependency of ci-ok; no cross-workflow polling or duplicate native compile is needed.
  • Other code pull requests: native-e2e-compile retains typecheck, lint and cargo check --locked --features native-e2e coverage. Docs-only PRs skip builds but still report ci-ok.
  • Main/tag pushes and manual dispatch: the standalone native workflow retains exact-commit receipts for release admission, with non-cancelling ref groups. These receipts certify the later main/tag SHA rather than a PR head. Docs-only main pushes skip; tag pushes still run. PR callers use separate native groups and cancel superseded PR runs.
  • A human must run: make release-preflight, which blocks on make test-e2e-native on every machine regardless of the verdict (D-03), and the checklist below.

Human-attended checklist

D-08's split, stated plainly: the synthetic-composition specs inside the runner guard the app's own composition-handling logic at the native level, and this checklist covers the OS input method itself — the half no synthetic event stream can exercise (per-surface evidence in ## IME sub-spike). Work every item and record a per-item observation, not a single overall yes.

macOS menu bar. Clicking a menu item is outside the webview and needs an Accessibility grant no unattended runner can give itself; the command path the menu emits into is covered by e2e-native/specs/menu.spec.ts (see ## macOS menu bar).

  1. Open the View menu, click Documents. Expected: the document list opens.
  2. Open the Terminal menu, click New Shell. Expected: a terminal pane opens with a live shell.
  3. With the terminal focused, click Split Terminal (or press ⌘D). Expected: a second terminal pane appears to the right of the first.

Korean IME, two-set (두벌식) input source. On both surfaces the sub-spike showed WebDriver key input never composes, so real OS-IME behavior is checked by a human:

  1. In a native terminal, switch to the macOS Korean 2-set input source and type 한글 (six raw jamo: ㅎㅏㄴㄱㅡㄹ). Expected: the jamo compose into syllables as you type, and the committed word appears exactly once — no trailing duplicate syllable after composition ends (the isTrailingCompositionDuplicate window).
  2. Still in the terminal, type one syllable and press Enter mid-composition. Expected: the in-progress syllable commits and no command executes; a second Enter sends the line. Then start a composition and click elsewhere to abandon it. Expected: no residue or doubled text on the screen when focus returns.
  3. In a rich editor document (open a document, switch to Rich mode), type 한글 with the same input source. Expected: the composed word lands in the document, and undo during/after composition plus mark/selection interaction behave as the engine specifies.

Cmd+Q quit route (plan 09-03, A1). e2e-native/specs/menu.spec.ts proves app.quit reaches the window-close guard through the debug bridge; only the OS delivering the physical keypress or menu click to that item is unproven by automation (same Accessibility-walled limit as items 1-3 above):

  1. With a dirty document open, press Cmd+Q. Expected: the unsaved-changes dialog appears instead of an immediate exit. Choose Cancel. Expected: Maru stays open. Then, with nothing dirty, press Cmd+Q again. Expected: Maru quits normally with no dialog.

Cmd+Q with the skill editor window open (review finding #2). Cmd+Q and the app menu's Quit Maru always route to main (see menu_command_target in src-tauri/src/app_menu.rs), regardless of which window has focus, and requestAppQuit (useDestructiveActionGuard.ts) asks the skill editor window's own guard before deciding whether to quit at all:

  1. Open a skill editor window, focus it, and press Cmd+Q. Expected: with nothing unsaved anywhere, the app quits and both windows close.
  2. With an unsaved edit in the skill editor, press Cmd+Q from either the skill editor or the main window. Expected: the skill editor's own unsaved-changes confirm appears; Cancel keeps both windows open and kills nothing.
  3. With an unsaved edit only in the main window, press Cmd+Q from the skill editor window. Expected: main's own unsaved-changes confirm appears (not the skill editor's); Cancel keeps both windows open.
  4. With a terminal running trap '' HUP; sleep 600 and a skill editor window open, press Cmd+Q. Expected: the app quits and pgrep -f "sleep 600" prints nothing.
  5. Regression: with no skill editor window open, edit the Scratchpad and press Cmd+Q. Expected: the app still quits within about 1 s and the edit is saved (unchanged from the single-window behavior above).

Quit flush of pending autosaves (plan 09-08, D-03/D-04/D-05/D-06). Every mounted autosave surface is flushed before quit, bounded to 3 s, with a failure keeping Maru open rather than dropping the edit:

  1. Open a Scratchpad memo, type a word, and press Cmd+Q immediately. Expected: Maru quits with no dialog, and reopening the memo shows the word saved.
  2. Run chmod a-w on the memo's folder, type a word, press Cmd+Q. Expected: Maru does not quit; a dialog says changes were not saved with Retry (primary), Quit anyway, and Cancel; a toast names the memo and the reason and offers Open recovery copy. Click Quit anyway. Expected: Maru quits and the recovery copy still exists. Restore permissions with chmod u+w afterwards.
  3. Open a shell tab, run trap '' HUP; sleep 600, and press Cmd+Q. Expected: Maru quits within about 3 s and pgrep -f "sleep 600" prints nothing (09-02's terminal sweep runs only after a guard-approved close).
  4. Open a shell tab, run trap '' HUP; while :; do sleep 1; done, then close the tab (not the whole app). Expected: within about 5 s pgrep -f "sleep 1" shows none of its processes. The trap makes the interactive shell itself ignore SIGHUP, and bash ignores SIGTERM, so it ends only at the ladder's SIGKILL step, about 4 s after the close.
  5. Open the Documents list after steps 8-9. Expected: no .maru/recovery file appears in it.

Observations: 2026-08-29 (plan 06-05 ratification)

Worked end to end on the current debug build under fixture isolation. Measurement note applying to all items: the debug binary and the installed Maru.app share a process name, and early probes that targeted the process by name hit the wrong app — every recorded result below used PID-targeted accessibility calls.

  1. View > Documents — PASS. Before the OS menu click: active mode 오늘, no .document-list. After: active mode 문서, .document-list present.
  2. Terminal > New Shell — PASS. Terminal canvas created; the text mirror shows a real zsh prompt running in the fixture workspace's cwd.
  3. Split Terminal — PASS. Panes 1 → 2, distinct session IDs.
  4. Terminal, Korean 2-set, 한글 — PASS WITH KNOWN ISSUE. Composition works and the committed word lands exactly once, BUT the first syllable typed immediately after switching the input source from English to Korean 2-set arrives decomposed (ㅎㅏㄴ as raw jamo instead of composed 한); from the second character onward composition is normal. Recorded as a known issue — suspected WKWebView IME-context attach race on input-source switch. Not a runner defect: the app's composition handlers are guarded by the 06-03 specs.
  5. Terminal, Enter mid-composition / abandon-by-click — PASS. No command executed on composition Enter; no residue or doubled text after abandoning a composition (user-observed).
  6. Rich editor, Korean 2-set, 한글 + undo/mark interactions — PASS. No anomalies (user-observed).

Known limits

  • The runner targets debug builds (cargo build --features native-e2e). A hardened-runtime, notarized .app restricts automation mechanisms a debug build permits, so a green run does not transfer to a signed release artifact without re-verification against that artifact.
  • The four D-13 surface specs are minimum-size proofs that each surface can be driven at all, not suite breadth; phases that need more (Phase 8's load test, Phase 9's SIGHUP test) attach their own specs to this runner (D-14).
  • IME coverage is the app's own composition handlers only; the OS input method itself is the human checklist above (D-08).
  • Disk fixtures and WKWebView stores are isolated between spec files. Within one spec, app-owned state is retained intentionally, including reloadSession persistence tests. Cleanup removes the owned disk fixture after app exit; isolated WebKit cache storage is managed by macOS rather than by recursive deletion of any developer browser profile.

Spike log

Scope of this section: the D-01 CI-viability verdict and the hosted evidence behind it. Attempt 1 (plan 06-01) covered session establishment only; attempt 2 (plan 06-05) ran the full four-spec suite on the hosted runner, covering D-01's remaining two conditions (PTY readability through the canvas surface, and a full run completed with no human present).

Rules (06-CONTEXT.md D-02): an observed interactive/TCC permission prompt would have settled the verdict as human-attended on the spot, with no further attempts. Every other failure class was retried within a cap of 3 hosted macOS job runs total. The cap and the observed failure class are both recorded here.

Retry cap: 3 hosted runs. Runs used: 2.

Attempts

# Date Run URL Result Failure class What changed before next attempt
1 2026-08-29 https://github.com/STAIxBWLB/maru/actions/runs/33243419439 pass — (no failure; no permission prompt) The three remaining D-13 surface specs landed (plans 06-02/06-03), so the suite grew from 1 spec to 4
2 2026-08-29 https://github.com/STAIxBWLB/maru/actions/runs/33250704926 pass — (no failure; no permission prompt) — (verdict settled; cap not needed)

Verdict

ci-viable — settled 2026-08-29; all three D-01 conditions held on hosted macos-14 runs:

  1. Session establishment with no interactive prompt — attempts 1 and 2 both established a WebDriver session with no interactive or TCC prompt: no dialog text appears in either run log, and the failure-path screencapture step never fired in either run.
  2. Real PTY output read through the canvas surface — attempt 2 ran e2e-native/specs/pty.spec.ts green on the hosted runner (1 passing (4.7s)): the shell-produced marker was read through the text mirror and the canvas ink check proved paint, under hosted WebKit rendering.
  3. A full run completed with no human present — attempt 2 ran the whole suite unattended: 4 passed, 4 total (webview 45.8s, pty 4.7s, ime 7.2s, menu 2.7s) inside a workflow_dispatch job, start to finish with no human present.

The original plan 06-05 / D-16 consequence was full main/tag runs with compile-only PR coverage and a retired spike. The current CI placement above adds selected full native PR runs under #512 while preserving that verdict.

Ratification

The plan 06-05 blocking human checkpoint ratified this verdict item by item on 2026-08-29, with the evidence named per condition:

  1. Session establishment with no interactive prompt — YES. Runs 33243419439 (1 spec) and 33250704926 (4 specs) both show zero TCC/permission lines; the failure-screencapture step was skipped in both (it only fires on failure/timeout), and timings were normal (4m47s and 4m09s). A modal would have surfaced as a hang-to-timeout.
  2. Real PTY output read through the canvas surface — YES. Run 33250704926's log shows pty.spec.ts 1 passing (4.7s) — the shell-produced marker (echo-spoof-proof) plus the canvas ink check.
  3. A full run completed with no human present — YES. Run 33250704926: 4 passed, 4 total in a workflow_dispatch job with no human present; the merged native-e2e-suite job carries ongoing proof.

Blocking local gate: make release-preflight exited 0 on macOS (2026-08-29, attempt 4; attempts 1-3 failed only on two pre-existing flaky tests — the editorSurfaceRenderIsolation vitest timing test and the graph.spec ghost-node test under parallel load — plus a missing Playwright browser install; each passed on rerun, and none implicate phase 06 changes).

CI placement reconfirmed at ratification: the ci.yml diff is purely additive (0 deleted lines, new native-e2e-compile job), native-e2e.yml exists, and native-e2e-spike.yml is deleted.

IME sub-spike

Recorded 2026-08-29 by e2e-native/specs/ime.spec.ts (D-07's two surfaces, each driven by both mechanisms; ROADMAP criterion 4). The spec asserts these observations, so a future WebDriver-provider or WebKit change that alters any row turns the suite red rather than silently changing what the suite covers.

The word driven is 한글 (two syllables; raw jamo ㅎㅏㄴㄱㅡㄹ).

Surface Mechanism Reached the app's handlers? Reproduced a real engine's sequence? Left to a human
Terminal hidden textarea (.native-terminal-input) WebDriver key input (raw jamo) Only the plain insertText path — the jamo landed literally (ㅎㅏㄴㄱㅡㄹ on screen); compositionstart/update/end never fired, so the composition handlers and the trailing-duplicate guard were never engaged No — no composition occurred at all Real OS-IME composition into the terminal, including the trailing-duplicate window under real timing
Terminal hidden textarea Synthetic composition events in-page Yes — onCompositionStart/onCompositionEnd ran, the word committed at compositionend, and a deliberately dispatched trailing insertText echo was dropped by isTrailingCompositionDuplicate; the word landed exactly once The event stream, yes; the OS engine that would emit it, no Everything above, plus Enter-during-composition and blur-abandonment under a real engine
Rich editor (.rich-editor-surface [contenteditable="true"], currently BlockNote's div.tiptap.ProseMirror.bn-editor) WebDriver key input (raw jamo) No observable change at all — the document text was unchanged after the keystrokes No Real OS-IME composition into the rich editor
Rich editor Synthetic composition events in-page Events alone: no (ProseMirror reads its DOM through a mutation observer, and a synthetic event mutates nothing). Events paired with the DOM mutation a real engine produces alongside them: yes — the composed word was adopted into the document and survived the observer flush The event stream plus the browser's DOM half, yes; the OS engine, no Real OS-IME composition, including undo behaviour and mark/selection interaction during composition

The boundary this draws, stated plainly: a synthetic stream can drive the app's own composition-handling code, but it cannot exercise the OS input method itself. Those prove different things. The working halves (terminal composition events, rich-editor composition events with DOM mutation) stay in the native suite as regression tests for the app's handler logic (D-08). The OS-level half — that a real macOS Korean IME composes correctly into both surfaces — is human-attended and is not claimed as covered anywhere in this document.

macOS menu bar

Recorded 2026-08-29 with e2e-native/specs/menu.spec.ts (D-13 surface 4).

The menu bar lives outside the webview and WebDriver cannot reach it; driving it needs an Accessibility grant no unattended runner can give itself. The surface therefore splits in two:

Automated half — the command path the menu emits into. The spec dispatches real command_item ids from src-tauri/src/app_menu.rs through window.__MARU_NATIVE_E2E__.menuCommand, which App.tsx registers into the same runMenuCommand the Tauri maru://menu-command listener calls — so the test exercises the handler the real menu reaches, not a parallel copy. Covered ids and their asserted DOM consequences:

Menu id Menu item Asserted DOM consequence
view.documents View → Documents The document list opens and shows the seeded document
terminal.shell Terminal → New Shell A native terminal view mounts with a live session
terminal.split Terminal → Split Terminal (⌘D) The terminal body enters split mode with a second, distinct session in the right pane
app.quit App menu → Quit Maru (Cmd+Q) With a dirty draft, the unsaved-changes dialog appears; Cancel keeps the app open

app.quit (plan 09-03, D-03) replaces tauri's predefined native quit item so Cmd+Q reaches the same guard as the red close button, instead of bypassing the webview through NSApplication terminate:. Review finding #2 (post-09-08) changed the routing: handle_menu_event always sends app.quit to main (menu_command_target in app_menu.rs), never the focused window, because a secondary window like skill-editor has no whole-app-quit orchestration of its own. main's requestAppQuit then asks the skill editor window's own guard (if open) before running its own, and only destroys the skill editor once both guards have passed; see src/lib/useDestructiveActionGuard.ts and src/lib/windowLayout.ts's requestSkillEditorQuitCheck/ closeSkillEditorForQuit.

Human-attended half: that the OS menu bar actually delivers the id, and that the multi-window routing above behaves per window. Clicking a menu item (or pressing a key equivalent the menu owns) is outside the webview and unscriptable here. The fixed checklist lives in ## Human-attended checklist above (items 1-3, plus item 7 for single-window app.quit and items 13-17 for the skill-editor multi-window cases), folded in by plan 06-05, extended by plan 09-03, and extended again by the review finding #2 fix.

Verified diagram export fixture

diagram-export.spec.ts uses a zero-argument, native-e2e-only bridge against the isolated fixture workspace. It renders a literal CJK architecture spec with the pinned engine, converts light SVG and dark PNG, writes fixed fixture paths through the native save bridge, and compares disk bytes with receipt hashes. It asserts conversion and readback gates separately from the unverified manual visual gate. This custom-protocol development build is not release-bundle proof.