Skip to content

fix(eval): #537 F01 — mot17.py --processes child failure fails the launch - #632

Merged
raylei50653 merged 18 commits into
mainfrom
fix/537-f01-child-failure
Oct 25, 2026
Merged

fix(eval): #537 F01 — mot17.py --processes child failure fails the launch#632
raylei50653 merged 18 commits into
mainfrom
fix/537-f01-child-failure

Conversation

@raylei50653

@raylei50653 raylei50653 commented Oct 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

#537 F01：mot17.py --processes 子程序失敗不再回傳 exit 0

來源：#538 D1 的 D1-F01（owner 裁決：#537 擁有修復契約）。完整契約與證據：docs/architecture/failure_degradation_537.md §25。

修復前（main@4380c2fa，GPU 重現）

D1 的 child_failure_probe（兩個 sequence 的 seqinfo.ini 沒有 [Sequence]）：兩個子程序都 failed with code 1（NoSectionError），parent 印 Completed all sequences 後 exit 0。

行為（fail-fast，src/saccade/perception/eval/sequence_launcher.py）

  • 第一個失敗（非零 exit、signal 死亡、無法啟動、launcher 自己出錯）讓 launch 失敗：待啟動的不啟動（也不建構命令），執行中的 SIGTERM →（送出後）grace 10 s → SIGKILL，所有直接子程序 reap 後才回傳；abort 之後不會再 spawn。
  • 原始失敗保留 sequence、code、stdout、stderr。輸出寫到暫存檔而不是 pipe，以 pread 讀到進入時的大小為止，讀取錯誤保留已讀部分。只有死於 launcher 實際送出的 signal 才標 terminated。
  • parent 先決定 exit code，再盡力印出結果（stream 已關閉或不存在也不會改掉 code），不算 metrics。exit code 由最早被觀察到的失敗決定：原 code；signal N → 128+N；無法啟動 → 127；launcher 自己出錯 → 70；KeyboardInterrupt 回收子程序後 → 130。
  • 未覆蓋（記錄為限制）：parent 自己收到 SIGTERM；子程序的後代程序。

launcher 放在 src/ 是因為 mot17.py import 時載入 TensorRT，CPU CI 無法 import。exporter 依賴的入口敘述未變（test_resolved_shipping_config.py 通過）。

驗證

  • CPU 故障注入（tests/unit/eval/test_sequence_launcher_537.py，44 cases，真實子程序；每案確認沒有未回收的直接子程序）：涵蓋要求的六種情境，加上多個子程序同時失敗時最早者決定 exit code且不留下孤兒，以及 Codex 審查找到的各種競態與錯誤路徑。4 個 pytest 同時執行也穩定。
  • Contract：test_mot17_producer_wiring.py 以 AST 檢查 launcher 區塊的直接敘述（呼叫與參數、中斷 exit 130、先決定 code 再回報、if exit_code != 0: sys.exit(exit_code)、metrics 最後）。
  • Mutation：40 個 launcher mutant ＋ 10 個 mot17.py mutant 全部攔住；worktree 檔案 SHA-256 前後相同。
  • GPU（同一 cwd／.venv／build/／models/，只換 checkout）：
    • 失敗路徑 @ fc935095：exit 1，兩個子程序各自保留 NoSectionError，沒有誤標 terminated，不算 metrics；連續 4 次相同。
    • 成功路徑：--processes 2 跑 MOT17-09/10-SDP，before／after 都 exit 0，結果檔逐位元組相同。
    • fc935095..1ae9f1f6 只改了回報方式、測試與文件。
  • Codex 審查 r1–r10：r1–r7、r9 NOT-READY，r8、r10 READY-WITH-CHANGES。每一項都已修正並加上測試／mutation，只有一項刻意不改：子程序處理 SIGTERM 後重新送出 SIGTERM 時記為 terminated；這不影響 exit code，也不遮住原始失敗。明細見 §25「審查過程的修正」。
  • pre_push @ 1ae9f1f6：6107 passed，只有 4 個 lagging-identity 測試失敗（test_runtime_identity_staleness ×2、test_research_lock ×2），由 docs(runtime): republish the runtime coordinate for #537 F01 (stacked on #632) #633 修正。

Runtime identity

mot17.py 與 sequence_launcher.py 是 decision_relevant；依 runbook §3.2 由 stacked #633 原子落地。implementation 4ad5ae2c… → 11204b8a…（317 → 318 檔），其他軸不動，probe 重跑 digest 相同，equivalence=unproven。

不在範圍

tracking 數值、online qualification、F06（另開 PR）、#538 的任何修復、mot17_all_sdp.py／run_eval_concurrent 的失敗傳播。

證據（repo 外）

~/.local/state/saccade/evidence/537-f01/20261025T145150/（README.md、SHA256SUMS）

🤖 Generated with Claude Code

https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx

raylei50653 and others added 17 commits October 25, 2026 15:01
@raylei50653 @claude
fix(eval): fail mot17.py --processes when a child fails (#537 F01)
ad4916d 
A child that exited non-zero used to be printed as ❌ while the parent
computed metrics from the remaining output and exited 0 (#538 D1-F01).

The launcher moves to saccade.perception.eval.sequence_launcher so CPU
tests can import it (mot17.py loads TensorRT at import). It is fail-fast:
the first failure (non-zero exit, signal death, or a child that cannot be
started) stops the launch; pending sequences are not started, running
children get SIGTERM then SIGKILL after a grace period, and all are
reaped. Each original failure keeps its sequence, code, stdout and
stderr; launcher-stopped children are reported as terminated. The parent
skips metrics and exits with the original failure's code (128+N for
signal N, 127 when it could not be started); KeyboardInterrupt reaps the
children and exits 130. SIGTERM to the parent is not handled.

The exporter's mot17.py anchor statements are unchanged; the producer
wiring contract now pins the launcher call and the exit-before-metrics
order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@raylei50653 @claude
fix(eval): keep a child's own failure when it exits during the abort
9908d21 
Found on the GPU child_failure_probe: both children failed by themselves
almost at once; the second one's thread had already set terminated_at and
reported it as "terminated by the launcher (code 1)", hiding its own
NoSectionError failure. Only a death by the launcher's own SIGTERM or
SIGKILL is TERMINATED now; any other exit keeps the child's own status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@raylei50653 @claude
docs(537): F01 child failure propagation contract (§25)
18271b5 
Records the fail-fast launch contract, exit codes (own code, 128+N,
127 start failure, 130 interrupt), the CPU fault-injection tests, 13
mutants, the before/after GPU child_failure_probe and the limits (parent
SIGTERM unhandled). #538 D1-F01 row links to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@raylei50653 @claude
fix(eval): close the Codex r1 gaps in the F01 launcher
4e917ca 
Codex r1 (NOT-READY) found, each confirmed with real children:
- a worker exception (invalid UTF-8 under text=True) neither aborted the
  launch nor kept the failure: output is now read as bytes and decoded
  with replacement, and any error while supervising a child fails that
  sequence (launcher_error, exit 70 = EX_SOFTWARE), aborts the launch and
  kills and reaps the child;
- "terminated" was decided from the return code alone, so a child that
  SIGKILLed itself in its SIGTERM handler looked stopped by the launcher:
  only a death by a signal the launcher actually sent counts now;
- the abort check and Popen could interleave with another worker's
  failure, so a sequence started after the abort: both now run under one
  lock, and spawned_at / failed_at are taken under it;
- the producer wiring contract was textual (`if False:` passed): it now
  checks the guard, its exit and the KeyboardInterrupt handler by AST;
- the interrupt test could read an empty pid file: pid files are written
  atomically and an autouse teardown SIGKILLs this test's leftover
  children (matched by command line).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
docs(537): update §25 for the Codex r1 fixes
92b2363 
Launcher errors (exit 70), the spawn lock, signal ownership for
"terminated", the AST wiring contract, 21 test cases, 19 + 5 mutants and
the GPU probe on 4e917ca.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
fix(eval): close the Codex r2 gaps in the F01 launcher
7397161 
Codex r2 (NOT-READY), each reproduced with real children:
- a signal was recorded as sent even when Popen.send_signal's own poll
  reaped the child and sent nothing: it is recorded only if the child was
  still unreaped after the call;
- an exception from command_for, or a non-OSError from Popen, escaped
  the worker without aborting: both now fail the sequence as a launcher
  error (exit 70) and abort; OSError stays a start failure (127);
- holding the spawn lock while building the command let a later failure
  win the lock and take precedence (127 instead of 5): the command is
  built outside the lock, and failures are ordered by when they were
  observed (failed_at) rather than when they set the abort (abort_at);
- the wiring contract searched the AST recursively, so an exit nested
  under `if False:` passed: only direct statements count now;
- the spawn-race test assumed B starts before A fails: it now forces both
  interleavings (slow Popen, slow command) and checks each outcome.
Also pins that a pending sequence builds no command after the abort.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
fix(eval): close the Codex r3 gaps in the F01 launcher
4c08a27 
Codex r3 (NOT-READY):
- a child's exit was only seen at pipe EOF, so a descendant holding its
  stdout/stderr (e.g. torch compile workers) hid the exit and let a later
  failure decide the code: output now goes to temporary files and the
  child is supervised with wait();
- the spawn message could raise (BrokenPipeError) outside every guard:
  one guard now covers the whole worker; any error kills and reaps its
  child and aborts the launch as a launcher error (70);
- tests: the spawn-race "command" scenario accepts both sound outcomes;
  the signal-death and start-failure tests synchronise A on B; the grace
  test measures SIGTERM->SIGKILL >= grace_s; new cases for a descendant
  holding the output and for a failing spawn message.

Not changed (documented in §25): a child that handles the launcher's
SIGTERM, cleans up and re-raises SIGTERM is counted as stopped by the
launcher; that is the usual graceful-shutdown idiom, and a death by the
same signal for an unrelated reason after the launcher sent it cannot be
told apart. It only affects how an already-aborted child is labelled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
fix(eval): close the Codex r4 gaps in the F01 launcher
ed84e8d 
Codex r4 (NOT-READY):
- capture-file creation (e.g. EMFILE) ran outside the worker guard: the
  files are now created inside it, so the error aborts as a launcher
  error;
- a failure was timestamped and aborted only after its output was read,
  so a slow or large read let a later failure win: classify, timestamp
  and abort first, then read;
- reading the capture with seek(0)+read moved the file offset shared with
  every descendant that inherited it, so a late writer could overwrite
  the child's own output: read with os.pread from offset 0;
- the wiring contract accepted a launcher call nested under a condition:
  the try body must be exactly the `results = ...run_sequence_children(`
  assignment;
- the grace test measured the launcher's return: it now records the
  actual send_signal times.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
fix(eval): close the Codex r5 gaps in the F01 launcher
63dca84 
Codex r5 (NOT-READY):
- a capture read error after a child's failure was observed (e.g. EIO on
  stderr) went to the worker guard and replaced the child's failure with
  a launcher error (70): collecting output now never raises; the read
  error is reported inside that stream and the child's code, timestamp
  and readable output are kept;
- the grace period started before SIGTERM was actually sent, so a slow
  send shortened it: it is timestamped after the send; the grace test
  slows the SIGTERM send by 1.3 s;
- the wiring contract accepted `run_sequence_children(...)[:0]`: the
  assignment value must be exactly that call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
fix(eval): close the Codex r6 gaps in the F01 launcher
314cbdd 
Codex r6 (NOT-READY):
- reading a capture followed a moving EOF, so a descendant that kept
  appending blocked collection after the child was reaped: reads stop at
  the size seen on entry;
- a launcher error was timestamped and aborted only after a possibly slow
  kill/reap, so a later child failure won: it is recorded and aborts
  first, then the child is cleaned up;
- a read error discarded the bytes already read: the readable prefix is
  kept and the error marker appended (_read itself never raises now);
- the wiring contract did not check the launcher's arguments
  (`seq_list[:0]` passed): seq_list, max_workers=args.processes and
  env=child_env are pinned;
- the signal-reap test assumed B starts before A fails: A waits for B.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
fix(eval): best-effort parent reporting keeps the F01 exit code (Code…
a410fde 
…x r7)

Codex r7 (NOT-READY):
- printing the captured child output to a stdout whose reader went away
  raised BrokenPipeError before sys.exit(exit_code), so the parent exited
  1 instead of the original code: the code is decided first, then
  sequence_launcher.report()/report_interrupt() print best effort and
  point a broken stream at /dev/null (so the interpreter's flush at exit
  cannot turn it into 120 either); the guard body is just
  sys.exit(exit_code);
- the wiring contract now pins the command-builder lambda, the report
  call between the decision and the guard, and the interrupt handler body;
- NIT: the precedence test makes B fail only after A has exited.

Test hardening found while stress-testing (4 concurrent runs): the
interrupt harness inherited SIGINT as ignored when pytest itself was
started in the background, so the KeyboardInterrupt never happened; it
installs Python's default SIGINT handler (a foreground run). The
launcher-error test now has B fail only once A is running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
test(eval): pin env forwarding and voluntary exit 0 after abort (Code…
fc93509 
…x r8)

Codex r8 was READY-WITH-CHANGES with two test gaps: dropping the explicit
env in Popen (losing parent_claim_environ) passed, and a child exiting 0
by itself after the abort could have been labelled terminated. A real
child now echoes an env var supplied only through `env=`, and the
voluntary-exit test runs with codes 0 (OK) and 4 (FAILED). Both targeted
mutants are caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
docs(537): §25 for the final F01 launcher
2927aa8 
Fail-fast contract after Codex r1-r8: spawn lock, signal ownership for
"terminated", file capture read with pread up to the size on entry,
worker-wide guard (launcher errors, exit 70), failures recorded before
output collection and cleanup, best-effort reporting after the exit code
is decided, exit-code table; 39 CPU cases, 36 + 10 mutants, the GPU
failure probe (4 repeats) and the unchanged success path at fc93509;
the review history and the one deliberately unchanged item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
fix(eval): reporting survives a missing stdout/stderr (Codex r9)
2d9f415 
Codex r9 (NOT-READY): a parent started with `1>&-` or `2>&-` has
sys.stdout / sys.stderr None; the unconditional flush in the best-effort
reporting raised AttributeError and the parent exited 1 instead of the
decided 5 (or 130). Reporting now absorbs any exception, not only
OSError, and a stream that cannot be flushed (None included) is skipped
or pointed at /dev/null. Tests run real harnesses with fd 1 or fd 2
closed, and with the reader of both pipes gone; the NITs widen two
scheduling margins in the grace and precedence tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
docs(537): §25 — reporting with a missing stream, r9 mutants and history
fb9d7f3 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
test(eval): pin the printed failure report; narrow the §25 timing cla…
1ae9f1f 
…im (Codex r10)

Codex r10 was READY-WITH-CHANGES with two MINORs: no test checked what is
printed on failure (dropping the child's stdout/stderr from describe()
passed), and §25 claimed the interleaving tests use no fixed delays
while the precedence test keeps a 1.5 s margin and the grace test an 8 s
upper bound. Both addressed; the describe mutant is caught.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 @claude
docs(runtime): republish the runtime coordinate for #537 F01
c9e33a4 
Fresh capture on the implementation head 1ae9f1f (clean tree, own
build/runtime_identity in the capture worktree, --run-probe
--require-complete under the gpu0 lease):

- implementation: 4ad5ae2c7f2a7d2c -> 11204b8a3246374f; 317 -> 318 files:
  src/saccade/perception/eval/sequence_launcher.py added (the fail-fast
  --processes launcher), scripts/eval/mot17.py changed (calls it).
- decision_surface, environment, identity_semantics, runtime_inputs:
  recomputed, unchanged.
- probe: rerun, not reused; digest 2dabed0b... unchanged, build witness
  unchanged. Probe equality is not an equivalence claim; equivalence
  stays unproven.

The outgoing canonical is archived as
runtime_identity.2026-10-25.4ad5ae2c7f2a7d2c.json (byte-identical to
origin/main's, sha256 a522eb76...). check_runtime_identity_staleness.py
--mode attested exits 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
@raylei50653 raylei50653 mentioned this pull request Oct 25, 2026
Merged
@raylei50653
Merge pull request #633 from raylei50653/chore/537-f01-runtime-identity
f67d4bf 
docs(runtime): republish the runtime coordinate for #537 F01 (stacked on #632)
@raylei50653
raylei50653 merged commit 07fdaf5 into main Oct 25, 2026
8 checks passed
This was referenced Oct 25, 2026
Merged
Merged
Open
Open
@raylei50653
raylei50653 deleted the fix/537-f01-child-failure branch November 13, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@raylei50653