Repository navigation
fix(eval): #537 F06 — refuse explicit sequences without seqinfo.ini; skip discovered ones with a reason - #634
Merged
Merged
Conversation
run_eval skipped a sequence without seqinfo.ini silently, and mot17.py exited 0 with no output for both single-process and --processes runs (#538 D1-F06); the C++ path built such a sequence with a 1920x1080 default instead. saccade.perception.eval.sequence_selection now owns the explicit vs discovered rule (parse_eval_config uses it unchanged for EvalConfig.seqs) and validates before an eval runs: an explicitly named sequence without seqinfo.ini is an error; a discovered one (empty --sequences, detector suffixes, or mot17.py --detector) is skipped with the reason logged; zero valid sequences is an error either way. - run_eval and run_eval_cpp validate first, before the config parse, the extension import or any detector load; run_eval_cpp only builds valid sequences and _build_cpp_seq_config raises instead of defaulting. - mot17.py: --detector discovery goes through select_sequences; the --processes parent validates before spawning. - The metrics sequence list and tracking values are unchanged; the exporter's mot17.py / run_eval anchor statements are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
Explicit vs discovered sequences, the validation points (run_eval, run_eval_cpp before parse / extension import; mot17.py --detector and --processes parent), the CPU tests, 11 + 3 mutants, the before/after GPU probe over normal/partial/all-missing trees (20 result files byte-identical) and the limits. #538 D1-F06 row links to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
… for other scripts Codex r1 on F06 (NOT-READY): - validation ran after the run claim (run_manifest.json written) and after teacher/Mamba detector builds: mot17.py now validates in both branches of the existing `if args.detector and not args.sequences:` statement (its head, the exporter anchor, is unchanged), before claim_or_join_run; the --processes parent check is therefore redundant and removed; - dancetrack.py, sportsmot.py and concurrent_mot17.py expand an empty --sequences into names themselves, which run_eval would now treat as explicit: they pass the names they found through the new filter_discovered() (skip and log, zero valid is an error); - --detector discovery now drops skipped sequences from EvalConfig.seqs and the metrics list (they never had output); §26 will say so instead of claiming the list is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
Validation before the run claim, filter_discovered for the other discovering scripts, the corrected EvalConfig.seqs claim, the duplicated skip log, 38 unit cases, 13 + 5 mutants, and the probe at 42661c3 (no run_manifest.json on refusal; 20 result files still byte-identical). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
…Codex r2) Codex r2 on F06 (NOT-READY): - DanceTrack scans data_root, so the usual `--data-root datasets/DanceTrack --split val` layout finds no names; the first filter_discovered raised on an empty list and broke the old fallback (an empty --sequences that run_eval discovers). filter_discovered([]) now returns [] and keeps it; names found but none valid is still an error. - §26 narrowed: mot17.py validates before any side effect, but concurrent_mot17.py writes its manifest first and its workers catch exceptions (listed as a boundary); metrics can also differ because the C++ path used to produce fresh results for sequences without seqinfo.ini, not only through stale result files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
Codex r3 on F06 was READY-TO-MERGE with one NIT: mot17_all_sdp.py hands each discovered SDP sequence to a child as an explicit name, so a directory without seqinfo.ini would now fail the batch instead of being skipped. Its discovery goes through filter_discovered as well; an explicit --sequences list is unchanged. Contract test pinned (fails on the previous file). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
Brings in F01 (fix/537-f01-child-failure @ f67d4bf, incl. its runtime coordinate publication) so F06 lands on top of it. A merge instead of a rebase keeps the F06 commit hashes that §26 and the evidence cite. Conflicts were append-only: §25 (F01) is kept before §26 (F06) in failure_degradation_537.md, and F06's wiring tests are appended after F01's in test_mot17_producer_wiring.py. mot17.py merged cleanly (F01 touches the launcher block, F06 the selection statement). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
Fresh capture on the implementation head 58a8fc7 (F06 merged on top of the F01 branch; clean tree, own build/runtime_identity in the capture worktree, --run-probe --require-complete under the gpu0 lease). The outgoing canonical is F01's publication: - implementation: 11204b8a3246374f -> 11263f7cae3f8bce; 318 -> 319 files: src/saccade/perception/eval/sequence_selection.py added (explicit vs discovered sequences, seqinfo.ini validation); scripts/eval/mot17.py, src/saccade/perception/eval/{config,cpp_runner,evaluator}.py changed (use it). - decision_surface, environment, identity_semantics, runtime_inputs: recomputed, unchanged. - probe: rerun, not reused; digest 2dabed0b... unchanged, build witness unchanged. Probe equality is not an equivalence claim; equivalence stays unproven. The outgoing canonical is archived as runtime_identity.2026-10-25.11204b8a3246374f.json (byte-identical to the F01 branch's). check_runtime_identity_staleness.py --mode attested exits 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#537 F06：評估 sequence 不再被靜默略過
來源：#538 D1 的 D1-F06（owner 裁決同 D1-F01：#537 擁有修復契約）。完整契約與證據：
docs/architecture/failure_degradation_537.md§26。以 merge（不是 rebase）疊在已落地的 F01（#632）之上，保留 §26 與證據引用的 commit hash；相對main的 diff 只有 F06。修復前（
main@4380c2fa，GPU 重現）run_eval對缺少seqinfo.ini的 sequence 靜默continue；全部缺少時沒有輸出，仍 exit 0（單一程序與--processes），並已寫出run_manifest.json。--cpp-threads）以 1920×1080 的預設值建構該 sequence，並寫出它的結果檔。行為（
src/saccade/perception/eval/sequence_selection.py）--sequences為名稱。缺少seqinfo.ini→SequenceSelectionError（列出全部缺少的名稱），exit 1，不執行任何 sequence。--sequences為空或只有 detector 後綴、--detector，以及 DanceTrack／SportsMOT／concurrent_mot17.py／mot17_all_sdp.py自己列目錄。缺少時記錄原因後略過；零個有效 → exit 1。mot17.py在 run claim、detector 建構、launcher 之前驗證（在既有的if args.detector and not args.sequences:敘述內，exporter 錨點不變）。run_eval／run_eval_cpp在解析與 extension import 之前驗證。_build_cpp_seq_config不再退回預設值。EvalConfig.seqs與 metrics 清單。它們沒有結果檔；只有舊結果檔、或修復前 C++ 以預設值產生的結果會讓 metrics 不同。驗證
tests/unit/eval/test_sequence_selection_537.py38 cases，涵蓋要求的三種情境——正常／部分缺失／全部缺失——各自在明確指定、自動發現、detector 後綴之下，加上入口在副作用之前報錯、C++ 不給預設值、requested_sequences與修復前規則逐字相同。test_mot17_producer_wiring.py另有 3 個 AST 案例。4380c2fa對 after58a8fc76（已含 F01）：run_manifest.json之前結束。MISSING-*.txt。mot17_all_sdp.py）已一併處理。58a8fc76：6148 passed，只有 4 個 lagging-identity 測試失敗，由 stacked publication PR 修正。Runtime identity
config.py、evaluator.py、cpp_runner.py、mot17.py與新的sequence_selection.py是decision_relevant；由 stacked publication PR 原子落地。implementation11204b8a…（F01 的出版）→11263f7c…（318 → 319 檔），其他軸不動，probe 重跑 digest 相同，equivalence=unproven。dancetrack.py／sportsmot.py／concurrent_mot17.py／mot17_all_sdp.py是unclassified。不在範圍
seqinfo.ini的內容檢查、tracking 數值、online qualification、#538、concurrent_mot17.py自己的失敗傳播（記為邊界）。證據（repo 外）
~/.local/state/saccade/evidence/537-f06/20261025T155523/（README.md、SHA256SUMS）🤖 Generated with Claude Code
https://claude.ai/code/session_019RVti7Gf7paqtEzERmZaCx