Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/architecture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
- 事件 / API / storage schema:看 [api_spec.md](../modules/storage/api_spec.md)
- 能力/REQ/Profile、具名 owner、inherited scope/proposed delta 與代表鏈狀態:看 [capability_requirements_535.md](capability_requirements_535.md)(#535;批准與待決狀態只在該檔維護)
- S-SHIP/S-EXPORT 的展開圖(現況與目標)、completion/identity/export 接口契約卡與 ID 索引:看 [ship_export_contracts_536.md](ship_export_contracts_536.md)(#536;設計狀態只在該檔維護)
- 失敗/fallback/降級(分 profile 矩陣、owner 裁決、已落地的 F07/C2/R-1、D2 host lifecycle 與 D2-GPU-0/1;WorkbenchPool 仍未 online qualified):看 [failure_degradation_537.md](failure_degradation_537.md);NVML 失敗的 4A 核准方案見 [§19](failure_degradation_537.md#19-c0-05nvml-失敗的重現與核准方案),實作與證據見 §20。
- 失敗/fallback/降級(分 profile 矩陣、owner 裁決、已落地的 F07/C2/R-1、D2 host lifecycle 與 D2-GPU-0/1;WorkbenchPool 仍未 online qualified):看 [failure_degradation_537.md](failure_degradation_537.md);NVML 失敗的 4A 核准方案見 [§19](failure_degradation_537.md#19-c0-05nvml-失敗的重現與核准方案),實作與證據見 §20;WorkbenchPool 明列未接監控見 §21。
- 安裝後 package 的 Completion/Gate A、entrypoint re-pin 與驗證證據:看 [package re-pin](../reference/native_runtime_package_repin_536.md)(#536;不擴大支持範圍)
- 模型 bundle 與 native runtime 的分離(現況/目標架構圖、信任根、責任分工、A/B/C 比較):看 [ADR 028](../decisions/028-model-bundle-runtime-separation.md);bundle manifest 契約、fail-closed 規則、G01–G07 處置與控制矩陣:看 [model_bundle_contract_549.md](model_bundle_contract_549.md)(#549 S1;設計狀態只在該檔維護)

Expand Down
24 changes: 20 additions & 4 deletions docs/architecture/failure_degradation_537.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,12 +86,12 @@ F07 修復後,第一個檔的 SHM 測試(C0-06~08)已改為目標語義
| C0-01/02 的拒絕點 | public facade 的拒絕點與相容性由 #441/#537 協調;eval entrypoint 的 preflight/結果政策仍歸 #536/#537。online 的 C0-03 已由 R-1 修復(§10) | REQ-535-02-01/02-03/02-04 |
| C0-05 | 4A 已核准(§19);初始化/probe/無效 sample 明確拒絕,實作與證據見 §20 | REQ-535-07-01;CAP owner:raylei50653 |
| C0-06~C0-08 | 已由 F07 修復(§7);T 預設值、多 writer、shutdown 仍待 #538 | #538 |
| C0-09 | WorkbenchPool 是否支援 resource degradation | REQ-535-07-02 |
| C0-09 | stats 已改為明列「未接監控」(§21);WorkbenchPool 是否支援 resource degradation 仍未裁決 | REQ-535-07-02 |
| C2 其他參數 | 重啟初值與決策頻率已裁決(R0、F0,見 §8);非法 usage(NaN、負值、>100)只在 C2 PR 內維持現況(O0),不構成支持契約,仍待裁決 | D-537-2 未涵蓋 |
| R-1(C0-11) | 呼叫契約已由 R-1 修復(§10);worker task 異常終止後的錯誤傳遞、health、stop 與 shutdown 語義依 §10 的切片順序處理 | #537 C0-11;#538 D2-AD、D2-WB-0/1、D2-GPU |
| health CLI、orchestrator 錯誤後的 ACK | 報告 unknown 的方式;ACK 語義 | REQ-535-07-03;REQ-535-06-01(#538) |

F07、C2、R-1、D2 的 host lifecycle 切片(§7~§16)與 D2-GPU-0/1(§17~§18)已落地;#538 其餘項目仍 OPEN,不取得 online-service qualification、不部署。本次 C0-05 的 4A 已核准(§19),實作與證據見 §20;C0-09 另外處理,C0-01/02 的 public facade 相容性與 eval consumer policy 依上表分工裁決。每個修復 PR 只處理一類行為。
F07、C2、R-1、D2 的 host lifecycle 切片(§7~§16)與 D2-GPU-0/1(§17~§18)已落地;#538 其餘項目仍 OPEN,不取得 online-service qualification、不部署。C0-05 的 4A 已核准(§19),實作與證據見 §20;C0-09 的「未接監控」回報見 §21,C0-01/02 的 public facade 相容性與 eval consumer policy 依上表分工裁決。每個修復 PR 只處理一類行為。

## 7. B0-F07 修復(C0-06/07/08 收斂)

Expand Down Expand Up @@ -386,7 +386,7 @@ function 標籤是 `diagnostic`。要修復時,必須在同一個 PR 把這些
- deadline 到時還有 worker 存活:**不 shutdown batcher**,狀態維持 STOPPING,拋 `WorkbenchPoolStopTimeout`(列出存活的 worker),不印「Stopped.」。之後再呼叫 `stop()` 會重試。
- 所有 worker 都結束後:清空 queue 並計入 `discarded_on_stop`,shutdown batcher,狀態改為 STOPPED。
- 這次執行如果失敗過,最後拋 `WorkbenchPoolFailed`。第二次呼叫 `stop()` 不再報告。
- `get_stats()` 新增 `state`、`failure`、`failed_stream`、`late_callbacks`、`discarded_on_stop`、`stop_timeouts`。`current_level` 仍固定為 `"NORMAL"`(C0-09 不變)。
- `get_stats()` 新增 `state`、`failure`、`failed_stream`、`late_callbacks`、`discarded_on_stop`、`stop_timeouts`。`current_level` 仍固定為 `"NORMAL"`(C0-09 不變;之後由 §21 改為 `"UNKNOWN"`/`not_wired`)。

**修復後的行為**(對照 §12):

Expand Down Expand Up @@ -928,7 +928,7 @@ flowchart LR
| 項目 | 保留的邊界與下一切片 | Owner/入口 |
|:--|:--|:--|
| C0-01/02 | ADR 025 與 README 仍允許 constructible stub;不能把 R-1 的 internal preflight 套到 public facade。public API 相容性與 eval entrypoint 的拒絕政策分開裁決 | raylei50653;#441/#537(public)、#536/#537(eval);REQ-535-02-01、02-03 |
| C0-09 | WorkbenchPool 沒有 resource manager/writer,stats 的固定 NORMAL 不構成監控證據。另提「明列未接監控」的回報切片;不在此接上降級 controller | raylei50653;#537/REQ-535-07-02;`--workbench` 仍不支援(#538) |
| C0-09 | WorkbenchPool 沒有 resource manager/writer,stats 的固定 NORMAL 不構成監控證據。另提「明列未接監控」的回報切片(已落地,見 §21);不在此接上降級 controller | raylei50653;#537/REQ-535-07-02;`--workbench` 仍不支援(#538) |
| O0 | 直接注入非法 usage 的決策語義仍待裁決;4A 僅驗證 NVML 原始 sample,不放寬 C2 契約 | raylei50653;#537 |
| freshness/ACK/ownership | T 預設、單一 writer 強制、storage 失敗仍 ACK 與 PEL 回收仍交 #538;4A 不替這些項目結案 | raylei50653;#538/REQ-535-06-01、07-01 |
| health CLI/telemetry | 另有直接 NVML/nvidia-smi probe,不是此 ResourceManager consumer;4A 不能推廣成全 profile 的 healthy 保證 | raylei50653;#537/REQ-535-07-03 |
Expand Down Expand Up @@ -967,3 +967,19 @@ dispatcher 的四個隔離 mutation 也都被攔住:移除 preflight(10 fail
**Runtime identity**:2026-10-20 在 `d6724f11` 基線的捕捉(`8b89ca8f… → e75f0067…`)因 #621 改了 `dispatcher.py`、`detector_trt.py` 而作廢,沒有出版。rebase 後在實作 head `791d85d8` 的乾淨 tree 重新捕捉(之後只改文件,不動任何已出版軸的 blob):本工作樹的獨立 `build/runtime_identity` 重新 build,`--run-probe --require-complete` 在 gpu0 lease 下 fresh capture 到 scratch,沒有沿用舊 probe。只有 implementation 改變:`583ede47… → d04e0183…`,317 個成員不增不減,只改 `resource_manager.py`、`dispatcher.py` 與 `run_8stream_perception.py`。其餘四軸重算後相同,probe digest 仍為 `2dabed0b…`;`equivalence=unproven`,probe 相同不構成 online monitor policy 或整體語義等價。promote 後 `--mode attested` exit 0。canonical/archive 以獨立的 stacked publication branch 提交,與實作原子落地。

**邊界**:這些測試沒有執行完整 online service、Redis/Chroma 或 VRAM 耗盡,亦不提供 GPU stream/buffer lifetime 的 D2-GPU qualification。失敗 probe 不會立即撤銷別的程序仍可讀到的舊快照,正常 retirement 也可能失敗;兩者在既有 T 後變成 STALE。health CLI、獨立 telemetry、C0-09、C0-01/02、C0-10、O0 與 #538 的其餘工作保持 §19.3 的分工。

## 21. C0-09:WorkbenchPool stats 明列未接監控

方案依 #550 §0A 的開工條件記錄於 [#537 comment](https://github.com/raylei50653/saccade/issues/537#issuecomment-6362121273)(選 B),範圍沿用 §19.3 的分工:只修正回報,不接上降級 controller。修復前的現況見 §2 C0-09 與 §13。

**實作**:

- `WorkbenchPool.get_stats()` 的 `current_level` 由固定的 `"NORMAL"` 改為 `"UNKNOWN"`,並新增 `resource_monitor_status="not_wired"` 與 `resource_monitor_error=""`。欄位名稱與 UNKNOWN 語義沿用 §20 的 AsyncDispatcher;`not_wired` 表示這條路徑沒有 resource manager 或 VRAM writer,與「監控失敗」的 `unavailable` 不同。所有 pool 狀態(CREATED/RUNNING/FAILED/STOPPING/STOPPED)回同一組值。
- `run_8stream_perception.py --workbench` 的文字報告加印 `monitor=<status>`。
- 不改 AsyncDispatcher、hysteresis、快照 v1、`_WorkbenchWorker` 或 batcher;WorkbenchPool 不讀 NVML、不發佈快照。

**驗收與負控制**:[test_workbench_pool_monitor_contract_537.py](../../tests/unit/test_workbench_pool_monitor_contract_537.py) 3 個案例。host:CPU fake 的 pool 在 CREATED、RUNNING、STOPPING(batcher 未釋放)、STOPPED、FAILED 及失敗後 STOPPED 都回 `UNKNOWN`/`not_wired`/空錯誤;`--workbench` caller 的單次報告含 `level=UNKNOWN monitor=not_wired`,且不出現 `level=NORMAL`,結束時 pool STOPPED、batcher 已回收、沒有殘留 task。`gpu`:沿用 D2-GPU-1 harness,真實 WorkbenchPool/Workbench/BatchingTRTDetector 在 CUDA 上送出 3 個結果時與 stop 後都回 `UNKNOWN`/`not_wired`(只有 TRT engine 是 fake)。五個隔離 mutation 都被 host 案例攔住:level 改回 `NORMAL`(2 failed)、移除 `resource_monitor_status`(2)、報告移除 `monitor=`(1)、status 改為 `available`(2)、只在 STOPPING 回 `NORMAL`(1;Codex r1 指出後補上 STOPPING 案例)。

**Runtime identity**:`dispatcher.py` 與 `run_8stream_perception.py` 是 `decision_relevant`,依 [republication runbook](../reference/runbooks/runtime_identity_republication.md) fresh capture 後以 stacked publication 原子落地;座標變化與證據記在 PR。

**邊界**:WorkbenchPool 仍沒有任何 resource degradation 能力;`--workbench` 仍 unsupported(#538),不取得 online-service qualification、不部署。`not_wired` 只說明沒有監控,不是健康判定。C0-01/02、C0-10、O0、health CLI 與 #538 的其餘工作不變。
2 changes: 2 additions & 0 deletions docs/modules/resource/architecture_degradation.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,5 @@ Owner 在 2026-10-20 核准 4A,裁決與原始重現見 [#537 §19](../../arch
- 沒有背景 init retry。init 失敗後要換新 manager;post-init 的暫時故障可在修復 NVML 後明確 stop/start,重新 preflight 成功才接受 frame,沿用該 manager 的 hysteresis。新 manager 則遵守 R0 的 NORMAL 初值。
- 初始化錯誤只在建構時診斷一次;讀取失敗保留在例外/stats,不按 frame 重複印 log。FAILED 也不會繼續消費 frame 來反覆故障。
- 本次不改快照 v1、T、事件持久化/ACK/RAG policy、tracker 數值或 public facade。D2-GPU 完成前不宣稱 online-service qualification、不部署。

**WorkbenchPool(C0-09)**:此路徑沒有接上 ResourceManager 或快照 writer。stats 固定回 `current_level="UNKNOWN"`、`resource_monitor_status="not_wired"`、`resource_monitor_error=""`,`--workbench` 報告印出 `monitor=not_wired`;這只說明沒有監控,不是健康判定([#537 §21](../../architecture/failure_degradation_537.md#21-c0-09workbenchpool-stats-明列未接監控))。
15 changes: 8 additions & 7 deletions docs/ownership/tests_inventory.generated.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!-- Generated by scripts/tools/build_tests_index.py; do not edit this file by hand. -->
# tests/ inventory (generated)

Total tracked test files: **269**. Source of truth = each `test_*.py`'s own docstring + `# scope:` / `# function:` / `# lifecycle:` header; regenerate with `scripts/tools/build_tests_index.py`.
Total tracked test files: **270**. Source of truth = each `test_*.py`'s own docstring + `# scope:` / `# function:` / `# lifecycle:` header; regenerate with `scripts/tools/build_tests_index.py`.

> **Snapshot, may lag.** CI regenerates this file on every run and publishes it as the `derived-indexes` artifact; the checked-in copy is refreshed whenever someone reruns the generator, and no gate requires that (#493 PR-2).

Expand All @@ -14,17 +14,17 @@ Guarantee coverage per module (behavior/contract/regression/acceptance/diagnosti
| `api` | 0 | 1 | 0 | 0 | 0 | 1 |
| `media` | 5 | 0 | 0 | 0 | 0 | 5 |
| `pipeline` | 1 | 1 | 0 | 0 | 0 | 2 |
| `resource` | 1 | 7 | 0 | 0 | 2 | 10 |
| `resource` | 1 | 8 | 0 | 0 | 2 | 11 |
| `cognition` | 2 | 0 | 0 | 0 | 0 | 2 |
| `storage` | 3 | 0 | 0 | 0 | 0 | 3 |
| `perception` | 11 | 7 | 1 | 0 | 3 | 22 |
| `perception` | 11 | 8 | 1 | 0 | 3 | 23 |
| `detection` | 15 | 8 | 0 | 0 | 0 | 23 |
| `reid` | 3 | 0 | 0 | 0 | 0 | 3 |
| `eval` | 49 | 46 | 1 | 0 | 0 | 96 |
| `tracking` | 8 | 23 | 0 | 0 | 1 | 32 |
| `cross-module` | 0 | 6 | 0 | 0 | 0 | 6 |
| `system` | 0 | 98 | 4 | 1 | 1 | 104 |
| **Total** | 98 | 197 | 6 | 1 | 7 | **309** |
| **Total** | 98 | 199 | 6 | 1 | 7 | **311** |

## Scope × lifecycle

Expand All @@ -35,17 +35,17 @@ Governance state per module. A multi-scope test counts once per scope.
| `api` | 1 | 0 | 0 | 0 | 0 | 0 | 1 |
| `media` | 5 | 0 | 0 | 0 | 0 | 0 | 5 |
| `pipeline` | 2 | 0 | 0 | 0 | 0 | 0 | 2 |
| `resource` | 10 | 0 | 0 | 0 | 0 | 0 | 10 |
| `resource` | 11 | 0 | 0 | 0 | 0 | 0 | 11 |
| `cognition` | 2 | 0 | 0 | 0 | 0 | 0 | 2 |
| `storage` | 3 | 0 | 0 | 0 | 0 | 0 | 3 |
| `perception` | 22 | 0 | 0 | 0 | 0 | 0 | 22 |
| `perception` | 23 | 0 | 0 | 0 | 0 | 0 | 23 |
| `detection` | 23 | 0 | 0 | 0 | 0 | 0 | 23 |
| `reid` | 3 | 0 | 0 | 0 | 0 | 0 | 3 |
| `eval` | 87 | 0 | 9 | 0 | 0 | 0 | 96 |
| `tracking` | 32 | 0 | 0 | 0 | 0 | 0 | 32 |
| `cross-module` | 6 | 0 | 0 | 0 | 0 | 0 | 6 |
| `system` | 104 | 0 | 0 | 0 | 0 | 0 | 104 |
| **Total** | 300 | 0 | 9 | 0 | 0 | 0 | **309** |
| **Total** | 302 | 0 | 9 | 0 | 0 | 0 | **311** |

## Disposition queue

Expand Down Expand Up @@ -252,6 +252,7 @@ Governance state per module. A multi-scope test counts once per scope.
| `test_workbench_pool_gpu_contract_538.py` | perception | contract | active | #538 D2-GPU-1: CUDA ordering, output ownership and batcher failure contracts. |
| `test_workbench_pool_lifecycle_contract_538.py` | perception | contract | active | #538 D2-WB-1: WorkbenchPool bounded stop, worker failure and safe release. |
| `test_workbench_pool_lifecycle_observed_538.py` | perception | diagnostic | active | #538 D2-WB-0: the WorkbenchPool normal path, plus the harness that the contract tests share. |
| `test_workbench_pool_monitor_contract_537.py` | perception,resource | contract | active | #537 C0-09: WorkbenchPool stats say that no resource monitor is wired. |

### `tests/unit/api/`

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/runtime_identity.generated.json

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions scripts/ops/run_8stream_perception.py
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,7 @@ def report() -> None:
"📈 WorkbenchPool Stats: "
f"state={stats['state']} "
f"level={stats['current_level']} "
f"monitor={stats['resource_monitor_status']} "
f"active={stats['active_streams']} "
f"queue={stats['queue_depth']} "
f"drops={stats['drops']} "
Expand Down
9 changes: 8 additions & 1 deletion src/saccade/perception/dispatcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,9 @@ class WorkbenchPool:
- True cross-thread parallelism (no GIL serialization in hot path)
- Per-stream isolation (tracker state, GPU scratch, CUDA stream)
- BatchingTRTDetector handles YOLO batching with CUDA event handshake

No resource monitoring is wired: stats report ``current_level="UNKNOWN"``
and ``resource_monitor_status="not_wired"`` (#537 C0-09).
"""

def __init__(
Expand Down Expand Up @@ -668,7 +671,11 @@ def get_stats(self) -> dict[str, Any]:
"late_callbacks": self._late_callbacks,
"discarded_on_stop": self._discarded_on_stop,
"stop_timeouts": self._stop_timeouts,
"current_level": "NORMAL",
# C0-09: no resource manager or VRAM writer is wired to this path,
# so the level is unknown rather than a healthy-looking NORMAL.
"current_level": "UNKNOWN",
"resource_monitor_status": "not_wired",
"resource_monitor_error": "",
"active_streams": self.n_streams,
"queue_depth": sum(q.qsize() for q in self._queues),
"drops": drops,
Expand Down
1 change: 1 addition & 0 deletions tests/unit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,5 +70,6 @@
| `test_workbench_pool_gpu_contract_538.py` | perception | contract | active | #538 D2-GPU-1: CUDA ordering, output ownership and batcher failure contracts. |
| `test_workbench_pool_lifecycle_contract_538.py` | perception | contract | active | #538 D2-WB-1: WorkbenchPool bounded stop, worker failure and safe release. |
| `test_workbench_pool_lifecycle_observed_538.py` | perception | diagnostic | active | #538 D2-WB-0: the WorkbenchPool normal path, plus the harness that the contract tests share. |
| `test_workbench_pool_monitor_contract_537.py` | perception,resource | contract | active | #537 C0-09: WorkbenchPool stats say that no resource monitor is wired. |

<!-- END generated test index -->
Loading
Loading