Skip to content

fix(resource): refuse out-of-domain usage before deciding (#537 O0) - #626

Merged
raylei50653 merged 5 commits into
mainfrom
fix/537-o0
Oct 23, 2026
Merged

raylei50653 merged 5 commits into
mainfrom
fix/537-o0

Conversation

@raylei50653

Copy link
Copy Markdown
Owner

#537 O0:拒絕非法 usage,不再當成一次有效決策

方案紀錄:依 #550 §0A 開工條件記錄在 #537 comment(候選 A/B/C,選 A)。風險:高(resource_manager.py 是 decision_relevant)。

修改

  • ResourceManager.decide_degradation_level():決策前要求 usage_percent 是 numbers.Real 且在 0~100(含端點)內。NaN、±inf、超出範圍、非實數型別(字串、None、Decimal)都拋 ResourceMonitorUnavailable,cause 為 ValueError;current_level 不變,也不算一次決策。dispatcher 走 4A 的既有通道(unavailable/UNKNOWN/FAILED、不寫新快照)。
  • 修復前:NaN 維持原級、負值降一級、>100 升到 EMERGENCY。production get_stats() 在 4A 後只會回 0~100 的有限值,所以只有被替換的 get_stats() 會走到這條路徑。
  • 不改 get_stats() 的 sample 驗證、門檻、D-537-2、R0/F0、快照 v1、dispatcher.py。已知缺口(寫在 §22):dispatcher 的啟動 probe 直接呼叫 get_stats(),被替換的 get_stats() 回非法值時,要到第一次 batch 決策才失敗。
  • 文件:failure_degradation_537.md 新增 §22,§6/§8/§19.3 對應列指向 §22;architecture_degradation.md §5/§7 改成拒絕契約。

驗證

  • test_resource_degradation_hysteresis_537.py:T7 的 3 個 O0「觀察」列改成 T8 契約(86 → 133 個案例)。4 個級別 × 11 個非法值都拒絕並保留 level;連續拒絕不推進遲滯;int、np.int64、np.float64、Fraction 有效。
  • test_resource_monitor_dispatcher_contract_537.py:+2 個案例(-1、Decimal("NaN")),驗證 worker FAILED、stats 為 UNKNOWN/unavailable、快照只有先前的 EMERGENCY。
  • 修復前(main 的實作)這兩檔有 47 個案例失敗;第一版實作 9e17133e 有 9 個失敗。六個隔離 mutation 都被抓到:移除驗證(47)、上界 < 100(15)、移除下界(14)、改拋 ValueError(47)、驗證移到寫入 state 之後(36)、移除 numbers.Real(17)。
  • 真實 NVML smoke(resctl gpu0,@ 4f3f4b30):真實 sample 2.3% 決策為 NORMAL;level 設為 EMERGENCY 後,NaN/-1/101/Decimal("NaN")/None 都被拒絕,EMERGENCY 保留;之後的真實 sample 只降一級到 FAST_PATH。
  • pre_push @ 4f3f4b30(只有實作、沒有座標):5952 passed、4 failed,都是 identity 落後的預期中間態(test_runtime_identity_staleness.py 2 個、test_research_lock.py 2 個)。座標 head 的 pre_push 見 stacked PR。

獨立審查

  • Codex r1 @ 9e17133e:NOT-READY,1 MAJOR:Decimal("NaN") 的 InvalidOperation 沒被 catch,dispatcher FAILED 但 stats 仍報 available → 4f3f4b30 改成先檢查 numbers.Real,並補 T8/dispatcher 回歸案例。
  • Codex r2(delta)@ 4f3f4b30:READY-TO-MERGE,沒有 findings。

Runtime identity

座標由 stacked PR(base 是本分支)提供,合併進本分支後一起原子落地。在 9e17133e 做的 capture 因 review 改了 resource_manager.py 而作廢,沒有出版。

不在範圍

不跑完整 online service,不取得 online-service qualification、不部署。C0-01/02、C0-10、health CLI、#538 其餘項目不變。

本機原始輸出(repo 外):~/.local/state/saccade/evidence/537-o0/20261023T194743/,sha256 見 stacked PR。

Refs #537

🤖 Generated with Claude Code

raylei50653 and others added 4 commits October 23, 2026 19:39
T7's three O0 rows recorded observed behavior only (NaN holds, negative
recovers one level, >100 escalates). T8 now pins the chosen contract
(#537 comment 6363972851, option A): usage outside 0..100, NaN,
infinities and non-comparable values raise ResourceMonitorUnavailable
with a ValueError cause and leave the level untouched; refusals do not
advance hysteresis. A dispatcher case checks the 4A failure channel
when a replaced get_stats() yields invalid usage mid-run.

Against the current implementation these 38 cases fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decide_degradation_level() trusted VRAMStats.usage_percent. A replaced
get_stats() could feed NaN, negative or >100 values that were taken as a
real decision (a negative sample recovered EMERGENCY one level). Usage
outside 0..100 is now an unavailable measurement, the same rule 4A
applies to raw NVML samples: it raises ResourceMonitorUnavailable with a
ValueError cause, keeps current_level and takes the existing dispatcher
failure channel. Production get_stats() cannot produce such a value.

Docs: failure_degradation_537.md §22 (plan, evidence, mutations) and
architecture_degradation.md §5/§7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… r1)

Codex r1 (NOT-READY, head 9e17133): Decimal("NaN") makes the range
comparison raise decimal.InvalidOperation, which escaped the
TypeError/ValueError handler; the dispatcher then failed with the monitor
still reported as available. Require numbers.Real before comparing, so
str, None and Decimal are refused without evaluating their comparisons;
numpy scalars and Fraction stay in domain.

T8 adds Decimal("NaN")/Decimal("50") and other real types; the dispatcher
case runs with -1 and Decimal("NaN"). 9 cases fail against 9e17133.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fresh capture (--run-probe --require-complete, gpu0 lease) at implementation
head 4f3f4b3, clean tree.

- implementation: 46d37ec50b215f21 -> ab1bc4ea82d70131. 317 -> 317 files;
  only src/saccade/resource/resource_manager.py (decide_degradation_level
  refuses usage that is not a real number in 0..100) changed.
- decision_surface, environment, identity_semantics, runtime_inputs:
  recomputed, unchanged.
- probe re-run, not reused: behavior digest 2dabed0b unchanged. Equal probe
  output is not an equivalence claim; equivalence stays unproven.

The outgoing canonical is archived append-only as
runtime_identity.2026-10-23.46d37ec50b215f21.json (byte-identical to
origin/main). --mode attested exits 0. A capture at the round-1 head
9e17133 was discarded unpublished after review changed resource_manager.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@raylei50653

Copy link
Copy Markdown
Owner Author

獨立審查紀錄(Codex,§0A 高風險)

輪次 head verdict findings 與處置
r1 9e17133ed5765745b63a88e7090b6b7044bf33bd NOT-READY 1. MAJOR resource_manager.py:113:Decimal("NaN") 比較時丟 decimal.InvalidOperation,沒被 TypeError/ValueError 接住;dispatcher FAILED 但 stats 仍報 available/EMERGENCY。→ 4f3f4b30:比較前先要求 numbers.Real,T8 補 Decimal("NaN")/Decimal("50") 與其他實數型別,dispatcher 案例參數化 Decimal("NaN");第一版在新測試下 9 個失敗
r2(delta) 4f3f4b30ea1c610f213c9cf00d07a429289941f5 READY-TO-MERGE none(148 個相關測試通過;NumPy NaN 在 strict floating-point error 設定下也沒有漏出例外)

Publication #627 的審查(r3 @ 9aca96f7,READY-TO-MERGE)記錄在 #627。原始 log:~/.local/state/saccade/evidence/537-o0/20261023T194743/codex_r{1,2}.log(sha256 見 #627)。

docs(runtime): republish the runtime coordinate for #537 O0
@raylei50653

Copy link
Copy Markdown
Owner Author

Self-check(#550 §0A 合併條件 1–5)

head:c625ce1b8136e6b4c50760114886b88a57a27d9c(= 4f3f4b30 實作 + #627 座標 9aca96f7 的 merge);base origin/main = 43432942,是 head 的祖先,所以合併後 tree 與 head tree 相同。

剩餘 OPEN:#537 的 C0-01/02、C0-10、health CLI;dispatcher 啟動 probe 不經過 O0 檢查(§22 已記錄,production 不可達);#538 其餘項目;不取得 online-service qualification、不部署。

Rollback/拒絕路徑:revert 本 PR 的 merge commit,再依 runbook 重新 capture 並出版座標(不直接恢復舊 canonical 冒充 current)。Owner 可否決方案 A,依此路徑回退。

合併指令(需 owner 執行):gh pr merge 626 --merge --match-head-commit c625ce1b8136e6b4c50760114886b88a57a27d9c

@raylei50653
raylei50653 marked this pull request as ready for review October 23, 2026 12:49
@raylei50653
raylei50653 merged commit 1f18cf7 into main Oct 23, 2026
8 checks passed
@raylei50653
raylei50653 deleted the fix/537-o0 branch October 23, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant