Skip to content

fix(omp): restore watcher continuity across superseded confirmations and dead arm children - #7208

Open
i098 wants to merge 3 commits into
kunchenguid:mainfrom
i098:fix/fm-omp-watch-fixes
Open

i098 wants to merge 3 commits into
kunchenguid:mainfrom
i098:fix/fm-omp-watch-fixes

Conversation

@i098

@i098 i098 commented Oct 13, 2026 •

Copy link
Copy Markdown

Closes #7207

Intent

Port the merged Pi watcher-continuity fix to the omp watch extension (.omp/extensions/fm-primary-omp-watch.ts). Today a handling confirmation that exits 3 (the recovery marker already moved to another generation, meaning a newer pipeline owns the episode) is reported to the primary as 'watcher: FAILED - handling delivery confirmation was rejected (status=3 ...)' on a wake that delivered fine, so the primary spends turns on a non-failure. It must deliver the wake plain, confirm and retry against the restoration's own recovery token, and retire the current arm only when the failed token names its exact watcher pid and generation and that pid is dead. Also, an arm child whose process is gone but whose close event has not fired (a straggler holds its stdio pipes) keeps the single-flight slot, so repair (fm_watch_arm_omp) and scheduled retry answer 'unchanged' and start nothing; such a child must count as an empty slot, as on Pi (liveArmChild). Restore only: no new knob and no default change; OpenCode keeps its current behavior.

What Changed

  • .omp/extensions/fm-primary-omp-watch.ts now ports the Pi contract from fix(pi): restore watcher continuity across successor gaps and make extension log opt-in #5489 (closing Pi watcher continuity: successor-gap confirmations and a dead-but-unclosed arm child block repair/retry recovery #5492).
    A handling confirmation that exits 3 means the recovery marker moved to another generation, so a newer pipeline owns the episode.
    omp delivers that wake plain, with no watcher: FAILED - handling delivery confirmation was rejected (status=3 ...) line.
  • omp confirms and retries against the restoration's own recovery token, not a fresh snapshot of the current arm child.
    After a failed confirmation it retires the current arm only when the failed token names that arm's exact watcher pid and generation and that pid is dead.
  • liveArmChild treats an arm child whose process is gone, but whose close event has not fired because a straggler holds its stdio, as an empty slot.
    Repair, scheduled retry, and deferred close start a fresh arm instead of answering "unchanged".
  • docs/watcher-continuity.md scopes those sentences to Pi and omp.
    OpenCode has the same gap and is not touched here.
  • Three tests in tests/fm-omp-harness.test.sh, each failing on main and passing here:
    a superseded confirmation (exit 3) is not a failure while a refused one (exit 1) still is;
    a failed confirmation for a stale successor spares a newer arm;
    a dead-but-unclosed arm child frees the slot for repair and for the scheduled retry.

Risk Assessment

✅ Low: This restores existing Pi behavior in the omp extension, with no new setting and no default change.
The retire guard now acts on fewer cases than before, never more.
The deferred-close path uses the same liveArmChild call as repair and retry, and it has no test of its own.

Testing

Run before and after, with a Node host running the real extension over a fake omp API and a stub bin/fm-watch-arm.sh:

Case Before After
confirmation exits 3 wake ends with watcher: FAILED - handling delivery confirmation was rejected (status=3 generation=gen-1 ...) plain wake, no failure text
confirmation exits 1 rejected (status=1 ...) unchanged
failed confirmation while a newer arm runs main retires the current arm once the failed token's watcher pid is dead, newer or not the newer healthy arm survives
arm exits while a straggler holds its pipes, then fm_watch_arm_omp unchanged - omp extension already owns an arm child started omp extension arm child

The retire-guard test also fails when only the pid and generation check is removed.

bin/fm-test-run.sh --all on Linux with no ruby and no agent harness installed, detached from omp: 254 scripts, 16 fail.
The same 16 fail the same way on clean main 20dcbf91, so this change adds no failure:
fm-backend-herdr-presentation-e2e, fm-ci-workflow, fm-control-relaunch, fm-extension-binding, fm-harness-liveness-drift-live-e2e, fm-nm-test-contract, fm-pr-check-security, fm-procevent-quota, fm-procevent, fm-public-followup, fm-remote-secondmate-lifecycle-e2e, fm-remote-transport-lanes, fm-secondmate-restart, fm-session-lock-ancestry, fm-supervision-host, fm-test-run.
Two of them print ruby is required, and one prints no verified harness is installed here.
tests/fm-omp-harness.test.sh passes when run detached from omp.
It fails when run from inside an omp session, on main and here alike, because its detection test sees the real omp ancestor.

Live validation: one scenario ran live, a real omp 18.7.0 that loaded the changed extension, armed a watcher through it, started a successor, and delivered one wake.
The other four scenarios (exit 3, exit 1, the newer-arm guard, the dead-child slot) ran only in the fixture host, with a fake omp API, real arm child processes, and the real confirm script.
The Test step therefore reports the live-validation verdict as inconclusive.
The merged Pi PR shipped with the same limitation.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

⏭️ **Rebase** - skipped

Step was skipped.

✅ **Review** - passed

✅ No issues found.

⚠️ **Test** - 2 warnings
  • ⚠️ tests/fm-omp-primary-live-e2e.test.sh - The live omp e2e (FM_OMP_LIVE_E2E=1 tests/fm-omp-primary-live-e2e.test.sh, real omp 18.12.1 over RPC, started from a clean tmux ancestor) fails at its step 2: 'the session-start digest did not reach model context before the first turn'. The model replied that no digest was present, and the log shows one extension_error frame. This is before the arm/deliver steps, so no live watch scenario could be reached. The diff changes only .omp/extensions/fm-primary-omp-watch.ts, docs/watcher-continuity.md and tests/fm-omp-harness.test.sh, so this looks like omp 18.12.1 drift or an environment issue. I did not run the script on the base commit to confirm that. The portable suite, which runs the same extension over a fake omp API, passes fully from a clean ancestor.
  • ⚠️ live validation verdict: inconclusive (0 of 4 scenarios were driven live against the product); untested: A handling confirmation exiting 3 delivers a plain wake with no 'watcher: FAILED' appendix, while an exit-1 refusal still reports the failure, A failed confirmation retries on its own recovery token and retires only an arm whose pid and generation match and whose pid is dead; a newer healthy arm survives, A scheduled retry and repair both treat a dead-but-unclosed arm child as an empty slot, Existing watch-extension behaviours unchanged: supervision host in away and quiet modes, opt-out, replacement replay, split host close
  • Live validation: ⚠️ inconclusive - 0 of 4 scenarios driven live against the product
Scenario Result Live Evidence
A handling confirmation exiting 3 delivers a plain wake with no 'watcher: FAILED' appendix, while an exit-1 refusal still reports the failure ⏸️ untested no The live omp e2e fails at its session-start-digest step (step 2) with real omp 18.12.1, before any arm or confirmation step. I tried a clean-ancestor tmux run, which reached the same result. Making th…
A failed confirmation retries on its own recovery token and retires only an arm whose pid and generation match and whose pid is dead; a newer healthy arm survives ⏸️ untested no Same blocker: the live omp e2e fails at its session-start-digest step before reaching any watch step, so a stale-successor lab case on the real omp could not be built on a passing live base.
A scheduled retry and repair both treat a dead-but-unclosed arm child as an empty slot ⏸️ untested no Same blocker: the live e2e does not get past its session-start-digest step, so SIGKILLing the arm child on the real omp could not be driven.
Existing watch-extension behaviours unchanged: supervision host in away and quiet modes, opt-out, replacement replay, split host close ⏸️ untested no Same blocker: the live e2e does not reach the arm-and-deliver step with real omp 18.12.1.
  • env -u NO_MISTAKES_GATE TMUX_TMPDIR=$LAB/tmux tmux -L fm-lab new-session ... FM_OMP_LIVE_E2E=1 bash tests/fm-omp-primary-live-e2e.test.sh (real omp 18.12.1): step 1 ok (both extensions auto-discovered), step 2 not ok (session-start digest), exit 1
  • bash tests/fm-omp-harness.test.sh full run from a clean tmux ancestor (fm-lab socket): 19 ok, 0 not ok, exit 0
  • Earlier: the nine watch-extension tests run with the ten earlier tests removed, all ok
✅ **Document** - passed

✅ No issues found.

⏭️ **Lint** - skipped
  • ⚠️ linter found issues (exit code 1)
✅ **Push** - passed

✅ No issues found.

I am new to open source, and this pull request was made with AI assistance.

@i098 i098 changed the title fix(omp): restore watcher continuity on superseded confirmations and dead arm children fix(omp): restore watcher continuity across superseded confirmations and dead arm children Oct 13, 2026
@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: thank you @i098 — porting the Pi continuity fix (#5489) to omp is useful work, and the issue write-up in #7207 made it easy to check.

  • Closes omp watcher continuity: a superseded handling confirmation is reported as a failure and a dead-but-unclosed arm child blocks repair/retry recovery #7207: verified. The diff covers all three defects: an exit-3 confirmation now routes as superseded, confirm/retry uses the restoration's own token with a narrower retire guard, and liveArmChild covers repair, scheduled retry, and deferred close. The three new tests fail on main and pass here. The no-mistakes attestation matches the tip (1902eaf1).
  • CI: I approved the fork CI runs for this head.
  • One gentle ask (live evidence): the attestation's live-validation verdict is inconclusive (0 of 5). The four changed behaviors ran only in the fixture host with a fake omp API. If you can, please drive at least the superseded-confirmation and dead-but-unclosed-child cases against a real omp primary and post what you saw. Re-raising through git push no-mistakes with live evidence works too. No rush, and thank you.
  • Heads-up for after that: docs/watcher-continuity.md on main documents omp's current confirm/retire path on purpose. Moving omp onto Pi's semantics changes that documented default, so once live evidence is in and CI is green, Kun will make the merge call himself. That part is not waiting on you.

Waiting on you right now: just the live evidence above. (main is currently red for unrelated reasons, so nothing lands this pass anyway.)

@i098 i098 changed the title fix(omp): restore watcher continuity across superseded confirmations and dead arm children fix(omp): harden watcher continuity restore against stale arms and superseded confirmations Oct 13, 2026
@i098 i098 changed the title fix(omp): harden watcher continuity restore against stale arms and superseded confirmations fix(omp): restore watcher continuity across superseded confirmations and dead arm children Oct 13, 2026
@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: thanks @i098. Moving the node heredocs into files for Bash 3.2 is a clean fix. The no-mistakes attestation matches the new tip (98cad9c3).

Re-look:

  • Delta since 1902eaf1: test-only (tests/fm-omp-harness.test.sh); the extension code is unchanged. I reviewed it and approved the fork CI runs for this head (CI 38360224257, plus the no-mistakes/body runs).
  • Live evidence: still inconclusive (1 of 5). The real omp 18.7.0 run proves auto-discovery, but the live e2e stops at the session-start digest stage before it reaches the watch stage. The four changed behaviors have only run against the fake omp API. If that digest stop also happens on main, a hand-driven real omp primary is fine instead: show one superseded (exit-3) confirmation delivering a plain wake, and one dead-but-unclosed arm child that repair replaces. No rush.
  • Contract-class: still new-default. It moves omp's documented confirm/retire path onto Pi's semantics, so once the live evidence is in and CI is green, Kun makes the merge call himself.

Waiting on you right now: just the live evidence. main is still red for unrelated reasons, so nothing lands this pass anyway.

@i098

i098 commented Oct 13, 2026

Copy link
Copy Markdown
Author

Live evidence for the two cases in the review. The later head 98cad9c3 changes only a test file for Bash 3.2, so the extension under test is the same. Both ran on a real omp primary (omp/18.7.0, interactive TUI) with the real bin/fm-watch-arm.sh and bin/fm-watch.sh, in an isolated FM_HOME with its own lock and state. "Before" is main at 6407052f. "After" is the PR head at 1902eaf1.

Case main PR at 1902eaf1
Superseded handling confirmation (exit 3) The wake carries watcher: FAILED - handling delivery confirmation was rejected (status=3 ...). The confirmation runs twice. The repair answers unchanged. The confirmation runs once and returns 3. It delivers a plain wake (no FAILED line).
Arm child dead, close event not fired The repair answers unchanged - omp extension already owns an arm child. The next wake never reaches main (83 s observed). The repair replaces the dead arm child (started omp extension arm child 5). The next wake reaches main in 5 s.

The main output is the same as on the author's own omp primary on 2026-10-13. There, a wake at 01:57Z and one at 02:00Z carried rejected (status=3 ...), and each repair answered unchanged. A second home logged the same line 24 times between 2026-10-11 and 2026-10-13, all status=3.

How the cases were driven:

  • Superseded confirmation. One lab-only hook moves the recovery marker to a newer generation just before the extension's --handling-delivered call. It uses fm_recovery_marker_ack and fm_recovery_marker_publish from bin/fm-wake-lib.sh. The same hook ran on both sides. Everything else is unmodified.
  • Dead arm child. kill -9 on the arm child. Its watcher keeps the arm's stderr socket open, so the close event cannot fire. No fake process is involved.
Superseded confirmation on main
confirm-start argv=--handling-delivered 1559193.1791873782.xCQqdL --watcher-pid 1569226 flag=armed
supersede     old=announced:downtime:1559193.1791873782.xCQqdL new=pending:downtime:1570018.1791873784.O7zsSc
confirm-end   exit=3
confirm-start argv=--handling-delivered 1559193.1791873782.xCQqdL --watcher-pid 1569226 flag=off
confirm-end   exit=3

Wake sent to main (2026-10-13T06:43:04Z):

FIRSTMATE WATCHER WAKE: signal: $LAB/state/lab-1.status

watcher: FAILED - handling delivery confirmation was rejected (status=3 generation=1559193.1791873782.xCQqdL watcherPid=1569226)

Repair, 06:43:45Z:

watcher: unchanged - omp extension already owns an arm child; no manual re-arm needed; call fm_watch_arm_omp again only after a later notification says the cycle is missing, failed, or unhealthy
Superseded confirmation on this PR
confirm-start argv=--handling-delivered 1369771.1791873617.iAe3e4 --watcher-pid 1409223 flag=armed
supersede     old=announced:downtime:1369771.1791873617.iAe3e4 new=pending:downtime:1410185.1791873620.SOLwNs
confirm-end   exit=3

Wake sent to main (2026-10-13T06:40:20Z). One confirmation call, no retry, no FAILED line:

FIRSTMATE WATCHER WAKE: signal: $LAB/state/lab-1.status

Run bin/fm-wake-drain.sh first and handle the queued wake. Watcher continuity is extension-owned.
Dead arm child on main
06:36:12  kill -9 arm 1117656 (watcher 1117751 keeps the stderr socket)
06:36:24  watcher: unchanged - omp extension already owns an arm child; no manual re-arm needed; ...
06:36:42  new wake appended
06:38:01  no wake reached main; state/.wake-queue still holds the two rows for it
Dead arm child on this PR
06:41:19  kill -9 arm 1409164 (watcher 1409223 keeps the stderr socket)
06:41:24  watcher: started omp extension arm child 5; future ordinary re-arms are automatic; ...
06:41:51  new wake appended
06:41:56  wake reached main: signal: $LAB/state/lab-2.status

Delivery log after the repair (state/.watch-deliveries.log, last column):

check: rearm-resurface
signal: $LAB/state/lab-2.status

Limits:

  • The marker advance in the first case is injected by the hook. The marker functions and the confirmation code are real.
  • Only the repair path of liveArmChild ran live. The scheduled-retry and deferred-close paths are covered by the new fixture test only.
  • The lab primary is a Haiku-class model. It was told to call the repair after the FAILED notice, as the main primary did in the incident.

@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: thank you @i098 — that hand-driven omp 18.7.0 evidence is exactly what was needed.

Re-look (tip still 98cad9c3, extension unchanged since the Bash 3.2 test-only commit):

  • Superseded (exit 3): main wakes with watcher: FAILED … status=3 + repair unchanged; tip delivers a plain wake, one confirm, no FAILED line.
  • Dead arm / straggler pipes: main repair stays unchanged and the wake sits; tip starts a fresh arm child and the wake reaches main in seconds.

Live ask: cleared (fixture-only rows in the attestation body are fine now that these two product traces are on the thread). Attestation MATCH at 98cad9c3.

Contract-class: still new-default. docs/watcher-continuity.md on main documented omp's confirm/retire path; this moves omp onto Pi's semantics. When live+CI+main are green, Kun makes the merge-or-leave call — not auto-merge.

CI: run 38360224257 failed only on the same main-red fm-pi-primary-types shard; Serial 2 cancelled. Not on you.

Waiting on: nothing from you right now. Next gate is green CI on a reasonably current base + green main, then captain merge-or-leave.

@i098
i098 force-pushed the fix/fm-omp-watch-fixes branch from 98cad9c to 0648e4a Compare October 22, 2026 00:46
@i098 i098 changed the title fix(omp): restore watcher continuity across superseded confirmations and dead arm children fix(omp): align watcher restoration with Pi recovery-token semantics Oct 22, 2026
@i098 i098 changed the title fix(omp): align watcher restoration with Pi recovery-token semantics fix(omp): restore watcher continuity across superseded confirmations and dead arm children Oct 22, 2026
@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: thanks @i098 for rebasing this onto current main. The attestation binds the new head 0648e4a3.

I re-read it against main bd866530 (0 behind, mergeable). It's the same omp extension change as before: superseded (exit 3) confirmations route as delivered, confirm and retry use the restoration's own token, retiring now needs a matching pid, generation and a dead process, and liveArmChild frees the slot of a dead-but-unclosed arm child. It also carries the matching docs/watcher-continuity.md lines and the Bash 3.2 test fix. There's no workflow, credential or network surface, so I approved this head's fork runs (CI 39701740447, Require no-mistakes 39704185325). Your hand-driven omp 18.7.0 evidence still covers the live ask.

contract-class: new-default (unchanged): omp's documented confirm/retire path moves onto Pi's semantics. VISION: every rule aligns (honest wakes, no false FAILED turns, repair that actually recovers).

There's nothing you need to do right now. If CI comes back green, Kun makes the merge-or-leave call.

@i098 i098 changed the title fix(omp): restore watcher continuity across superseded confirmations and dead arm children fix(omp): align omp watch handoff confirmation and arm-slot recovery with Pi Oct 22, 2026
@i098 i098 changed the title fix(omp): align omp watch handoff confirmation and arm-slot recovery with Pi fix(omp): restore watcher continuity across superseded confirmations and dead arm children Oct 22, 2026
i098 added 3 commits October 27, 2026 00:56
…and dead arm children

Port the Pi watcher-continuity contract to the omp watch extension.

A handling confirmation that exits 3 means the recovery marker moved past this restoration's generation, so a newer pipeline owns the episode.
omp reported it as "watcher: FAILED - handling delivery confirmation was rejected (status=3)" on a wake that delivered fine.
It now delivers the wake plain, confirms and retries against the restoration's own recovery token, and retires the current arm only when the failed token names its exact watcher pid and generation and that pid is dead.

An arm child whose process is gone but whose close event has not fired (a straggler holds its stdio) kept the single-flight slot, so repair and scheduled retry answered "unchanged" and started nothing.
liveArmChild treats such a child as an empty slot.

docs/watcher-continuity.md scopes the token confirm, retry, retire guard, and superseded delivery to Pi and omp.
OpenCode keeps its current behavior.
@i098
i098 force-pushed the fix/fm-omp-watch-fixes branch from 0648e4a to b740983 Compare October 27, 2026 02:18
@i098 i098 changed the title fix(omp): restore watcher continuity across superseded confirmations and dead arm children fix(omp): harden watcher restoration against dead arm children and superseded handoffs Oct 27, 2026
@i098 i098 changed the title fix(omp): harden watcher restoration against dead arm children and superseded handoffs fix(omp): restore watcher continuity across superseded confirmations and dead arm children Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

omp watcher continuity: a superseded handling confirmation is reported as a failure and a dead-but-unclosed arm child blocks repair/retry recovery

2 participants