From d1b86fa53ed6ed7460f4de7cf2307a1ef6a624ee Mon Sep 17 00:00:00 2001 From: Rey Date: Fri, 17 Jul 2026 09:04:55 +0000 Subject: [PATCH 1/3] fix(fifo): record _ever_delivered at publish, not at raw read MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The cold-start liveness check (harness-control#93) exists to re-arm a pipe-pane forwarder that never started delivering. It is gated on `not ever_delivered`, but `_ever_delivered` was set where bytes are pulled off the FIFO — not where they are published to the event bus. Those are not the same moment. Bytes sit in `pending` until the coalesce window closes, and when the forwarder cold-starts dead, the only bytes ever read are flushed by the `finally:` block at thread teardown. So `ever_delivered` flipped True while the StatusMonitor buffer stayed empty, permanently blinding the one check that would have healed the pipe. The divergence path cannot cover for it either: an idle shell's pane is static, so it never accumulates strikes. Net effect was a dead pipe that nothing re-armed, and wait_for_shell() timing out after 60s on every single `cao launch`. Move the flag to the publish site so it means what the check reads it as — "this pipeline has delivered to consumers". `_last_data_at` stays on the raw read: the divergence check genuinely wants "did the FIFO yield bytes since the last tick", and its semantics are unchanged. Verified with a probe replicating terminal_service's exact sequence: before, 0 events / 0 re-arms across 3/3 runs (matching 4/4 real launch failures); after, events delivered 3/3 with the watchdog firing and self-healing. Co-Authored-By: Claude Opus 4.8 --- .../services/fifo_reader.py | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/src/cli_agent_orchestrator/services/fifo_reader.py b/src/cli_agent_orchestrator/services/fifo_reader.py index 7c3cfed2f..2bb80def4 100644 --- a/src/cli_agent_orchestrator/services/fifo_reader.py +++ b/src/cli_agent_orchestrator/services/fifo_reader.py @@ -340,7 +340,6 @@ def _reader_loop(self, terminal_id: str, fifo_path, stop_flag: threading.Event) with self._lock: if terminal_id in self._readers: self._last_data_at[terminal_id] = time.monotonic() - self._ever_delivered[terminal_id] = True if not pending: batch_start = time.monotonic() pending.extend(raw) @@ -357,6 +356,22 @@ def _reader_loop(self, terminal_id: str, fifo_path, stop_flag: threading.Event) ): bus.publish(topic, {"data": pending.decode("utf-8", errors="replace")}) pending.clear() + # Cold-start liveness (harness-control#93) asks "has this + # pipeline delivered to CONSUMERS yet" — so it must be + # recorded here, at the publish, not where bytes are pulled + # off the FIFO. Recording it on the raw read let a reader + # that read bytes but never published them (they sit in + # `pending` until the coalesce window closes) satisfy + # `ever_delivered` while the StatusMonitor buffer stayed + # empty — permanently blinding the cold-start check, whose + # whole purpose is to re-arm a forwarder that never + # started. The divergence path cannot cover for it either: + # an idle shell's pane is static, so it never strikes. + # Net effect was a dead pipe that nothing ever re-armed and + # wait_for_shell() timing out at 60s on every launch. + with self._lock: + if terminal_id in self._readers: + self._ever_delivered[terminal_id] = True except Exception as e: if not stop_flag.is_set(): logger.error("FIFO reader for terminal %s exiting on error: %s", terminal_id, e) From 2f666e3a7b57bc38c808d0775bac15860f8d33d8 Mon Sep 17 00:00:00 2001 From: Rey Date: Fri, 17 Jul 2026 09:05:09 +0000 Subject: [PATCH 2/3] fix(status): fall back to live pane when the pushed pipeline says UNKNOWN MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tmux silently stops forwarding a pane's output to the FIFO after a burst of alternate-screen redraws (#388) — Claude Code's Ink TUI is exactly that shape. The rolling buffer then freezes on whatever it last saw, typically the pre-launch shell prompt, which detects as UNKNOWN, while the pane itself renders a healthy agent idling at a ready prompt. The liveness watchdog structurally cannot recover this. Once the TUI finishes painting, the pane is STATIC, so "pane advanced but FIFO silent" never trips — a settled frame is indistinguishable from a genuinely idle terminal. Live-reproduced: exactly one cold-start re-arm, then the watchdog sat silent for 66s (alive and ticking the whole time) while wait_until_status timed out at 60s on an agent that was ready the entire time. Detection was never the problem — on the real captured output, get_status_from_screen returns IDLE and raw get_status returns COMPLETED. CAO simply never saw the bytes. So when the pushed pipeline knows nothing, ask tmux directly. capture-pane is immune to the stall (it reads tmux's own screen) and the providers' detectors already understand composited screen content — it is what _handle_startup_prompts() has always used to spot trust/bypass dialogs while the FIFO was dead. Gated on a cached UNKNOWN, mirroring the PROCESSING escape hatch already in get_status() for the same class of "debounced detection is stuck" problem. It therefore only ever upgrades a non-answer into a real one, and never forks tmux on the hot per-chunk path the class docstring warns about. Verified against real Claude Code running in tmux with no CAO or FIFO involved: get_status_from_screen(capture-pane) -> COMPLETED in 4s. Co-Authored-By: Claude Opus 4.8 --- .../services/status_monitor.py | 75 +++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/src/cli_agent_orchestrator/services/status_monitor.py b/src/cli_agent_orchestrator/services/status_monitor.py index d77ce7e7e..dc9629dcd 100644 --- a/src/cli_agent_orchestrator/services/status_monitor.py +++ b/src/cli_agent_orchestrator/services/status_monitor.py @@ -531,6 +531,65 @@ def reset_buffer(self, terminal_id: str) -> None: handle = self._quiesce_handle.pop(terminal_id, None) self._cancel_quiesce_handle(handle) + def _detect_from_live_pane(self, terminal_id: str) -> Optional[TerminalStatus]: + """Detect status from tmux's LIVE pane instead of the FIFO-fed buffer. + + Ground-truth escape hatch for a stalled pipe-pane forwarder (#388). tmux + can silently stop forwarding a pane's output to the FIFO after a burst of + alternate-screen redraws — Claude Code's Ink TUI is exactly that shape. + The rolling buffer then freezes on whatever it last saw (typically the + pre-launch shell prompt, which detects as UNKNOWN) while the pane itself + renders a perfectly healthy, idle agent. + + The liveness watchdog cannot recover this case: once the TUI finishes + painting, the pane is STATIC, so its "pane advanced but FIFO silent" + divergence test never trips — a settled frame is indistinguishable from a + genuinely idle terminal. Live-reproduced: agent idle at a ready prompt, + status pinned at UNKNOWN, wait_until_status timing out at 60s. + + capture-pane is immune to the stall (it reads tmux's own screen), and the + providers' detectors already understand composited screen content — it is + what _handle_startup_prompts() has always used to spot trust/bypass + dialogs while the FIFO was dead. So when the pushed pipeline knows + nothing, ask tmux directly. + + Only consulted on a cached UNKNOWN (see get_status), so this costs one + capture-pane per poll on a terminal we know nothing about — never on the + hot per-chunk path this class's docstring warns about forking from. + """ + from cli_agent_orchestrator.backends.registry import get_backend + + try: + provider = provider_manager.get_provider(terminal_id) + except Exception: + return None + if provider is None: + return None + + session_name = getattr(provider, "session_name", None) + window_name = getattr(provider, "window_name", None) + if not session_name or not window_name: + return None + + try: + content = get_backend().get_history(session_name, window_name, strip_escapes=True) + except Exception as e: + logger.debug(f"live-pane detect [{terminal_id}]: capture-pane failed: {e}") + return None + + if not content.strip(): + return None + + try: + if getattr(provider, "supports_screen_detection", False): + # capture-pane output IS a composited viewport — exactly what + # get_status_from_screen expects (escape-free rows). + return provider.get_status_from_screen(content.split("\n")) + return provider.get_status(content) + except Exception as e: + logger.debug(f"live-pane detect [{terminal_id}]: detection failed: {e}") + return None + def get_status(self, terminal_id: str) -> TerminalStatus: """Get current terminal status — the single source of truth for both backends. @@ -585,6 +644,22 @@ def get_status(self, terminal_id: str) -> TerminalStatus: if fresh != TerminalStatus.PROCESSING and fresh != TerminalStatus.UNKNOWN: self._apply_detection(terminal_id, fresh) return fresh + + # A cached UNKNOWN on a pipe-pane backend means the pushed pipeline has + # told us nothing at all — either it has not started yet, or the + # forwarder stalled and the buffer is frozen on stale content. Both are + # indistinguishable from here, and in both cases tmux's live pane is the + # ground truth. Ask it directly rather than waiting on a stream that may + # never resume. Same escape-hatch shape as the PROCESSING case above: + # only ever upgrades a non-answer into a real one. + if cached == TerminalStatus.UNKNOWN: + fresh = self._detect_from_live_pane(terminal_id) + if fresh is not None and fresh != TerminalStatus.UNKNOWN: + logger.debug( + f"get_status [{terminal_id}]: cached=UNKNOWN, live pane -> {fresh.value}" + ) + self._apply_detection(terminal_id, fresh) + return fresh return cached def get_buffer(self, terminal_id: str) -> str: From 04ee8a09ff1c443dc5954120b788b322ee74c20d Mon Sep 17 00:00:00 2001 From: Rey Date: Fri, 17 Jul 2026 09:06:13 +0000 Subject: [PATCH 3/3] docs: runbook for the three cao launch --provider claude_code failures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records all three bugs, why each hid the next, the evidence behind each fix, and the install-layout trap that reverts them (uv tool install is NON-editable — site-packages is a copy, so reinstalling from `main` silently undoes both code fixes). Includes setup steps for a new machine, the debugging notes that were expensive to rediscover (bus.publish silently no-ops without set_loop; _apply_detection only logs on change; the reader spins rather than blocks in os.read), and the wrong turns worth not repeating — chiefly the stale-TUI-regex theory, which is false: both failures occur before or independently of TUI parsing. Co-Authored-By: Claude Opus 4.8 --- LOCAL-PATCHES.md | 202 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 202 insertions(+) create mode 100644 LOCAL-PATCHES.md diff --git a/LOCAL-PATCHES.md b/LOCAL-PATCHES.md new file mode 100644 index 000000000..b7385b122 --- /dev/null +++ b/LOCAL-PATCHES.md @@ -0,0 +1,202 @@ +# Local patches: `cao launch --provider claude_code` fails + +Diagnosed and fixed 2026-07-17 against CAO `main` (a614f32) + Claude Code CLI v2.1.212 on Linux / Python 3.14. + +Three independent bugs stacked on top of each other. Each one hid the next, so fixing only the first +just moves the failure one stage later. All three must be applied. + +--- + +## TL;DR — setting CAO up on a new machine + +```bash +# 1. Clone + install (uv tool install is NON-editable — it copies the source) +git clone https://github.com/awslabs/cli-agent-orchestrator.git ~/cli-agent-orchestrator +cd ~/cli-agent-orchestrator +git checkout fix/fifo-pipeline-stalls # <- the branch carrying fixes 2 & 3 +uv tool install . + +# 2. Fix 1 is config, not code. Must be > provider_init_timeout (60). +cao config set server.mcp_request_timeout 120 + +# 3. Start the server and launch +nohup cao-server > ~/.aws/cli-agent-orchestrator/logs/server-stdout.log 2>&1 & +cao launch --agents developer --provider claude_code --auto-approve +``` + +If you install from upstream `main` instead of the branch, bugs 2 and 3 come back. + +--- + +## The install layout (read this first — it causes the most confusion) + +CAO is a **`uv tool` install from a local git checkout**, and it is **NOT editable**: + +``` +~/cli-agent-orchestrator <- git checkout (source of truth) +~/.local/share/uv/tools/cli-agent-orchestrator/lib/python3.14/site-packages/cli_agent_orchestrator + <- a COPY; this is what cao-server actually runs +``` + +Consequences that will bite you: + +- **Editing the checkout does nothing until you reinstall.** The server runs the copy. +- **`uv tool install .` rebuilds the copy from whatever the checkout is currently on.** Reinstall while + sitting on `main` and you silently revert all patches. +- **PyPI is the wrong place to check versions.** PyPI's 2.3.0 lags `main` by many commits. + Use `git -C ~/cli-agent-orchestrator log main..origin/main`. + +Verify the two copies agree: + +```bash +diff -rq ~/.local/share/uv/tools/cli-agent-orchestrator/lib/python3.14/site-packages/cli_agent_orchestrator \ + ~/cli-agent-orchestrator/src/cli_agent_orchestrator +``` + +Any difference means the running server is not the code you think it is. **Check this first** if launches +break again after an update. + +--- + +## Bug 1 — inverted timeout defaults (config fix) + +**Symptom** + +``` +Error: Failed to connect to cao-server: HTTPConnectionPool(host='127.0.0.1', port=9889): +Read timed out. (read timeout=30) +``` + +**Cause** + +The launch client POSTs with `timeout=mcp_request_timeout` (default **30s**, `cli/commands/launch.py:290`), +while the server legitimately blocks up to `provider_init_timeout` (default **60s**) waiting for the agent +to reach idle. Any init taking 30–60s therefore *always* times out client-side — while the server is +working correctly and would have succeeded. The defaults are simply inverted; `get_server_settings()`'s own +docstring example shows the intended shape (`mcp_request_timeout: 120` *above* `provider_init_timeout: 90`). + +This error is a **red herring**: it masks the real failure. Fixing it doesn't make launch work, it makes +the actual error visible (a 500). + +**Fix** — config only, no code, no server restart (the value is read client-side per invocation): + +```bash +cao config set server.mcp_request_timeout 120 +``` + +Persists to `~/.aws/cli-agent-orchestrator/settings.json`. Not versioned in git — re-apply per machine. + +--- + +## Bug 2 — `_ever_delivered` blinded the cold-start watchdog (commit `d1b86fa`) + +**Symptom** + +``` +ERROR - Failed to create terminal: Shell initialization timed out after 60s +WARNING - FIFO reader thread for terminal did not exit within 2s; leaking a daemon thread +``` + +Note this fails in `wait_for_shell`, which runs **before Claude Code is ever launched** — it is waiting for +the plain *shell prompt*. Any theory about Claude's TUI is therefore irrelevant here. + +**Cause** + +1. tmux's initial `pipe-pane` attach often forwards nothing (the known cold-start case, harness-control#93). +2. The reader pulls some bytes, but they sit in `pending` until the coalesce window closes — when the pipe + is cold-dead, the only flush is the `finally:` block at thread teardown. +3. `_ever_delivered` was set where bytes are **read**, not where they are **published**. +4. The cold-start check requires `not ever_delivered` → permanently blinded by bytes that never reached a + consumer. +5. The divergence path can't cover for it: an idle shell's pane is static, so it never strikes. +6. Nothing re-arms the dead pipe → buffer stays empty → `wait_for_shell` times out. + +**Fix** — move `_ever_delivered = True` to the publish site so it means "delivered to consumers". +`_last_data_at` stays on the raw read (divergence semantics unchanged). + +**Evidence** — probe replicating `terminal_service`'s exact sequence: before, 0 events / 0 re-arms in 3/3 +runs (matching 4/4 real launch failures); after, events delivered 3/3, watchdog self-heals. + +--- + +## Bug 3 — FIFO stalls on Claude's alternate-screen TUI (commit `2f666e3`) + +**Symptom** (only visible after bug 2 is fixed) + +``` +INFO - Shell ready for (buffer stable, 557 bytes) <- bug 2 fixed, init got further +INFO - wait_until_status []: waiting for {idle, completed}, timeout=60s +WARNING - wait_until_status []: timeout waiting for {idle, completed} +ERROR - Failed to create terminal: Claude Code initialization timed out after 60s +``` + +Meanwhile Claude is **completely healthy** — banner rendered, trust dialog accepted, idling at a ready prompt. + +**Cause** + +tmux silently stops forwarding to the FIFO after a burst of alternate-screen redraws (issue #388) — Claude's +Ink TUI is exactly that shape. The rolling buffer freezes on the pre-launch shell prompt (detects UNKNOWN) +while the pane renders a healthy agent. + +The liveness watchdog **structurally cannot** recover this: once the TUI finishes painting, the pane is +STATIC, so its "pane advanced but FIFO silent" divergence test never trips — a settled frame is +indistinguishable from a genuinely idle terminal. Observed: exactly one cold-start re-arm, then the watchdog +sat silent for 66s while alive and ticking. + +Detection was never at fault. On the real captured output, `get_status_from_screen` → IDLE and raw +`get_status` → COMPLETED. CAO just never saw the bytes. + +**Fix** — `StatusMonitor._detect_from_live_pane()` + a hook in `get_status()`: on a cached UNKNOWN, detect +from tmux's live `capture-pane` instead of the frozen buffer. Mirrors the pre-existing PROCESSING escape +hatch in the same function. Gated on UNKNOWN, so it never forks tmux on the hot per-chunk path. + +**Evidence** — real Claude in tmux with no CAO/FIFO involved: `get_status_from_screen(capture-pane)` → +COMPLETED in 4s. Tests: 146 passed. + +--- + +## The load-bearing insight + +**`capture-pane` is reliable. The FIFO / pipe-pane pipeline is not.** + +`_handle_startup_prompts()` has always used `capture-pane` and has always worked — even while the FIFO was +stone dead and every FIFO-fed consumer was blind. That pipeline has four upstream issues against it (#382 +blocked opens / leaked threads, #388 stalled forwarder, harness-control#93 cold start, #148 burst-then-settle) +and elaborate self-healing machinery that still didn't cover this case. + +**If you need something to be correct, read `capture-pane`.** Bug 3's fix is an application of exactly this. + +--- + +## Debugging notes + +- **Server logs**: `~/.aws/cli-agent-orchestrator/logs/cao_.log`. + **Per-terminal raw output**: `logs/terminal/.log` (written by LogWriter off the same event + bus — if this file has content but StatusMonitor saw nothing, the data arrived in the teardown flush). +- **Tracebacks name site-packages, not the checkout** — proof the server runs the copy. +- `bus.publish` **silently no-ops** when `bus._loop is None` (`event_bus.py:61`). A standalone repro script + must call `bus.set_loop()` or every publish vanishes with no error. +- The FIFO reader **spins** in `os.read`, it does not block — `O_NONBLOCK` is genuinely set (verified via + `F_GETFL` on the live fd). A stack snapshot showing `os.read` is a spin artifact, not a wedge. +- The watchdog thread is named `fifo-pipe-watchdog`; check it exists and its state: + `for t in /proc/$(pgrep -f bin/cao-server)/task/*; do cat $t/comm; done` + Parked in `futex_do_wait` = healthy, ticking its 4s `Event.wait`. +- `_apply_detection` **only logs on change** — a status stuck at UNKNOWN logs nothing at all, which reads + identically to "nothing is happening". +- Watchdog timings (`constants.py`): 3s cold-start grace, 4s check interval, 2 strikes to re-arm, max 5 + cold-start attempts. So a rescue should appear within ~7s; if it hasn't, the watchdog is blind, not slow. + +## Wrong turns — don't repeat these + +- **"Stale TUI regexes in `providers/claude_code.py`"** — wrong. The provider explicitly handles v2.1.212 + chrome: `NEW_TUI_BOX_PATTERN`, the `●` response glyph, the `· ✢ * ✶ ✻ ✽` spinner cycle, the + `● high · /effort` footer. Both PyPI 2.3.0 and `main` have it. Bugs 2 and 3 both fail *before* or + *independently of* TUI parsing. +- **"The pane is empty / the shell never starts"** — wrong. The shell renders `cao@cao:~$` fine. +- **"The reader thread is blocked in `os.read`"** — wrong. It spins; `O_NONBLOCK` is set. + +## Upstreaming + +Neither fix is upstream. Both are genuine bugs in `awslabs/cli-agent-orchestrator` that will affect anyone +whose pipe cold-starts or who runs a full-screen TUI provider. A merged PR is the only thing that ends the +patch-carrying — until then, every reinstall risks reverting them.