fix(launch): stop dropping the initial message when provider init outlives the client

[tedswinyar]

Problem

cao launch --agents <profile> --async "<message>" created the session and terminal but silently never delivered the message. The CLI reported Failed to connect to cao-server: ... Read timed out. (read timeout=30) while a healthy idle TUI sat there with last_active frozen at creation time. Reproduced on codex workers in a concurrent fan-out; claude_code/kiro_cli workers in the same batch were unaffected.

Root cause

launch delivers the message from the client, as a second HTTP request that only runs if the first (POST /sessions) returns. But POST /sessions runs the provider's full initialize() synchronously server-side (session_service.create_session → create_terminal → await provider.initialize()), bounded by provider_init_timeout applied twice (wait_for_shell + wait_until_status) plus the startup-prompt handler — up to ~85s. The client only allowed mcp_request_timeout (30s).

When the client's budget expired, requests raised ReadTimeout, launch caught it as a generic RequestException → "Failed to connect to cao-server", and the create-then-send branch never ran. Server-side init completed normally, so nothing retried — from the server's point of view the launch had succeeded. The manual POST /terminals/<id>/inbox/messages workaround worked because it was the missing second request.

Why codex specifically, with claude_code/kiro_cli fine in the same fan-out: codex had the slowest initialize() and was the last provider still running its init's blocking subprocess calls directly on cao-server's single shared event loop, so concurrent codex inits inflated each other past the budget.

Fix

1. _create_session_timeout() — give POST /sessions a read budget covering the init it waits on, not the generic tool budget.
2. _effective_init_timeout() = max(global, profile_override) — providers disagree on where the init cap comes from: claude_code/antigravity_cli/kimi_cli honour a per-profile provider_init_timeout override via BaseProvider.get_init_timeout(profile), while codex/copilot_cli read the global directly. max is the only bound covering whichever the target provider uses, and ensures a lowered override can't shrink the budget below what a global-reading provider will actually spend.
3. _CREATE_OVERHEAD_MARGIN = 30 — covers server-side time neither init window bounds: pane/window creation runs before initialize(), plus codex's asyncio.sleep(2.0) warm-up, send_keys latency, and poll-interval tails.
4. Offload codex initialize()/_handle_trust_prompt backend calls to asyncio.to_thread — the #451 parity fix codex never received. It was the last provider blocking the shared event loop, which is why a concurrent fan-out blew the budget.
5. settings.get("startup_prompt_handler_timeout", 20) — a hand-edited partial settings.json omitting the key must not turn the widened budget into a KeyError that aborts launch.

Tests

12 new/updated in test_launch.py + test_codex_provider_unit.py:

• create budget covers provider init; never below the readiness-wait floor; scales with configured init timeout
• per-profile override honoured, incl. a lowered-override case pinning that it can't shrink the budget
• worst-case server path encoded as an explicit sum with a drift guard
• unloadable/partial-settings survival (no crash)
• event-loop non-starvation, measured via the worst gap between ticks of a concurrent heartbeat (~0.21s loop-side vs ~0.01s offloaded)
• trust-prompt timeout reads the configured setting

All fail before this change, pass after. test_launch.py + test_codex_provider_unit.py: 226 passed, 3 skipped.

Known limitation (deferred)

If the client's settings/env differ from the server's (e.g. different CAO_PROVIDER_INIT_TIMEOUT, or different profile files on disk in a cross-host split), the client can still under-budget. max() mitigates the common cases; the robust fix is server-authoritative — have the server telegraph its own budget (return it from POST /sessions, or expose it on /health). That is an API change, left for a follow-up.

Review provenance

Diagnosed, then adversarially reviewed by three independent reviewers across two model families (Claude ×2 + GPT-5.6 Sol) and two harnesses. The first revision was a partial fix (missed per-profile overrides and fixed overhead); review caught it and it was hardened. Every reviewer claim was reproduced against real code before acting on it.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@2a6f20c). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #566   +/-   ##
=======================================
  Coverage        ?   90.98%           
=======================================
  Files           ?      179           
  Lines           ?    23281           
  Branches        ?        0           
=======================================
  Hits            ?    21182           
  Misses          ?     2099           
  Partials        ?        0           

Flag	Coverage Δ
unittests	90.98% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Adjusts cao launch and Codex provider initialization to avoid client-side timeouts that prevent the initial message from being delivered when provider initialization is slow, and reduces event-loop blocking during Codex startup.

Changes:

• Add a dedicated POST /sessions timeout computation that covers server-side provider initialization (including profile overrides and fixed overhead), with a readiness-wait floor.
• Offload Codex backend subprocess-backed calls in initialize() / _handle_trust_prompt() via asyncio.to_thread to avoid starving the server event loop.
• Add/extend unit tests covering session-create budgeting, profile override behavior, partial settings resilience, and Codex init non-starvation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/cli_agent_orchestrator/cli/commands/launch.py	Adds _create_session_timeout / _effective_init_timeout and applies widened timeout only to POST /sessions, plus centralizes readiness timeout constant.
src/cli_agent_orchestrator/providers/codex.py	Offloads blocking backend calls during trust-prompt handling and initialization; reads trust-prompt timeout from server settings.
test/cli/commands/test_launch.py	Adds tests asserting correct create-session budgeting, readiness floor, scaling, and profile override handling.
test/providers/test_codex_provider_unit.py	Adds heartbeat-based tests to assert Codex init/trust handling does not starve the event loop; tests configured trust-prompt timeout wiring.

Suppressed comments (1)

src/cli_agent_orchestrator/providers/codex.py:635

• The TimeoutError message is now misleading: the init timeout is configurable via server settings, but this hard-codes "60 seconds". If provider_init_timeout is raised, the error should report the actual configured value to aid debugging.

        if not await wait_until_status(
            self.terminal_id,
            {TerminalStatus.IDLE, TerminalStatus.COMPLETED},
            timeout=float(get_server_settings()["provider_init_timeout"]),
            polling_interval=1.0,
        ):
            raise TimeoutError("Codex initialization timed out after 60 seconds")

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[haofeif]

The Codex event-loop fix looks sound, but the new create timeout still expires before several supported providers can finish a successful initialization, preserving the initial-message loss this PR is intended to fix.

[haofeif]

[P1] Budget the providers' actual initialization paths. startup_prompt_handler_timeout is only the idle gap after a prompt; the Claude handler's hard outer cap is another full provider_init_timeout. A successful Claude path can therefore take shell(T) + handler(T) + ready(T) + the 5s input settle: 185s with defaults, while this returns 170s (and about 545s versus 410s for a profile with T=180). This is not Claude-specific: Kimi can take T + 2max(120,T) = 300s by default, Kiro's legacy fallback can take 4T = 240s, and Antigravity/OpenCode/Hermes also have successful paths beyond 170s. In each case requests can still time out before /sessions responds, so the second /input request is skipped and the initial message is silently lost—the exact failure this PR addresses. Please derive a bound from the real provider path, or avoid the race by putting the message in the existing CreateSessionBody.initial_message field so /sessions uses deferred initialization and delivery.

[call-me-ram]

The bug is real and the root-cause trace is correct — nice find, and the diagnosis is unusually well-evidenced. I confirmed every link in the chain against head: launch sends message only as a second request (launch.py:433-447), POST /sessions → session_service.create_session → create_terminal runs await provider_instance.initialize() inline (terminal_service.py:445) because defer_init is False when no initial_message is present, and requests.exceptions.ReadTimeout is swallowed by the blanket except requests.exceptions.RequestException at launch.py:459 into "Failed to connect to cao-server" — so the create-then-send branch never runs while the server happily finishes init and keeps the session. The codex event-loop offload (item 4) is also correct, is a genuine parity gap versus #451, and its three tests are real regression guards (I reverted codex.py to upstream/main in a worktree and all three fail).

Where I land differently is the fix. The timeout widening does not close the hole it claims to close — it closes it for codex and leaves it open for most other providers on default settings — and POST /sessions already has a purpose-built mechanism for this exact problem that cao launch is the last client not using. Details below.

Must-fix

1. launch.py:106-140 — the new budget is below the worst-case successful init for 7 of 9 providers on default settings, so the silent drop survives for everything except the provider that was reported.

I derived each provider's worst-case successful initialize() wall clock directly from head by stubbing every bounded wait to report the timeout it was handed. With defaults (provider_init_timeout=60, startup_prompt_handler_timeout=20), _create_session_timeout returns 170s, against:

provider	worst-case successful init	vs 170s
antigravity_cli	10 + max(180,T) + max(180,T) = 370s	+200
kimi_cli	T + max(120,T) + max(120,T) = 300s	+130
kiro_cli (non-yolo TUI→--legacy-ui fallback)	4T = 240s	+70
copilot_cli (shell-ready fallback)	T + T + 10 + 60 = 190s	+20
claude_code	T + T + T + 5 = 185s	+15
opencode_cli	T + 120 = 180s	+10
hermes	T + 120 = 180s	+10
codex	T + 20 + T = 140s	OK
cursor_cli	(under)	OK

The formula 2*T + startup_prompt_handler_timeout + 30 encodes codex's shape specifically, and two of its three assumptions are false elsewhere:

• startup_prompt_handler_timeout is only the idle gap, not the handler's cap. claude_code._handle_startup_prompts and kimi_cli/antigravity_cli._handle_startup_dialog take outer_timeout — a full provider_init_timeout (or max(120/180, T)) — as the hard cap (claude_code.py:565-575, kimi_cli.py:523-533, antigravity_cli.py:563-576). So the middle term is T-or-larger, not 20.
• "the init timeout applies TWICE per init" isn't a general invariant: kimi_cli floors its two readiness waits at max(120, T) (kimi_cli.py:607), antigravity_cli at max(180, T) (antigravity_cli.py:660-668), and kiro_cli has a four-window path when the TUI attempt times out and it retries with --legacy-ui (kiro_cli.py:285, 349, 363, 381).

With a profile override the gap widens rather than closes: for provider_init_timeout: 180, claude_code can legitimately spend 3*180+5 = 545s against a 410s budget. _CREATE_OVERHEAD_MARGIN doesn't help — the shortfall is structural, not marginal.

I confirmed the residual failure is byte-identical to the reported one at head:

Error: Failed to connect to cao-server: HTTPConnectionPool(host='127.0.0.1', port=8001): Read timed out. (read timeout=170)
posts made: ['http://127.0.0.1:9889/sessions']

One POST, message never sent, exit 1 — the original bug, just with a bigger number in it.

This is haofeif's P1 and I confirm it independently, with the same numbers for claude_code (185) and kimi_cli (300); antigravity_cli is worse than their "also beyond 170s" — it's 370s.

2. launch.py:390-394 — wrong layer: POST /sessions already accepts initial_message and defers init + delivery server-side, and cao launch is the only client that doesn't use it.

CreateSessionBody.initial_message exists (api/main.py:215, 219), and session_service.create_session sets defer_init=initial_message is not None (session_service.py:88), so the response returns as soon as the session and terminal record exist and _schedule_deferred_init (terminal_service.py:768-896) owns init and delivery. Both sibling clients already do this: mcp_server/server.py:321-335 and ops_mcp_server/server.py:123-125 pass {"initial_message": ...} — and mcp_server keeps timeout=_mcp_timeout() (30s) on that call, which is direct evidence that 30s is the right budget once the body field is used.

The layer matters beyond aesthetics, because client-side create-then-send is unrecoverable by construction: on ReadTimeout the client has no terminal_id — the identity of the thing it would need to retry against only arrives in the response that timed out. So the create call can never be made safe by widening it; it can only be made less likely to fail, and its failure is always a silent orphan. The deferred path is also strictly more robust than what the client can do: it confirms the worker actually started and re-submits (_confirm_worker_started_or_resubmit), and on failure notifies the caller's inbox and tears the worker down instead of leaving a healthy idle TUI with a frozen last_active.

Concretely: send {"initial_message": message} in the create body when message is set, keep mcp_request_timeout on the create call, and drop the client-side /input POST. Two things to handle: the deferred path returns TerminalStatus.UNKNOWN (terminal_service.py:460), so the --async branch should just report and return rather than waiting for IDLE; and the non---async branch should go straight to poll_until_done. Note also that message is silently ignored today on the attach (non---headless) path — worth deciding deliberately rather than inheriting.

No double-delivery risk in either direction here, since the client send would be removed in the same change — but please don't ship both paths concurrently, because _schedule_deferred_init + a client /input would deliver twice.

3. launch.py:393 — the widened value is a scalar, so it also raises the connect timeout from 30s to 170s (unbounded with a profile override), and a genuine connection failure is still reported with the same string as a slow init.

requests applies a scalar timeout to both connect and read. So the change trades a slow-init failure for a new one: with cao-server unreachable rather than refused (filtered port, wrong host, stale SERVER_HOST), cao launch now blocks for 170s with zero output after the confirmation prompt, where it used to fail in 30s. And AgentProfile.provider_init_timeout is an unbounded Optional[int] (models/agent_profile.py:66) — I measured provider_init_timeout: 3600 producing a 7250s (121 min) budget.

Minimal fix, whichever layer you land on: pass a tuple, timeout=(connect, read) with a small fixed connect (5-10s) and the computed read budget; cap the derived read budget; and split the handler so ReadTimeout is distinguishable from ConnectionError — right now launch.py:459 flattens "server is initialising, your message was NOT delivered, a session may now exist" into "Failed to connect to cao-server", which is the misleading message the PR body itself calls out as part of the bug.

Non-blocking

1. codex.py:458-465 and launch.py:16-27 — "codex was the last provider still running its init's blocking subprocess calls directly on cao-server's single shared event loop" is not true, and the claim is now baked into source comments. After this PR, kiro_cli.initialize() still makes four loop-side backend calls (kiro_cli.py:289, 344, 361, 380), plus opencode_cli.py:160 and cursor_cli.py:589. kiro_cli is DEFAULT_PROVIDER. The offload is still right; please just reword the causal claim, since a future reader will otherwise trust "codex was last" and skip these.

2. Item 5 in the PR body (settings.get("startup_prompt_handler_timeout", 20)) guards a state that cannot occur, and the guard is self-defeating anyway. get_server_settings() starts from dict(_SERVER_DEFAULTS) and merges only keys already in it (settings_service.py:249-250), so a hand-edited partial settings.json can never yield a missing key. And within the same expression _effective_init_timeout hard-indexes settings["provider_init_timeout"] — I passed {"startup_prompt_handler_timeout": 20} and got KeyError('provider_init_timeout'), so the guard protects one of two keys against an impossible input. test_create_session_timeout_survives_partial_settings asserts against a synthetic dict get_server_settings() cannot return. Either drop both, or make it consistent. Related: codex.py:626 hard-indexes the same key that launch.py:138 guards — pick one convention.

3. The codex non-starvation tests belong in test/providers/test_startup_handler_nonblocking.py. That module already exists for exactly this property, is parametrized over _COROUTINE_TARGETS / _HEARTBEAT_CASES, and covers kimi_cli/antigravity_cli/copilot_cli from #494. Adding codex there is ~2 list entries instead of a bespoke 50-line _heartbeat_gap helper in the codex file, and it gives the still-loop-side providers from item 1 an obvious home. That module also does cancel() → await → suppress CancelledError (test_startup_handler_nonblocking.py:87-91), which is Copilot's inline point on test_codex_provider_unit.py:2410 — that one is correct, beat.cancel() without an await can emit "Task was destroyed but it is pending". Moving the tests fixes it for free.

4. codex.py:635 still raises TimeoutError("Codex initialization timed out after 60 seconds") while the timeout is configurable. Copilot suppressed this one; it's worth taking, since this PR is what makes the neighbouring timeout configurable — interpolate the value like claude_code.py:701 does.

5. _READINESS_WAIT_TIMEOUT as a floor penalises deliberately-low configs. An operator or CI run that sets provider_init_timeout: 5 for fast failure now still waits 120s on create. Also, the floor's stated rationale ("the create call gives up on work the very next step is still willing to wait for") doesn't hold on the synchronous path — the readiness wait runs after create returns and init has already completed, so the two waits are sequential, not competing.

6. _effective_init_timeout re-loads the profile that launch already loaded at launch.py:288 on the non---yolo path. Pass it through. Also note the two load sites disagree on what they catch — (FileNotFoundError, RuntimeError) vs bare Exception.

Asks

• mcp_server/app_tools.py:404 calls _post_json("/sessions", params) with no initial_message and timeout=MCP_REQUEST_TIMEOUT, so it has the same synchronous-init-outlives-the-client hole. Not this PR's job to fix, but the body's "This closes the hole for every provider" should be scoped to "for cao launch" — and if the answer to must-fix 2 is initial_message, this becomes a natural follow-up.
• Whatever the create budget ends up being, cao launch prints nothing between "Proceed?" and the result. A 170s+ silent wait needs some progress feedback.
• The "Known limitation (deferred)" section is honest and I agree with the diagnosis there — but adopting initial_message dissolves it rather than deferring it, since the client stops needing to guess the server's init budget at all.

On haofeif's review: their single P1 is correct and I reproduce it with the same numbers. I'd add that it's not only an arithmetic shortfall — their suggested alternative (CreateSessionBody.initial_message) is the one that removes the race rather than resizing it, and it's already the pattern used by both other POST /sessions clients.

What I verified

• git fetch upstream pull/566/head → d041a7d; all reading and testing at that rev in an isolated worktree.
• Diagnosis trace: launch.py:433-447 (second request), launch.py:459 (blanket RequestException), session_service.py:88 (defer_init=initial_message is not None), terminal_service.py:435-445 (inline initialize()), api/main.py:215-232, 1919-1976 (CreateSessionBody.initial_message), terminal_service.py:768-896 (_schedule_deferred_init + resubmit + failure notification). Confirmed.
• Per-provider budget derivation: script stubbing wait_for_shell / wait_until_status / each startup handler / wait_until_input_ready to report the timeout each was handed, run against real initialize() for claude_code, kimi_cli, codex, opencode_cli (antigravity_cli, hermes, kiro_cli, copilot_cli read statically where the probe aborted on a missing binary). Output: client budget 170s; claude_code 185, kimi_cli 300, opencode_cli 180, codex 140. Table above.
• Residual failure repro: requests.post stubbed to raise ReadTimeout at head → exit 1, Failed to connect to cao-server: ... (read timeout=170), exactly one POST, message never sent, no terminal id in output.
• Scalar/unbounded timeout: _create_session_timeout returns int (scalar → governs connect too); with profile.provider_init_timeout = 3600 it returns 7250.
• Self-defeating settings guard: _create_session_timeout({"startup_prompt_handler_timeout": 20}, "any") → KeyError('provider_init_timeout').
• Loop-side survey: grep for un-offloaded get_backend(). across all 9 providers → kiro_cli ×4, opencode_cli ×1, cursor_cli ×1 remain; copilot_cli's _send_enter/_send_key are correctly offloaded inside _accept_trust_prompts.
• Tests: test/cli/commands/test_launch.py → 58 passed. test/providers/test_codex_provider_unit.py + test/services/test_session_service.py → 188 passed, 3 skipped. Reverting providers/codex.py to upstream/main and re-running TestCodexInitEventLoopBlocking → 3 failed (genuine guards).