Skip to content

fix(coding-agent): snapshot transfer ids from the materialized cursor; mismatches settle the transfer, not the worker channel - #2044

Closed
snimu wants to merge 4 commits into
mainfrom
sebastian/worker-snapshot-cursor
Closed

fix(coding-agent): snapshot transfer ids from the materialized cursor; mismatches settle the transfer, not the worker channel#2044
snimu wants to merge 4 commits into
mainfrom
sebastian/worker-snapshot-cursor

Conversation

@snimu

@snimu snimu commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Part of the worker-state single-truth program (Linear RES-1270); squashes discussion #1662 (both halves).

Purpose

Two halves of one duplicate-truth defect in chunked snapshot transfers:

  1. Worker side, the transfer id ${activeSessionId}-${eventGeneration}-${lastEventSequence} was computed from the live session cursor at a different time than the message-array capture it labels (after createAttachResult's awaits on attach/catchup, before them on replacement). Events arriving during materialization let two different byte streams legitimately share one snapshot id.
  2. Supervisor side, the resulting duplicate-transfer disagreement (reentrant begin, mismatched duplicate envelope / chunk bytes / end metadata) was treated as protocol corruption and handled with closeWorkerChannel=true: handleWorkerClose + client.close() on a healthy worker, failing every in-flight request for every session on that worker. Since fix(coding-agent): harden daemon startup and recovery ownership #1929 this is a control-plane bounce rather than a terminal brick, but it can recur every time the race re-fires on a big busy transcript.

Change

  • daemon-mode.ts: new snapshotTransferId(snapshot) derives the id from snapshot.lastEventCursor — the cursor captured synchronously with the message array in createSessionSnapshot — and all three call sites use it. For the replacement path the id computation moves after materialization (parameter deleted).
  • daemon-supervisor.ts: new failSnapshotTransfer settles a transfer anomaly with the transfer as the blast radius: fail that one generation (existing failWorkerSnapshotCache with closeWorkerChannel=false) and queue attached clients for a resync from a fresh snapshot. The four mismatch sites use it, and the existing session_snapshot_failed handler plus the end-frame catchup loop are consolidated through the same helpers (they previously duplicated the loop inline). Malformed-frame handling (undecodable payloads) still closes the channel.

Net src LOC: +33/-27 in daemon-supervisor (mostly the extracted helpers replacing two inline copies), +14/-5 in daemon-mode. No wire-shape change: snapshot ids are opaque strings and the mismatch handling is supervisor-internal, so this is backward-compatible with old workers and clients.

Tests

In 4602-snapshot-transfer-idempotency.test.ts (the existing file for exactly this machinery):

  • New pin: the attach response's snapshotStream.id names the materialized snapshot cursor even when the live session cursor has advanced (fail-unfixed verified: reverting the worker-side hunk yields the live-cursor id).
  • Updated the three pins that asserted the old channel-close behavior: reentrant begin, mismatched duplicate envelope, mismatched chunk bytes, and mismatched end metadata now assert the worker client stays connected, lifecycle stays ready, no recovery run, caches dropped — and for a published replacement-purpose mismatch, that a fresh attach reload is requested from the worker (the queued resync).

Ran locally: 4602, 4601, 4677, snapshot-transcript-cache, agent-connection-daemon, daemon-supervisor-monitor — 224/224 pass; npm run check green.


Note

Medium Risk
Changes daemon worker/supervisor snapshot protocol behavior; incorrect resync paths could leave clients stale, but blast radius is intentionally narrowed from whole-worker bounce to per-transfer recovery.

Overview
Chunked session snapshot transfers now use a transfer id tied to the materialized snapshot cursor (snapshot.lastEventCursor) instead of the live session generation/sequence, so late-arriving events cannot label two different byte streams with the same id. Replacement snapshots compute that id only after createAttachResult materializes the snapshot.

On the supervisor, duplicate or invalid snapshot transfers (restarted begin, envelope mismatch, bad chunks/end, worker-reported failure) are handled via failSnapshotTransfer / queueSnapshotResync: drop the affected cache generation, keep the worker channel open, and queue attached clients for a fresh attach/catch-up when the failed transfer was published. Malformed frame handling that still closes the channel is unchanged.

Tests in ENG-4602 pin the materialized-cursor id and expect resync instead of worker channel close for those anomaly paths.

Reviewed by Cursor Bugbot for commit 5af3bbe. Bugbot is set up for automated code reviews on this repo. Configure here.

LOC

Total src: +61/−39 (net +22); tests: +61/−26 (net +35).

Note

Fix snapshot transfer IDs to use materialized cursor and scope transfer failures away from worker channel

  • Snapshot transfer IDs for attach, replacement, and catch-up now derive from the snapshot's materialized event cursor via the new snapshotTransferId helper in daemon-mode.ts, instead of the live session's current generation and sequence.
  • The DaemonSupervisor.failSnapshotTransfer helper in daemon-supervisor.ts contains anomalies to the affected transfer: it discards the matching snapshot cache and queues a fresh catch-up or replacement for attached clients.
  • Restart, mismatch, and validation failures during chunked snapshot transfers no longer close or recover the worker channel; the worker stays ready while clients retry.
  • Risk: handleWorkerFrame now routes restart, completed-transfer mismatch, chunk-byte validation, end-frame validation, and worker-reported snapshot failures through failSnapshotTransfer instead of worker-level recovery; any snapshot protocol path not covered by this routing will still trigger the old worker-close path.

Macroscope summarized 5af3bbe.

…nd settle transfer mismatches transfer-scoped

Worker side, the chunked-snapshot transfer id was computed from the live
session cursor at a different time than the message-array capture it
labels, so two transfers could carry the same id for different bytes
whenever events arrived during snapshot materialization. Derive the id
from the snapshot's own lastEventCursor at all three call sites (attach,
replacement, catchup) so the id always names the captured cut.

Supervisor side, a duplicate-transfer disagreement (reentrant begin,
mismatched duplicate envelope, mismatched duplicate chunk bytes, or
mismatched duplicate end metadata) was punished with
closeWorkerChannel=true, bouncing a healthy worker and every session on
it. Settle these anomalies with the transfer as the blast radius
instead: fail the one transfer and queue attached clients for a resync
from a fresh snapshot, mirroring the existing session_snapshot_failed
handling, which now shares the same helper. Malformed-frame handling
(undecodable payloads) still closes the channel.
…drop, not the failing frame's purpose

A failed published transfer can be serving any attached client's
catch-up wait: drainClientCatchups clears its queue entry before
draining, so when duplicate validation rejects the waiter it was logged
and dropped, and an attach-purpose failure never requeued it — the
client stayed attached with a stale transcript while the worker stayed
ready. Queue the resync whenever the published cache is dropped; the
frame purpose only picks the replacement/resync flavor.
…omments

One workerAttachHarness builder serves both worker-side attach pins, and
multi-line comment blocks collapse toward one-line invariant guards. No
behavior or coverage change.
sethkarten added a commit that referenced this pull request Sep 7, 2026
…; mismatches settle the transfer, not the worker channel

Incorporates #2044 at 5af3bbe.
@sethkarten

Copy link
Copy Markdown
Contributor

Included in #2028: #2028

@sethkarten sethkarten closed this Sep 7, 2026
sethkarten added a commit that referenced this pull request Sep 7, 2026
)

* refactor(coding-agent): move the semantic-edge ledger onto the event-log substrate

The recorder's private append/replay/repair IO is deleted; EventLog owns it, the same move #1987 made for the RLM spawn ledger. One durability rule is unified in the substrate rather than dropped: an unterminated final line is an uncommitted append, skipped on read and truncated before the next append — never newline-completed and never surfaced to a consumer whose next append destroys it.

* fix(coding-agent): make the explicit ledger reader's ENOENT contract atomic

readSemanticEdgeLedger probed with statSync before reading through EventLog, which swallows ENOENT; a ledger deleted between the two returned [] instead of throwing. The missing-file decision now lives at the single open (replaySync missingFileThrows), so no check-then-read window exists.

* docs(coding-agent): state the event-log tail rule once

The unterminated-tail contract was restated four times (module doc, replaySync doc, two test comments). It now lives once in the module doc; the method doc keeps only its own parse/missing-file semantics and the test comments reference the contract.

* fix(coding-agent): write event-log appends fully and gate appends on tail repair

writeSync may write short (ENOSPC after a prefix); appendSync now loops until the payload is fully on disk so write-before-action callers never act on a torn record reported as success. A tail-repair failure (e.g. append-only ACL permitting O_APPEND but not r+) now propagates instead of being swallowed: writing through an unrepaired torn tail would weld it to the new record as permanent interior corruption. ENOENT and the concurrent-writer instability path keep their existing semantics.

* fix(coding-agent): reclaim short event-log writes instead of completing them

The rlm spawn ledger is multi-writer by documented design (supervisor plus each worker over one file), so completing a short O_APPEND write with a second write could interleave with a rival append and weld two records. A short write now truncates its own torn prefix back off (only while this writer still owns the tail) and fails the append; a torn tail is read-tolerated, a weld is permanent corruption. The append fd opens a+ so the ownership check can read the tail.

* fix(coding-agent): leave the torn tail on a short write instead of reclaiming it

The tail-match reclaim could truncate a rival's committed record whose final bytes coincide with our torn prefix - committed-data loss, strictly worse than the torn tail it prevented. A short write now just fails the append: the torn tail is the one tolerated shape, skipped on read and truncated by any writer's next repair (verified for both topologies: a resumed single-writer recorder repairs on its first append; every rlm-ledger writer repairs before each append).

* refactor(coding-agent): compress event-log comments

* fix(ai): omit the default service tier, reprice cache writes from message_delta, repoint the zai default

Incorporates #2032 at f82c7fa.

* fix(tui,coding-agent): survive lone surrogates in table cells and terminate the WebP EXIF scan

Incorporates #2033 at a3d1139.

* fix(coding-agent): restart dead kernels on ensure() and read mcp>=2 tool schemas

Incorporates #2034 at 749e216.

* fix: one crash-safe owner for durable state writes

Incorporates #2035 at f0f02d2.

* fix(coding-agent): one zombie-aware process-liveness probe

Incorporates #2041 at 92a0eac.

* fix(coding-agent): snapshot transfer ids from the materialized cursor; mismatches settle the transfer, not the worker channel

Incorporates #2044 at 5af3bbe.

* fix(coding-agent): failed workers recover on touch; roster gaps answer a structured recovering error

Incorporates #2047 at 77b747a.

* fix(coding-agent): seven session and IO correctness defects

Incorporates #2037 at 41b5d72.

* fix(coding-agent): coalesce child-usage attribution and gate agent-status persistence on real changes

Incorporates #2050 at 6b0af5d.

* fix(coding-agent): incremental single-flight session metadata scans

Incorporates #2043 at df032c1.

* fix(coding-agent): memoize the passive RLM topology derivation

Incorporates #2051 at 0ee114c.

* fix(coding-agent): preserve accounting and metadata across deferred updates

Keep durable child-usage aggregates separate from pending sibling usage. Retry optional topology metadata after transient reads. Completes #2050 and #2051 integration.

* fix: preserve session accounting and read-only persistence boundaries

---------

Co-authored-by: Seth <seth@primeintellect.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants