🐛 Bug Report: `std::terminate() called with no exception` when a tail consumer's target process dies abruptly (Windows)

[petebacondarwin]

Summary

On Windows, a workerd process aborts with

*** std::terminate() called with no exception

when a worker in that process declares a tailConsumer whose target lives in a second workerd process (reached via an external service over a loopback socket), and that second process dies abruptly (SIGKILL / taskkill /T /F).

There is no exception, no JS stack, and the abort happens on a non-main thread. The process dies immediately rather than surfacing a delivery error for the tail event.

Version / platform

• workerd 1.20260731.1
• compatibility_date 2025-05-01
• Windows only. Not reproducible on macOS or Linux across many hundreds of cycles.
• Reached via Miniflare (workers-sdk), which is what wires the cross-process tail consumer up; see "How it's configured" below.

Signature

*** std::terminate() called with no exception
stack: 7ffd4a141249 7ff692d863a5 7ffd4a15baff 7ffd4a143a9c 7ffd4a15ae8c 7ffd4a142ce1 7ffd4a15b8f7
7ffd583a700e 7ffd5824e592 7ffd4a15b3d5 7ffd4a141c1f 7ffd4a142dd7 7ffd4a15b8f7 7ffd583a6f8e
7ffd58252326 7ffd5824a960 7ffd555a75f9 7ffd4a1455a8 7ff69441045f 7ff692d85f63 7ff694415512
7ff69441c416 7ff694415a6b 7ff69441c416 7ff694412fbb 7ff692d71e77 7ff69624219e 7ffd56a1e8d6
7ffd582cc53b

The stack is byte-identical across every occurrence once ASLR base addresses are normalised, so this is a single deterministic path. We could not symbolize it — $LLVM_SYMBOLIZER is unset on the CI runners and the release binary is stripped — which is the main reason we're filing rather than proposing a fix.

How it's configured

Two independent workerd processes, each with its own config, cross-referencing each other:

• Process 1 hosts worker A, which has assets configured (so its incoming requests are fronted by an asset/router worker pair and its user worker is reached over RPC).
• Process 2 hosts worker B, which declares tailConsumers: [A].

B's tail consumer is not a local service — it's an external service that proxies over a loopback socket into process 1.

When process 1 dies abruptly, process 2 aborts with the signature above.

Notably the abort also happens when both sessions are completely idle — no request ever traverses the binding. So this is not triggered by an in-flight tail event from live traffic; merely having the consumer configured and the peer die is enough.

Reproduction rate and discriminators

Measured on Windows CI, isolating one variable at a time (each "arm" is repeated kill/restart cycles against an otherwise identical setup):

Arm	Crash rate
B tails A, where A has assets	4/4 (and 12/13 in an earlier round)
B has no tail consumer	0/4 (16 cycles)
B tails a peer without assets	0/4 (18 cycles)
B tails A (assets), service binding between them removed	4/4

So:

• The tail consumer edge is necessary.
• The peer having assets configured is necessary.
• A service binding between the two is irrelevant — removing it does not help, and the crash occurs with no traffic at all.
• Killing an assets-enabled worker that is not anyone's tail consumer: 0/6 (28 cycles). Killing a non-assets worker that is bound as a service: 0/6 (23 cycles).

What we ruled out

• Traffic over the binding — idle sessions still crash.
• Process restarts by themselves — 0/3 with 7–10 restarts and no peer present.
• Registry/config churn with no live peer — 0/3.
• Peer count, port contention, start order — 0/41 combined.
• CPU starvation — 0/3 with a live peer under load.
• Reporting tail-forward failures at the JS layer (workers-sdk#14993) — 8/8 still crashed, i.e. the abort is below the JS-visible layer.
• Bypassing the extra RPC hop that assets introduces for tail delivery — 4/4 and 3/4 still crashed. This is worth emphasising because it's the obvious explanation for why assets is a discriminator, and it appears to be wrong. We don't have a mechanism that explains the assets dependency.

Impact

This is the direct cause of a persistent CI flake in workers-sdk (multi-session local dev tests), where abruptly terminating one vite dev / wrangler dev session kills a sibling session's runtime. For users running several local dev sessions with tail consumers pointed at each other, stopping one session can take down another.

The workers-sdk side already surfaces the crash report and restarts the runtime (workers-sdk#14989), so it's recoverable — but the abort itself looks like it should be a delivery error rather than a process kill.

What would help

Even without a fix: if a maintainer recognises this stack shape, or can point at where tail-event delivery to an unreachable external service could reach std::terminate on a non-main thread, we can narrow it from our side. We're happy to run instrumented builds on Windows CI — that's where we have a reliable ~100% repro.

[petebacondarwin]

Dug into this. Two of the premises in the report are wrong in ways that matter, and the chain now
closes end to end. Summary first, evidence below.

The abort is on the main thread, not a worker thread. The stack trace is structurally incapable of
identifying the cause — only the discarded kj::Exception message can. The most likely origin is
kj::FdConnectionReceiver::accept() on Windows, which has three unguarded throw sites that the Unix
implementation explicitly defends against.

---

Correction 1: it's the main thread

Every Windows thread bottoms out at BaseThreadInitThunk / RtlUserThreadStart, including the main
one — the main thread runs RtlUserThreadStart → BaseThreadInitThunk → CRT startup → main.
Normalising the reported stack against module bases (the two captured runs differ by a constant EXE
slide of 0x4D30000, which confirms the module grouping):

#	Module	Reading
28	ntdll	RtlUserThreadStart
27	kernel32	BaseThreadInitThunk
26	workerd	CRT startup (__scrt_common_main_seh)
25	workerd	main()
24-20	workerd	kj::runMainAndExit + kj::MainBuilder dispatch (…c416 twice = subcommand recursion, not re-entrancy)
19	workerd	CliMain::serve() (with serveImpl inlined)
18	kj	re-throw site
17	vcruntime	_CxxThrowException
16	KERNELBASE	RaiseException
15-2	ntdll / vcruntime	SEH dispatch, __CxxFrameHandler4, FindHandler
1	workerd	EH funclet, +0x442 from frame 19 — same function
0	vcruntime	terminate

Correction 2: the stack cannot identify the bug

There is exactly one frame between serve() (19) and _CxxThrowException (17). If the exception
had originated in the event loop there would be a dozen kj promise-continuation frames there.
There aren't.

Frame 18 is Promise::wait() re-throwing an exception that was captured asynchronously.
kj::_::waitImpl (kj/async.c++) runs the loop in a nested scope and returns before the throw, so
the originating stack is already gone. The origin file:line only exists in the kj::Exception
payload — which is precisely what's being discarded.

So: no amount of symbolication of this trace will help. That's why it's been a dead end.

---

The chain

peer taskkill /T /F  →  RST on a connection pending in our accept queue
  → kj FdConnectionReceiver::accept() throws        kj/async-io-win32.c++:861 / :864 / :873
  → accept loop has no error tolerance              server.c++:6086 / :6294 / :6358
  → Server::taskFailed → fatalFulfiller->reject()   server.c++:5780   (unlogged)
  → Server::run() promise rejects                   server.c++:6538
  → promise.wait(io.waitScope) re-throws            workerd.c++:1448  ← frame 18
  → escapes `serveImpl(...) noexcept`               workerd.c++:1406  ← frame 1 funclet
  → std::terminate(); MSVC reports no current exception

The diagnostic black hole

kj::runMainAndExit already wraps the main function and would print the full exception
(kj/main.c++:219 → *** Uncaught exception ***\n<kj::Exception>) and exit non-zero. Both
CliMain::serveImpl (workerd.c++:1406) and CliMain::serve (workerd.c++:1465) are declared
noexcept, which pre-empts that and turns it into std::terminate() instead.

git log -L suggests this is accidental: noexcept was paired with [[noreturn]] in e14287e,
both were dropped in a2d5d18 for KJ_CLEAN_SHUTDOWN, and ebb5b48 restored noexcept alone.

Server::taskFailed also doesn't log:

// server.c++:5780
void Server::taskFailed(kj::Exception&& exception) {
  fatalFulfiller->reject(kj::mv(exception));
}

This has been fatal-for-any-exception since the initial import (a265d5c), when the only tasks were
listen loops that genuinely shouldn't fail.

Why it must be an accept loop

Server::tasks has exactly five add-sites: server.c++:6136, :6301, :6497, :6860, :6883.
None is outbound, so an outbound failure cannot be fatal. :6136 (per-HTTP-connection) is explicitly
try/caught with the comment "do not consider exceptions from a specific connection to be fatal".
:6497 only runs during drain. That leaves the accept loops:

• HttpListener::run() — server.c++:6086
• TcpListener::run() — server.c++:6294
• DebugPortListener::run() — server.c++:6358, via capnp::TwoPartyServer::listen()

None has any error handling around accept. Read-side failures are contained
(TwoPartyServer::taskFailed logs), so it is the accept path by elimination.

The Windows accept path

kj/async-io-win32.c++, FdConnectionReceiver::accept():

Line	Site	Fails when
861	KJ_FAIL_WIN32("AcceptEx()", ioResult.errorCode)	peer RSTs before the pending AcceptEx completes → ERROR_NETNAME_DELETED (64), ERROR_CONNECTION_ABORTED (1236), ERROR_SEM_TIMEOUT (121)
864 → 326	setsockopt(SO_UPDATE_ACCEPT_CONTEXT) via KJ_WINSOCK	socket already dead
873 → 674	getpeername() via KJ_WINSOCK	WSAENOTCONN on a reset socket

The Unix implementation defends against exactly this race in three separate places
(kj/async-io-unix.c++):

• :1398-1414 — retries on EINTR/ENETDOWN/EPROTO/EHOSTDOWN/EHOSTUNREACH/ENETUNREACH/
  ECONNABORTED/ETIMEDOUT, with the comment "accept() may report an error if the accepted
  connection is already broken. In this case, we really ought to just ignore it and keep waiting."
• :1342 — retries when accept() returns addrlen == 0 ("The socket is already dead and should
  be discarded")
• :1363-1375 — KJ_SYSCALL_HANDLE_ERRORS around TCP_NODELAY, tolerating "EINVAL … for sockets
  that are already dead (due to a race with RST)"

Windows has none of it, and calls getpeername() where Unix doesn't need to (accept() returns the
address atomically). Upstream capnproto main has the same gap.

Independent corroboration that this error family is live in this exact setup — workers-sdk already
carries two workarounds naming it by number:

• packages/miniflare/src/index.ts — dev-registry-proxy was given its own socket because "the
  internal Cap'n Proto service-to-service forwarding from the entry worker can break (WSARecv error
  64)"
• packages/miniflare/src/workers/browser-rendering/binding.worker.ts — "kj/async-io-win32.c++
  WSARecv (Support locales Chromium disables #64) — the connection went away mid-read"

---

Why the tail edge is necessary and the service binding isn't

ExternalServiceProxy's constructor
(packages/miniflare/src/workers/core/dev-registry-proxy.worker.ts) calls
DEV_REGISTRY_DEBUG_PORT.connect() twice — once in resolve(), once for core:entry. Merely
constructing the entrypoint opens two TCP connections to the peer's debug port. A service binding
only connects when a request actually traverses it. That matches the measured arms exactly: tail edge
necessary, binding irrelevant, idle sessions still affected.

Both processes accept from each other

fixtures/dev-registry/wrangler.*.jsonc — the tail edges are mutual pairs:

exported-handler              ──tails──▶  worker-entrypoint-with-assets
worker-entrypoint-with-assets ──tails──▶  exported-handler
exported-handler-with-assets  ──tails──▶  worker-entrypoint
worker-entrypoint             ──tails──▶  exported-handler-with-assets

Per packages/miniflare/src/shared/DEV_REGISTRY.md, every session gets --debug-port=127.0.0.1:N
and reaches peers by connecting to their debug port. With mutual tail edges both processes are
simultaneously client and server on each other's debug ports, so an abrupt kill RSTs connections
sitting in the survivor's accept queue. This closes the gap left by the deduction above.

Still unexplained: the assets discriminator

The only concrete difference is defaultEntrypointService — core:user:<name> for a plain worker
versus assets:rpc-proxy:<name> for an assets worker (DEV_REGISTRY.md). Tail forwarding uses
defaultEntrypointService. Given that bypassing the extra hop still reproduced (4/4 and 3/4), my
reading is that this widens the race window rather than being causal. I could not do better than
that, and it remains the weakest part of the analysis.

Ruled out

• Outbound connection failures — cannot reach Server::tasks, so cannot be fatal.
• Read-side failures on accepted connections — contained by TwoPartyServer::taskFailed and the
  per-connection try/catch at server.c++:6136.
• A worker thread — see Correction 1.
• Blocking DNS on parseAddress — moot, debugPortAddress is a literal 127.0.0.1:N.

---

Fix in progress

Two parts, one PR:

Diagnostics — so this class of failure can never be silent again:

• Server::taskFailed logs the exception before rejecting fatalFulfiller
• Drop noexcept from CliMain::serveImpl and CliMain::serve so kj::runMainAndExit reports
  the exception and exits non-zero
• Test coverage asserting a fatal task failure produces a readable message

Accept resilience — workerd-side, no upstream dependency:

• A shared tolerant acceptor used by all three accept loops. A failure to accept one connection
  is logged and retried with backoff; consecutive failures escalate to fatal, so a listener that
  is genuinely broken still fails the server rather than spinning.

No error classification for now. kj::Exception carries no OS error code, and typeOfWin32Error
only maps WSAE* codes — so ERROR_NETNAME_DELETED (64) and friends arrive as FAILED, not
DISCONNECTED. Filtering on DISCONNECTED would miss the most likely site. The retry bound, not
the classification, is what preserves "a broken listener is fatal".

Bind/listen failures remain fatal — they throw before any accept, and Miniflare depends on
EADDRINUSE killing the process.

Follow-ups, deferred until the diagnostics confirm the exception on Windows CI:

• Upstream KJ fix for kj/async-io-win32.c++ (mirror the Unix tolerance at all three sites), and/or
  map ERROR_NETNAME_DELETED / ERROR_CONNECTION_ABORTED / ERROR_SEM_TIMEOUT to
  Exception::Type::DISCONNECTED in typeOfWin32Error. Once that lands, the workerd-side tolerance
  can be narrowed to DISCONNECTED with backoff kept purely as spin protection.
• workers-sdk: ExternalServiceProxy opens two debug-port TCP connections per construction, i.e.
  per tail dispatch. Caching per debugPortAddress, as the DO proxy already does, would cut the
  churn.

Note

This investigation was carried out by an AI agent: opencode, claude-opus-5.