Explicit Connection Lifecycle States for TCPConnection

[SeanTAllen]

Explicit Connection Lifecycle States for TCPConnection

Problem

TCPConnection encodes its connection lifecycle as boolean flags: _connected, _closed, _shutdown, _shutdown_peer, plus SSL-related flags _ssl_ready, _ssl_failed, _tls_upgrade. These flags form an implicit state machine where valid combinations aren't documented and invalid combinations aren't prevented.

This creates several maintenance challenges:

1. Dense dispatch in _event_notify: The method is ~90 lines with nested branching on event is _event, not _connected and not _closed, and platform ifdef blocks. Different lifecycle phases (client doing Happy Eyeballs, open connection doing I/O, connection shutting down) are interleaved in the same method.

2. Complex callback routing in hard_close(): Which lifecycle callback fires depends on a multi-way branch: SSL present? SSL ready? TLS upgrade? Client or server? These checks inspect 4+ flags to determine the correct callback.

3. Scattered state transitions: _connected = true is set in _event_notify (Happy Eyeballs success) and _complete_server_initialization. _closed = true is set in hard_close(), _close(), and _complete_server_initialization. Understanding the lifecycle requires tracing all flag mutations across the file.

Inspiration: ponylang/postgres

The ponylang/postgres library addresses the same class of problem with state objects: each lifecycle state is a separate class, and the Session actor holds a single var state: _SessionState field. Every incoming event is delegated to whatever state object is current. The session actor itself contains no conditional logic about what state it's in — it is purely a dispatcher.

A trait hierarchy provides defaults for operations that are illegal in each state. Capability traits implement valid behavior; denial traits panic on impossible transitions. Each concrete state mixes in the appropriate combination, so it only implements the methods that are valid transitions. Everything else panics by default.

This pattern also ties data ownership to state — per-query accumulation data lives inside the in-flight query state object and is structurally destroyed on state transition.

Analysis: Which of Lori's state machines benefit?

TCPConnection has five interleaved state machines:

Machine	Flags	Candidate?
Connection lifecycle	_connected, _closed, _shutdown, _shutdown_peer	Yes — gates most operations, has complex dispatch
SSL/TLS	_ssl, _ssl_ready, _ssl_failed, _tls_upgrade	Later — tightly coupled to lifecycle, but benefits from it being clearer first
Backpressure	_throttled, _writeable	No — flow control within the open state
Read readiness	_readable, _muted	No — _readable is a read loop condition, no operations are illegal when unreadable
Happy Eyeballs	_inflight_connections	Absorbed into the connecting state

Readable and writeable were evaluated as candidates. _readable is checked in exactly one place — the _read() loop condition. No operation becomes illegal when the connection is unreadable. _writeable has slightly more gatekeeping (send() returns SendErrorNotWriteable), but it's a transient flow-control flag, not a lifecycle boundary. Both remain as flags.

Conclusion: The connection lifecycle is the right starting point. It's where operations become structurally illegal (you can't send on a closed connection), where dispatch is most complex (_event_notify, hard_close()), and where flag combinations are hardest to reason about.

Proposed Design

States

_ConnectionNone ──(_finish_initialization)──► _ClientConnecting  (client path)
                ──(_finish_initialization)──► _Open              (server path)
                ──(_finish_initialization)──► _Closed            (SSL creation failed)

_ClientConnecting ──(connection established)──► _Open
                  ──(all attempts failed)────► _Closed

_Open ──(graceful close)──► _Closing
      ──(hard close)──────► _Closed

_Closing ──(shutdown + peer shutdown)──► _Closed

_ConnectionNone: The universal initial state. All constructors (including none()) start here. No behaviors can fire between construction and _finish_initialization (it's the first behavior queued), so _ConnectionNone is safe as the starting state. When _finish_initialization fires, it transitions to the appropriate state (_ClientConnecting, _Open, or _Closed). For TCPConnection.none() (the field initializer placeholder), _finish_initialization is never called because the actor's constructor body replaces the placeholder before any behaviors fire. Every operation on _ConnectionNone is _Unreachable().

_ClientConnecting: Happy Eyeballs in progress. Handles incoming connection-attempt events. Contains the _connecting_callback() logic — firing _on_connecting(count) as attempts resolve, or _on_connection_failure() when all fail. Also handles the _ssl_failed check during _finish_initialization. send() returns SendErrorNotConnected. start_tls() returns StartTLSNotConnected.

_Open: Connected and alive. Handles own-event I/O, send, close, start_tls. Straggler Happy Eyeballs events (connection attempts that resolved after we picked a winner) are cleaned up here. SSL handshake and flow control flags live within this state.

_Closing: Graceful close initiated. No new sends accepted (is_open() returns false). Continues reading to detect peer shutdown. Drains pending writes. Contains the _try_shutdown() logic. Transitions to _Closed when both local shutdown and peer shutdown are complete.

_Closed: Terminal. send() returns SendErrorNotConnected. close() and hard_close() are no-ops. Events are destroyed.

Server initialization

Servers receive an fd from the listener and transition directly to _Open during _complete_server_initialization. If SSL session creation failed in the constructor (_ssl_failed), the server transitions to _Closed instead, firing _on_start_failure().

Initialization dispatch

_finish_initialization() is a behavior on TCPConnectionActor that fires asynchronously after construction. All constructors (including none()) start with _state = _ConnectionNone. No behaviors can fire between construction and _finish_initialization (it's the first behavior queued), so _ConnectionNone is safe as the universal initial state.

When _finish_initialization fires, TCPConnection's _finish_initialization() method performs the transition:

• Client path: Checks _ssl_failed (transitions to _Closed with _on_connection_failure() if true). Otherwise transitions to _ClientConnecting and calls PonyTCP.connect() to initiate Happy Eyeballs.
• Server path: Checks _ssl_failed (transitions to _Closed with _on_start_failure() if true). Otherwise transitions to _Open, creates the ASIO event, and starts reading.

The _finish_initialization method itself remains on TCPConnection — it's not dispatched through the state interface. It's the bootstrap that sets the first real state.

For TCPConnection.none() (the field initializer placeholder), _finish_initialization is never called because the actor's constructor body replaces the placeholder with a real TCPConnection before any behaviors fire.

State interface

trait _ConnectionState
  fun ref own_event(conn: TCPConnection ref, flags: U32, arg: U32)
  fun ref foreign_event(conn: TCPConnection ref, event: AsioEventID,
    flags: U32, arg: U32)
  fun ref send(conn: TCPConnection ref, data: ByteSeq): (SendToken | SendError)
  fun ref close(conn: TCPConnection ref)
  fun ref hard_close(conn: TCPConnection ref)
  fun ref start_tls(conn: TCPConnection ref, ssl_ctx: SSLContext val,
    host: String): (None | StartTLSError)
  fun is_open(): Bool
  fun is_closed(): Bool

TCPConnection's _event_notify handles event ownership dispatch and disposable cleanup before delegating lifecycle-relevant events to the state:

// TCPConnection
fun ref _event_notify(event: AsioEventID, flags: U32, arg: U32) =>
  // Disposable events are orthogonal to lifecycle state
  if AsioEvent.disposable(flags) then
    PonyAsio.destroy(event)
    if event is _event then
      _event = AsioEvent.none()
    end
    return
  end

  if event is _event then
    _state.own_event(this, flags, arg)
  else
    _state.foreign_event(this, event, flags, arg)
  end

This replaces the current nested if event is _event / if not _connected and not _closed structure with per-state dispatch:

• _ClientConnecting.foreign_event() handles Happy Eyeballs resolution (the current inner if not _connected and not _closed block)
• _Open.own_event() handles normal I/O (the current if event is _event block)
• _Open.foreign_event() cleans up straggler Happy Eyeballs events
• _Closing.own_event() handles I/O with shutdown tracking
• _Closing.foreign_event() cleans up straggler Happy Eyeballs events (stragglers are asynchronous and can arrive after close())
• _Closed no-ops (events already handled by disposable check above)

Asynchronous operations and state safety

Several behaviors on TCPConnectionActor dispatch to TCPConnection methods that are state-dependent:

• _read_again: Calls _connection()._read() to re-enter the read loop. Can arrive after the connection has closed (it's asynchronous). Currently there's no guard — _read() would try to read from fd -1. With state objects, _read_again should dispatch through the state, which no-ops in _Closed. Similarly, unmute() calls _queue_read() which calls _read_again() — the same concern applies if unmute() is called in a subsequent behavior after close.

• _notify_sent / _notify_send_failed: These fire deferred callbacks. They call _fire_on_sent() / _fire_on_send_failed() on TCPConnection. These can arrive after close and must still deliver their callbacks (by design — _on_send_failed is expected after _on_closed). They remain on TCPConnection, not gated by state.

• _register_spawner: Server-only. Currently gates on _closed. With state objects, it can use _state.is_closed() directly on TCPConnection without being part of the state interface.

Other state-dependent methods

keepalive() currently gates on _connected. With state objects, it gates on _state.is_open(). It remains on TCPConnection — not part of the state interface, since it's a simple one-line delegation to PonyTCP.keepalive.

State transition ordering

hard_close() currently sets _connected = false and _closed = true at the top (lines 227-228), before firing any callbacks. The state-object equivalent must preserve this: set _state = _Closed before firing _on_closed(), _on_connection_failure(), etc. This ensures that if a callback re-enters TCPConnection (e.g., calling send()), it sees the closed state and gets the appropriate error.

Data ownership

Recommended: states are logic-only. All data (_fd, _event, _read_buffer, _pending, SSL fields, flow control flags) stays on TCPConnection. State objects provide dispatch and operation gating. Transitions are _state = _Open rather than flag mutations.

The alternative — states owning per-state data — was considered. _ClientConnecting could own _host, _port, _from, _inflight_connections (only relevant while connecting). But _Open and _Closing share almost all data (fd, event, buffers, SSL), making ownership between them awkward. Data ownership can be added incrementally later where it proves valuable.

Trait hierarchy

The postgres library uses a rich hierarchy of capability traits and denial traits. This pays off when there are ~20 methods and many states need the same defaults.

For Lori's smaller surface (~8 methods, 5 states), each state can directly implement the full interface. Illegal operations are one-liners:

class _Closed is _ConnectionState
  fun ref send(conn: TCPConnection ref, data: ByteSeq): (SendToken | SendError) =>
    SendErrorNotConnected

  fun ref close(conn: TCPConnection ref) => None
  fun ref hard_close(conn: TCPConnection ref) => None

  fun ref start_tls(conn: TCPConnection ref, ssl_ctx: SSLContext val,
    host: String): (None | StartTLSError)
  =>
    StartTLSNotConnected

  fun is_open(): Bool => false
  fun is_closed(): Bool => true
  // ...

If the method count grows (e.g., when SSL becomes a sub-state machine), a trait hierarchy can be introduced then.

Platform differences (IOCP)

POSIX and IOCP have different I/O models (edge-triggered notification vs completion callbacks) but the same state transitions. The platform difference is in how I/O is performed within a state, not which states exist.

With logic-only state objects, platform-specific I/O methods stay on TCPConnection where they are today (_read() for POSIX, _read_completed() for Windows). State objects call into these methods, and the ifdef blocks remain in the I/O layer. The state machine pattern doesn't resolve the IOCP complexity, but it doesn't worsen it either — platform branching stays where it is.

SSL

SSL remains as fields on TCPConnection for this iteration. The _ssl, _ssl_ready, _ssl_failed, _tls_upgrade flags and associated logic in _ssl_poll() stay unchanged.

A future iteration could model SSL as a sub-state machine within _Open (e.g., NoSSL | SSLHandshaking | SSLReady). This would simplify the callback routing in hard_close() — the SSL sub-state would know whether to fire _on_connection_failure, _on_tls_failure, or _on_closed. But that's a separate step that builds on the foundation this change lays.

What changes

1. _event_notify splits from one ~90-line method into per-state handlers, each handling only its relevant events. Disposable event handling stays in TCPConnection as a pre-dispatch step.
2. send(), close(), start_tls() gate on state type rather than boolean flag checks.
3. State transitions become explicit assignments (_state = _Open) rather than scattered flag mutations.
4. hard_close() callback routing could be partially simplified — the state knows whether we're in the connecting phase vs open.
5. _read_again becomes state-aware, preventing reads on closed connections.

What stays the same

• All flow control flags (_readable, _writeable, _throttled, _muted) remain on TCPConnection.
• SSL fields and logic remain on TCPConnection.
• I/O methods (_read, _send_pending_writes, etc.) remain on TCPConnection.
• The public API of TCPConnection is unchanged.
• TCPConnectionActor and lifecycle event receiver traits are unchanged.

Pony implementation notes

State objects must be class (not primitive) because the _ConnectionState trait uses fun ref methods. While states like _Closed and _ConnectionNone have no per-instance data and could theoretically be primitives, primitives are val — they can't satisfy fun ref method signatures. Since the trait needs uniformity across all states (and some states like _Open may need ref for future data ownership), all states use class. The allocation cost is minimal — state transitions are infrequent relative to I/O operations.

Open questions

1. hard_close() during _ClientConnecting: Currently, hard_close() checks if not _connected then return end, making it a no-op during connecting. The actual failure-during-connecting path goes through _connecting_callback(): when a connection attempt fails in _event_notify, it calls _connecting_callback(), which fires _on_connection_failure() when all attempts are exhausted and then calls hard_close() (which no-ops). For graceful cancellation, close() sets _closed = true and _try_shutdown() waits for inflight connections to resolve naturally. In the state-object model, should _ClientConnecting.hard_close() actively cancel inflight connection attempts (unsubscribe events, close fds), or should it remain a no-op that relies on close() and natural event resolution? This is a behavioral decision, not just a refactoring question.

2. _Closing as a separate state? From the user's perspective, _Closing behaves identically to _Closed — all public operations return errors. The difference is internal: pending write draining and the shutdown handshake. A separate state makes the shutdown sequence explicit and avoids reintroducing a boolean flag within _Open. A flag within _Open keeps the state count lower but partially defeats the purpose of the refactoring.

3. Client/server dimension: Currently encoded as the type of _lifecycle_event_receiver (client vs server). The _ClientConnecting state is inherently client-only. Within _Open, the client/server distinction only affects callback names (_on_connected vs _on_started, _on_connection_failure vs _on_start_failure). Keeping client/server orthogonal to lifecycle state (state objects match on the receiver type for callbacks) avoids duplicating nearly identical _ClientOpen/_ServerOpen classes.

[SeanTAllen]

Decision: hard_close() during _ClientConnecting (Open Question 1)

Decision: Preserve current behavior — _ClientConnecting.hard_close() is a no-op.

Currently hard_close() guards on if not _connected then return end, making it a no-op during connecting. The cancellation path that works is close() → _close() → sets _closed = true → _try_shutdown() waits for inflight events to resolve naturally.

Changing hard_close() to actively cancel inflight connection attempts would be a behavioral change, not a refactoring. There's an existing inconsistency worth noting: calling hard_close() during connecting doesn't set _closed = true, so if a connection attempt later succeeds via Happy Eyeballs, the connection goes live as if hard_close() was never called. close() handles this correctly; hard_close() doesn't. But that's a pre-existing issue — a separate issue should be filed to track making hard_close() during connecting actually cancel.

[SeanTAllen]

Decision: _Closing as a separate state (Open Question 2)

Decision: Yes, _Closing is its own state.

Explicit is better than implicit. Reintroducing a boolean flag within _Open to track "closing in progress" would partially defeat the purpose of the refactoring. A separate _Closing state makes the shutdown sequence (draining pending writes, waiting for peer shutdown, calling _try_shutdown()) explicit in the type system.

[SeanTAllen]

Decision: Client/server dimension (Open Question 3)

Decision: Keep client/server orthogonal to lifecycle state.

The _ClientConnecting state is inherently client-only. Within _Open and _Closing, the client/server distinction only affects callback names (_on_connected vs _on_started, _on_connection_failure vs _on_start_failure). State objects match on the _lifecycle_event_receiver type for these differences. No _ClientOpen/_ServerOpen split — it would duplicate nearly identical classes for a small callback routing difference.

[SeanTAllen]

This is out of date. Closing. Will create a new one.