Skip to content

fix(server): reconcile stopped sessions after restart#4562

Open
PixPMusic wants to merge 3 commits into
pingdotgg:mainfrom
PixPMusic:pixpmusic/fix-stale-working-session-recovery
Open

fix(server): reconcile stopped sessions after restart#4562
PixPMusic wants to merge 3 commits into
pingdotgg:mainfrom
PixPMusic:pixpmusic/fix-stale-working-session-recovery

Conversation

@PixPMusic

@PixPMusic PixPMusic commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • reconcile durable stopped provider bindings with thread session projections during the initial reaper sweep
  • clear abandoned pending turns through the existing stopped-session projection path so restarted threads can be settled
  • reject reconciliation atomically if the projected session changes before dispatch, so a concurrent restart cannot be overwritten
  • use a fresh command ID for each reconciliation attempt so a rejected receipt cannot poison a later retry
  • cover shutdown divergence, stale-binding ordering, concurrent session updates, and rejected-command retries with regression tests

Fixes #4561

Verification

  • pnpm exec vp test run apps/server/src/provider/Layers/ProviderSessionReaper.test.ts apps/server/src/orchestration/decider.settled.test.ts — 21 tests passed
  • pnpm exec vp run --filter t3 typecheck
  • pnpm exec vp run --filter @t3tools/contracts typecheck
  • targeted formatting checks for changed files
  • targeted lint for changed files
  • read-only production-state inspection confirmed the failure shape: projected starting with no active turn, durable runtime stopped, and an abandoned pending turn; the sidebar was initially stuck Working, then looked idle after a recovery message while settling remained blocked

Note

Medium Risk
Changes thread session projection paths after restart and adds server-initiated orchestration dispatches on every reaper sweep for stopped bindings; guarded by optional concurrency checks but still affects settling/working UI state.

Overview
Fixes threads that stay stuck in a transient session state (e.g. starting / working) when shutdown persisted a durable stopped provider binding but never updated the thread projection.

Provider session reaper now treats stopped bindings as reconciliation targets on each sweep: it reads the projected session, skips when already stopped or when the binding’s lastSeenAt is older than session.updatedAt, and otherwise dispatches thread.session.set with a server-generated command id to project stopped (clearing activeTurnId). Invariant rejections are logged and retried on later sweeps with a new command id.

Orchestration adds optional expectedSessionUpdatedAt on thread.session.set; the decider rejects the command with OrchestrationCommandInvariantError when the projected session’s updatedAt no longer matches, so reconciliation cannot clobber a concurrent restart or newer session write.

Regression tests cover shutdown divergence, stale-binding ordering, concurrent session updates, and retry behavior.

Reviewed by Cursor Bugbot for commit 56da49d. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Reconcile stopped provider sessions after server restart

  • Adds reconcileStoppedBinding to ProviderSessionReaper, which dispatches a thread.session.set command for bindings persisted as stopped so their projected session status is updated after a restart.
  • Skips dispatch when the binding's lastSeenAt is older than the projected session's updatedAt, preventing stale writes from overwriting newer state.
  • Adds an optional expectedSessionUpdatedAt field to the thread.session.set command schema in orchestration.ts for conditional write semantics.
  • The decider in decider.ts now returns OrchestrationCommandInvariantError when expectedSessionUpdatedAt is provided but does not match the projected session's updatedAt.
  • Risk: The reaper now depends on OrchestrationEngineService and Crypto; misconfigured environments missing these services will fail at startup.

Macroscope summarized 56da49d.

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 97dc1069-3d74-4a74-918a-448242c60f84

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:M 30-99 changed lines (additions + deletions). labels Jul 26, 2026
Comment thread apps/server/src/provider/Layers/ProviderSessionReaper.ts
Comment thread apps/server/src/provider/Layers/ProviderSessionReaper.ts
@macroscopeapp

macroscopeapp Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs human review

This PR adds new session reconciliation behavior after server restart, including ~80 lines of new logic that dispatches orchestration commands to update session states. Despite the 'fix' label, this introduces significant new runtime behavior affecting session lifecycle management, warranting careful review.

You can customize Macroscope's approvability policy. Learn more.

@github-actions github-actions Bot added size:L 100-499 changed lines (additions + deletions). and removed size:M 30-99 changed lines (additions + deletions). labels Jul 26, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit dca27c9. Configure here.

Comment thread apps/server/src/provider/Layers/ProviderSessionReaper.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Stopped sessions remain active after restart and block settling

1 participant