Skip to content

docs(server): document provider turn startup recovery - #249

Open
patroza wants to merge 94 commits into
fork/changesfrom
docs/provider-turn-recovery-runbook
Open

docs(server): document provider turn startup recovery#249
patroza wants to merge 94 commits into
fork/changesfrom
docs/provider-turn-recovery-runbook

Conversation

@patroza

@patroza patroza commented Jul 31, 2026

Copy link
Copy Markdown
Owner

What Changed

  • add an operator runbook for diagnosing and recovering stuck provider turns
  • document the global command-worker head-of-line failure mode and startup reconciliation behavior
  • record the 2026-07-31 incident pattern, safe SQLite diagnostics, and normal/emergency systemd recovery
  • define prevention priorities and acceptance tests for timeouts, per-thread isolation, sweeping, and readiness
  • cross-link the runbook from the docs index and composer lifecycle analysis

Why

One wedged provider session start can currently block unrelated turn starts while the process and HTTP listener remain alive. Recovery is reliable only through service restart and startup replay. This documents the present operator path and the product work needed to make recovery automatic.

Downstream Fork Relationship

Base: fork/changes. Affected area: documentation only. No dependency on another open PR. Deployment-specific hostnames, credentials, monitoring, and restart automation remain assigned to the private operations repository.

Verification

  • git diff --check
  • vp check (zero errors; existing repository warnings only)
  • ELECTRON_SKIP_BINARY_DOWNLOAD=1 vp run -r --cache --log labeled typecheck

Checklist

  • This PR is small and focused
  • I explained what changed and why
  • UI screenshots are not applicable
  • Animation/interaction video is not applicable

tim-smart and others added 30 commits July 31, 2026 11:08
Add custom "Open with" applications

Source: tim-smart#4
Source head: 8c4bdfb
Source commits: 08e1a4f,cd43a401c6c148f1fe26cff72104ac527ea189f3,a8370e7502c552ebb064436e42e1c00f86f0946b,8c4bdfbc5b57f6b600233244d330f9efa41dc498
Imported: complete product delta from the source PR.

(cherry picked from commit 9fae005)
Load direnv environments for provider sessions

Source: tim-smart#5
Source head: 8f5fc87
Source commits: e4f0701,0d1463af61e0bd174f698b2519ebf3b207a2eaca,a66e4160d5f4b79140ec8fbcbc6aa66af750a991,8f5fc87c13f4628c179cda44d4f32f7fe4d316b2
Imported: complete product delta from the source PR.

(cherry picked from commit 0da8bfe)
Add unsigned retry for commit signing failures

Source: tim-smart#6
Source head: 7d65c5a
Source commits: 18ee567,72a6fae23c86708080c4fed346d5bf0f136f0221,6614b28239ed2330a8f601357a413f2d50da195a,ec169369daa554541511aa28f551b36f3dd26485,7d65c5a224e97a6b811b0a84892f1fda065c5963
Imported: complete product delta from the source PR.

(cherry picked from commit 03671a2)
Add /new command for contextual threads

Source: tim-smart#7
Source head: 2051a80
Source commits: 2051a80
Imported: complete product delta from the source PR.

(cherry picked from commit 4d94f31)
Add session dashboard board

Source: tim-smart#8
Source head: d9f8e4d
Source commits: 268fb8d,dde20f271f674da22dd8f3a08201c2acf5e58ee5,df4a145e7b2cd2dc17a7a595267d2d8eb0a2a3f0,ce5723ddb0bf630a18d4cb8227b5344d12626e72,ad8c1a6af41161e1fc38a52f681b306517c7b918,6281887e6125317da0c7b4252d59bfd41c9bf35e,550db6316c634febdbe1cb27334d1347c23c7b2a,d9f8e4d0a8dc22231ca315f3c595c3597f3b13e5
Imported: complete product delta from the source PR.

(cherry picked from commit cd0e281)
Recover interrupted provider turns after server restarts

Source: tim-smart#9
Source head: b181832
Source commits: 7f69028,1d52bce1376766d804ef884d7d50b8b6d1b48cf7,b181832560177250b90bbfe07b0882c9e5b93493
Imported: complete product delta from the source PR.

(cherry picked from commit 83de8f5)
Avoid repeated thread snapshot loads during subscription retries

Source: tim-smart#10
Source head: c8c9ead
Source commits: c8c9ead
Imported: complete product delta from the source PR.

(cherry picked from commit 9e400c3)
Add image upload button to compact chat composer

Source: tim-smart#11
Source head: 1ff63f9
Source commits: 1ff63f9
Imported: complete product delta from the source PR.

(cherry picked from commit 720ec65)
Truncate mobile branch toolbar controls

Source: tim-smart#12
Source head: 1b7d444
Source commits: 1b7d444
Imported: complete product delta from the source PR.

(cherry picked from commit dc2bbb4)
Clean up worktrees when archiving threads

Source: tim-smart#13
Source head: a23f42d
Source commits: 4a19470,1b6688aa5b641010cb2e9dad23d36d87257403ad,9ed32aa3923fb674380564b1ffcb3268290069b9,a23f42d6ac671ea36b8db5d03934c089a31be448
Imported: complete product delta from the source PR.

(cherry picked from commit 7e02dc9)
Pass hosted app channel into Vercel web builds

Source: tim-smart#14
Source head: de6966a
Source commits: de6966a
Imported: complete product delta from the source PR.

(cherry picked from commit 6333d8d)
Allow worktrees to reuse the selected branch

Source: tim-smart#15
Source head: 2d3900b
Source commits: cd60531,2d3900ba36c9397dc4fbe879c613a809f6b45384
Imported: complete product delta from the source PR.

(cherry picked from commit 5e7dff2)
Add optional worktree removal confirmation

Source: tim-smart#16
Source head: c3f509f
Source commits: 76f063e,c3f509fe8f690b704bb34692d9c132c0644db777
Imported: complete product delta from the source PR.

(cherry picked from commit 9886109)
Stop retrying unavailable thread subscriptions

Source: tim-smart#17
Source head: 1359af8
Source commits: 1359af8
Imported: complete product delta from the source PR.

(cherry picked from commit 7b37a7a)
Compatibility fix for running the selected Tim stack on the fork CI matrix. Source adaptation review: #31.

(cherry picked from commit 6edd39a)
Keep the selected Tim Open With feature portable on non-macOS builders and avoid treating custom app definitions as macOS bundles. Source adaptation review: #33.

(cherry picked from commit 15a7d2a)
…nd; green tip

Bring Tim layer tip to typecheck green by joining main ref-refresh VCS client
state with fork failureKind/worktree-cleanup contracts, restoring
filterBrowseEntries/reuse-base-branch surfaces Tim dropped, and fixing
ChatView/Board call-site type errors left by incomplete Tim joins.

(cherry picked from commit 0e24917)
Bring fork/tim typecheck/test green after main pingdotgg#2679 + Tim client-runtime
rewrite: rejoin EnvironmentSubscriptionRpcTag/localApi/ws scopes, wire
BackgroundPolicy/ResourceTelemetry layers, force openpgp for signing tests
on hosts with gpg.format=ssh, and treat TRACE2 child_exit without
child_class as hook finish (git 2.55+).
Source: pingdotgg#4018
Source SHA: de8fd65

Imported: bounded server activity snapshots, cursor pagination, lazy web history loading, reconnect-safe reset/dedup, and disabled eager browser sidebar hydration.
Adapted: preserved Tim thread lifecycle handling and Omega composer/minimap behavior while resolving current-stack conflicts.
Excluded: none of the source PR behavior; native mobile pagination remains separate because pingdotgg#4018 intentionally excludes it.
…#3510) (#35)

Source: pingdotgg#3510
Source SHA: 034f493

Imported: native mobile lazy loading for older thread activity, a 1,000-event subscription catch-up ceiling with snapshot fallback, and synchronized stale snapshot watermarks.
Adapted: applied above the refreshed pingdotgg#4018 web/server candidate and preserved Tim lifecycle handling plus our mobile composer changes.
Excluded: pingdotgg#3510 server/web pagination duplicated by pingdotgg#4018, the later shared-hook refactor, formatting-only commits, and contract comments. The shared refactor can be revisited independently after production validation.
Source: pingdotgg#4176
Source SHA: 56b6615

Imported: O(1) command read-model maps, deleted-thread eviction, VCS cache cleanup, browser surface cleanup, preview idle TTL, and per-thread UI cleanup.
Adapted: preserved our thread settlement, snooze, and sequential worktree deletion actions while wiring upstream cleanup into the current hook.
Excluded: none.

Co-authored-by: Rusiru Sadathana <27785781+RusiruSadathana@users.noreply.github.com>
#44)

Source: pingdotgg#4506

Source SHA: f7eaa00

Imported unchanged as one candidate provenance commit.
…tgg#4558)

Imported from https://github.com/pingdotgg/t3code/pull/4558\n\nAdapted to retain our provider restart-recovery constants while replacing the local default-title check with the shared policy.
After rebasing candidates onto the green tim tip, restore missing
EnvironmentThread loading fields, ChatView sendDisabledReason/threadSyncPhase
wiring, and orchestration.getThreadActivities auth coverage.
github-actions Bot and others added 15 commits July 31, 2026 11:11
That durable conflict resolution reintroduced the pre-pingdotgg#2679 RPC auth Map
on every restack, undoing typed requiredScopeForRpcMethod. Tim product
deltas in ws.ts must 3-way merge with main's auth path instead.
Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
* fix(stack): forbid blind product conflict resolution

* refactor(stack): protect future workspace sources by structure

---------

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
* Revert "fix(mobile): keep conversation feed visible (#232)"

This reverts commit bb01fe8.

* Revert "fix(mobile): reduce thread feed scroll jank (pingdotgg#4874)"

This reverts commit 9e3e9bb.

---------

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
…ng up (pingdotgg#4867)" (#235)

This reverts commit 90f3913.

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
Upstream restack of tim-smart#5 conflicts on GitVcsDriverCore.test.ts.
Prefer the Tim-side test file (theirs during replay) on every restack.
The #207 compact command was dropped from fork/changes during the upstream
restack while the Discord overlay still dispatches thread.context.compact.
Restore contracts, decider, provider adapters, and client-runtime handling
so integration typechecks and compact works end-to-end.
* chore(identity): reapply identity product after upstream restack

* fix(identity): keep upstream Clerk catalog after reapply

The path-filtered reapply pulled stale pnpm-workspace.yaml clerk pins
(3.13.x) while the lockfile still reflected fork/changes (3.14.x), which
breaks frozen install on the permanent overlay PR.

* fix(identity): restore shared product files dropped by reapply

The path-filtered identity reapply overwrote fork/changes product with a
stale overlay tree, dropping content search contracts/UI, file picker
query helpers, desktop update helpers, ForwardCompatibleArray keybindings,
and related tests. Restore those shared files from fork/changes so recent
product work is not lost under the identity layer.

* fix(identity): drop obsolete requestSingleInstanceLock test mock

ElectronApp no longer exposes requestSingleInstanceLock (Clerk bridge
owns the single-instance lock). Remove it from DesktopUpdates test stubs
so desktop typecheck is green on the identity overlay.

* fix(identity): restore project content search under identity layer

The identity reapply dropped projects.searchContents contracts, RPC
wiring, workspace search implementation, and related auth scopes while
keeping the web UI that depends on them. Re-add the shared product
surface without removing identity RPCs so typecheck and search stay green.

* fix(identity): allow ReactNode command palette descriptions

Project file picker (from fork/changes) renders fuzzy-highlighted path
descriptions as elements; the identity reapply left description typed as
string only. Widen it to ReactNode so web typecheck matches the UI.

* fix(identity): restore os-jank hydratePosixHome from fork/changes

Identity reapply dropped hydratePosixHome while leaving its unit test,
breaking server typecheck.

* fix(identity): restore ProjectFavicon cache behavior from fork/changes

Identity reapply left a stale ProjectFavicon/test combo that fails in CI
with useContext outside providers. Restore the fork/changes implementation
and tests so Fork CI Test is green on the overlay.

---------

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
@patroza
patroza force-pushed the fork/changes branch 15 times, most recently from 46a7aa8 to d8ee8ae Compare August 1, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants