fix(opal-server): git resilience — never stuck on an offline repo (PR3)

[dshoen619]

PR3 — Git resilience + the deferred items from the leak series

Closes PER-15157.

Ships PR3's original charter (no git operation can hang the server) plus the six items deferred out of PR2 (#923), and — because the resilience fix required it — the parallel scope loading originally scoped as PR4 (see §6; PR4 is closed as redundant). Design: docs/superpowers/specs/2026-07-19-pr3-deferred-items-design.md.

1. Git resilience — hung remotes can't starve healthy scopes

Scope clone/fetch used pygit2 with no timeout, on the shared default executor. POLICY_REPO_CLONE_TIMEOUT is wired only to the legacy non-scopes path, so scopes mode had no bound at all: boot (preload_scopes()) blocked indefinitely on one unreachable repo, and in steady state a hung op held the per-source lock and a shared-pool thread.

• Hard per-operation timeout (SCOPES_GIT_FETCH_TIMEOUT) on clone and fetch.
• Zombie-aware executor. Each op runs on its own single-use daemon-thread executor; SCOPES_GIT_MAX_WORKERS bounds only live ops via a semaphore. A timed-out op releases its capacity slot immediately — its pygit2 call lingers on a private thread until the OS gives up, but no longer consumes capacity. (A fixed pool starves once lingering threads exceed its size: 40 blackholed repos vs. 10 slots meant healthy scopes never got a thread. That's the test_offline_repo_does_not_block_healthy_scopes gate.)
• git_op_in_flight(source_id) guards every path that would free a pygit2 handle or delete a clone dir while a thread may still be using it.
• Bounded-concurrency scope sweep (per-scope failures isolated) — this is also the parallel-boot fix; see §6.

2. Fleet-wide cache purge — two-phase (deferred items 1 + 3)

PR2's purge was process-local: only the worker serving the DELETE dropped its caches, so the leader — which populates most of repos/repo_locks/repos_last_fetched — leaked until restart. A non-leader DELETE also rmtree'd the shared clone tree, breaking the "only the leader mutates the clone tree" invariant.

New every-worker channel SCOPES_PURGE_CHANNEL (__opal_scope_purge__, following the __opal_stats_* convention), carrying a two-phase purge:

1. Routes publish a purge request (confirmed=False).
2. Only the leader acts on requests: sibling-check under lock_source, then disk work — in a background task, never on the publish path (publish() awaits subscriber callbacks inline, so an inline handler would put a lock wait on DELETE/PUT latency).
3. The leader broadcasts the confirmation (confirmed=True), and that is what every worker's memory handler acts on.

Phase 2 exists because purging on the raw request over-purges: a source shared by a sibling scope had its cache dropped while the sibling still used it. Only the leader can sibling-check, so only the leader may authorize a memory purge.

• Repoint (item 3): PUT /scopes with a changed source_id publishes a purge for the old source. Same channel, same handlers.
• Deferred-purge split: when a git op is still in flight, the clone dir and pygit2 handle wait for the sweep, but the lock/timestamp entries — which the thread never touches — drain immediately and the purge still confirms.
• Mixed-fleet safe by construction: old workers aren't subscribed; confirmed is additive with a False default.

3. Scope-liveness check before clone (item 2)

A sync that loaded a scope before its DELETE and reached it mid-sync re-cloned the dead scope and re-populated the caches. GitPolicyFetcher now takes a liveness probe, checked under lock_source immediately before the clone; ScopesService supplies one that re-reads from Redis and also confirms the scope still points at this source (so a repointed scope's stale sync doesn't resurrect the old clone either). Fails open on store errors.

4. Orphan clone-dir sweep (item 6)

Leader-only reconciliation after boot, periodic, and refresh-triggered syncs: clone dirs under git_sources/ referencing no live scope are reclaimed. One set-difference covers crash orphans, redis-wiped boots, and old-shard dirs after a SCOPES_REPO_CLONES_SHARDS change. A store-scan error aborts the sweep (a transient Redis error must never read as "no scopes exist"); each orphan is re-checked under its own lock and skipped while a git op is in flight.

5. ⚠️ Behavior change — retryable 503 for a live scope's broken clone (item 5)

GET /scopes/{scope_id}/policy:

Case	Before	After
Scope record missing	default scope's bundle	unchanged
Record present, clone invalid/vanished	default scope's bundle	503 + Retry-After: 5
Record present, make_bundle raises raw OSError	unhandled 500	503 + Retry-After: 5

A live tenant was briefly served another tenant's policy. The condition is transient by construction (recovery or the next sync re-creates the clone), so a retryable error is the honest answer. opal-client tolerates this: its PolicyFetcher retries with tenacity backoff and a failed cycle is skipped, not fatal — it does not read Retry-After (uses its own backoff). Direct consumers of this endpoint should expect 503 and retry.

6. Parallel scope loading — the ~20-minute boot fix (folds in PR4)

sync_scopes (boot preload and every periodic/refresh sweep) was serial: one await sync_scope(...) per scope. With a fleet of repos that meant a ~20-minute boot, and — once §1 added a per-fetch timeout — a serial loop would still stall healthy scopes behind each unreachable repo's timeout. So the resilience fix in §1 requires concurrency to actually deliver "one offline repo can't block the others"; the two ship together and PR4 (which planned this same work as a standalone change) is closed as redundant.

The sweep runs in two phases, each asyncio.gather under its own semaphore:

• Phase 1 — distinct repos (network clone/fetch). Bounded by SCOPES_GIT_MAX_WORKERS. Combined with the zombie-aware executor (§1), a hung remote consumes only its own slot until its timeout, then releases it.
• Phase 2 — scopes reusing an already-handled repo (local change-check only). In the common case _should_fetch returns False, so there's no network fetch — just a disk open + change-check + notify. It therefore gets a separate, wider bound (max(SCOPES_GIT_MAX_WORKERS, 32)) instead of inheriting phase 1's network cap; 32 matches asyncio's default thread-pool ceiling, which is what actually limits those disk opens. Any rare re-fetch phase 2 does trigger (a branch missing after phase 1) is still throttled by the inner live-ops semaphore, so the network is never over-driven.

Per-scope failures stay isolated (ScopeNotFoundError from a mid-sweep delete is skipped; other exceptions are logged), and sync_scope re-reads each scope by id right before use, so a delete that lands after the snapshot never re-clones a dead scope.

No new config key — the split is internal. SCOPES_GIT_MAX_WORKERS remains the single operator lever for git concurrency; a dedicated sync-concurrency knob was considered (PR4's SCOPES_SYNC_CONCURRENCY) and deliberately dropped to avoid a redundant, easily-misconfigured second bound.

⚠️ Caveat — the timeout is soft, not a hard kill

The timeout unblocks the event loop and the awaiting coroutine, but the underlying pygit2 call keeps running on its private daemon thread until the OS network timeout. Daemon threads never block shutdown, and git_op_in_flight keeps anything from freeing that repo meanwhile — but worst-case thread count during an outage is SCOPES_GIT_MAX_WORKERS plus the number of lingering timed-out ops. Hard-kill via subprocess is explicitly out of scope.

New config keys (opal-server, server-only, additive)

Env var	Type	Default	Purpose
OPAL_SCOPES_GIT_FETCH_TIMEOUT	float (s)	120.0	Hard timeout for a single scope git clone/fetch. 0 = no limit.
OPAL_SCOPES_GIT_MAX_WORKERS	int	10	Bounds live git ops. Timed-out ops release their slot, so hung remotes never starve healthy scopes.
OPAL_SCOPES_PURGE_CHANNEL	str	__opal_scope_purge__	Worker-to-worker channel for the fleet-wide cache purge.

Invariants (enforced and tested)

1. repo_locks entries are popped only while holding that source's lock.
2. forget_repo / rmtree never run while a git op is in flight for that source.
3. Only the leader mutates the clone tree.

The purge confirmation is published while holding lock_source — it frees this process's cached handle via the inline local subscriber, so releasing first opened a use-after-free against a re-created scope's _notify_on_changes (which holds the handle across an await, then calls set_target() on it).

Verification

• opal-server unit suite: 151 passed. The §6 per-pass split adds sync_scopes_perpass_test.py (2 tests: phase 2 runs wider than the git cap; phase 1 still respects it) — both green, and the related sync/delete/preload suites pass unchanged.
• app-tests/git-leak docker bed: all 10 acceptance gates green — 19/19 in the main phase plus test_offline_repo_does_not_block_healthy_scopes. The bed (not the unit suite) is the real gate: it caught three product bugs unit tests could not — the purge blocking DELETE/PUT on the publish path, the shared-source over-purge, and the thread-pool starvation.
• Bed changes: OPAL_SCOPES_GIT_FETCH_TIMEOUT=10 for realistic serve windows; the delete-vs-inflight-sync churn exclusion lifted (closed by the liveness probe); the repoint gate rewritten as a green guard (it was racing a purge that now completes in ~4ms); allow_worker_restart on the force-recreating boot test; chown after the restoring compose cp.

Known limitation (pre-existing, not introduced here)

test_server_recovers_after_postgres_bounce can fail in a full-file bed run. A worker receives backbone messages only if its broadcaster reader is running, which happens via STATISTICS_ENABLED (default False) or a connected websocket client — subscribe() alone does not start it. The bed has no opal-client service and statistics off, so a non-leader worker there is deaf to the backbone and a publish it buffers during an outage never replays. With clients connected (or statistics on) the reader runs and the replay works — verified in the logs of a passing run. Unrelated to this PR's changes; tracked separately.

Consumer surface

packages/opal-client and packages/opal-common are untouched. No OPAL_* key renamed or removed; all three new keys are additive with behavior-preserving defaults. The 503 above is the one intentional contract change.

🤖 Generated with Claude Code

[linear-code]

PER-15157

[netlify]

✅ Deploy Preview for opal-docs canceled.

Name	Link
🔨 Latest commit	d2ad045
🔍 Latest deploy log	https://app.netlify.com/projects/opal-docs/deploys/6a60de75d5b2680009c0a6e6

[Copilot]

Pull request overview

This PR improves opal-server resilience when syncing scope policy repos by ensuring git clone/fetch operations can’t block indefinitely or starve the server’s shared executor.

Changes:

• Add a dedicated, bounded ThreadPoolExecutor and run_in_git_executor(...) helper to run blocking pygit2 operations with an asyncio.wait_for timeout.
• Apply the helper + new timeout config to scope repo clone and fetch paths.
• Add server config keys for timeout and executor sizing, plus focused unit tests for timeout behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
packages/opal-server/opal_server/git_fetcher.py	Introduces dedicated git executor + timeout helper; routes scope clone/fetch through it.
packages/opal-server/opal_server/config.py	Adds SCOPES_GIT_FETCH_TIMEOUT and SCOPES_GIT_MAX_WORKERS configuration.
packages/opal-server/opal_server/tests/git_executor_test.py	Tests config defaults and run_in_git_executor basic behavior.
packages/opal-server/opal_server/tests/fetch_timeout_test.py	Tests that a hanging git op times out quickly (doesn’t block).
.claude/plans/docs/05-config-reference.md	Internal config reference entry for the new env vars and caveat.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zeevmoney]

Review notes — overlap + gate location (planned with the opal-development skill: references/add-config-key.md, references/debug-pubsub.md)

Faithful to the PR3 plan, with two improvements over it: normalizing asyncio.TimeoutError → builtin TimeoutError (so callers catch the builtin on 3.9/3.10 where they're distinct), and moving repos_last_fetched to after a successful fetch — a timed-out fetch no longer falsely marks the source "fresh," and it's lock-safe because _should_fetch runs inside the per-source repo_lock. The two config keys follow references/add-config-key.md (server-only OpalServerConfig, bare names, mandatory descriptions, no double-prefix).

Three things to resolve:

1. Direct overlap with Fix git clone/fetch hanging indefinitely on unreachable repos #875 (PER-13817). Same root cause — pygit2 clone_repository / remotes.fetch going through run_sync with no timeout — and the same edited blocks in git_fetcher.py (_clone and fetch_and_notify_on_changes). They cannot both merge; whichever lands second will conflict. This PR is the stronger of the two:

   • Default-on protection: OPAL_SCOPES_GIT_FETCH_TIMEOUT=120 vs Fix git clone/fetch hanging indefinitely on unreachable repos #875 reusing POLICY_REPO_CLONE_TIMEOUT (default 0 = no timeout, so Fix git clone/fetch hanging indefinitely on unreachable repos #875 protects nothing unless an operator opts in).
   • Thread isolation: a dedicated bounded ThreadPoolExecutor for scope git work, so a hang can't starve bundle-serving — Fix git clone/fetch hanging indefinitely on unreachable repos #875 stays on the shared default executor. (This is the starvation concern from the offline-repo test in test(opal-server): git leak/resilience test environment (PR1) #922.)
   • Best-effort boot: _clone catches (GitError, TimeoutError), logs, and skips the scope rather than letting the caller crash.

   Recommend closing Fix git clone/fetch hanging indefinitely on unreachable repos #875 in favor of this PR.

2. Its regression gate lives in PR1 (test(opal-server): git leak/resilience test environment (PR1) #922). test_offline_repo_does_not_block_healthy_scopes is the fail-now/pass-after gate for this fix and only exists on test(opal-server): git leak/resilience test environment (PR1) #922. So this can't be validated end-to-end until test(opal-server): git leak/resilience test environment (PR1) #922 merges (or is merged into this branch). Suggested order: test(opal-server): git leak/resilience test environment (PR1) #922 → this.

3. This PR is currently check-blocked by branch protection (required checks / review), not by a merge conflict — needs CI green + an approval.

Minor:

• The dedicated pool reads SCOPES_GIT_MAX_WORKERS once, lazily, and caches the executor for the process lifetime — it isn't runtime-reconfigurable and is never shut down. Matches the plan's design; worth a one-line note in the docstring.
• The line numbers cited in .claude/plans/docs/05-config-reference.md for the two keys are stale vs where they actually land on this branch (cosmetic).

[dshoen619]

Thanks @zeevmoney — addressed in bad21c1.

Minor items

• Executor lifetime docstring — added a note to _get_git_executor: SCOPES_GIT_MAX_WORKERS is read once on first use, the executor is cached for the process lifetime (not runtime-reconfigurable) and is never explicitly shut down, matching the PR3 design.
• Stale line numbers — good catch. They'd drifted after the master merge shifted config.py. Fixed the 05-config-reference.md refs: SCOPES_GIT_FETCH_TIMEOUT 150-156 → 196-202, SCOPES_GIT_MAX_WORKERS 157-163 → 203-209.

Things to resolve

1. Overlap with Fix git clone/fetch hanging indefinitely on unreachable repos #875 (PER-13817) — agreed this is the stronger fix (default-on OPAL_SCOPES_GIT_FETCH_TIMEOUT=120, dedicated bounded pool, best-effort boot). Plan is to close Fix git clone/fetch hanging indefinitely on unreachable repos #875 in favor of this.
2. Regression gate in test(opal-server): git leak/resilience test environment (PR1) #922 — agreed; the fail-now/pass-after gate (test_offline_repo_does_not_block_healthy_scopes) lives on PR1, so the plan is to land test(opal-server): git leak/resilience test environment (PR1) #922 → this and validate end-to-end there.
3. Check-blocked — required checks (E2E, builds 3.9–3.12, pre-commit) were green on the prior head and are re-running on bad21c1e; the remaining blocker is the required approving review. A review once it's green would unblock it.

Also confirmed the two improvements you flagged are in place: asyncio.TimeoutError → builtin normalization, and repos_last_fetched moved to after a successful fetch.

[zeevmoney]

Review (PER-15157 / PR3) — git resilience: never stuck on an offline repo

What this PR does. Makes scope git clone/fetch resilient to unreachable repos. It moves scope git work off the shared default executor onto a dedicated daemon-thread ThreadPoolExecutor (SCOPES_GIT_MAX_WORKERS, default 10), wraps each clone/fetch in a soft per-op timeout (SCOPES_GIT_FETCH_TIMEOUT, default 120s) via run_in_git_executor using asyncio.wait (so a timeout unblocks the event loop without cancelling the still-running pygit2 call), adds a per-repo in-flight guard so a lingering timed-out op is not touched concurrently (pygit2 Repository is not thread-safe), records repos_last_fetched only on fetch success, makes scope sync concurrent (bounded by the pool via a semaphore), and adds fork-safety (os.register_at_fork reset + shutdown_git_executor() after pre-fork preload). Plus two unit-test files and a private config-reference doc.

Verdict: REQUEST_CHANGES. Per the severity rule, there is one Postable HIGH finding (credential-redaction bypass in the new log lines) → REQUEST_CHANGES. Independently, the PR is not mergeable: GitHub reports CONFLICTING and git_fetcher.py has a content conflict with master (see Blockers).

The core design is sound and a real improvement over master: isolating git work onto a dedicated pool means a hung clone/fetch can no longer starve bundle serving or the event loop (on master these share the default executor via run_sync, so one offline repo hangs the whole server). The soft-timeout + single-flight + fork-safety mechanics are correct and well-tested (test_busy_key_stays_in_flight_until_call_returns, test_hanging_git_op_raises_timeout, the config-default tests). The two prior open Copilot threads are already addressed by the current code (asyncio.get_running_loop() replaces get_event_loop; repos_last_fetched is now written only after a successful fetch) — not re-raised.

Findings

Postable:

#	Severity	File:Line	Category	Description
1	HIGH	packages/opal-server/opal_server/git_fetcher.py:412 (also 376, 465)	Security	New skip/timeout/clone-error log lines log the raw self._source.url; master redacts every repo URL in this file via redact_url(). Once merged these are the only un-redacted URL logs → credential exposure. redact_url isn't imported.
2	MEDIUM	.claude/plans/docs/05-config-reference.md:27	Doc accuracy	The ceil(offline / workers) × timeout boot/poll bound is optimistic: timed-out ops keep their pool thread until the OS network timeout, so with offline >= workers healthy repos queue behind lingering threads longer than the stated bound.

Informational (not posted inline):

#	Severity	File:Line	Category	Description
3	LOW	packages/opal-server/opal_server/git_fetcher.py:52-91	Maintainability	_DaemonThreadPoolExecutor._adjust_thread_count reimplements CPython concurrent.futures.thread internals. It falls back to super() if _worker/_threads_queues disappear, but a signature change to _worker (name kept, args changed) would pass the hasattr check yet break. Acceptable given the fallback + # pragma: no cover, but a fragility to track across Python upgrades (repo targets 3.9–3.12).
4	LOW	.claude/plans/docs/05-config-reference.md (new tracked file)	Cross-PR coordination	PR #922 adds .claude/ to .gitignore while this PR tracks a file under .claude/plans/docs/. Not a conflict on master today (.claude/ isn't ignored), but once both land the tracked doc sits under a gitignored path — coordinate.

Design note (not a blocker). The "never stuck" guarantee delivered is: the event loop / HTTP surface / bundle serving never block, and each sync slot stalls at most SCOPES_GIT_FETCH_TIMEOUT. It is not that a fixed pool immediately reclaims capacity on timeout — a timed-out op lingers on its thread until the OS network timeout (by design; pygit2 can't be cancelled). For the realistic case (a few offline repos among many healthy) this is fine — the offline ops each hold one lingering thread and the rest of the pool serves healthy repos. The pathological case (offline repos >= pool size) can saturate the git pool; the daemon threads still let the process exit promptly and other server work is unaffected. Finding #2 asks the doc to reflect this precisely.

Blast radius: Production opal-server git-fetcher + scopes sync path — affects every scoped deployment's boot and poll behavior. No client-facing symbol from references/pdp-impact.md §3 is renamed/removed (new module-level helpers + two config keys only; GitPolicyFetcher public shape unchanged), so no PDP import-surface break. Two new OPAL_* keys are additive with sane defaults, no env-name collision, no OPAL_ double-prefix. Pub/sub topology unchanged — this only changes how the leader fetches git; the scope publish path is untouched, so PDP policy-update propagation is unaffected except that offline repos now fail fast (skip + retry) instead of hanging.

Isolation / scope: Well-isolated. All changes serve the stated purpose (git resilience); no unrelated refactors, no half-done work.

Blockers:

• CONFLICTING / not mergeable. git_fetcher.py has a content conflict with master (master added redact_url() to the log lines this PR also edits). Rebase/merge master and resolve — and when doing so, apply redact_url() to the new log lines too (finding #1). Reviewed here against the merge-base three-dot diff (d30e462...d31a0b63), which is unaffected by the conflict.

[Zivxx]

Adversarial review round — findings + fixes (all pushed, CI green)

Ran a multi-agent review over the whole branch (correctness, security, unit-test coverage, docker-bed coverage, code quality) and acted on everything confirmed. Summarizing here so the earlier CHANGES_REQUESTED can be re-reviewed against current code — all of your inline comments were already addressed by the rework (get_running_loop, redacted URLs, start-time-on-success timestamp, monotonic clock in the timeout test).

Two confirmed blockers — fixed

1. Arbitrary path deletion via a forged purge message (d577103d). ScopePurgeCommand.clone_path arrived over the __opal_scope_purge__ pub/sub channel and was fed unvalidated into shutil.rmtree (leader) and Repository.free() (workers). With the topic ACL fail-open by default, any client token could publish a forged purge and delete an arbitrary path as the opal user. Fix: the path is now derived locally from the validated source_id (^[0-9a-f]{64}-\d+$, confined to git_sources/); the wire clone_path is never used for I/O. Forged-path regression test included and mutation-verified.

2. Hung fetch could block graceful shutdown (85dc358b). The daemon git executor still registered its threads in concurrent.futures' global _threads_queues, which the stdlib atexit handler join()s regardless of daemon=True — so a lingering (timed-out) fetch would wedge a rolling restart, i.e. the exact "stuck on an offline repo" hang, relocated to shutdown. Reproduced empirically on 3.11/3.12/3.13. Fix: drop the registration (normal shutdown uses self._threads + queue sentinels, unaffected).

Should-fixes — fixed

• Fleet-purge topology gap (842a0988): gunicorn preloads clones in the master pre-fork, so every worker inherits GitPolicyFetcher.repos handles; a client-less non-leader worker (broadcaster reader not started — stats off by default) never receives the purge confirmation and would pin those handles for life. Since non-leaders never populate the cache otherwise (sync is leader-only; bundle-serving opens fresh handles), the fix is to clear the fetcher caches in the master before fork — complete regardless of worker count. On-disk clones are preserved.
• Semaphore leak on cancellation (0c52dd85): the live-op permit is now released via try/finally, so a cancelled await can't permanently shrink git concurrency.
• Repoint PUT ambiguous-outcome parity (2c02c53c): the old-source purge now publishes even if scopes.put() commits then raises, matching the delete path's existing hardening.
• Coverage + clarity (c3eb19ba): added the liveness-probe not-found path test through the real closure, a boot-sweep logs-on-failure test, corrected the fetch-timeout config description ("soft", not "hard"), and commented the divergent in-flight branches.

What held up under review (no change needed)

The 503-on-clone-vanish except tuple (empirically probed), the liveness probe gating both clone paths, sibling-delete convergence, the lock-identity recheck, the confirmation-under-lock UAF fix, credential redaction, and the two-phase purge routing all verified sound. Unit mutation audit: 0 weak tests among existing ones. Docker bed: all 10 acceptance gates green.

Deferred (out of this PR's scope — flagging for a follow-up)

• Fail-open pub/sub topic ACL: a client token without a permitted_topics claim can publish to any topic, including internal __-prefixed ones. Pre-existing and affects every internal channel, not just this one — blocker WIP: Opal refactor (policy fetcher, tree structure) #1's fix neutralizes the purge-channel consequence, but hardening the ACL itself is a broader change worth its own ticket.
• test_server_recovers_after_postgres_bounce can flake in a full bed run: a non-leader, client-less worker's broadcaster reader never starts, so a publish it buffers during an outage has no flush path (same reader-liveness mechanism as the topology gap above). Pre-existing, not introduced here; README note pending.

Full CI matrix (3.9–3.12), both E2E suites, and pre-commit are green.

[zeevmoney]

Review — PR3 (git resilience + fleet-wide purge) @ 8400a75e

21 inline comments: 4 HIGH, 8 MEDIUM, 9 LOW. Requesting changes on the HIGH set. Everything below was verified against the source on this branch (and against the installed fastapi_websocket_pubsub / CPython internals where the design depends on library behaviour), not against the PR description.

HIGH — must resolve

1. purge.py:73 — purge commands are accepted from any connected pub/sub peer. The new control channel rides the same PubSubEndpoint clients connect to; RpcEventServerMethods.publish is client-callable and _verify_permitted_topics returns immediately when the token has no permitted_topics claim — which the default POST /token mint omits. An ordinary client token can therefore publish confirmed: true for any known repo URL and drive Repository.free() on every worker in the fleet, or spend the leader's Redis + broadcast budget with confirmed: false. The sibling-check does keep a live scope's clone from being rmtree'd — that part holds.
2. purge.py:62 — handle_purge_message frees the pygit2 handle with no lock_source. This is exactly the use-after-free the leader path documents at 215-221; that protection only covers same-process inline delivery. A confirmation from another pod's leader (per-host leader lock ⇒ one leader per pod) or a replayed confirmation after a re-create both land on the reader task unserialized, and git_op_in_flight is already False during the _notify_on_changes window that holds the handle across an await.
3. purge.py:213 — the confirmation is published after repo_locks.pop, and lock_source treats a missing entry as free (fresh setdefault → uncontended acquire → identity re-check passes). The pop is equivalent to releasing, so the await publish yield point is open to the exact race the comment says it prevents. sweep_orphans has the same ordering; origin/master popped last.
4. task.py:117 — the pre-fork teardown frees handles a lingering timed-out git thread is still using (reset_caches has no git_op_in_flight guard, and shutdown_git_executor erases the evidence first), and gunicorn's when_ready forks the workers while that libgit2 thread is alive. Both are reachable in the offline-repo scenario this PR targets.

MEDIUM — worth fixing before merge

api.py:157 a failed PUT publishes a repoint purge whose "sibling-check makes it safe" argument inverts under the Redis outage that caused the failure (fail-open → delete a live scope's clone) · service.py:213 the DELETE disk purge now depends on the leader's broadcaster reader, which does not run without STATISTICS_ENABLED or a client connected to that worker — master did it locally, so this is a regression for that shape · purge.py:195 the deferred-purge backstop (sweep_orphans) only runs at boot under the default POLICY_REFRESH_INTERVAL=0 · purge.py:265 O(M²) full-store rescans + is_dir on the event loop · git_fetcher.py:229 zombie threads/FDs bounded only by source count, with no libgit2 transport timeout so a stalled remote pins them forever · plus three test/doc gaps (dropped PR2 regression tests, untested start()/periodic-sweep wiring, a self-declared private doc as the only home for three new operator-facing keys while configuration.mdx goes stale).

Notes

• Blast radius: server-only — opal-client/opal-common untouched, no OPAL_* key renamed or removed. The 503 flip is the one intentional contract change and opal-client's tenacity retry does tolerate it; direct consumers of GET /scopes/{id}/policy need to expect it. The purge channel is new fleet-wide state on the broadcaster, which is where findings 1–3 land.
• Scope: in charter overall. The one overreach is committing .claude/plans/docs/05-config-reference.md (a file that calls itself private) into the OSS tree while the public config reference gains none of the three new keys.
• The app-tests/git-leak README gate matrix is now stale in rows this PR made green (rows 48, 62-63 still say the orphan sweep does not exist), along with several "RED until PR3" docstrings in test_transitions.py / test_boot_states.py. Not flagged inline since those lines are untouched by the diff — happy to push the doc fix if you want it in this PR.

[zeevmoney]

[HIGH] Purge commands are accepted from any connected pub/sub peer

Problem: handle_purge_message trusts confirmed as it arrives on the wire, and the channel rides the same PubSubEndpoint that clients connect to. RpcEventServerMethods.publish(topics, data) is exposed to every websocket peer, and EventNotifier.notify enforces channel restrictions only through PubSub._verify_permitted_topics (pubsub.py:340-351), which returns immediately when the token carries no permitted_topics claim — the default POST /token mint omits it (security/api.py:26, AccessTokenRequest.claims defaults to {}). So any holder of an ordinary OPAL client token can publish

{"source_id": "<sha256(repo_url)>-0", "clone_path": "x", "scope_id": "x", "reason": "x", "confirmed": true}

to __opal_scope_purge__ and make every worker in the fleet run purge_local_memory -> Repository.free() for a live source at a moment of their choosing — the broadcaster's ALL_TOPICS subscriber re-broadcasts it fleet-wide. With confirmed: false they instead cost the leader a full scopes.all() Redis scan plus a fleet-wide confirmation broadcast per message, unrate-limited, behind an unbounded _pending_purges set.

Client-publishable topics are pre-existing OPAL behaviour; what is new here is that a filesystem/cache control plane now listens on that bus. (Credit where due: the sibling-check does stop a forged request from rmtree-ing a live scope's clone — that part holds.)

Suggestion: do not accept purge commands from a client-reachable endpoint. Either require a server-minted marker inside ScopePurgeCommand and verify it in both handlers, or move the fleet purge onto a server-only broadcaster channel that RpcEventServerMethods.publish cannot reach. At minimum, make _verify_permitted_topics fail closed for internal __opal_* topics regardless of the claim.

[zeevmoney]

[HIGH] Worker handler frees the shared pygit2 handle outside lock_source — the exact use-after-free the leader path guards against

Problem: purge_source_if_unshared goes out of its way to publish the confirmation while still holding lock_source, and the comment at purge.py:215-221 says why: _notify_on_changes holds the cached handle across an await and then calls set_target() on it, so freeing it underneath is a use-after-free. That argument only covers same-process, inline delivery. A confirmation arriving over the broadcaster is dispatched on the reader task with no lock at all, and handle_purge_message takes none.

git_op_in_flight is not a substitute: _mark_git_op_done runs in run_in_git_executor._runner's finally, so it is already False while _notify_on_changes (git_fetcher.py:663-686) still holds repo and local_branch across await self.callbacks.on_update(...) — which for NewCommitsCallbacks builds a bundle and publishes over the backbone (tens to hundreds of ms).

Two reachable triggers:

1. Leader election is a per-host fcntl lock, so a multi-pod deployment without a shared volume has one leader per pod, each holding a warm handle for the same source_id (same key, same relative clone path). Pod A's confirmed=True lands on pod B mid-sync.
2. The reconnect replay buffer (pubsub_resilience.py) can redeliver a confirmed purge after the same URL/branch has been re-created — source_id is deterministic from url+branch — so a stale confirmation frees a live handle.

Suggestion: take GitPolicyFetcher.lock_source(cmd.source_id) inside handle_purge_message around purge_local_memory. Note the leader publishes its own confirmation while holding that lock, so inline local delivery would self-deadlock — split local from remote delivery, or drop the publish-under-lock trick and serialize on the receiving side only (see the comment on line 213). A generation/epoch field on ScopePurgeCommand, checked before applying, would also close the replay case.

[zeevmoney]

[HIGH] repo_locks is popped before the confirmation publish, so the "published under the lock" protection does not hold

Problem: the comment two lines below states the confirmation must be published while the source lock is held, because releasing first would let a re-created scope's sync cache a fresh handle and enter _notify_on_changes while this confirmation frees it underneath.

But lock_source (git_fetcher.py:379-384) treats a missing dict entry as free:

while True:
    lock = GitPolicyFetcher.repo_locks.setdefault(source_id, asyncio.Lock())
    async with lock:
        if GitPolicyFetcher.repo_locks.get(source_id) is lock:
            yield
            return

After the pop at line 213 (and line 200 in the in-flight branch), a newcomer's setdefault mints a fresh lock, acquires it uncontended, passes the identity re-check and enters. The pop is therefore equivalent to releasing the lock, and the await ...publish(...) at line 223 is a yield point where exactly the scenario the comment warns about can run. sweep_orphans has the same ordering (pop at 292, publish at 294). origin/master's _purge_source_cache_if_unshared did the pop last, after all other work.

Suggestion: publish the confirmation first and make the repo_locks.pop the last statement in the critical section, in both purge_source_if_unshared branches and in sweep_orphans.

[zeevmoney]

[HIGH] Pre-fork teardown frees handles that a lingering timed-out git thread is still using, and forks with that thread alive

Problem: two distinct hazards, both in the block starting here, and both reachable in precisely the offline-repo scenario this PR exists for (any restart with a persistent BASE_DIR and an unreachable remote):

1. Free-while-in-use. After a preload fetch hits SCOPES_GIT_FETCH_TIMEOUT, its pygit2 call keeps running on a private daemon thread — by design. shutdown_git_executor() then does _git_busy.clear(), destroying the only record that the thread is alive, and GitPolicyFetcher.reset_caches() (line 127) calls forget_repo -> Repository.free() on every cached handle with no git_op_in_flight guard — unlike purge_local_memory (purge.py:61-62), which the PR guards precisely because "freeing a pygit2 handle a pool thread still uses ... is a crash risk". The submitted callable is the bound method of that cached handle (repo.remotes[self._remote].fetch, git_fetcher.py:448-453), so this frees the repository the thread is inside. A SIGSEGV here is not containable by the except Exception around free().

2. Fork with a live thread. scripts/gunicorn_conf.py calls preload_scopes() from when_ready, which runs in the arbiter before any worker is forked. These threads are deliberately not registered in concurrent.futures.thread._threads_queues and asyncio.run's executor shutdown no longer joins them, so the fork can happen while a libgit2/OpenSSL thread holds internal mutexes — the classic fork-in-threaded-process hazard, which is the same "stuck at boot" failure class this series is fixing, relocated to the children.

Suggestion: skip forget_repo for any path whose source_id is still git_op_in_flight, and clear _git_busy only in the child (_reset_git_executor_after_fork already does). Before returning from preload_scopes, wait a bounded time for _git_busy to drain and leave still-busy sources alone; running the preload in a short-lived subprocess removes both hazards outright.

[zeevmoney]

[MEDIUM] A failed PUT publishes a repoint purge whose safety argument does not hold

Problem: the publish is in a finally, so it fires even when await scopes.put(scope_in) raised. The comment above says over-publishing self-heals "since the leader sibling-checks and a source still shared by another scope survives". That does not hold for the case that produced the failed put: find_scope_sharing_source (purge.py:119-124) catches any exception from scopes.all() and returns None, which purge_source_if_unshared treats as "no sharer" -> forget_repo + rmtree. ScopeRepository.put and .all hit the same Redis, so the outage that makes put() raise is exactly the one that makes the sibling check fail open into a purge. The leader also ignores cmd.reason, so it cannot distinguish "record known gone" (delete — purging defensively is right) from "record known live" (failed repoint — purging is wrong).

Net effect: a Redis blip during a scope PUT can delete a still-live scope's clone dir.

Suggestion: make the fail-open bias depend on cmd.reason — for reason == "repoint" a raising sibling scan should KEEP the clone (the record is known live, and the orphan sweep is the backstop); keep today's defensive purge for reason == "delete". Alternatively publish the repoint purge only on the success path.

[zeevmoney]

[LOW] register_at_fork installs only after_in_child, so a fork while _git_busy_lock is held deadlocks the child

Problem: _reset_git_executor_after_fork acquires _git_busy_lock (line 130). threading.Lock state is inherited across fork(): if another thread held it at fork time, the child's copy is permanently locked and the handler blocks forever in the post-fork path — the worker never starts. That lock is taken by non-forking threads: _mark_git_op_done runs on the git daemon thread (from _runner's finally), and preload_scopes can leave such threads alive in the gunicorn arbiter at exactly the moment it forks workers.

The stdlib module this file already imports does it the complete way:

os.register_at_fork(before=_global_shutdown_lock.acquire,
                    after_in_child=_global_shutdown_lock._at_fork_reinit,
                    after_in_parent=_global_shutdown_lock.release)

Suggestion: register the same triple — before=_git_busy_lock.acquire, after_in_parent=_git_busy_lock.release, and in the child reinit the lock before touching _git_busy.

[zeevmoney]

[LOW] The fallback guard checks that CPython internals exist, not that they still have the shape this code assumes

Problem: the docstring promises "If a future CPython changes the internals we rely on, we fall back to the stdlib (non-daemon) behaviour", but the guard only tests hasattr(cf_thread, "_worker") and hasattr(cf_thread, "_threads_queues"). On CPython 3.14 both still exist, while _worker's signature changed and ThreadPoolExecutor.__init__ no longer sets _initializer/_initargs — so the override raises AttributeError out of submit(), which lands in the except BaseException at 236-240 and makes every scope clone/fetch fail. 3.9-3.13 are unaffected (verified: 3.13 still has the 4-arg _worker), and setup.py classifiers stop at 3.12 — but python_requires is ">=3.9" with no upper bound, so a 3.14 install is permitted and would silently never sync.

Suggestion: validate what is actually used — e.g. inspect.signature(cf_thread._worker) parameter count plus hasattr(self, "_initializer") — and fall back to super()._adjust_thread_count() on any mismatch. Adding an upper bound to python_requires would also make the constraint explicit.

[zeevmoney]

[LOW] Description is a single 438-character line

Problem: awk 'length > 100' flags this line at 438 characters — 4.4x the project's 100-char limit and the longest line in the file by ~3x. It is also inconsistent with SCOPES_GIT_FETCH_TIMEOUT immediately above (lines 217-221), which is wrapped into implicit-concatenated segments. Black does not reflow string literals, so nothing catches this automatically.

Suggestion: wrap it the same way the adjacent key is wrapped, and move the multi-clause caveat about worst-case thread count into a code comment next to the declaration.

[zeevmoney]

[LOW] Dangling spec reference and two off-by-one file:line citations

Problem: this line ends "See spec §6", but no such file exists on this branch: git ls-files matches nothing for pr3 or superpowers, and docs/superpowers/specs/ does not exist (the PR description links docs/superpowers/specs/2026-07-19-pr3-deferred-items-design.md, which is not in the diff either). test_leak.py:198 adds a second dangling reference to .superpowers/sdd/bed-deep-dive.md.

Separately, the doc's stated purpose is letting a contributor jump straight to the Confi declaration, but two of its three citations land on the previous key's closing paren: SCOPES_GIT_MAX_WORKERS is at config.py:223-227 (cited 222-226) and SCOPES_PURGE_CHANNEL at 312-319 (cited 311-318).

Suggestion: land or drop the referenced spec/deep-dive files, and cite symbol names rather than line ranges — the ranges go stale on the next edit to config.py.

[zeevmoney]

[LOW] The replacement route test pins a configuration production cannot reach

Problem: test_delete_route_purges_fetcher_caches was replaced by a test that asserts the caches are LEFT populated, which only holds because _client() wires pubsub_endpoint=None (line 51). Production always supplies self.pubsub.endpoint to both ScopesService and init_scope_router (server.py:275-287), so this locks in the behaviour of a branch only test code reaches — and the assertions would keep passing if the DELETE route stopped publishing a purge command entirely. The service-level publish is covered (delete_scope_cache_purge_test.py:71), but nothing covers it through the route.

Suggestion: wire a fake pubsub endpoint into _client() and assert the route publishes ScopePurgeCommand(source_id=..., reason="delete", confirmed=False) on SCOPES_PURGE_CHANNEL; keep the pubsub_endpoint=None case as a separate "degraded mode does not crash" test.