fix(pce): PCE lock, failed-onboarding quarantine, and clientId-only PCE

[oblinder]

Summary

This branch fixes the PCE concurrency bugs and the cleanup of a failed onboarding (handoff 11). It also makes the PCE take only the clientId (handoff 15).

Always DENY by default

• Remove the configurable default effect. The deployed Rego denies every (role, scope) pair that no rule mentions.

PCE lock

• One module-level lock serializes compute_and_apply, decommission and quarantine. Before this, two runs that wrote to one shared SPM could lose the rules of the first run. The PRB (LLM) work stays outside the lock.

Quarantine a failed onboarding (UC1)

• On a build failure, the orchestrator rolls back what Provision created and disables the client. Then the PCE quarantine deletes the SPM of the service, removes its roles from the other SPMs (also the roles that the rollback deleted), and replaces the CR of an agent with a no-rules CR that denies every request.
• quarantine runs in a finally, so a failed rollback cannot leave a first onboarding fail-open.
• The routing guard drops every later rule that touches a disabled or deleted service, except the rules of the focus service. Only a successful re-onboarding lifts a quarantine.

The PCE takes only the clientId

• quarantine, decommission and compute_and_apply(focus_service=…) take the clientId (Service.serviceId, the SPM key). The Keycloak UUID is only for IdP reads.
• onboard_service reads the Service once, before Provision, and returns (rules, False, client_id). The route and the NATS consumer pass that clientId as focus_service.
• New ServiceUuid / ClientId NewTypes mark the two kinds of id.
• The consumer's _handle is now its only subject switch.

PDP and cluster

• The agent outbound allows the MCP session messages. By-id policy routes accept a slash-bearing agent id.
• Fixes to k8s/opa-kind-enable.sh (current operator, plugin build tags).

Tests and docs

• New system rung: test_uc1_onboard_failure_rollback.py (quarantine and lift).
• New unit tests fail if a UUID reaches a clientId parameter.
• Updated the PCE, UC1, agent and PDP specs and CONTEXT.md (Quarantine, Focus service).

Test plan

• .venv/bin/pytest (unit lane): 954 passed
• .venv/bin/pytest -m system --collect-only -q: 85 collected, no errors
• .venv/bin/pytest -m system -k failure_rollback on the Kind cluster, after a redeploy of the Controller (running now)

🤖 Generated with Claude Code

[oblinder]

Code review: 4 findings. Findings 2 and 4 are high-confidence. Findings 1 and 3 are plausible, but they depend on edge-case state.

[oblinder]

(Plausible) quarantine removes every role that X still has from the policy models of all other services. This includes a role that X reused by name and that another live service Z also holds. Z then loses its own grants on each failed onboarding of X. Before this change, this occurred only on an explicit offboard.

[oblinder]

Fixed in 7409457. _remove_footprint (shared by quarantine and decommission) now skips a role that another service in the catalog also holds. Thus Z keeps its grants. X stays denied: a quarantined agent has the no-rules CR, and an offboarded client cannot authenticate. Tests: test_quarantine_keeps_the_grants_of_a_role_another_service_also_holds, test_decommission_keeps_the_grants_of_a_role_another_service_also_holds.

[oblinder]

(High confidence) The new routing guard filters rules before the code builds the set of roles to purge. In override mode, if every new rule for a role touches a disabled service, the purge for that role does not run, and its old grants stay. At this time this is latent, because update_role and rebuild_policy are stubs that return []. Fix: build the purge set from the unfiltered input rules.

[oblinder]

Fixed in 7409457. _run now builds distinct_roles (the override purge set and the affected seed) from the input rules before the routing guard. Test: test_override_purges_a_role_whose_every_new_rule_the_guard_drops.

[oblinder]

(Plausible) This PR stops the normal apply path from re-deriving an agent that is not in the catalog (affected & catalog.keys()). _quarantine and _decommission (also see line 500) do not apply that filter. Thus, an agent whose client was deleted without an offboard, and whose stored policy model remains, can get its policy CR back.

[oblinder]

Fixed in 7409457. All three paths now derive only live agents through _live_services() (in the catalog and not disabled; the focus service counts as live). This also stops a re-derive from writing over the no-rules CR of a disabled agent. Tests: test_quarantine_never_derives_an_agent_that_is_absent_from_the_catalog, test_quarantine_never_derives_a_disabled_agent, test_decommission_never_derives_an_agent_that_is_absent_from_the_catalog.

[oblinder]

(High confidence) The new config.get_service() call runs before Provision, outside the error handling of Provision. When the IdP is down or the UUID is unknown, the caller now gets a raw RuntimeError (a generic 500). Before this change, the caller got the sanitized HTTPException(502).

[oblinder]

Fixed in 7409457. onboard_service now maps a failed get_service() to HTTPException(502), the same boundary as Provision's classify_service. The test now raises a raw RuntimeError and asserts the 502.

[oblinder]

All 4 review findings are fixed in 7409457. Each inline thread has a reply with the details and the test names.

1. Shared role on quarantine/offboard (plausible): the shared teardown skips a role that another service in the catalog also holds. Thus that service keeps its grants. This also corrects the offboard path, which had the same behavior before this PR.
2. Override purge after the routing guard (high confidence): the purge set now comes from the input rules before the guard.
3. Re-derive of a deleted agent in quarantine/offboard (plausible): all three paths now derive only live agents (in the catalog and not disabled; the focus service counts as live). Thus a re-derive also cannot write over the no-rules CR of a quarantined agent.
4. Raw RuntimeError before Provision (high confidence): a failed get_service() now raises HTTPException(502), as Provision does.

The PCE and UC1 specs are updated to match. Verification: 6 new engine tests fail on the old code and pass with the fix. The 960 offline unit tests pass, and ruff reports no problems. I did not run the system suite (-m system).

[oblinder]

Re-review after 7409457: all 4 findings are fixed. I found no new problems. At the new head, the unit suite passes (960 passed, 305 deselected).

1. Shared role purged on quarantine. Fixed. _remove_footprint now gets the catalog and does not purge a role that another service holds. This is correct because Service.roles lists the roles on each service account. X stays denied: a quarantined agent gets the no-rules CR, and an offboarded client cannot authenticate. Tests cover quarantine and decommission.
2. Override purge set built after the routing guard. Fixed. distinct_roles is now built from the input before the guard. A test shows that a role still loses its old grants when the guard drops all of its new rules.
3. Quarantine and decommission can re-derive a deleted agent. Fixed. All three apply paths now use _live_services, which returns the services that are in the catalog and not disabled. The fix also covers a case I did not report: a disabled agent now keeps its no-rules CR when another run re-derives it. Tests cover an agent that is not in the catalog and an agent that is disabled.
4. Raw 500 when get_service() fails. Fixed. The call now raises HTTPException(502) with the same message format that Provision's classify_service uses. The test now uses a raw RuntimeError.

Note (does not block the merge): the new except Exception in the orchestrator also changes a bug that is not an IdP error into a 502. Provision does the same, so the behavior is consistent.