From 0ed16f62bd203a32564980b1d603848c54d47186 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Wed, 14 Oct 2026 19:50:37 +0300 Subject: [PATCH 01/15] fix(test): system fixture enables Direct Access Grants itself The system lane skipped all 102 tests when the login client `rossoctl` had no Direct Access Grants, and the skip text pointed to the demo realm fix-up in demo.md. A test must not depend on a demo doc. - Add `ensure_direct_access_grants` and call it in the session fixture before `verify_login_subject`. It is idempotent and does not create the client. - Remove the `demo.md` pointers from `launcher.py`. - Update docs/agents/test.md, docs/testing and the PRD to match. - Add unit tests for the new function. - Record the dependency direction in CLAUDE.md: tests do not use `demo/`, except `demo/assets/`. Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Oleg Blinder --- CLAUDE.md | 7 +++++ docs/agents/test.md | 6 ++--- docs/specs/PRD.md | 4 +-- docs/testing/policy-pipeline.md | 3 ++- docs/testing/uc1-onboarding-pipeline.md | 5 ++-- test/system/launcher.py | 10 +++---- test/system/uc1_onboard.py | 23 +++++++++++++--- .../uc/onboarding/test_uc1_provision_users.py | 27 +++++++++++++++++++ 8 files changed, 68 insertions(+), 17 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index ee319fa5..1019f867 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,6 +21,13 @@ offline unit lane. - To write a new test (placement, markers, clean-skip), see `docs/testing/testing-strategy.md`. +**Dependency direction.** Tests of any type (`test/`, `eval/`) must not depend on +`demo/` or point to a demo doc (for example `demo.md`). They set up what they need +themselves. The demo may use the test harness, but the harness must not use the demo. +One exception: `demo/assets/` (the `github-agent` and `github-tool` workloads, their +manifests, Dockerfiles and `kind-load.sh`). The system tests deploy these two +workloads, so they may use `demo/assets/` and nothing else under `demo/`. + ## Python environment Always use the `.venv` virtual environment (`.venv/bin/python`, `.venv/bin/pytest`) diff --git a/docs/agents/test.md b/docs/agents/test.md index 27442b32..7c5a5e21 100644 --- a/docs/agents/test.md +++ b/docs/agents/test.md @@ -114,9 +114,9 @@ test both sides, run the lane, do a clean redeploy with the other side (see `docs/agents/deploy.md`), and run the lane again. The platform prerequisites, wiring, verification and the setting of the side: -`k8s/opa-kind-runbook.md`. The realm fix-up (Direct Access Grants) and the manual -probes of the scenario: `demo/use-cases/onboarding/demo.md`. The per-loop shape: -`test/system/uc1_onboard.py`. +`k8s/opa-kind-runbook.md`. The session fixture enables Direct Access Grants on the +login client `rossoctl` itself, so the lane needs no manual realm step. The per-loop +shape: `test/system/uc1_onboard.py`. ```bash k8s/opa-kind-enable.sh # one time: wire the OPA plugin and the AIAC combiner into Kind diff --git a/docs/specs/PRD.md b/docs/specs/PRD.md index cf66a452..7829166f 100644 --- a/docs/specs/PRD.md +++ b/docs/specs/PRD.md @@ -819,7 +819,7 @@ The Controller reads `AIAC_ENFORCEMENT_SIDE` at start, and an unknown value stop - **The namespace pipelines.** The `authbridge-runtime-config` ConfigMap of each service namespace must have `opa` in the inbound pipeline, and also `mcp-parser` for a tool. Under agent side, the agents' namespaces also need `mcp-parser` and `opa` in the outbound pipeline. Onboarding check #2 finds a missing plugin. - **Pod restarts.** The webhook injects the sidecar only on pod CREATE. After a change to the pipeline or to `injectTools`, restart the agent pods and the tool pods (`rossoctl.io/type=agent` and `rossoctl.io/type=tool`). - **The username precondition (D31).** The CRs key users by username, and `sub` is the username. So usernames must be unique, must not change and must never be used again. Keep `editUsernameAllowed: false` and `registrationAllowed: false` in the realm (the operator's realm template sets `registrationAllowed: false`; it does not set `editUsernameAllowed`, so that value is `false` only by the Keycloak default). An admin must never create a deleted username again: the new user would get the grants of the old user until AIAC writes the CRs again. -- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The only manual step on `rossoctl` is Direct Access Grants, and only the password grant of the runbook probes and of the test harness needs it. +- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the runbook probes and of the test harness needs Direct Access Grants on `rossoctl`; the system-test fixture enables it (`ensure_direct_access_grants`), and the runbook probes need it set by hand. - **The legacy token exchange (optional hardening).** AIAC does not need the Keycloak feature `token-exchange` (V1) to be off: the AuthBridge requests go to the standard exchange (V2), and a request that V1 handles still gets `sub` = the username for an onboarded target (§6). Keycloak recommends to turn V1 off, because then a request that V2 declines fails with `400` ("Standard token exchange is not enabled for the requested client") and does not silently go to a different engine. `KC_FEATURES` is in the Keycloak deployment of the rossoctl platform, not in this repo. If V1 stays on, AIAC needs no change. Detail: [`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md). @@ -874,7 +874,7 @@ Require a live rossoctl/Kind cluster with the AuthBridge OPA pipeline wired in ( | `LLM_BASE_URL` / `LLM_MODEL` / `LLM_API_KEY` | LLM endpoint for onboarding | | `AIAC_TEST_REALM` | Optional: realm of the live stack (default from the scenario) | -System tests onboard through the in-cluster Controller and assert the deployed OPA plugin's allow/deny on real HTTP requests through AuthBridge. They skip cleanly when the cluster or the env is missing. Under target side, github-tool's own inbound OPA decides the tool calls, and the agent's outbound is a pass-through. A service that has no CR is denied, and a quarantine deletes the CR. The demo passes under each enforcement side; each side is a separate deploy (a clean redeploy changes the side). After each workload converges, `require_subject_scope` checks that its client links `aiac-username-sub` as a default scope (D31); a missing link is a failure, not a skip. `require_subject_scope` also checks that the login client `rossoctl` links the scope as a default scope (a missing login client, a missing link or an optional link only is a failure). Rung 2 also checks the scope and the links of `rossoctl`, `github-agent` and `github-tool`, that `rossoctl` has no client mapper that writes `sub`, and that the `rossoctl` login token of `dev-user` has `sub` = `dev-user` (`test_subject_scope_linked`), and the `sub` of an exchanged token (`test_exchanged_token_subject_is_username`: the harness exchanges as the agent client with its client secret, the identity that `k8s/opa-kind-enable.sh` gives AuthBridge; it skips only if the agent client uses another client authenticator). `verify_login_subject` checks the login token through `rossoctl`: it skips when it cannot mint the token (Direct Access Grants), and fails when `sub` is not the username, because the link is an AIAC step. Rung 5 checks that the rollback keeps the scope (`test_rollback_keeps_the_subject_scope`). +System tests onboard through the in-cluster Controller and assert the deployed OPA plugin's allow/deny on real HTTP requests through AuthBridge. They skip cleanly when the cluster or the env is missing. Under target side, github-tool's own inbound OPA decides the tool calls, and the agent's outbound is a pass-through. A service that has no CR is denied, and a quarantine deletes the CR. The demo passes under each enforcement side; each side is a separate deploy (a clean redeploy changes the side). After each workload converges, `require_subject_scope` checks that its client links `aiac-username-sub` as a default scope (D31); a missing link is a failure, not a skip. `require_subject_scope` also checks that the login client `rossoctl` links the scope as a default scope (a missing login client, a missing link or an optional link only is a failure). Rung 2 also checks the scope and the links of `rossoctl`, `github-agent` and `github-tool`, that `rossoctl` has no client mapper that writes `sub`, and that the `rossoctl` login token of `dev-user` has `sub` = `dev-user` (`test_subject_scope_linked`), and the `sub` of an exchanged token (`test_exchanged_token_subject_is_username`: the harness exchanges as the agent client with its client secret, the identity that `k8s/opa-kind-enable.sh` gives AuthBridge; it skips only if the agent client uses another client authenticator). `verify_login_subject` checks the login token through `rossoctl`: the fixture first enables Direct Access Grants on `rossoctl`; it skips when it still cannot mint the token, and fails when `sub` is not the username, because the link is an AIAC step. Rung 5 checks that the rollback keeps the scope (`test_rollback_keeps_the_subject_scope`). Selection is marker-only — no path argument. `integration` (several units in-process, no cluster), `system`, `llm` (real LLM, no cluster) and `eval` are opt-in markers: diff --git a/docs/testing/policy-pipeline.md b/docs/testing/policy-pipeline.md index 6591035a..88b72f41 100644 --- a/docs/testing/policy-pipeline.md +++ b/docs/testing/policy-pipeline.md @@ -93,7 +93,8 @@ over the fully onboarded stack. provisions these; UC-1 does not), `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its `sub` is the username, and `ensure_agent_policy` mounts the single abstract `policy.md` on the Controller pod. `verify_login_subject` skips when it cannot mint the - token (Direct Access Grants is not on), and fails when `sub` is not the username. The `sub` comes + token (the fixture enables Direct Access Grants first, so a skip means for example that the login + client is absent), and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, which the IdP Configuration Service links to `rossoctl` at its startup and at each onboarding. That link is an AIAC step, not a prerequisite, so a wrong `sub` is a failure. The exchanged token gets the same mapping from the same scope, diff --git a/docs/testing/uc1-onboarding-pipeline.md b/docs/testing/uc1-onboarding-pipeline.md index dda31cc5..8998cca4 100644 --- a/docs/testing/uc1-onboarding-pipeline.md +++ b/docs/testing/uc1-onboarding-pipeline.md @@ -224,8 +224,9 @@ cleanly** when any of it is absent (they never stand it up, and never false-pass *[Scenario](#scenario)* — via `KeycloakAdmin` into `AIAC_TEST_REALM`, **before** deploying (the PRB reads the realm role universe when the event fires `onboard_and_apply`); idempotent; left in place. `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its - `sub` is the username. It skips when it cannot mint the token (Direct Access Grants is a - prerequisite), and fails when `sub` is not the username. The `sub` comes from the client scope + `sub` is the username. The fixture first enables Direct Access Grants on `rossoctl` + (`ensure_direct_access_grants`). It skips when it still cannot mint the token (for example the + login client is absent), and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, the one source of the rule (D31). The IdP Configuration Service links it to `rossoctl` at its startup and at each onboarding, so the link is an AIAC step, not a prerequisite. There is no `sub` client mapper on `rossoctl`. The exchanged token gets the same diff --git a/test/system/launcher.py b/test/system/launcher.py index ae90c5e6..cfc388af 100644 --- a/test/system/launcher.py +++ b/test/system/launcher.py @@ -1032,9 +1032,9 @@ def verify_login_subject( and ``pytest.fail`` unless its ``sub`` equals ``user``. The live loop keys OPA decisions on ``input.identity.subject`` (the token ``sub``). To mint the - token, Direct Access Grants must be enabled on ``client_id`` — a one-time Keycloak prerequisite the - fixture does **not** provision (``demo/use-cases/onboarding/demo.md`` realm fix-up), so a mint - failure is a skip. The ``sub`` = username comes from the client scope ``aiac-username-sub``, which + token, Direct Access Grants must be enabled on ``client_id``. The fixture enables it before this call + (``uc1_onboard.ensure_direct_access_grants``), so a mint failure here means the login client is + absent or the user cannot log in, and it is a skip. The ``sub`` = username comes from the client scope ``aiac-username-sub``, which the IdP Configuration Service links as a default scope to the login client (``client_id``, by default ``rossoctl``) at its startup and at each onboarding (D31). That is an AIAC step, not a prerequisite, so a wrong ``sub`` fails. ``uc1_onboard.require_subject_scope`` checks the links again after each onboarding. Returns the @@ -1046,8 +1046,8 @@ def verify_login_subject( except Exception as exc: # noqa: BLE001 — any mint failure is a skippable prerequisite gap pytest.skip( f"cannot mint a {user!r} token in realm {realm!r} via client {client_id!r}: {exc}. " - "Enable Direct Access Grants on the client and set the user's password " - "(see the realm fix-up in demo/use-cases/onboarding/demo.md)." + "Check that the login client exists in the realm, has Direct Access Grants on, and that " + "the user has a password and a full profile." ) sub = jwt_claim(token, "sub") if sub != user: diff --git a/test/system/uc1_onboard.py b/test/system/uc1_onboard.py index 62c299ab..712f09ad 100644 --- a/test/system/uc1_onboard.py +++ b/test/system/uc1_onboard.py @@ -1034,6 +1034,19 @@ def login_client(admin) -> dict | None: return next((c for c in admin.get_clients() if c.get("clientId") == KEYCLOAK_CLIENT_ID), None) +def ensure_direct_access_grants(admin) -> bool: + """Enable Direct Access Grants on the login client (``KEYCLOAK_CLIENT_ID``) of ``TEST_REALM``, so + that ``mint_token`` can use the password grant. Idempotent: a client that has it already is not + written. Returns ``True`` if the client exists and has the grant on, ``False`` if the client is + absent (nothing is created: the realm owns its login client).""" + client = login_client(admin) + if client is None: + return False + if not client.get("directAccessGrantsEnabled"): + admin.update_client(client["id"], {**client, "directAccessGrantsEnabled": True}) + return True + + def _workload_list(workloads: str | Sequence[str]) -> list[str]: """One workload name or a sequence of them, as a list.""" return [workloads] if isinstance(workloads, str) else list(workloads) @@ -1416,10 +1429,12 @@ def onboarded_stack( ) provision_realm_and_users(admin, TEST_REALM) # BEFORE deploying (PRB reads the role universe when the event fires) - # Direct Access Grants on the login client is a one-time realm prereq the fixture does NOT provision: - # skip if a token can't be minted. The login token's ``sub`` comes from ``aiac-username-sub``, which - # the IdP Configuration Service links to the login client at startup (D31): fail if it isn't the - # username. ``require_subject_scope`` below checks the links again after each onboarding. + # The fixture enables Direct Access Grants on the login client (the password grant of ``mint_token`` + # needs it), then skips if a token still can't be minted. The login token's ``sub`` comes from + # ``aiac-username-sub``, which the IdP Configuration Service links to the login client at startup + # (D31): fail if it isn't the username. ``require_subject_scope`` below checks the links again after + # each onboarding. + ensure_direct_access_grants(admin) verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) tool_onboarded = scn.TOOL_WORKLOAD in workloads diff --git a/test/unit/agent/uc/onboarding/test_uc1_provision_users.py b/test/unit/agent/uc/onboarding/test_uc1_provision_users.py index 4697e09b..a97da0b1 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_provision_users.py +++ b/test/unit/agent/uc/onboarding/test_uc1_provision_users.py @@ -78,3 +78,30 @@ def test_an_existing_full_profile_is_not_changed() -> None: admin = _admin(profiles) uc1.provision_realm_and_users(admin, "rossoctl") admin.update_user.assert_not_called() + + +def _login_admin(client: dict | None) -> MagicMock: + admin = MagicMock(name="KeycloakAdmin") + admin.get_clients.return_value = [] if client is None else [{"clientId": "other", "id": "x"}, client] + return admin + + +def test_direct_access_grants_is_enabled_on_the_login_client() -> None: + admin = _login_admin({"clientId": "rossoctl", "id": "uuid-1", "directAccessGrantsEnabled": False}) + assert uc1.ensure_direct_access_grants(admin) is True + ((client_uuid, payload),) = [c.args for c in admin.update_client.call_args_list] + assert client_uuid == "uuid-1" + assert payload["directAccessGrantsEnabled"] is True and payload["clientId"] == "rossoctl" + + +def test_direct_access_grants_already_on_is_not_written() -> None: + admin = _login_admin({"clientId": "rossoctl", "id": "uuid-1", "directAccessGrantsEnabled": True}) + assert uc1.ensure_direct_access_grants(admin) is True + admin.update_client.assert_not_called() + + +def test_a_missing_login_client_is_not_created() -> None: + admin = _login_admin(None) + assert uc1.ensure_direct_access_grants(admin) is False + admin.update_client.assert_not_called() + admin.create_client.assert_not_called() From c15c69338283e1006da6a32625d70e2e515bf89f Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Wed, 14 Oct 2026 22:31:10 +0300 Subject: [PATCH 02/15] fix(test): enable Direct Access Grants in the shared realm setup Move ensure_direct_access_grants into provision_realm_and_users, so that every setup path (onboarded_stack, pristine_stack and the disabled cross-service test) enables the password grant on the login client. - Send a partial update ({"directAccessGrantsEnabled": true}), not the full client, so a concurrent IdP Configuration Service change is kept. - Fail on a missing login client (as require_subject_scope does); log a refused update, so verify_login_subject skips with the cause. - Document that the grant stays after the run (leave-in-place). - Move the grant unit tests to test_uc1_login_client.py. - Add the manual grant step to k8s/opa-kind-runbook.md and point the PRD and docs/agents/test.md to it; correct the demo teardown note. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- demo/use-cases/onboarding/demo.md | 4 +- docs/agents/test.md | 6 +- docs/specs/PRD.md | 4 +- docs/testing/policy-pipeline.md | 5 +- docs/testing/uc1-onboarding-pipeline.md | 7 +- k8s/opa-kind-runbook.md | 26 ++++++ test/system/launcher.py | 13 +-- test/system/uc1_onboard.py | 82 +++++++++++++------ .../uc/onboarding/test_uc1_login_client.py | 70 ++++++++++++++++ .../uc/onboarding/test_uc1_provision_users.py | 30 +------ 10 files changed, 178 insertions(+), 69 deletions(-) create mode 100644 test/unit/agent/uc/onboarding/test_uc1_login_client.py diff --git a/demo/use-cases/onboarding/demo.md b/demo/use-cases/onboarding/demo.md index d596ff53..9f711600 100644 --- a/demo/use-cases/onboarding/demo.md +++ b/demo/use-cases/onboarding/demo.md @@ -817,8 +817,8 @@ make teardown ARGS=--include-opa # also revert step 2's overlay and combiner ( - **the `team1` namespace itself** — the installer owns it ([`INSTALL.md`](../../assets/INSTALL.md): "a precondition, not an output"). A fresh install gives you an *empty* `team1`, so emptying it *is* the post-install state. -- **the realm fix-up** (Direct Access Grants) — one-time cluster state that this demo and the - `-m system` suite both depend on. The client scope `aiac-username-sub` and its links (D31) are +- **the realm fix-up** (Direct Access Grants) — one-time cluster state that this demo depends on. + (The `-m system` suite enables the grant itself.) The client scope `aiac-username-sub` and its links (D31) are AIAC's; AIAC makes them again at its next start and onboarding. - **the operator's `*-aud` audience client scopes**, which it owns and recreates. - **step 2's OPA overlay**, unless you pass `--include-opa`. diff --git a/docs/agents/test.md b/docs/agents/test.md index 7c5a5e21..bbebff51 100644 --- a/docs/agents/test.md +++ b/docs/agents/test.md @@ -114,8 +114,10 @@ test both sides, run the lane, do a clean redeploy with the other side (see `docs/agents/deploy.md`), and run the lane again. The platform prerequisites, wiring, verification and the setting of the side: -`k8s/opa-kind-runbook.md`. The session fixture enables Direct Access Grants on the -login client `rossoctl` itself, so the lane needs no manual realm step. The per-loop +`k8s/opa-kind-runbook.md`. The fixture enables Direct Access Grants on the login +client `rossoctl` itself and leaves it on, so the lane needs no manual realm step. A +manual probe with the password grant outside the lane needs the grant set by hand: +see the runbook section "Direct Access Grants on the login client". The per-loop shape: `test/system/uc1_onboard.py`. ```bash diff --git a/docs/specs/PRD.md b/docs/specs/PRD.md index 7829166f..a91db7a3 100644 --- a/docs/specs/PRD.md +++ b/docs/specs/PRD.md @@ -819,7 +819,7 @@ The Controller reads `AIAC_ENFORCEMENT_SIDE` at start, and an unknown value stop - **The namespace pipelines.** The `authbridge-runtime-config` ConfigMap of each service namespace must have `opa` in the inbound pipeline, and also `mcp-parser` for a tool. Under agent side, the agents' namespaces also need `mcp-parser` and `opa` in the outbound pipeline. Onboarding check #2 finds a missing plugin. - **Pod restarts.** The webhook injects the sidecar only on pod CREATE. After a change to the pipeline or to `injectTools`, restart the agent pods and the tool pods (`rossoctl.io/type=agent` and `rossoctl.io/type=tool`). - **The username precondition (D31).** The CRs key users by username, and `sub` is the username. So usernames must be unique, must not change and must never be used again. Keep `editUsernameAllowed: false` and `registrationAllowed: false` in the realm (the operator's realm template sets `registrationAllowed: false`; it does not set `editUsernameAllowed`, so that value is `false` only by the Keycloak default). An admin must never create a deleted username again: the new user would get the grants of the old user until AIAC writes the CRs again. -- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the runbook probes and of the test harness needs Direct Access Grants on `rossoctl`; the system-test fixture enables it (`ensure_direct_access_grants`), and the runbook probes need it set by hand. +- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`provision_realm_and_users` → `ensure_direct_access_grants`) and leaves it on; a manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). - **The legacy token exchange (optional hardening).** AIAC does not need the Keycloak feature `token-exchange` (V1) to be off: the AuthBridge requests go to the standard exchange (V2), and a request that V1 handles still gets `sub` = the username for an onboarded target (§6). Keycloak recommends to turn V1 off, because then a request that V2 declines fails with `400` ("Standard token exchange is not enabled for the requested client") and does not silently go to a different engine. `KC_FEATURES` is in the Keycloak deployment of the rossoctl platform, not in this repo. If V1 stays on, AIAC needs no change. Detail: [`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md). @@ -874,7 +874,7 @@ Require a live rossoctl/Kind cluster with the AuthBridge OPA pipeline wired in ( | `LLM_BASE_URL` / `LLM_MODEL` / `LLM_API_KEY` | LLM endpoint for onboarding | | `AIAC_TEST_REALM` | Optional: realm of the live stack (default from the scenario) | -System tests onboard through the in-cluster Controller and assert the deployed OPA plugin's allow/deny on real HTTP requests through AuthBridge. They skip cleanly when the cluster or the env is missing. Under target side, github-tool's own inbound OPA decides the tool calls, and the agent's outbound is a pass-through. A service that has no CR is denied, and a quarantine deletes the CR. The demo passes under each enforcement side; each side is a separate deploy (a clean redeploy changes the side). After each workload converges, `require_subject_scope` checks that its client links `aiac-username-sub` as a default scope (D31); a missing link is a failure, not a skip. `require_subject_scope` also checks that the login client `rossoctl` links the scope as a default scope (a missing login client, a missing link or an optional link only is a failure). Rung 2 also checks the scope and the links of `rossoctl`, `github-agent` and `github-tool`, that `rossoctl` has no client mapper that writes `sub`, and that the `rossoctl` login token of `dev-user` has `sub` = `dev-user` (`test_subject_scope_linked`), and the `sub` of an exchanged token (`test_exchanged_token_subject_is_username`: the harness exchanges as the agent client with its client secret, the identity that `k8s/opa-kind-enable.sh` gives AuthBridge; it skips only if the agent client uses another client authenticator). `verify_login_subject` checks the login token through `rossoctl`: the fixture first enables Direct Access Grants on `rossoctl`; it skips when it still cannot mint the token, and fails when `sub` is not the username, because the link is an AIAC step. Rung 5 checks that the rollback keeps the scope (`test_rollback_keeps_the_subject_scope`). +System tests onboard through the in-cluster Controller and assert the deployed OPA plugin's allow/deny on real HTTP requests through AuthBridge. They skip cleanly when the cluster or the env is missing. Under target side, github-tool's own inbound OPA decides the tool calls, and the agent's outbound is a pass-through. A service that has no CR is denied, and a quarantine deletes the CR. The demo passes under each enforcement side; each side is a separate deploy (a clean redeploy changes the side). After each workload converges, `require_subject_scope` checks that its client links `aiac-username-sub` as a default scope (D31); a missing link is a failure, not a skip. `require_subject_scope` also checks that the login client `rossoctl` links the scope as a default scope (a missing login client, a missing link or an optional link only is a failure). Rung 2 also checks the scope and the links of `rossoctl`, `github-agent` and `github-tool`, that `rossoctl` has no client mapper that writes `sub`, and that the `rossoctl` login token of `dev-user` has `sub` = `dev-user` (`test_subject_scope_linked`), and the `sub` of an exchanged token (`test_exchanged_token_subject_is_username`: the harness exchanges as the agent client with its client secret, the identity that `k8s/opa-kind-enable.sh` gives AuthBridge; it skips only if the agent client uses another client authenticator). `verify_login_subject` checks the login token through `rossoctl`: the fixture first enables Direct Access Grants on `rossoctl` (a missing `rossoctl` is a failure); it skips when it still cannot mint the token (for example the admin cannot manage clients), and fails when `sub` is not the username, because the link is an AIAC step. Rung 5 checks that the rollback keeps the scope (`test_rollback_keeps_the_subject_scope`). Selection is marker-only — no path argument. `integration` (several units in-process, no cluster), `system`, `llm` (real LLM, no cluster) and `eval` are opt-in markers: diff --git a/docs/testing/policy-pipeline.md b/docs/testing/policy-pipeline.md index 88b72f41..46e18319 100644 --- a/docs/testing/policy-pipeline.md +++ b/docs/testing/policy-pipeline.md @@ -93,8 +93,9 @@ over the fully onboarded stack. provisions these; UC-1 does not), `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its `sub` is the username, and `ensure_agent_policy` mounts the single abstract `policy.md` on the Controller pod. `verify_login_subject` skips when it cannot mint the - token (the fixture enables Direct Access Grants first, so a skip means for example that the login - client is absent), and fails when `sub` is not the username. The `sub` comes + token (`provision_realm_and_users` enables Direct Access Grants first and fails if the login client + is absent, so a skip means for example that the admin cannot manage clients), and fails when `sub` + is not the username. The `sub` comes from the client scope `aiac-username-sub`, which the IdP Configuration Service links to `rossoctl` at its startup and at each onboarding. That link is an AIAC step, not a prerequisite, so a wrong `sub` is a failure. The exchanged token gets the same mapping from the same scope, diff --git a/docs/testing/uc1-onboarding-pipeline.md b/docs/testing/uc1-onboarding-pipeline.md index 8998cca4..1563895d 100644 --- a/docs/testing/uc1-onboarding-pipeline.md +++ b/docs/testing/uc1-onboarding-pipeline.md @@ -224,9 +224,10 @@ cleanly** when any of it is absent (they never stand it up, and never false-pass *[Scenario](#scenario)* — via `KeycloakAdmin` into `AIAC_TEST_REALM`, **before** deploying (the PRB reads the realm role universe when the event fires `onboard_and_apply`); idempotent; left in place. `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its - `sub` is the username. The fixture first enables Direct Access Grants on `rossoctl` - (`ensure_direct_access_grants`). It skips when it still cannot mint the token (for example the - login client is absent), and fails when `sub` is not the username. The `sub` comes from the client scope + `sub` is the username. `provision_realm_and_users` first enables Direct Access Grants on + `rossoctl` (`ensure_direct_access_grants`, left on) and fails if `rossoctl` is absent. It skips + when it still cannot mint the token (for example the admin cannot manage clients), and fails when + `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, the one source of the rule (D31). The IdP Configuration Service links it to `rossoctl` at its startup and at each onboarding, so the link is an AIAC step, not a prerequisite. There is no `sub` client mapper on `rossoctl`. The exchanged token gets the same diff --git a/k8s/opa-kind-runbook.md b/k8s/opa-kind-runbook.md index fe30001e..3de409e6 100644 --- a/k8s/opa-kind-runbook.md +++ b/k8s/opa-kind-runbook.md @@ -221,6 +221,32 @@ such pods, it skips this check. --- +## Direct Access Grants on the login client + +A manual probe that mints a user token with the password grant through the +login client `rossoctl` needs Direct Access Grants on that client. In a fresh +realm it is off, and the mint fails with `unauthorized_client`. The system +tests do not need this step: their fixture enables the grant itself +(`ensure_direct_access_grants` in `test/system/uc1_onboard.py`). To enable it by +hand (a partial update: only this field changes): + +```bash +KC=http://keycloak.localtest.me:8080 +ADMIN=$(curl -s -X POST "$KC/realms/master/protocol/openid-connect/token" \ + -d client_id=admin-cli -d username=admin -d password=admin -d grant_type=password \ + | python3 -c 'import sys,json;print(json.load(sys.stdin)["access_token"])') +CID=$(curl -s -H "Authorization: Bearer $ADMIN" "$KC/admin/realms/rossoctl/clients?clientId=rossoctl" \ + | python3 -c 'import sys,json;print(json.load(sys.stdin)[0]["id"])') +curl -s -o /dev/null -w "enable DAG HTTP %{http_code}\n" -X PUT \ + -H "Authorization: Bearer $ADMIN" -H "Content-Type: application/json" \ + "$KC/admin/realms/rossoctl/clients/$CID" -d '{"directAccessGrantsEnabled": true}' +``` + +Expect `enable DAG HTTP 204`. The grant stays on; set the field to `false` the +same way to turn it off again. + +--- + ## Restore ```bash diff --git a/test/system/launcher.py b/test/system/launcher.py index cfc388af..844cf6e7 100644 --- a/test/system/launcher.py +++ b/test/system/launcher.py @@ -1033,10 +1033,11 @@ def verify_login_subject( The live loop keys OPA decisions on ``input.identity.subject`` (the token ``sub``). To mint the token, Direct Access Grants must be enabled on ``client_id``. The fixture enables it before this call - (``uc1_onboard.ensure_direct_access_grants``), so a mint failure here means the login client is - absent or the user cannot log in, and it is a skip. The ``sub`` = username comes from the client scope ``aiac-username-sub``, which - the IdP Configuration Service links as a default scope to the login client (``client_id``, by - default ``rossoctl``) at its startup and at each onboarding (D31). That is an AIAC step, not a prerequisite, so a wrong ``sub`` fails. + (``uc1_onboard.provision_realm_and_users`` -> ``ensure_direct_access_grants``, which fails on a missing + login client), so a mint failure here means the grant could not be written (the admin cannot manage + clients) or the user cannot log in, and it is a skip. The ``sub`` = username comes from the client + scope ``aiac-username-sub``, which the IdP Configuration Service links as a default scope to the + login client (``client_id``, by default ``rossoctl``) at its startup and at each onboarding (D31). That is an AIAC step, not a prerequisite, so a wrong ``sub`` fails. ``uc1_onboard.require_subject_scope`` checks the links again after each onboarding. Returns the minted token on success.""" import pytest @@ -1046,8 +1047,8 @@ def verify_login_subject( except Exception as exc: # noqa: BLE001 — any mint failure is a skippable prerequisite gap pytest.skip( f"cannot mint a {user!r} token in realm {realm!r} via client {client_id!r}: {exc}. " - "Check that the login client exists in the realm, has Direct Access Grants on, and that " - "the user has a password and a full profile." + "Check that the login client has Direct Access Grants on (the fixture logs a warning if it " + "cannot enable it), and that the user has a password and a full profile." ) sub = jwt_claim(token, "sub") if sub != user: diff --git a/test/system/uc1_onboard.py b/test/system/uc1_onboard.py index 712f09ad..2bcd6d9e 100644 --- a/test/system/uc1_onboard.py +++ b/test/system/uc1_onboard.py @@ -30,7 +30,8 @@ → drive REAL HTTP requests through AuthBridge and assert the real plugin's allow/deny → capture every AIAC CR + the global combiner to the pytest host (``teardown``) — the first teardown step, on the success and the failure path, best-effort (it never raises) - → teardown full-to-pristine (undeploy + delete all registrations + delete CRs, verified) + → teardown full-to-pristine (undeploy + delete all registrations + delete CRs, verified); the + realm users, roles and Direct Access Grants on the login client stay (leave-in-place) The only thing that differs between rungs is *which workloads are onboarded and in what order*, so all the machinery lives here and each ``test_uc1_onboard_*.py`` supplies just its own oracle @@ -39,7 +40,8 @@ This module owns: * **Config** (env, spec § Configuration) — single stack, no variants. -* **Keycloak** — ``connect_admin`` / ``provision_realm_and_users`` (the fixture UC-1 does *not* do) / +* **Keycloak** — ``connect_admin`` / ``provision_realm_and_users`` (the fixture UC-1 does *not* do, + with ``ensure_direct_access_grants`` on the login client) / ``cleanup_provisioned`` / ``delete_workload_registrations`` (teardown). * **CRs** — ``delete_workload_crs`` (the agent's and the tool's CR) / ``sweep_authpolicies`` (every other CR in the namespace) / ``delete_authpolicy`` (one CR, raising). @@ -329,8 +331,10 @@ def provision_realm_and_users(admin, realm: str) -> None: """Idempotently ensure ``realm`` holds ``scenario_uc1``'s users + realm roles with the descriptions the PRB reads. Realm roles carry the ``aiac.managed`` marker so the IdP populates each role's ``actorIds`` (member usernames) — the PCE needs them to build the ``subject_roles`` - map the inbound/outbound gates key on. Never deletes/recreates — reruns converge (spec: shared - realm, leave-in-place).""" + map the inbound/outbound gates key on. Also turns on Direct Access Grants on the login client + (``ensure_direct_access_grants``), so that ``verify_login_subject`` and ``mint_token`` can use the + password grant. Never deletes/recreates — reruns converge (spec: shared realm, leave-in-place: the + users, the roles and the grant stay after the run).""" from keycloak.exceptions import KeycloakError try: @@ -351,6 +355,8 @@ def provision_realm_and_users(admin, realm: str) -> None: admin.set_user_password(user_id, scn.USER_PASSWORD, temporary=False) admin.assign_realm_roles(user_id, [admin.get_realm_role(role_name)]) + ensure_direct_access_grants(admin, realm) + def _user_profile(username: str) -> dict: """The profile fields that a fresh ``rossoctl`` realm requires (``email``, ``firstName``, @@ -1034,17 +1040,41 @@ def login_client(admin) -> dict | None: return next((c for c in admin.get_clients() if c.get("clientId") == KEYCLOAK_CLIENT_ID), None) -def ensure_direct_access_grants(admin) -> bool: - """Enable Direct Access Grants on the login client (``KEYCLOAK_CLIENT_ID``) of ``TEST_REALM``, so - that ``mint_token`` can use the password grant. Idempotent: a client that has it already is not - written. Returns ``True`` if the client exists and has the grant on, ``False`` if the client is - absent (nothing is created: the realm owns its login client).""" - client = login_client(admin) +def ensure_direct_access_grants(admin, realm: str) -> None: + """Turn on Direct Access Grants on the login client (``KEYCLOAK_CLIENT_ID``) of ``realm``, so that + ``mint_token`` can use the password grant. ``provision_realm_and_users`` calls it, so each setup path + that provisions the users also gets the grant. + + Idempotent: a client that has the grant already is not written. The write is a partial update (only + ``directAccessGrantsEnabled``), so it does not write back a stale copy of the client over a change + that the IdP Configuration Service made (for example its D31 scope link). The grant stays after the + run (leave-in-place, like the users); the change is logged, so the owner of a realm that had it off + on purpose can see it. + + Raises ``RuntimeError`` when the login client does not exist: the realm owns its login client, so + nothing is created, and a missing login client is a failure (``require_subject_scope``), not a skip. + A ``KeycloakError`` from the update (for example the admin cannot manage clients) is logged, not + raised: ``verify_login_subject`` then cannot mint the token and skips with the cause.""" + from keycloak.exceptions import KeycloakError + + admin.change_current_realm(realm) + client = next((c for c in admin.get_clients() if c.get("clientId") == KEYCLOAK_CLIENT_ID), None) if client is None: - return False - if not client.get("directAccessGrantsEnabled"): - admin.update_client(client["id"], {**client, "directAccessGrantsEnabled": True}) - return True + raise RuntimeError( + f"the login client {KEYCLOAK_CLIENT_ID!r} does not exist in realm {realm!r}, so the harness cannot " + "mint a user token through it (the realm owns its login client; the harness does not create it)" + ) + if client.get("directAccessGrantsEnabled"): + return + try: + admin.update_client(client["id"], {"directAccessGrantsEnabled": True}) + log.warning( + "enabled Direct Access Grants on the login client %r in realm %r (left on after the run)", + KEYCLOAK_CLIENT_ID, + realm, + ) + except KeycloakError as exc: + log.warning("cannot enable Direct Access Grants on the login client %r: %s", KEYCLOAK_CLIENT_ID, exc) def _workload_list(workloads: str | Sequence[str]) -> list[str]: @@ -1373,8 +1403,10 @@ def onboarded_stack( Teardown first captures every AIAC CR and the global combiner to the pytest host (``capture_aiac_crs``, phase ``teardown``; best-effort, it never masks a failure), then is **full-to-pristine** — the workloads and every registration are removed and the removal is - verified. The capture directories are named for the test module that started the stack. The - workload order is the rung's identity — e.g. rung 2 passes ``[agent, tool]`` so tool onboarding + verified. The realm setup of ``provision_realm_and_users`` (the users, the roles and Direct Access + Grants on the login client) is shared realm state and stays. The capture directories are named for + the test module that started the stack. The workload order is the rung's identity — e.g. rung 2 + passes ``[agent, tool]`` so tool onboarding completes the tool check (under target side it writes github-tool's own CR with both gates; under agent side it completes the agent's outbound gate); rung 3 passes ``[tool, agent]`` and must converge to the same live decisions. @@ -1429,12 +1461,11 @@ def onboarded_stack( ) provision_realm_and_users(admin, TEST_REALM) # BEFORE deploying (PRB reads the role universe when the event fires) - # The fixture enables Direct Access Grants on the login client (the password grant of ``mint_token`` - # needs it), then skips if a token still can't be minted. The login token's ``sub`` comes from - # ``aiac-username-sub``, which the IdP Configuration Service links to the login client at startup - # (D31): fail if it isn't the username. ``require_subject_scope`` below checks the links again after - # each onboarding. - ensure_direct_access_grants(admin) + # ``provision_realm_and_users`` also enables Direct Access Grants on the login client (the password + # grant of ``mint_token`` needs it); skip if a token still can't be minted. The login token's ``sub`` + # comes from ``aiac-username-sub``, which the IdP Configuration Service links to the login client at + # startup (D31): fail if it isn't the username. ``require_subject_scope`` below checks the links again + # after each onboarding. verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) tool_onboarded = scn.TOOL_WORKLOAD in workloads @@ -2393,9 +2424,10 @@ def pristine_stack(workloads: Sequence[str], *, policy_md: str = scn.POLICY_ABST Same skip gates (pipeline wiring, env, event path — before any mutation), same no-workloads slate (a ``pre-run-slate`` CR capture, then undeploy + ``_scrub_to_pristine`` + - ``reenable_provisioned_clients`` + clients-gone poll), same realm/users provisioning + ``sub`` - mapper gate, ``policy.md`` mount and image load for ``workloads``; and the same teardown: a - ``teardown`` CR capture first, then full-to-pristine, verified (clients gone, no CR left).""" + ``reenable_provisioned_clients`` + clients-gone poll), same realm/users provisioning (with Direct + Access Grants on the login client) + ``sub`` mapper gate, ``policy.md`` mount and image load for + ``workloads``; and the same teardown: a ``teardown`` CR capture first, then full-to-pristine, + verified (clients gone, no CR left).""" require_pipeline(namespace=NAMESPACE, workloads=[]) creds = require_env_or_skip("KEYCLOAK_URL", "KEYCLOAK_ADMIN_USERNAME", "KEYCLOAK_ADMIN_PASSWORD") keycloak_url = creds["KEYCLOAK_URL"] diff --git a/test/unit/agent/uc/onboarding/test_uc1_login_client.py b/test/unit/agent/uc/onboarding/test_uc1_login_client.py new file mode 100644 index 00000000..d21ffd09 --- /dev/null +++ b/test/unit/agent/uc/onboarding/test_uc1_login_client.py @@ -0,0 +1,70 @@ +"""``uc1_onboard.ensure_direct_access_grants`` turns on the password grant of the login client — pure +unit tests. + +The harness mints user tokens through the login client ``rossoctl`` with the password grant, so the +client needs Direct Access Grants on. ``provision_realm_and_users`` turns it on, so each setup path +(``onboarded_stack``, ``pristine_stack`` and the disabled cross-service test) gets it. These tests use a +``MagicMock`` admin (no Keycloak). They import the system harness like ``test_uc1_subject_scope.py`` +does. +""" + +from __future__ import annotations + +import sys +from pathlib import Path +from unittest.mock import MagicMock + +import pytest +from keycloak.exceptions import KeycloakError + +HERE = Path(__file__).resolve().parent # test/unit/agent/uc/onboarding/ +REPO_ROOT = HERE.parents[4] # -> aiac/ +sys.path.insert(0, str(REPO_ROOT)) # so ``import test.system.*`` resolves + +from test.system import uc1_onboard as uc1 # noqa: E402 + + +def _admin(client: dict | None) -> MagicMock: + admin = MagicMock(name="KeycloakAdmin") + admin.get_clients.return_value = [{"clientId": "other", "id": "x"}] + ([] if client is None else [client]) + return admin + + +def _login(enabled: bool) -> dict: + return {"clientId": uc1.KEYCLOAK_CLIENT_ID, "id": "uuid-1", "directAccessGrantsEnabled": enabled} + + +def test_the_grant_is_enabled_with_a_partial_update() -> None: + admin = _admin(_login(False)) + uc1.ensure_direct_access_grants(admin, "rossoctl") + admin.change_current_realm.assert_called_with("rossoctl") + admin.update_client.assert_called_once_with("uuid-1", {"directAccessGrantsEnabled": True}) + + +def test_a_grant_that_is_on_is_not_written() -> None: + admin = _admin(_login(True)) + uc1.ensure_direct_access_grants(admin, "rossoctl") + admin.update_client.assert_not_called() + + +def test_a_missing_login_client_fails_and_is_not_created() -> None: + admin = _admin(None) + with pytest.raises(RuntimeError, match=uc1.KEYCLOAK_CLIENT_ID): + uc1.ensure_direct_access_grants(admin, "rossoctl") + admin.update_client.assert_not_called() + admin.create_client.assert_not_called() + + +def test_a_refused_update_is_logged_not_raised(caplog: pytest.LogCaptureFixture) -> None: + admin = _admin(_login(False)) + admin.update_client.side_effect = KeycloakError(error_message="forbidden", response_code=403) + uc1.ensure_direct_access_grants(admin, "rossoctl") # verify_login_subject then skips with the cause + assert "cannot enable Direct Access Grants" in caplog.text + + +def test_provision_realm_and_users_enables_the_grant() -> None: + admin = _admin(_login(False)) + admin.create_user.side_effect = lambda payload, exist_ok=False: f"id-{payload['username']}" + admin.get_user.return_value = {"email": "e", "firstName": "f", "lastName": "l"} + uc1.provision_realm_and_users(admin, "rossoctl") + admin.update_client.assert_called_once_with("uuid-1", {"directAccessGrantsEnabled": True}) diff --git a/test/unit/agent/uc/onboarding/test_uc1_provision_users.py b/test/unit/agent/uc/onboarding/test_uc1_provision_users.py index a97da0b1..012571c1 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_provision_users.py +++ b/test/unit/agent/uc/onboarding/test_uc1_provision_users.py @@ -38,6 +38,9 @@ def create_user(payload, exist_ok=False): admin.create_user.side_effect = create_user admin.get_user.side_effect = lambda user_id: {**(existing.get(user_id[3:]) or created[user_id[3:]])} + # The login client, with Direct Access Grants on (``test_uc1_login_client.py`` covers that step). + login = {"clientId": uc1.KEYCLOAK_CLIENT_ID, "id": "c", "directAccessGrantsEnabled": True} + admin.get_clients.return_value = [login] return admin @@ -78,30 +81,3 @@ def test_an_existing_full_profile_is_not_changed() -> None: admin = _admin(profiles) uc1.provision_realm_and_users(admin, "rossoctl") admin.update_user.assert_not_called() - - -def _login_admin(client: dict | None) -> MagicMock: - admin = MagicMock(name="KeycloakAdmin") - admin.get_clients.return_value = [] if client is None else [{"clientId": "other", "id": "x"}, client] - return admin - - -def test_direct_access_grants_is_enabled_on_the_login_client() -> None: - admin = _login_admin({"clientId": "rossoctl", "id": "uuid-1", "directAccessGrantsEnabled": False}) - assert uc1.ensure_direct_access_grants(admin) is True - ((client_uuid, payload),) = [c.args for c in admin.update_client.call_args_list] - assert client_uuid == "uuid-1" - assert payload["directAccessGrantsEnabled"] is True and payload["clientId"] == "rossoctl" - - -def test_direct_access_grants_already_on_is_not_written() -> None: - admin = _login_admin({"clientId": "rossoctl", "id": "uuid-1", "directAccessGrantsEnabled": True}) - assert uc1.ensure_direct_access_grants(admin) is True - admin.update_client.assert_not_called() - - -def test_a_missing_login_client_is_not_created() -> None: - admin = _login_admin(None) - assert uc1.ensure_direct_access_grants(admin) is False - admin.update_client.assert_not_called() - admin.create_client.assert_not_called() From a589c4f8679c9e8652be3af8ebefbe49511abdac Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Thu, 15 Oct 2026 08:23:25 +0300 Subject: [PATCH 03/15] docs(deploy): reload the aiac-agent ConfigMap on every agent rebuild The redeploy section of the guide now has a flow for the aiac-agent image: rebuild, kind load, apply agent-deployment.yaml, patch LLM_BASE_URL and LLM_MODEL from .env, then restart last. It also notes that the apply resets AIAC_ENFORCEMENT_SIDE and LOG_LEVEL, and that the Secret is not changed. Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Oleg Blinder --- k8s/aiac-deployment-guide.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/k8s/aiac-deployment-guide.md b/k8s/aiac-deployment-guide.md index 6108dc5d..ca9feb08 100644 --- a/k8s/aiac-deployment-guide.md +++ b/k8s/aiac-deployment-guide.md @@ -313,6 +313,37 @@ kind load docker-image localhost/aiac-pdp-config:local --name kubectl rollout restart deployment/aiac-interface -n aiac-system ``` +### Rebuilding the AIAC Agent: always reload its ConfigMap + +Every time you rebuild `localhost/aiac-agent:local`, also reload the `aiac-agent-config` +ConfigMap. Do not skip it, also when only code changed. It makes sure the Agent runs with the +LLM endpoint and model of the repo-root `.env`, and not with an older value in the cluster. + +```bash +docker build -f src/aiac/agent/controller/Dockerfile -t localhost/aiac-agent:local src/ +kind load docker-image localhost/aiac-agent:local --name + +# Reload the ConfigMap: the apply writes the manifest values (placeholders), the patch +# puts the real LLM values from .env back (step 3b). +set -a; . .env; set +a +kubectl apply -f k8s/agent-deployment.yaml +kubectl patch configmap aiac-agent-config -n aiac-system --type merge \ + -p "{\"data\":{\"LLM_BASE_URL\":\"${LLM_BASE_URL}\",\"LLM_MODEL\":\"${LLM_MODEL}\"}}" + +# Restart AFTER the patch. The Agent reads the ConfigMap at start only. +kubectl rollout restart deployment/aiac-agent -n aiac-system +kubectl rollout status deployment/aiac-agent -n aiac-system +``` + +- The apply also writes the manifest values of `AIAC_ENFORCEMENT_SIDE` (`target-side`) and + `LOG_LEVEL` (`INFO`). On an `agent-side` deployment, patch `AIAC_ENFORCEMENT_SIDE` back + **before** the restart. See `docs/agents/deploy.md`, `ac-side`. Patch `LOG_LEVEL` again if you + had raised it. +- `.env` must have `LLM_BASE_URL` and `LLM_MODEL`. If one is missing, stop and ask. Do not leave + the placeholder in the ConfigMap. +- The Secret `aiac-agent-secret` (`LLM_API_KEY`) is not changed by this flow. To change it, see + step 3b. + --- ## Phase 2: Upgrading the OPA PDP Policy Writer to the CR-backed implementation From 3c34d8f9ebf6ba289a1a81feab6193a432cbbe34 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Thu, 15 Oct 2026 08:40:47 +0300 Subject: [PATCH 04/15] fix(test): skip with the cause when Direct Access Grants cannot be set, and restore it at teardown ensure_direct_access_grants now calls pytest.skip with the cause when the login client is missing or the admin API refuses the read or the update. Before, these cases raised an error or wrote only a log line. It returns True when it turned the grant on, and each fixture teardown then turns it off again (restore_direct_access_grants, best-effort). verify_login_subject moves into the fixture try block, so a skip there also restores the grant. The lookup reuses login_client (one copy). Docs: the Agent rebuild block in the deployment guide checks the LLM values in .env, keeps the live enforcement side and log level, builds the patch JSON with jq, and runs in a set -e subshell. The runbook snippet for Direct Access Grants reads the harness env and stops with a clear message when the admin login fails or the client is missing. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- docs/agents/test.md | 3 +- docs/specs/PRD.md | 4 +- docs/testing/policy-pipeline.md | 6 +- docs/testing/uc1-onboarding-pipeline.md | 6 +- k8s/aiac-deployment-guide.md | 26 ++-- k8s/opa-kind-runbook.md | 29 +++-- ...test_uc1_onboard_cross_service_conflict.py | 5 +- test/system/launcher.py | 10 +- test/system/uc1_onboard.py | 111 ++++++++++++------ .../uc/onboarding/test_uc1_login_client.py | 52 ++++++-- 10 files changed, 171 insertions(+), 81 deletions(-) diff --git a/docs/agents/test.md b/docs/agents/test.md index bbebff51..ffd02045 100644 --- a/docs/agents/test.md +++ b/docs/agents/test.md @@ -115,7 +115,8 @@ test both sides, run the lane, do a clean redeploy with the other side (see The platform prerequisites, wiring, verification and the setting of the side: `k8s/opa-kind-runbook.md`. The fixture enables Direct Access Grants on the login -client `rossoctl` itself and leaves it on, so the lane needs no manual realm step. A +client `rossoctl` itself and turns it off again at teardown if it was off, so the +lane needs no manual realm step. A manual probe with the password grant outside the lane needs the grant set by hand: see the runbook section "Direct Access Grants on the login client". The per-loop shape: `test/system/uc1_onboard.py`. diff --git a/docs/specs/PRD.md b/docs/specs/PRD.md index a91db7a3..7be98c83 100644 --- a/docs/specs/PRD.md +++ b/docs/specs/PRD.md @@ -819,7 +819,7 @@ The Controller reads `AIAC_ENFORCEMENT_SIDE` at start, and an unknown value stop - **The namespace pipelines.** The `authbridge-runtime-config` ConfigMap of each service namespace must have `opa` in the inbound pipeline, and also `mcp-parser` for a tool. Under agent side, the agents' namespaces also need `mcp-parser` and `opa` in the outbound pipeline. Onboarding check #2 finds a missing plugin. - **Pod restarts.** The webhook injects the sidecar only on pod CREATE. After a change to the pipeline or to `injectTools`, restart the agent pods and the tool pods (`rossoctl.io/type=agent` and `rossoctl.io/type=tool`). - **The username precondition (D31).** The CRs key users by username, and `sub` is the username. So usernames must be unique, must not change and must never be used again. Keep `editUsernameAllowed: false` and `registrationAllowed: false` in the realm (the operator's realm template sets `registrationAllowed: false`; it does not set `editUsernameAllowed`, so that value is `false` only by the Keycloak default). An admin must never create a deleted username again: the new user would get the grants of the old user until AIAC writes the CRs again. -- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`provision_realm_and_users` → `ensure_direct_access_grants`) and leaves it on; a manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). +- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`provision_realm_and_users` → `ensure_direct_access_grants`) and turns it off again at teardown if it was off; a manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). - **The legacy token exchange (optional hardening).** AIAC does not need the Keycloak feature `token-exchange` (V1) to be off: the AuthBridge requests go to the standard exchange (V2), and a request that V1 handles still gets `sub` = the username for an onboarded target (§6). Keycloak recommends to turn V1 off, because then a request that V2 declines fails with `400` ("Standard token exchange is not enabled for the requested client") and does not silently go to a different engine. `KC_FEATURES` is in the Keycloak deployment of the rossoctl platform, not in this repo. If V1 stays on, AIAC needs no change. Detail: [`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md). @@ -874,7 +874,7 @@ Require a live rossoctl/Kind cluster with the AuthBridge OPA pipeline wired in ( | `LLM_BASE_URL` / `LLM_MODEL` / `LLM_API_KEY` | LLM endpoint for onboarding | | `AIAC_TEST_REALM` | Optional: realm of the live stack (default from the scenario) | -System tests onboard through the in-cluster Controller and assert the deployed OPA plugin's allow/deny on real HTTP requests through AuthBridge. They skip cleanly when the cluster or the env is missing. Under target side, github-tool's own inbound OPA decides the tool calls, and the agent's outbound is a pass-through. A service that has no CR is denied, and a quarantine deletes the CR. The demo passes under each enforcement side; each side is a separate deploy (a clean redeploy changes the side). After each workload converges, `require_subject_scope` checks that its client links `aiac-username-sub` as a default scope (D31); a missing link is a failure, not a skip. `require_subject_scope` also checks that the login client `rossoctl` links the scope as a default scope (a missing login client, a missing link or an optional link only is a failure). Rung 2 also checks the scope and the links of `rossoctl`, `github-agent` and `github-tool`, that `rossoctl` has no client mapper that writes `sub`, and that the `rossoctl` login token of `dev-user` has `sub` = `dev-user` (`test_subject_scope_linked`), and the `sub` of an exchanged token (`test_exchanged_token_subject_is_username`: the harness exchanges as the agent client with its client secret, the identity that `k8s/opa-kind-enable.sh` gives AuthBridge; it skips only if the agent client uses another client authenticator). `verify_login_subject` checks the login token through `rossoctl`: the fixture first enables Direct Access Grants on `rossoctl` (a missing `rossoctl` is a failure); it skips when it still cannot mint the token (for example the admin cannot manage clients), and fails when `sub` is not the username, because the link is an AIAC step. Rung 5 checks that the rollback keeps the scope (`test_rollback_keeps_the_subject_scope`). +System tests onboard through the in-cluster Controller and assert the deployed OPA plugin's allow/deny on real HTTP requests through AuthBridge. They skip cleanly when the cluster or the env is missing. Under target side, github-tool's own inbound OPA decides the tool calls, and the agent's outbound is a pass-through. A service that has no CR is denied, and a quarantine deletes the CR. The demo passes under each enforcement side; each side is a separate deploy (a clean redeploy changes the side). After each workload converges, `require_subject_scope` checks that its client links `aiac-username-sub` as a default scope (D31); a missing link is a failure, not a skip. `require_subject_scope` also checks that the login client `rossoctl` links the scope as a default scope (a missing login client, a missing link or an optional link only is a failure). Rung 2 also checks the scope and the links of `rossoctl`, `github-agent` and `github-tool`, that `rossoctl` has no client mapper that writes `sub`, and that the `rossoctl` login token of `dev-user` has `sub` = `dev-user` (`test_subject_scope_linked`), and the `sub` of an exchanged token (`test_exchanged_token_subject_is_username`: the harness exchanges as the agent client with its client secret, the identity that `k8s/opa-kind-enable.sh` gives AuthBridge; it skips only if the agent client uses another client authenticator). `verify_login_subject` checks the login token through `rossoctl`: the fixture first enables Direct Access Grants on `rossoctl` and skips with the cause when it cannot (a missing `rossoctl`, or an admin that cannot manage clients); it skips when it still cannot mint the token, and fails when `sub` is not the username, because the link is an AIAC step. Rung 5 checks that the rollback keeps the scope (`test_rollback_keeps_the_subject_scope`). Selection is marker-only — no path argument. `integration` (several units in-process, no cluster), `system`, `llm` (real LLM, no cluster) and `eval` are opt-in markers: diff --git a/docs/testing/policy-pipeline.md b/docs/testing/policy-pipeline.md index 46e18319..930497b4 100644 --- a/docs/testing/policy-pipeline.md +++ b/docs/testing/policy-pipeline.md @@ -93,9 +93,9 @@ over the fully onboarded stack. provisions these; UC-1 does not), `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its `sub` is the username, and `ensure_agent_policy` mounts the single abstract `policy.md` on the Controller pod. `verify_login_subject` skips when it cannot mint the - token (`provision_realm_and_users` enables Direct Access Grants first and fails if the login client - is absent, so a skip means for example that the admin cannot manage clients), and fails when `sub` - is not the username. The `sub` comes + token (`provision_realm_and_users` enables Direct Access Grants first, and skips with the cause if + the login client is absent or the admin cannot manage clients; the teardown turns the grant off + again if it was off), and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, which the IdP Configuration Service links to `rossoctl` at its startup and at each onboarding. That link is an AIAC step, not a prerequisite, so a wrong `sub` is a failure. The exchanged token gets the same mapping from the same scope, diff --git a/docs/testing/uc1-onboarding-pipeline.md b/docs/testing/uc1-onboarding-pipeline.md index 1563895d..2e45f4b2 100644 --- a/docs/testing/uc1-onboarding-pipeline.md +++ b/docs/testing/uc1-onboarding-pipeline.md @@ -225,9 +225,9 @@ cleanly** when any of it is absent (they never stand it up, and never false-pass reads the realm role universe when the event fires `onboard_and_apply`); idempotent; left in place. `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its `sub` is the username. `provision_realm_and_users` first enables Direct Access Grants on - `rossoctl` (`ensure_direct_access_grants`, left on) and fails if `rossoctl` is absent. It skips - when it still cannot mint the token (for example the admin cannot manage clients), and fails when - `sub` is not the username. The `sub` comes from the client scope + `rossoctl` (`ensure_direct_access_grants`; the teardown turns it off again if it was off) and + skips with the cause if `rossoctl` is absent or the admin cannot manage clients. It skips when it + still cannot mint the token, and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, the one source of the rule (D31). The IdP Configuration Service links it to `rossoctl` at its startup and at each onboarding, so the link is an AIAC step, not a prerequisite. There is no `sub` client mapper on `rossoctl`. The exchanged token gets the same diff --git a/k8s/aiac-deployment-guide.md b/k8s/aiac-deployment-guide.md index ca9feb08..353021bb 100644 --- a/k8s/aiac-deployment-guide.md +++ b/k8s/aiac-deployment-guide.md @@ -324,23 +324,33 @@ docker build -f src/aiac/agent/controller/Dockerfile -t localhost/aiac-agent:loc kind load docker-image localhost/aiac-agent:local --name # Reload the ConfigMap: the apply writes the manifest values (placeholders), the patch -# puts the real LLM values from .env back (step 3b). +# puts the real LLM values from .env back (step 3b). A subshell: a failed check or step +# stops the rest of the block, not your shell. +( set -e set -a; . .env; set +a +: "${LLM_BASE_URL:?missing in .env}" "${LLM_MODEL:?missing in .env}" +# Read the live side and log level BEFORE the apply: the apply resets them to the manifest values. +SIDE=$(kubectl get configmap aiac-agent-config -n aiac-system -o jsonpath='{.data.AIAC_ENFORCEMENT_SIDE}') +LEVEL=$(kubectl get configmap aiac-agent-config -n aiac-system -o jsonpath='{.data.LOG_LEVEL}') kubectl apply -f k8s/agent-deployment.yaml -kubectl patch configmap aiac-agent-config -n aiac-system --type merge \ - -p "{\"data\":{\"LLM_BASE_URL\":\"${LLM_BASE_URL}\",\"LLM_MODEL\":\"${LLM_MODEL}\"}}" +# jq builds the JSON, so a quote or a backslash in a value cannot break the patch. +kubectl patch configmap aiac-agent-config -n aiac-system --type merge -p "$(jq -n \ + --arg url "$LLM_BASE_URL" --arg model "$LLM_MODEL" \ + --arg side "${SIDE:-target-side}" --arg level "${LEVEL:-INFO}" \ + '{data: {LLM_BASE_URL: $url, LLM_MODEL: $model, AIAC_ENFORCEMENT_SIDE: $side, LOG_LEVEL: $level}}')" # Restart AFTER the patch. The Agent reads the ConfigMap at start only. kubectl rollout restart deployment/aiac-agent -n aiac-system kubectl rollout status deployment/aiac-agent -n aiac-system +) ``` - The apply also writes the manifest values of `AIAC_ENFORCEMENT_SIDE` (`target-side`) and - `LOG_LEVEL` (`INFO`). On an `agent-side` deployment, patch `AIAC_ENFORCEMENT_SIDE` back - **before** the restart. See `docs/agents/deploy.md`, `ac-side`. Patch `LOG_LEVEL` again if you - had raised it. -- `.env` must have `LLM_BASE_URL` and `LLM_MODEL`. If one is missing, stop and ask. Do not leave - the placeholder in the ConfigMap. + `LOG_LEVEL` (`INFO`). The block reads the live values before the apply and patches them back + before the restart, so an `agent-side` deployment stays `agent-side`. To change the side, see + `docs/agents/deploy.md`, `ac-side`. +- `.env` must have `LLM_BASE_URL` and `LLM_MODEL`. If one is missing, the `:?` check stops the + block before the apply. Then stop and ask. Do not leave the placeholder in the ConfigMap. - The Secret `aiac-agent-secret` (`LLM_API_KEY`) is not changed by this flow. To change it, see step 3b. diff --git a/k8s/opa-kind-runbook.md b/k8s/opa-kind-runbook.md index 3de409e6..c2fe4386 100644 --- a/k8s/opa-kind-runbook.md +++ b/k8s/opa-kind-runbook.md @@ -227,23 +227,32 @@ A manual probe that mints a user token with the password grant through the login client `rossoctl` needs Direct Access Grants on that client. In a fresh realm it is off, and the mint fails with `unauthorized_client`. The system tests do not need this step: their fixture enables the grant itself -(`ensure_direct_access_grants` in `test/system/uc1_onboard.py`). To enable it by -hand (a partial update: only this field changes): +(`ensure_direct_access_grants` in `test/system/uc1_onboard.py`) and turns it off +again at teardown if it was off. To enable it by hand (a partial update: only +this field changes), source the repo-root `.env` first. The snippet uses the +same variables as the harness: ```bash -KC=http://keycloak.localtest.me:8080 +( set -e # a subshell: a failed check stops the snippet, not your shell +set -a; . .env; set +a +KC=${KEYCLOAK_URL:?}; REALM=${AIAC_TEST_REALM:-rossoctl} ADMIN=$(curl -s -X POST "$KC/realms/master/protocol/openid-connect/token" \ - -d client_id=admin-cli -d username=admin -d password=admin -d grant_type=password \ - | python3 -c 'import sys,json;print(json.load(sys.stdin)["access_token"])') -CID=$(curl -s -H "Authorization: Bearer $ADMIN" "$KC/admin/realms/rossoctl/clients?clientId=rossoctl" \ - | python3 -c 'import sys,json;print(json.load(sys.stdin)[0]["id"])') + -d client_id=admin-cli -d grant_type=password \ + --data-urlencode "username=${KEYCLOAK_ADMIN_USERNAME:?}" \ + --data-urlencode "password=${KEYCLOAK_ADMIN_PASSWORD:?}" \ + | python3 -c 'import sys,json;print(json.load(sys.stdin).get("access_token",""))') +[ -n "$ADMIN" ] || { echo "admin login failed: check KEYCLOAK_ADMIN_*"; exit 1; } +CID=$(curl -s -H "Authorization: Bearer $ADMIN" "$KC/admin/realms/$REALM/clients?clientId=rossoctl" \ + | python3 -c 'import sys,json;c=json.load(sys.stdin);print(c[0]["id"] if c else "")') +[ -n "$CID" ] || { echo "login client rossoctl not found in realm $REALM"; exit 1; } curl -s -o /dev/null -w "enable DAG HTTP %{http_code}\n" -X PUT \ -H "Authorization: Bearer $ADMIN" -H "Content-Type: application/json" \ - "$KC/admin/realms/rossoctl/clients/$CID" -d '{"directAccessGrantsEnabled": true}' + "$KC/admin/realms/$REALM/clients/$CID" -d '{"directAccessGrantsEnabled": true}' +) ``` -Expect `enable DAG HTTP 204`. The grant stays on; set the field to `false` the -same way to turn it off again. +Expect `enable DAG HTTP 204`. The grant stays on; set the field +to `false` the same way to turn it off again. --- diff --git a/test/system/disabled/test_uc1_onboard_cross_service_conflict.py b/test/system/disabled/test_uc1_onboard_cross_service_conflict.py index c307ab73..4e3d73ee 100644 --- a/test/system/disabled/test_uc1_onboard_cross_service_conflict.py +++ b/test/system/disabled/test_uc1_onboard_cross_service_conflict.py @@ -167,11 +167,12 @@ def test_cross_service_conflict_is_surfaced_as_422_conflict_report() -> None: # Open the cleanup guard BEFORE the first shared-state mutation: the clean-slate steps below # (Agent CR delete, Keycloak cleanup, Policy Store clear, provisioning) already mutate shared # cluster state, so a failure mid-setup must still hit the ``finally`` teardown. + grant_enabled = False try: uc1.delete_workload_crs() # clean policy slate (drop any prior run's CRs) uc1.cleanup_provisioned(admin, TEST_REALM) # clean slate (Keycloak) uc1.clear_policy_store() # clean slate ONCE — NOT between phases (the agent must see the tool's rule) - uc1.provision_realm_and_users(admin, TEST_REALM) # PRB reads the role universe + grant_enabled = uc1.provision_realm_and_users(admin, TEST_REALM) # PRB reads the role universe uc1.verify_login_subject( keycloak_url=keycloak_url, realm=TEST_REALM, user="test-user", password=scn.USER_PASSWORD ) @@ -187,6 +188,8 @@ def test_cross_service_conflict_is_surfaced_as_422_conflict_report() -> None: finally: uc1.delete_workload_crs() # after — drop any CR (there should be none on the raising path) uc1.cleanup_provisioned(admin, TEST_REALM) # restore the pre-run Keycloak state + if grant_enabled: + uc1.restore_direct_access_grants(admin, TEST_REALM) # Phase 1 persisted a tool-side deny; clear it so later integration tests don't read stale # inbound rules and become order-dependent. uc1.clear_policy_store() diff --git a/test/system/launcher.py b/test/system/launcher.py index 844cf6e7..9b405346 100644 --- a/test/system/launcher.py +++ b/test/system/launcher.py @@ -1033,9 +1033,9 @@ def verify_login_subject( The live loop keys OPA decisions on ``input.identity.subject`` (the token ``sub``). To mint the token, Direct Access Grants must be enabled on ``client_id``. The fixture enables it before this call - (``uc1_onboard.provision_realm_and_users`` -> ``ensure_direct_access_grants``, which fails on a missing - login client), so a mint failure here means the grant could not be written (the admin cannot manage - clients) or the user cannot log in, and it is a skip. The ``sub`` = username comes from the client + (``uc1_onboard.provision_realm_and_users`` -> ``ensure_direct_access_grants``, which itself skips with + the cause on a missing login client or a refused read or write), so a mint failure here means the + user cannot log in, and it is a skip. The ``sub`` = username comes from the client scope ``aiac-username-sub``, which the IdP Configuration Service links as a default scope to the login client (``client_id``, by default ``rossoctl``) at its startup and at each onboarding (D31). That is an AIAC step, not a prerequisite, so a wrong ``sub`` fails. ``uc1_onboard.require_subject_scope`` checks the links again after each onboarding. Returns the @@ -1047,8 +1047,8 @@ def verify_login_subject( except Exception as exc: # noqa: BLE001 — any mint failure is a skippable prerequisite gap pytest.skip( f"cannot mint a {user!r} token in realm {realm!r} via client {client_id!r}: {exc}. " - "Check that the login client has Direct Access Grants on (the fixture logs a warning if it " - "cannot enable it), and that the user has a password and a full profile." + "Check that the user exists, has a password and a full profile, and that the login client " + "has Direct Access Grants on." ) sub = jwt_claim(token, "sub") if sub != user: diff --git a/test/system/uc1_onboard.py b/test/system/uc1_onboard.py index 2bcd6d9e..0a5b78dd 100644 --- a/test/system/uc1_onboard.py +++ b/test/system/uc1_onboard.py @@ -41,7 +41,7 @@ * **Config** (env, spec § Configuration) — single stack, no variants. * **Keycloak** — ``connect_admin`` / ``provision_realm_and_users`` (the fixture UC-1 does *not* do, - with ``ensure_direct_access_grants`` on the login client) / + with ``ensure_direct_access_grants`` on the login client, ``restore_direct_access_grants`` at teardown) / ``cleanup_provisioned`` / ``delete_workload_registrations`` (teardown). * **CRs** — ``delete_workload_crs`` (the agent's and the tool's CR) / ``sweep_authpolicies`` (every other CR in the namespace) / ``delete_authpolicy`` (one CR, raising). @@ -327,14 +327,15 @@ def connect_admin(): ) -def provision_realm_and_users(admin, realm: str) -> None: +def provision_realm_and_users(admin, realm: str) -> bool: """Idempotently ensure ``realm`` holds ``scenario_uc1``'s users + realm roles with the descriptions the PRB reads. Realm roles carry the ``aiac.managed`` marker so the IdP populates each role's ``actorIds`` (member usernames) — the PCE needs them to build the ``subject_roles`` map the inbound/outbound gates key on. Also turns on Direct Access Grants on the login client (``ensure_direct_access_grants``), so that ``verify_login_subject`` and ``mint_token`` can use the password grant. Never deletes/recreates — reruns converge (spec: shared realm, leave-in-place: the - users, the roles and the grant stay after the run).""" + users and the roles stay after the run). Returns ``True`` when it turned the grant on: the caller + turns it off again at teardown (``restore_direct_access_grants``).""" from keycloak.exceptions import KeycloakError try: @@ -355,7 +356,7 @@ def provision_realm_and_users(admin, realm: str) -> None: admin.set_user_password(user_id, scn.USER_PASSWORD, temporary=False) admin.assign_realm_roles(user_id, [admin.get_realm_role(role_name)]) - ensure_direct_access_grants(admin, realm) + return ensure_direct_access_grants(admin, realm) def _user_profile(username: str) -> dict: @@ -1033,48 +1034,78 @@ def subject_scope_problems( return problems -def login_client(admin) -> dict | None: +def login_client(admin, realm: str = TEST_REALM) -> dict | None: """The Keycloak client representation of the login client ``rossoctl`` (``KEYCLOAK_CLIENT_ID``, the - client the harness mints user tokens through), keyed on its ``clientId``; ``None`` if absent.""" - admin.change_current_realm(TEST_REALM) + client the harness mints user tokens through) in ``realm``, keyed on its ``clientId``; ``None`` if + absent.""" + admin.change_current_realm(realm) return next((c for c in admin.get_clients() if c.get("clientId") == KEYCLOAK_CLIENT_ID), None) -def ensure_direct_access_grants(admin, realm: str) -> None: +def ensure_direct_access_grants(admin, realm: str) -> bool: """Turn on Direct Access Grants on the login client (``KEYCLOAK_CLIENT_ID``) of ``realm``, so that ``mint_token`` can use the password grant. ``provision_realm_and_users`` calls it, so each setup path that provisions the users also gets the grant. Idempotent: a client that has the grant already is not written. The write is a partial update (only ``directAccessGrantsEnabled``), so it does not write back a stale copy of the client over a change - that the IdP Configuration Service made (for example its D31 scope link). The grant stays after the - run (leave-in-place, like the users); the change is logged, so the owner of a realm that had it off - on purpose can see it. - - Raises ``RuntimeError`` when the login client does not exist: the realm owns its login client, so - nothing is created, and a missing login client is a failure (``require_subject_scope``), not a skip. - A ``KeycloakError`` from the update (for example the admin cannot manage clients) is logged, not - raised: ``verify_login_subject`` then cannot mint the token and skips with the cause.""" + that the IdP Configuration Service made (for example its D31 scope link). Returns ``True`` when this + call turned the grant on: the caller then turns it off again at teardown + (``restore_direct_access_grants``), so a realm that had it off on purpose stays as it was. + + ``pytest.skip`` (with the cause) when the login client does not exist (the realm owns its login + client, so nothing is created) or when the admin API refuses the read or the update (for example the + admin cannot manage clients). Both are prerequisite gaps, not AIAC failures.""" + import pytest from keycloak.exceptions import KeycloakError - admin.change_current_realm(realm) - client = next((c for c in admin.get_clients() if c.get("clientId") == KEYCLOAK_CLIENT_ID), None) + try: + client = login_client(admin, realm) + except KeycloakError as exc: + pytest.skip( + f"cannot read the login client {KEYCLOAK_CLIENT_ID!r} in realm {realm!r} to enable Direct Access " + f"Grants: {exc}. Give the admin user the view-clients and manage-clients roles." + ) if client is None: - raise RuntimeError( + pytest.skip( f"the login client {KEYCLOAK_CLIENT_ID!r} does not exist in realm {realm!r}, so the harness cannot " "mint a user token through it (the realm owns its login client; the harness does not create it)" ) if client.get("directAccessGrantsEnabled"): - return + return False try: admin.update_client(client["id"], {"directAccessGrantsEnabled": True}) - log.warning( - "enabled Direct Access Grants on the login client %r in realm %r (left on after the run)", + except KeycloakError as exc: + pytest.skip( + f"cannot enable Direct Access Grants on the login client {KEYCLOAK_CLIENT_ID!r} in realm {realm!r}: " + f"{exc}. Give the admin user the manage-clients role, or enable the grant by hand " + "(k8s/opa-kind-runbook.md, 'Direct Access Grants on the login client')." + ) + log.warning( + "enabled Direct Access Grants on the login client %r in realm %r (turned off again at teardown)", + KEYCLOAK_CLIENT_ID, + realm, + ) + return True + + +def restore_direct_access_grants(admin, realm: str) -> None: + """Turn Direct Access Grants off again on the login client of ``realm``: the teardown half of + ``ensure_direct_access_grants``, called only when that call turned it on. Best-effort: a missing + client or a ``KeycloakError`` is logged, never raised, so it does not mask a teardown failure.""" + from keycloak.exceptions import KeycloakError + + try: + client = login_client(admin, realm) + if client is not None: + admin.update_client(client["id"], {"directAccessGrantsEnabled": False}) + except KeycloakError as exc: + log.error( + "cannot turn Direct Access Grants off again on the login client %r in realm %r: %s", KEYCLOAK_CLIENT_ID, realm, + exc, ) - except KeycloakError as exc: - log.warning("cannot enable Direct Access Grants on the login client %r: %s", KEYCLOAK_CLIENT_ID, exc) def _workload_list(workloads: str | Sequence[str]) -> list[str]: @@ -1403,8 +1434,8 @@ def onboarded_stack( Teardown first captures every AIAC CR and the global combiner to the pytest host (``capture_aiac_crs``, phase ``teardown``; best-effort, it never masks a failure), then is **full-to-pristine** — the workloads and every registration are removed and the removal is - verified. The realm setup of ``provision_realm_and_users`` (the users, the roles and Direct Access - Grants on the login client) is shared realm state and stays. The capture directories are named for + verified. The users and the roles of ``provision_realm_and_users`` are shared realm state and stay; + Direct Access Grants on the login client is turned off again if the setup turned it on. The capture directories are named for the test module that started the stack. The workload order is the rung's identity — e.g. rung 2 passes ``[agent, tool]`` so tool onboarding completes the tool check (under target side it writes github-tool's own CR with both gates; under @@ -1460,17 +1491,19 @@ def onboarded_stack( "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." ) - provision_realm_and_users(admin, TEST_REALM) # BEFORE deploying (PRB reads the role universe when the event fires) - # ``provision_realm_and_users`` also enables Direct Access Grants on the login client (the password - # grant of ``mint_token`` needs it); skip if a token still can't be minted. The login token's ``sub`` - # comes from ``aiac-username-sub``, which the IdP Configuration Service links to the login client at - # startup (D31): fail if it isn't the username. ``require_subject_scope`` below checks the links again - # after each onboarding. - verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) + # BEFORE deploying (PRB reads the role universe when the event fires). It also enables Direct Access + # Grants on the login client (the password grant of ``mint_token`` needs it; skip if it cannot); + # ``grant_enabled`` = it was off, so the teardown turns it off again. + grant_enabled = provision_realm_and_users(admin, TEST_REALM) tool_onboarded = scn.TOOL_WORKLOAD in workloads signals = list(ready_signals) if ready_signals is not None else _default_ready_signals(tool_onboarded) try: + # Inside the ``try``, so a skip here still restores the grant. Skip if a token still can't be + # minted. The login token's ``sub`` comes from ``aiac-username-sub``, which the IdP Configuration + # Service links to the login client at startup (D31): fail if it isn't the username. + # ``require_subject_scope`` below checks the links again after each onboarding. + verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) # mount this run's policy.md BEFORE deploying # Minimal probe ctx for the per-workload agent convergence gate: the inbound leg reaches the @@ -1641,6 +1674,8 @@ def _ready() -> bool: undeploy_workload(workload) # Explicit scrub — do not trust an unverified operator cascade. Same reset the pre-run slate runs. _scrub_to_pristine(admin) + if grant_enabled: # best-effort, before the pristine check (which can raise) + restore_direct_access_grants(admin, TEST_REALM) # Verify pristine: both clients gone AND no AuthorizationPolicy CR remains. A leaked footprint is # a test failure, not silent drift — but do not mask a failure already propagating out of the try. if not poll_until( @@ -2426,8 +2461,8 @@ def pristine_stack(workloads: Sequence[str], *, policy_md: str = scn.POLICY_ABST (a ``pre-run-slate`` CR capture, then undeploy + ``_scrub_to_pristine`` + ``reenable_provisioned_clients`` + clients-gone poll), same realm/users provisioning (with Direct Access Grants on the login client) + ``sub`` mapper gate, ``policy.md`` mount and image load for - ``workloads``; and the same teardown: a ``teardown`` CR capture first, then full-to-pristine, - verified (clients gone, no CR left).""" + ``workloads``; and the same teardown: a ``teardown`` CR capture first, then full-to-pristine (and + Direct Access Grants off again if the setup turned it on), verified (clients gone, no CR left).""" require_pipeline(namespace=NAMESPACE, workloads=[]) creds = require_env_or_skip("KEYCLOAK_URL", "KEYCLOAK_ADMIN_USERNAME", "KEYCLOAK_ADMIN_PASSWORD") keycloak_url = creds["KEYCLOAK_URL"] @@ -2446,10 +2481,10 @@ def pristine_stack(workloads: Sequence[str], *, policy_md: str = scn.POLICY_ABST f"pre-run cleanup left Keycloak client(s) {workload_clients_present(admin)} for {NAMESPACE!r} — the " "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." ) - provision_realm_and_users(admin, TEST_REALM) - verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) + grant_enabled = provision_realm_and_users(admin, TEST_REALM) try: + verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) load_workload_images(list(workloads)) yield {"admin": admin, "namespace": NAMESPACE, "keycloak_url": keycloak_url, "realm": TEST_REALM, "side": side} @@ -2458,6 +2493,8 @@ def pristine_stack(workloads: Sequence[str], *, policy_md: str = scn.POLICY_ABST for workload in reversed(list(workloads)): undeploy_workload(workload) _scrub_to_pristine(admin) + if grant_enabled: + restore_direct_access_grants(admin, TEST_REALM) if not poll_until( lambda: not workload_clients_present(admin) and no_authpolicies_remain(), timeout=DEPLOY_TIMEOUT, interval=5 ): diff --git a/test/unit/agent/uc/onboarding/test_uc1_login_client.py b/test/unit/agent/uc/onboarding/test_uc1_login_client.py index d21ffd09..a0d3bba3 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_login_client.py +++ b/test/unit/agent/uc/onboarding/test_uc1_login_client.py @@ -1,9 +1,11 @@ -"""``uc1_onboard.ensure_direct_access_grants`` turns on the password grant of the login client — pure -unit tests. +"""``uc1_onboard.ensure_direct_access_grants`` turns on the password grant of the login client, and +``restore_direct_access_grants`` turns it off again — pure unit tests. The harness mints user tokens through the login client ``rossoctl`` with the password grant, so the client needs Direct Access Grants on. ``provision_realm_and_users`` turns it on, so each setup path -(``onboarded_stack``, ``pristine_stack`` and the disabled cross-service test) gets it. These tests use a +(``onboarded_stack``, ``pristine_stack`` and the disabled cross-service test) gets it, and each +teardown turns it off again when the setup turned it on. A grant that cannot be written is a skip with +the cause. These tests use a ``MagicMock`` admin (no Keycloak). They import the system harness like ``test_uc1_subject_scope.py`` does. """ @@ -36,35 +38,63 @@ def _login(enabled: bool) -> dict: def test_the_grant_is_enabled_with_a_partial_update() -> None: admin = _admin(_login(False)) - uc1.ensure_direct_access_grants(admin, "rossoctl") + assert uc1.ensure_direct_access_grants(admin, "rossoctl") is True admin.change_current_realm.assert_called_with("rossoctl") admin.update_client.assert_called_once_with("uuid-1", {"directAccessGrantsEnabled": True}) def test_a_grant_that_is_on_is_not_written() -> None: admin = _admin(_login(True)) - uc1.ensure_direct_access_grants(admin, "rossoctl") + assert uc1.ensure_direct_access_grants(admin, "rossoctl") is False admin.update_client.assert_not_called() -def test_a_missing_login_client_fails_and_is_not_created() -> None: +def test_a_missing_login_client_skips_and_is_not_created() -> None: admin = _admin(None) - with pytest.raises(RuntimeError, match=uc1.KEYCLOAK_CLIENT_ID): + with pytest.raises(pytest.skip.Exception, match=uc1.KEYCLOAK_CLIENT_ID): uc1.ensure_direct_access_grants(admin, "rossoctl") admin.update_client.assert_not_called() admin.create_client.assert_not_called() -def test_a_refused_update_is_logged_not_raised(caplog: pytest.LogCaptureFixture) -> None: +def test_a_refused_read_skips_with_the_cause() -> None: + admin = _admin(_login(False)) + admin.get_clients.side_effect = KeycloakError(error_message="forbidden", response_code=403) + with pytest.raises(pytest.skip.Exception, match="cannot read the login client.*forbidden"): + uc1.ensure_direct_access_grants(admin, "rossoctl") + admin.update_client.assert_not_called() + + +def test_a_refused_update_skips_with_the_cause() -> None: admin = _admin(_login(False)) admin.update_client.side_effect = KeycloakError(error_message="forbidden", response_code=403) - uc1.ensure_direct_access_grants(admin, "rossoctl") # verify_login_subject then skips with the cause - assert "cannot enable Direct Access Grants" in caplog.text + with pytest.raises(pytest.skip.Exception, match="cannot enable Direct Access Grants.*forbidden"): + uc1.ensure_direct_access_grants(admin, "rossoctl") def test_provision_realm_and_users_enables_the_grant() -> None: admin = _admin(_login(False)) admin.create_user.side_effect = lambda payload, exist_ok=False: f"id-{payload['username']}" admin.get_user.return_value = {"email": "e", "firstName": "f", "lastName": "l"} - uc1.provision_realm_and_users(admin, "rossoctl") + assert uc1.provision_realm_and_users(admin, "rossoctl") is True admin.update_client.assert_called_once_with("uuid-1", {"directAccessGrantsEnabled": True}) + + +def test_the_restore_turns_the_grant_off_with_a_partial_update() -> None: + admin = _admin(_login(True)) + uc1.restore_direct_access_grants(admin, "rossoctl") + admin.change_current_realm.assert_called_with("rossoctl") + admin.update_client.assert_called_once_with("uuid-1", {"directAccessGrantsEnabled": False}) + + +def test_a_refused_restore_is_logged_not_raised(caplog: pytest.LogCaptureFixture) -> None: + admin = _admin(_login(True)) + admin.update_client.side_effect = KeycloakError(error_message="forbidden", response_code=403) + uc1.restore_direct_access_grants(admin, "rossoctl") # never masks a teardown failure + assert "cannot turn Direct Access Grants off again" in caplog.text + + +def test_the_restore_of_a_missing_login_client_writes_nothing() -> None: + admin = _admin(None) + uc1.restore_direct_access_grants(admin, "rossoctl") + admin.update_client.assert_not_called() From 4f8043bc3e152c4e8d8a18dcd4ff42cbbb894e55 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Thu, 15 Oct 2026 13:18:51 +0300 Subject: [PATCH 05/15] fix(test): address review on #238 - Add the context manager direct_access_grants. It turns on Direct Access Grants on the login client and turns it off again in its own finally, so a teardown step that raises cannot leave the grant on. - onboarded_stack, pristine_stack and the disabled cross-service test enter it after the skip gates and before the first mutation, so a skip at entry changes nothing. - provision_realm_and_users does users and roles only again. - login_client uses a query filtered by clientId; the restore uses the UUID that the enable found. - Docs: module docstring, the ConfigMap reload keeps all live keys, and the runbook snippet reads KEYCLOAK_ADMIN_REALM. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- docs/specs/PRD.md | 2 +- docs/testing/policy-pipeline.md | 2 +- docs/testing/uc1-onboarding-pipeline.md | 6 +- k8s/aiac-deployment-guide.md | 32 +- k8s/opa-kind-runbook.md | 8 +- ...test_uc1_onboard_cross_service_conflict.py | 58 +- test/system/launcher.py | 2 +- test/system/uc1_onboard.py | 607 +++++++++--------- .../uc/onboarding/test_uc1_login_client.py | 148 ++++- .../uc/onboarding/test_uc1_provision_users.py | 3 - .../uc/onboarding/test_uc1_subject_scope.py | 5 + 11 files changed, 500 insertions(+), 373 deletions(-) diff --git a/docs/specs/PRD.md b/docs/specs/PRD.md index 7be98c83..2dae8f97 100644 --- a/docs/specs/PRD.md +++ b/docs/specs/PRD.md @@ -819,7 +819,7 @@ The Controller reads `AIAC_ENFORCEMENT_SIDE` at start, and an unknown value stop - **The namespace pipelines.** The `authbridge-runtime-config` ConfigMap of each service namespace must have `opa` in the inbound pipeline, and also `mcp-parser` for a tool. Under agent side, the agents' namespaces also need `mcp-parser` and `opa` in the outbound pipeline. Onboarding check #2 finds a missing plugin. - **Pod restarts.** The webhook injects the sidecar only on pod CREATE. After a change to the pipeline or to `injectTools`, restart the agent pods and the tool pods (`rossoctl.io/type=agent` and `rossoctl.io/type=tool`). - **The username precondition (D31).** The CRs key users by username, and `sub` is the username. So usernames must be unique, must not change and must never be used again. Keep `editUsernameAllowed: false` and `registrationAllowed: false` in the realm (the operator's realm template sets `registrationAllowed: false`; it does not set `editUsernameAllowed`, so that value is `false` only by the Keycloak default). An admin must never create a deleted username again: the new user would get the grants of the old user until AIAC writes the CRs again. -- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`provision_realm_and_users` → `ensure_direct_access_grants`) and turns it off again at teardown if it was off; a manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). +- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`direct_access_grants`, before its first mutation) and turns it off again at exit if it was off; a manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). - **The legacy token exchange (optional hardening).** AIAC does not need the Keycloak feature `token-exchange` (V1) to be off: the AuthBridge requests go to the standard exchange (V2), and a request that V1 handles still gets `sub` = the username for an onboarded target (§6). Keycloak recommends to turn V1 off, because then a request that V2 declines fails with `400` ("Standard token exchange is not enabled for the requested client") and does not silently go to a different engine. `KC_FEATURES` is in the Keycloak deployment of the rossoctl platform, not in this repo. If V1 stays on, AIAC needs no change. Detail: [`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md). diff --git a/docs/testing/policy-pipeline.md b/docs/testing/policy-pipeline.md index 930497b4..724240bc 100644 --- a/docs/testing/policy-pipeline.md +++ b/docs/testing/policy-pipeline.md @@ -93,7 +93,7 @@ over the fully onboarded stack. provisions these; UC-1 does not), `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its `sub` is the username, and `ensure_agent_policy` mounts the single abstract `policy.md` on the Controller pod. `verify_login_subject` skips when it cannot mint the - token (`provision_realm_and_users` enables Direct Access Grants first, and skips with the cause if + token (the fixture enables Direct Access Grants before the slate, and skips with the cause if the login client is absent or the admin cannot manage clients; the teardown turns the grant off again if it was off), and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, which the IdP Configuration Service links to diff --git a/docs/testing/uc1-onboarding-pipeline.md b/docs/testing/uc1-onboarding-pipeline.md index 2e45f4b2..98912d23 100644 --- a/docs/testing/uc1-onboarding-pipeline.md +++ b/docs/testing/uc1-onboarding-pipeline.md @@ -224,9 +224,9 @@ cleanly** when any of it is absent (they never stand it up, and never false-pass *[Scenario](#scenario)* — via `KeycloakAdmin` into `AIAC_TEST_REALM`, **before** deploying (the PRB reads the realm role universe when the event fires `onboard_and_apply`); idempotent; left in place. `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its - `sub` is the username. `provision_realm_and_users` first enables Direct Access Grants on - `rossoctl` (`ensure_direct_access_grants`; the teardown turns it off again if it was off) and - skips with the cause if `rossoctl` is absent or the admin cannot manage clients. It skips when it + `sub` is the username. Before the pre-run slate, the fixture enables Direct Access Grants on + `rossoctl` (the context manager `direct_access_grants`; at exit its own `finally` turns it off again + if it was off) and skips with the cause if `rossoctl` is absent or the admin cannot manage clients. It skips when it still cannot mint the token, and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, the one source of the rule (D31). The IdP Configuration Service links it to `rossoctl` at its startup and at each onboarding, so the link is an AIAC step, not a diff --git a/k8s/aiac-deployment-guide.md b/k8s/aiac-deployment-guide.md index 353021bb..be303849 100644 --- a/k8s/aiac-deployment-guide.md +++ b/k8s/aiac-deployment-guide.md @@ -323,21 +323,20 @@ LLM endpoint and model of the repo-root `.env`, and not with an older value in t docker build -f src/aiac/agent/controller/Dockerfile -t localhost/aiac-agent:local src/ kind load docker-image localhost/aiac-agent:local --name -# Reload the ConfigMap: the apply writes the manifest values (placeholders), the patch -# puts the real LLM values from .env back (step 3b). A subshell: a failed check or step -# stops the rest of the block, not your shell. -( set -e +# Reload the ConfigMap: the apply writes the manifest value of each key (the LLM keys are +# placeholders). The patch puts every live key back and then the real LLM values from .env +# (step 3b). A subshell: a failed check or step stops the rest of the block, not your shell. +( set -eo pipefail set -a; . .env; set +a : "${LLM_BASE_URL:?missing in .env}" "${LLM_MODEL:?missing in .env}" -# Read the live side and log level BEFORE the apply: the apply resets them to the manifest values. -SIDE=$(kubectl get configmap aiac-agent-config -n aiac-system -o jsonpath='{.data.AIAC_ENFORCEMENT_SIDE}') -LEVEL=$(kubectl get configmap aiac-agent-config -n aiac-system -o jsonpath='{.data.LOG_LEVEL}') +# Read ALL live keys BEFORE the apply: the apply resets each key to its manifest value. +LIVE=$(kubectl get configmap aiac-agent-config -n aiac-system -o json | jq -c '.data // {}') kubectl apply -f k8s/agent-deployment.yaml -# jq builds the JSON, so a quote or a backslash in a value cannot break the patch. +# Live keys first, the .env LLM values on top. jq builds the JSON, so a quote or a backslash in +# a value cannot break the patch. kubectl patch configmap aiac-agent-config -n aiac-system --type merge -p "$(jq -n \ - --arg url "$LLM_BASE_URL" --arg model "$LLM_MODEL" \ - --arg side "${SIDE:-target-side}" --arg level "${LEVEL:-INFO}" \ - '{data: {LLM_BASE_URL: $url, LLM_MODEL: $model, AIAC_ENFORCEMENT_SIDE: $side, LOG_LEVEL: $level}}')" + --argjson live "$LIVE" --arg url "$LLM_BASE_URL" --arg model "$LLM_MODEL" \ + '{data: ($live + {LLM_BASE_URL: $url, LLM_MODEL: $model})}')" # Restart AFTER the patch. The Agent reads the ConfigMap at start only. kubectl rollout restart deployment/aiac-agent -n aiac-system @@ -345,10 +344,13 @@ kubectl rollout status deployment/aiac-agent -n aiac-system ) ``` -- The apply also writes the manifest values of `AIAC_ENFORCEMENT_SIDE` (`target-side`) and - `LOG_LEVEL` (`INFO`). The block reads the live values before the apply and patches them back - before the restart, so an `agent-side` deployment stays `agent-side`. To change the side, see - `docs/agents/deploy.md`, `ac-side`. +- The apply writes the manifest value of each key, for example `AIAC_ENFORCEMENT_SIDE` + (`target-side`), `LOG_LEVEL` (`INFO`) and `LLM_REQUEST_TIMEOUT` (`120`). The block reads all + live keys before the apply and patches them back before the restart. Thus an `agent-side` + deployment stays `agent-side`, and a tuned key (for example `LLM_MAX_RETRIES` or + `LLM_RETRY_BACKOFF_MAX`) keeps its live value. Only a key that is new in the manifest gets its + manifest value. To set a key back to its manifest default, patch it by hand. To change the + side, see `docs/agents/deploy.md`, `ac-side`. - `.env` must have `LLM_BASE_URL` and `LLM_MODEL`. If one is missing, the `:?` check stops the block before the apply. Then stop and ask. Do not leave the placeholder in the ConfigMap. - The Secret `aiac-agent-secret` (`LLM_API_KEY`) is not changed by this flow. To change it, see diff --git a/k8s/opa-kind-runbook.md b/k8s/opa-kind-runbook.md index c2fe4386..d5262071 100644 --- a/k8s/opa-kind-runbook.md +++ b/k8s/opa-kind-runbook.md @@ -227,7 +227,7 @@ A manual probe that mints a user token with the password grant through the login client `rossoctl` needs Direct Access Grants on that client. In a fresh realm it is off, and the mint fails with `unauthorized_client`. The system tests do not need this step: their fixture enables the grant itself -(`ensure_direct_access_grants` in `test/system/uc1_onboard.py`) and turns it off +(`direct_access_grants` in `test/system/uc1_onboard.py`) and turns it off again at teardown if it was off. To enable it by hand (a partial update: only this field changes), source the repo-root `.env` first. The snippet uses the same variables as the harness: @@ -235,13 +235,13 @@ same variables as the harness: ```bash ( set -e # a subshell: a failed check stops the snippet, not your shell set -a; . .env; set +a -KC=${KEYCLOAK_URL:?}; REALM=${AIAC_TEST_REALM:-rossoctl} -ADMIN=$(curl -s -X POST "$KC/realms/master/protocol/openid-connect/token" \ +KC=${KEYCLOAK_URL:?}; REALM=${AIAC_TEST_REALM:-rossoctl}; ADMIN_REALM=${KEYCLOAK_ADMIN_REALM:-master} +ADMIN=$(curl -s -X POST "$KC/realms/$ADMIN_REALM/protocol/openid-connect/token" \ -d client_id=admin-cli -d grant_type=password \ --data-urlencode "username=${KEYCLOAK_ADMIN_USERNAME:?}" \ --data-urlencode "password=${KEYCLOAK_ADMIN_PASSWORD:?}" \ | python3 -c 'import sys,json;print(json.load(sys.stdin).get("access_token",""))') -[ -n "$ADMIN" ] || { echo "admin login failed: check KEYCLOAK_ADMIN_*"; exit 1; } +[ -n "$ADMIN" ] || { echo "admin login at realm $ADMIN_REALM failed: check KEYCLOAK_ADMIN_*"; exit 1; } CID=$(curl -s -H "Authorization: Bearer $ADMIN" "$KC/admin/realms/$REALM/clients?clientId=rossoctl" \ | python3 -c 'import sys,json;c=json.load(sys.stdin);print(c[0]["id"] if c else "")') [ -n "$CID" ] || { echo "login client rossoctl not found in realm $REALM"; exit 1; } diff --git a/test/system/disabled/test_uc1_onboard_cross_service_conflict.py b/test/system/disabled/test_uc1_onboard_cross_service_conflict.py index 4e3d73ee..e00f3075 100644 --- a/test/system/disabled/test_uc1_onboard_cross_service_conflict.py +++ b/test/system/disabled/test_uc1_onboard_cross_service_conflict.py @@ -164,35 +164,35 @@ def test_cross_service_conflict_is_surfaced_as_422_conflict_report() -> None: keycloak_url = creds["KEYCLOAK_URL"] admin = uc1.connect_admin() - # Open the cleanup guard BEFORE the first shared-state mutation: the clean-slate steps below - # (Agent CR delete, Keycloak cleanup, Policy Store clear, provisioning) already mutate shared - # cluster state, so a failure mid-setup must still hit the ``finally`` teardown. - grant_enabled = False - try: - uc1.delete_workload_crs() # clean policy slate (drop any prior run's CRs) - uc1.cleanup_provisioned(admin, TEST_REALM) # clean slate (Keycloak) - uc1.clear_policy_store() # clean slate ONCE — NOT between phases (the agent must see the tool's rule) - grant_enabled = uc1.provision_realm_and_users(admin, TEST_REALM) # PRB reads the role universe - uc1.verify_login_subject( - keycloak_url=keycloak_url, realm=TEST_REALM, user="test-user", password=scn.USER_PASSWORD - ) - - # Phase 1 — tool onboarding under the exclusivity policy: clean (200), persists the tool-side - # DENY(tester -> github-tool.source-*) on SPM(github-tool). - _onboard_via_fresh_controller(POLICY_TOOL_EXCLUSIVE, scn.TOOL_WORKLOAD, expect_conflict=False) - - # Phase 2 — agent onboarding under the source-granting policy: the outbound subject gate's fresh - # ALLOW(tester -> github-tool.source-*) collides with the tool's already-applied DENY on the same - # (role, scope). The #2504 store read surfaces it -> 422 + ConflictReport. - report = _onboard_via_fresh_controller(POLICY_AGENT_GRANTS_SOURCE, scn.AGENT_WORKLOAD, expect_conflict=True) - finally: - uc1.delete_workload_crs() # after — drop any CR (there should be none on the raising path) - uc1.cleanup_provisioned(admin, TEST_REALM) # restore the pre-run Keycloak state - if grant_enabled: - uc1.restore_direct_access_grants(admin, TEST_REALM) - # Phase 1 persisted a tool-side deny; clear it so later integration tests don't read stale - # inbound rules and become order-dependent. - uc1.clear_policy_store() + # The password grant on the login client, before the first mutation: a skip at entry changes nothing, + # and its own ``finally`` turns the grant off again if it was off, whatever cleanup step raises. + with uc1.direct_access_grants(admin, TEST_REALM): + # Open the cleanup guard BEFORE the first shared-state mutation: the clean-slate steps below + # (Agent CR delete, Keycloak cleanup, Policy Store clear, provisioning) already mutate shared + # cluster state, so a failure mid-setup must still hit the ``finally`` teardown. + try: + uc1.delete_workload_crs() # clean policy slate (drop any prior run's CRs) + uc1.cleanup_provisioned(admin, TEST_REALM) # clean slate (Keycloak) + uc1.clear_policy_store() # clean slate ONCE — NOT between phases (the agent must see the tool's rule) + uc1.provision_realm_and_users(admin, TEST_REALM) # PRB reads the role universe + uc1.verify_login_subject( + keycloak_url=keycloak_url, realm=TEST_REALM, user="test-user", password=scn.USER_PASSWORD + ) + + # Phase 1 — tool onboarding under the exclusivity policy: clean (200), persists the tool-side + # DENY(tester -> github-tool.source-*) on SPM(github-tool). + _onboard_via_fresh_controller(POLICY_TOOL_EXCLUSIVE, scn.TOOL_WORKLOAD, expect_conflict=False) + + # Phase 2 — agent onboarding under the source-granting policy: the outbound subject gate's fresh + # ALLOW(tester -> github-tool.source-*) collides with the tool's already-applied DENY on the same + # (role, scope). The #2504 store read surfaces it -> 422 + ConflictReport. + report = _onboard_via_fresh_controller(POLICY_AGENT_GRANTS_SOURCE, scn.AGENT_WORKLOAD, expect_conflict=True) + finally: + uc1.delete_workload_crs() # after — drop any CR (there should be none on the raising path) + uc1.cleanup_provisioned(admin, TEST_REALM) # restore the pre-run Keycloak state + # Phase 1 persisted a tool-side deny; clear it so later integration tests don't read stale + # inbound rules and become order-dependent. + uc1.clear_policy_store() # The 422 body is the shared ConflictReport shape (same as the deterministic routes test), carrying # at least one cross-service conflict with a real role + scope. Structural (not id-pinned) so it is diff --git a/test/system/launcher.py b/test/system/launcher.py index 9b405346..e6946f07 100644 --- a/test/system/launcher.py +++ b/test/system/launcher.py @@ -1033,7 +1033,7 @@ def verify_login_subject( The live loop keys OPA decisions on ``input.identity.subject`` (the token ``sub``). To mint the token, Direct Access Grants must be enabled on ``client_id``. The fixture enables it before this call - (``uc1_onboard.provision_realm_and_users`` -> ``ensure_direct_access_grants``, which itself skips with + (``uc1_onboard.direct_access_grants`` -> ``ensure_direct_access_grants``, which itself skips with the cause on a missing login client or a refused read or write), so a mint failure here means the user cannot log in, and it is a skip. The ``sub`` = username comes from the client scope ``aiac-username-sub``, which the IdP Configuration Service links as a default scope to the diff --git a/test/system/uc1_onboard.py b/test/system/uc1_onboard.py index 0a5b78dd..5887543d 100644 --- a/test/system/uc1_onboard.py +++ b/test/system/uc1_onboard.py @@ -31,7 +31,9 @@ → capture every AIAC CR + the global combiner to the pytest host (``teardown``) — the first teardown step, on the success and the failure path, best-effort (it never raises) → teardown full-to-pristine (undeploy + delete all registrations + delete CRs, verified); the - realm users, roles and Direct Access Grants on the login client stay (leave-in-place) + realm users and roles stay (leave-in-place) + → Direct Access Grants on the login client: the fixture turns it on before the pre-run slate and + turns it off again at the end if it was off (``direct_access_grants``) The only thing that differs between rungs is *which workloads are onboarded and in what order*, so all the machinery lives here and each ``test_uc1_onboard_*.py`` supplies just its own oracle @@ -40,8 +42,8 @@ This module owns: * **Config** (env, spec § Configuration) — single stack, no variants. -* **Keycloak** — ``connect_admin`` / ``provision_realm_and_users`` (the fixture UC-1 does *not* do, - with ``ensure_direct_access_grants`` on the login client, ``restore_direct_access_grants`` at teardown) / +* **Keycloak** — ``connect_admin`` / ``direct_access_grants`` (the password grant on the login client, + for the life of the fixture) / ``provision_realm_and_users`` (the fixture UC-1 does *not* do) / ``cleanup_provisioned`` / ``delete_workload_registrations`` (teardown). * **CRs** — ``delete_workload_crs`` (the agent's and the tool's CR) / ``sweep_authpolicies`` (every other CR in the namespace) / ``delete_authpolicy`` (one CR, raising). @@ -327,15 +329,13 @@ def connect_admin(): ) -def provision_realm_and_users(admin, realm: str) -> bool: +def provision_realm_and_users(admin, realm: str) -> None: """Idempotently ensure ``realm`` holds ``scenario_uc1``'s users + realm roles with the descriptions the PRB reads. Realm roles carry the ``aiac.managed`` marker so the IdP populates each role's ``actorIds`` (member usernames) — the PCE needs them to build the ``subject_roles`` - map the inbound/outbound gates key on. Also turns on Direct Access Grants on the login client - (``ensure_direct_access_grants``), so that ``verify_login_subject`` and ``mint_token`` can use the - password grant. Never deletes/recreates — reruns converge (spec: shared realm, leave-in-place: the - users and the roles stay after the run). Returns ``True`` when it turned the grant on: the caller - turns it off again at teardown (``restore_direct_access_grants``).""" + map the inbound/outbound gates key on. Never deletes/recreates — reruns converge (spec: shared + realm, leave-in-place: the users and the roles stay after the run). The password grant on the login + client is not set here: see ``direct_access_grants``.""" from keycloak.exceptions import KeycloakError try: @@ -356,8 +356,6 @@ def provision_realm_and_users(admin, realm: str) -> bool: admin.set_user_password(user_id, scn.USER_PASSWORD, temporary=False) admin.assign_realm_roles(user_id, [admin.get_realm_role(role_name)]) - return ensure_direct_access_grants(admin, realm) - def _user_profile(username: str) -> dict: """The profile fields that a fresh ``rossoctl`` realm requires (``email``, ``firstName``, @@ -1036,26 +1034,27 @@ def subject_scope_problems( def login_client(admin, realm: str = TEST_REALM) -> dict | None: """The Keycloak client representation of the login client ``rossoctl`` (``KEYCLOAK_CLIENT_ID``, the - client the harness mints user tokens through) in ``realm``, keyed on its ``clientId``; ``None`` if - absent.""" + client the harness mints user tokens through) in ``realm``; ``None`` if absent. A query filtered by + ``clientId`` finds its UUID, so the call does not list every client of the realm.""" admin.change_current_realm(realm) - return next((c for c in admin.get_clients() if c.get("clientId") == KEYCLOAK_CLIENT_ID), None) + client_uuid = admin.get_client_id(KEYCLOAK_CLIENT_ID) + return admin.get_client(client_uuid) if client_uuid else None -def ensure_direct_access_grants(admin, realm: str) -> bool: +def ensure_direct_access_grants(admin, realm: str) -> str | None: """Turn on Direct Access Grants on the login client (``KEYCLOAK_CLIENT_ID``) of ``realm``, so that - ``mint_token`` can use the password grant. ``provision_realm_and_users`` calls it, so each setup path - that provisions the users also gets the grant. + ``mint_token`` can use the password grant. Use it through ``direct_access_grants``, which turns the + grant off again. Idempotent: a client that has the grant already is not written. The write is a partial update (only ``directAccessGrantsEnabled``), so it does not write back a stale copy of the client over a change - that the IdP Configuration Service made (for example its D31 scope link). Returns ``True`` when this - call turned the grant on: the caller then turns it off again at teardown - (``restore_direct_access_grants``), so a realm that had it off on purpose stays as it was. + that the IdP Configuration Service made (for example its D31 scope link). Returns the client UUID + when this call turned the grant on, else ``None``. ``pytest.skip`` (with the cause) when the login client does not exist (the realm owns its login client, so nothing is created) or when the admin API refuses the read or the update (for example the - admin cannot manage clients). Both are prerequisite gaps, not AIAC failures.""" + admin cannot manage clients). Both are prerequisite gaps, not AIAC failures. A refused update writes + nothing, so each skip leaves the realm as it was.""" import pytest from keycloak.exceptions import KeycloakError @@ -1064,7 +1063,8 @@ def ensure_direct_access_grants(admin, realm: str) -> bool: except KeycloakError as exc: pytest.skip( f"cannot read the login client {KEYCLOAK_CLIENT_ID!r} in realm {realm!r} to enable Direct Access " - f"Grants: {exc}. Give the admin user the view-clients and manage-clients roles." + f"Grants: {exc}. Make sure that the realm exists and that the admin user has the view-clients and " + "manage-clients roles." ) if client is None: pytest.skip( @@ -1072,7 +1072,7 @@ def ensure_direct_access_grants(admin, realm: str) -> bool: "mint a user token through it (the realm owns its login client; the harness does not create it)" ) if client.get("directAccessGrantsEnabled"): - return False + return None try: admin.update_client(client["id"], {"directAccessGrantsEnabled": True}) except KeycloakError as exc: @@ -1086,20 +1086,17 @@ def ensure_direct_access_grants(admin, realm: str) -> bool: KEYCLOAK_CLIENT_ID, realm, ) - return True + return client["id"] -def restore_direct_access_grants(admin, realm: str) -> None: - """Turn Direct Access Grants off again on the login client of ``realm``: the teardown half of - ``ensure_direct_access_grants``, called only when that call turned it on. Best-effort: a missing - client or a ``KeycloakError`` is logged, never raised, so it does not mask a teardown failure.""" - from keycloak.exceptions import KeycloakError - +def restore_direct_access_grants(admin, realm: str, client_uuid: str) -> None: + """Turn Direct Access Grants off again on the login client ``client_uuid`` of ``realm`` (the UUID that + ``ensure_direct_access_grants`` returned): a partial update, like the enable. Best-effort: any error + is logged, never raised, so it does not mask a teardown failure that is propagating.""" try: - client = login_client(admin, realm) - if client is not None: - admin.update_client(client["id"], {"directAccessGrantsEnabled": False}) - except KeycloakError as exc: + admin.change_current_realm(realm) + admin.update_client(client_uuid, {"directAccessGrantsEnabled": False}) + except Exception as exc: # noqa: BLE001 — best-effort teardown: log, never mask the real failure log.error( "cannot turn Direct Access Grants off again on the login client %r in realm %r: %s", KEYCLOAK_CLIENT_ID, @@ -1108,6 +1105,23 @@ def restore_direct_access_grants(admin, realm: str) -> None: ) +@contextmanager +def direct_access_grants(admin, realm: str) -> Iterator[None]: + """Direct Access Grants on the login client of ``realm`` for the ``with`` body. + + At entry: ``ensure_direct_access_grants`` (turn it on, or skip with the cause). At exit: if the entry + turned it on, ``restore_direct_access_grants`` turns it off again, so a realm that had it off stays + as it was. The restore is in this context manager's own ``finally``: it runs when the body raises, + skips or returns, and also when a teardown step of the body raises. Enter it before the first + mutation of shared cluster state, so a skip at entry changes nothing.""" + client_uuid = ensure_direct_access_grants(admin, realm) + try: + yield + finally: + if client_uuid is not None: + restore_direct_access_grants(admin, realm, client_uuid) + + def _workload_list(workloads: str | Sequence[str]) -> list[str]: """One workload name or a sequence of them, as a list.""" return [workloads] if isinstance(workloads, str) else list(workloads) @@ -1434,10 +1448,11 @@ def onboarded_stack( Teardown first captures every AIAC CR and the global combiner to the pytest host (``capture_aiac_crs``, phase ``teardown``; best-effort, it never masks a failure), then is **full-to-pristine** — the workloads and every registration are removed and the removal is - verified. The users and the roles of ``provision_realm_and_users`` are shared realm state and stay; - Direct Access Grants on the login client is turned off again if the setup turned it on. The capture directories are named for - the test module that started the stack. The workload order is the rung's identity — e.g. rung 2 - passes ``[agent, tool]`` so tool onboarding + verified. The users and the roles of ``provision_realm_and_users`` are shared realm state and stay. + Direct Access Grants on the login client is on for the whole flow (``direct_access_grants``, entered + after the skip gates and before the pre-run slate) and is turned off again at the end if it was off. + The capture directories are named for the test module that started the stack. The workload order is + the rung's identity — e.g. rung 2 passes ``[agent, tool]`` so tool onboarding completes the tool check (under target side it writes github-tool's own CR with both gates; under agent side it completes the agent's outbound gate); rung 3 passes ``[tool, agent]`` and must converge to the same live decisions. @@ -1470,226 +1485,229 @@ def onboarded_stack( require_event_path(admin=admin, realm=TEST_REALM) side = live_enforcement_side() # read-only; the suite runs under the live side - # Pre-run: start from a NO-WORKLOADS slate. The trigger is event-driven — deploying a workload only - # re-fires ``CLIENT_CREATED`` if there is no client for it yet, so a leftover workload + its Keycloak - # client from a prior run would silently swallow the event and onboarding would never trigger. Remove - # both workloads and every registration first (best-effort, tolerant of already-absent), then poll the - # clients actually gone before deploying. Capture the CRs first: a prior run's leftovers are evidence. - label = _capture_label() # the rung, read once here (a session fixture is torn down after the session) - capture_aiac_crs("pre-run-slate", label=label) - undeploy_workload(scn.AGENT_WORKLOAD) - undeploy_workload(scn.TOOL_WORKLOAD) - _scrub_to_pristine( - admin - ) # clients + *-aud scopes + workload CRs + stray AuthorizationPolicy CRs + roles/scopes + store - reenable_provisioned_clients( - admin, TEST_REALM - ) # undo any prior run's failed-service disable (a no-op once scrubbed) - if not poll_until(lambda: not workload_clients_present(admin), timeout=DEPLOY_TIMEOUT, interval=5): - raise RuntimeError( - f"pre-run cleanup left Keycloak client(s) {workload_clients_present(admin)} for {NAMESPACE!r} — the " - "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." - ) - - # BEFORE deploying (PRB reads the role universe when the event fires). It also enables Direct Access - # Grants on the login client (the password grant of ``mint_token`` needs it; skip if it cannot); - # ``grant_enabled`` = it was off, so the teardown turns it off again. - grant_enabled = provision_realm_and_users(admin, TEST_REALM) + # Direct Access Grants on the login client (the password grant of ``mint_token`` needs it). Enter it + # before the first mutation, so a skip (missing client, refused write) leaves the cluster as it was. + # Its own ``finally`` turns the grant off again if it was off, whatever teardown step below raises. + with direct_access_grants(admin, TEST_REALM): + # Pre-run: start from a NO-WORKLOADS slate. The trigger is event-driven — deploying a workload only + # re-fires ``CLIENT_CREATED`` if there is no client for it yet, so a leftover workload + its Keycloak + # client from a prior run would silently swallow the event and onboarding would never trigger. Remove + # both workloads and every registration first (best-effort, tolerant of already-absent), then poll the + # clients actually gone before deploying. Capture the CRs first: a prior run's leftovers are evidence. + label = _capture_label() # the rung, read once here (a session fixture is torn down after the session) + capture_aiac_crs("pre-run-slate", label=label) + undeploy_workload(scn.AGENT_WORKLOAD) + undeploy_workload(scn.TOOL_WORKLOAD) + _scrub_to_pristine( + admin + ) # clients + *-aud scopes + workload CRs + stray AuthorizationPolicy CRs + roles/scopes + store + reenable_provisioned_clients( + admin, TEST_REALM + ) # undo any prior run's failed-service disable (a no-op once scrubbed) + if not poll_until(lambda: not workload_clients_present(admin), timeout=DEPLOY_TIMEOUT, interval=5): + raise RuntimeError( + f"pre-run cleanup left Keycloak client(s) {workload_clients_present(admin)} for {NAMESPACE!r} — the " + "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." + ) - tool_onboarded = scn.TOOL_WORKLOAD in workloads - signals = list(ready_signals) if ready_signals is not None else _default_ready_signals(tool_onboarded) - try: - # Inside the ``try``, so a skip here still restores the grant. Skip if a token still can't be - # minted. The login token's ``sub`` comes from ``aiac-username-sub``, which the IdP Configuration - # Service links to the login client at startup (D31): fail if it isn't the username. - # ``require_subject_scope`` below checks the links again after each onboarding. + # BEFORE deploying (PRB reads the role universe when the event fires). + provision_realm_and_users(admin, TEST_REALM) + # Skip if a token can't be minted (the grant is on: ``direct_access_grants`` above). The login token's + # ``sub`` comes from ``aiac-username-sub``, which the IdP Configuration Service links to the login + # client at startup (D31): fail if it isn't the username. ``require_subject_scope`` below checks the + # links again after each onboarding. verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) - ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) # mount this run's policy.md BEFORE deploying - - # Minimal probe ctx for the per-workload agent convergence gate: the inbound leg reaches the - # agent through its pod-agnostic Service, so no ``agent_pod`` is needed yet (it is resolved after - # Part B for the outbound leg). The full ``ctx`` is assembled below once ``agent_pod`` is known. - probe_ctx = {"admin": admin, "namespace": NAMESPACE, "keycloak_url": keycloak_url, "realm": TEST_REALM} - - # Fulfill the images precondition BEFORE any deploy: build-if-absent + ``kind load`` the demo - # image(s) for exactly the workloads this rung deploys. Loading is order-independent, so it is - # done once here rather than per-iteration; the loop below then only ``kubectl apply``s. - load_workload_images(workloads) - - # Deploy in the rung's ``workloads`` order, one at a time, converging before the next — each - # ``deploy_workload`` fires the event-driven trigger (deploy -> operator -> CLIENT_CREATED -> - # SPI -> NATS -> consumer). The order is the rung's ordering proof. - for workload in workloads: - deployed_at = datetime.now(timezone.utc) # the log window of the race hint (handoff 20) - deploy_workload(workload) - if not wait_for_registration(admin, workload): - raise RuntimeError( - f"operator did not register Keycloak client {NAMESPACE}/{workload!r} within " - f"{DEPLOY_TIMEOUT:.0f}s of deploying it — is the event path wired (NATS broker + " - "aiac-event-listener SPI)? (The rollout already passed, so the image is loaded.) " - "See docs/testing/uc1-onboarding-pipeline.md Preconditions." - ) - if workload == scn.AGENT_WORKLOAD: - # Option A′ — one representative ENFORCED decision proves OPA loaded the agent's bundle. - # A bare AuthorizationPolicy CR proves only that the operator reconciled, not that the - # bundle is in force; a live inbound dev-user=allow through AuthBridge->OPA is positive - # proof. dev-user inbound is deterministically allow on every rung. The changed combiner - # denies the agent pod until its own CR is loaded (D20), so the allow cannot come from a - # missing CR. - gate = ReadySignal("inbound", "dev-user", "allow") - if not poll_until( - lambda g=gate: g.decide(probe_ctx) == g.expected, - timeout=BUNDLE_TIMEOUT, - interval=BUNDLE_POLL_INTERVAL, - ): - # The hint says when the Controller log shows the event-before-commit race (handoff 20): - # then the onboarding had not run yet, and OPA had no CR of the agent to load. + + tool_onboarded = scn.TOOL_WORKLOAD in workloads + signals = list(ready_signals) if ready_signals is not None else _default_ready_signals(tool_onboarded) + try: + # Mount this run's policy.md BEFORE deploying. + ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) + + # Minimal probe ctx for the per-workload agent convergence gate: the inbound leg reaches the + # agent through its pod-agnostic Service, so no ``agent_pod`` is needed yet (it is resolved after + # Part B for the outbound leg). The full ``ctx`` is assembled below once ``agent_pod`` is known. + probe_ctx = {"admin": admin, "namespace": NAMESPACE, "keycloak_url": keycloak_url, "realm": TEST_REALM} + + # Fulfill the images precondition BEFORE any deploy: build-if-absent + ``kind load`` the demo + # image(s) for exactly the workloads this rung deploys. Loading is order-independent, so it is + # done once here rather than per-iteration; the loop below then only ``kubectl apply``s. + load_workload_images(workloads) + + # Deploy in the rung's ``workloads`` order, one at a time, converging before the next — each + # ``deploy_workload`` fires the event-driven trigger (deploy -> operator -> CLIENT_CREATED -> + # SPI -> NATS -> consumer). The order is the rung's ordering proof. + for workload in workloads: + deployed_at = datetime.now(timezone.utc) # the log window of the race hint (handoff 20) + deploy_workload(workload) + if not wait_for_registration(admin, workload): raise RuntimeError( - append_hint( - f"agent did not converge after deploy: {gate.label()}={gate.decide(probe_ctx)!r} " - f"(want {gate.expected!r}) within {BUNDLE_TIMEOUT:.0f}s — OPA had not loaded the " - "agent's bundle. See k8s/opa-kind-runbook.md and issue #139.", - controller_commit_race_hint(deployed_at, admin=admin, workload=workload), - ) + f"operator did not register Keycloak client {NAMESPACE}/{workload!r} within " + f"{DEPLOY_TIMEOUT:.0f}s of deploying it — is the event path wired (NATS broker + " + "aiac-event-listener SPI)? (The rollout already passed, so the image is loaded.) " + "See docs/testing/uc1-onboarding-pipeline.md Preconditions." ) - else: - # Tool — it has its own CR (D20): under target side its inbound decides the tool calls, - # under agent side it is a pass-through CR. A live decision on its inbound needs the - # outbound leg (Part B, below), so the per-workload gate reads state instead: the scopes - # provisioned, SPM(github-tool) stored (the PRB ran, not only the bootstrap), and the CR - # present with both request packages (``tool_onboarding_state``). - seen = {"state": {}} - - def _tool_converged() -> bool: - seen["state"] = tool_onboarding_state(admin) - return all(seen["state"].values()) - - if not poll_until(_tool_converged, timeout=ONBOARD_TIMEOUT, interval=5): - raise RuntimeError( - append_hint( - f"tool did not converge after deploy within {ONBOARD_TIMEOUT:.0f}s: {seen['state']} " - f"(scopes = all of {sorted(scn.TOOL_SCOPES)} provisioned; spm = " - f"SPM({scn.TOOL_WORKLOAD}) stored; cr = its AuthorizationPolicy CR with both request " - "packages). A failed onboarding check (D30: the tool pod has no authbridge-proxy " - "sidecar — injectTools; no opa / mcp-parser in the namespace inbound pipeline; an " - "httpGet probe) leaves no CR — see the Controller log and k8s/opa-kind-enable.sh.", - controller_commit_race_hint(deployed_at, admin=admin, workload=workload), + if workload == scn.AGENT_WORKLOAD: + # Option A′ — one representative ENFORCED decision proves OPA loaded the agent's bundle. + # A bare AuthorizationPolicy CR proves only that the operator reconciled, not that the + # bundle is in force; a live inbound dev-user=allow through AuthBridge->OPA is positive + # proof. dev-user inbound is deterministically allow on every rung. The changed combiner + # denies the agent pod until its own CR is loaded (D20), so the allow cannot come from a + # missing CR. + gate = ReadySignal("inbound", "dev-user", "allow") + if not poll_until( + lambda g=gate: g.decide(probe_ctx) == g.expected, + timeout=BUNDLE_TIMEOUT, + interval=BUNDLE_POLL_INTERVAL, + ): + # The hint says when the Controller log shows the event-before-commit race (handoff 20): + # then the onboarding had not run yet, and OPA had no CR of the agent to load. + raise RuntimeError( + append_hint( + f"agent did not converge after deploy: {gate.label()}={gate.decide(probe_ctx)!r} " + f"(want {gate.expected!r}) within {BUNDLE_TIMEOUT:.0f}s — OPA had not loaded the " + "agent's bundle. See k8s/opa-kind-runbook.md and issue #139.", + controller_commit_race_hint(deployed_at, admin=admin, workload=workload), + ) + ) + else: + # Tool — it has its own CR (D20): under target side its inbound decides the tool calls, + # under agent side it is a pass-through CR. A live decision on its inbound needs the + # outbound leg (Part B, below), so the per-workload gate reads state instead: the scopes + # provisioned, SPM(github-tool) stored (the PRB ran, not only the bootstrap), and the CR + # present with both request packages (``tool_onboarding_state``). + seen = {"state": {}} + + def _tool_converged() -> bool: + seen["state"] = tool_onboarding_state(admin) + return all(seen["state"].values()) + + if not poll_until(_tool_converged, timeout=ONBOARD_TIMEOUT, interval=5): + raise RuntimeError( + append_hint( + f"tool did not converge after deploy within {ONBOARD_TIMEOUT:.0f}s: {seen['state']} " + f"(scopes = all of {sorted(scn.TOOL_SCOPES)} provisioned; spm = " + f"SPM({scn.TOOL_WORKLOAD}) stored; cr = its AuthorizationPolicy CR with both request " + "packages). A failed onboarding check (D30: the tool pod has no authbridge-proxy " + "sidecar — injectTools; no opa / mcp-parser in the namespace inbound pipeline; an " + "httpGet probe) leaves no CR — see the Controller log and k8s/opa-kind-enable.sh.", + controller_commit_race_hint(deployed_at, admin=admin, workload=workload), + ) ) - ) - # The workload converged, so its Provision ran and linked the subject scope (D31). Fail fast - # (not skip) when it did not: a missing link is AIAC's own fault, and without it the exchanged - # token keeps sub = the user ID, so every tool call would be denied later with no clear cause. - require_subject_scope(admin, workload) - - # Part B — enable the outbound token-exchange leg so the tool check is actually reached (under - # target side github-tool's inbound OPA, which accepts only a token with its audience; under - # agent side the agent's outbound OPA). Done after onboarding so the restarted agent (and its - # OPA sidecar) picks up both the new route and, on its next poll, the recomposed bundle. - # - # Prepared ONLY when this run actually probes outbound (``has_outbound``). The agent-only rung - # (rung 1) has no tool onboarded, so there is no real ``agent -> tool`` call to make and no - # outbound signal in its set — its outbound leg has no real-life counterpart, and token-exchange - # would short-circuit before OPA anyway (no ``github-tool`` audience grant, since the operator - # creates ``agent-{ns}-github-tool-aud`` only on TOOL deploy). So on a tool-less rung Part B is - # skipped entirely: no outbound route, no restart. Its inbound gate already converged in the - # per-workload loop above, and skipping the restart means rung 1 asserts the exact bundle the - # live event-driven onboard produced (no restart papering over a bad compose). - # - # When there IS an outbound leg: add the route so the forward proxy intercepts the github-tool - # call and runs token-exchange, grant the agent client the ``*-aud`` scope (only ever present - # once the tool is deployed) so the exchange succeeds and OPA is reached, then restart the agent - # so it reloads the route and re-fetches the recomposed bundle. - has_outbound = any(sig.kind == "outbound" for sig in signals) - if has_outbound: - ensure_github_tool_route(NAMESPACE) - if tool_onboarded: - grant_exchange_scope(admin) - restart_agent(NAMESPACE) - # Resolved once for the ctx contract, but the live outbound path re-resolves per probe - # (``resolve_agent_pod``) so a pod replaced after this point can't poison the whole run (#139). - agent_pod = resolve_agent_pod() - - ctx = { - "admin": admin, - "namespace": NAMESPACE, - "agent_pod": agent_pod, - "keycloak_url": keycloak_url, - "realm": TEST_REALM, - "tool_onboarded": tool_onboarded, - "side": side, - } + # The workload converged, so its Provision ran and linked the subject scope (D31). Fail fast + # (not skip) when it did not: a missing link is AIAC's own fault, and without it the exchanged + # token keeps sub = the user ID, so every tool call would be denied later with no clear cause. + require_subject_scope(admin, workload) + + # Part B — enable the outbound token-exchange leg so the tool check is actually reached (under + # target side github-tool's inbound OPA, which accepts only a token with its audience; under + # agent side the agent's outbound OPA). Done after onboarding so the restarted agent (and its + # OPA sidecar) picks up both the new route and, on its next poll, the recomposed bundle. + # + # Prepared ONLY when this run actually probes outbound (``has_outbound``). The agent-only rung + # (rung 1) has no tool onboarded, so there is no real ``agent -> tool`` call to make and no + # outbound signal in its set — its outbound leg has no real-life counterpart, and token-exchange + # would short-circuit before OPA anyway (no ``github-tool`` audience grant, since the operator + # creates ``agent-{ns}-github-tool-aud`` only on TOOL deploy). So on a tool-less rung Part B is + # skipped entirely: no outbound route, no restart. Its inbound gate already converged in the + # per-workload loop above, and skipping the restart means rung 1 asserts the exact bundle the + # live event-driven onboard produced (no restart papering over a bad compose). + # + # When there IS an outbound leg: add the route so the forward proxy intercepts the github-tool + # call and runs token-exchange, grant the agent client the ``*-aud`` scope (only ever present + # once the tool is deployed) so the exchange succeeds and OPA is reached, then restart the agent + # so it reloads the route and re-fetches the recomposed bundle. + has_outbound = any(sig.kind == "outbound" for sig in signals) + if has_outbound: + ensure_github_tool_route(NAMESPACE) + if tool_onboarded: + grant_exchange_scope(admin) + restart_agent(NAMESPACE) + # Resolved once for the ctx contract, but the live outbound path re-resolves per probe + # (``resolve_agent_pod``) so a pod replaced after this point can't poison the whole run (#139). + agent_pod = resolve_agent_pod() + + ctx = { + "admin": admin, + "namespace": NAMESPACE, + "agent_pod": agent_pod, + "keycloak_url": keycloak_url, + "realm": TEST_REALM, + "tool_onboarded": tool_onboarded, + "side": side, + } - # Wait for bundle-service + OPA to reflect THIS run's CR (and token-exchange to settle) before - # any assertion. The ``signals`` set (this policy's ``ready_signals``, or the Policy-A default) - # is polled until every probe reaches its terminal verdict — each deterministic for its - # scenario regardless of a stale CR (which the run replaced), and each polled to a *definitive* - # allow/deny (not ``error``) so we also wait out the post-restart token-exchange 503 window. - def _ready() -> bool: - return all(sig.decide(ctx) == sig.expected for sig in signals) - - if not poll_until(_ready, timeout=BUNDLE_TIMEOUT, interval=BUNDLE_POLL_INTERVAL): - # Surface the RAW outbound (code, body) — not just the classified outcome — so a stalled - # run is self-diagnosing (issue #139). The classifier collapses three very different - # failures into ``"error"``; the raw ``(code, body)`` tells them apart in one shot: - # * ``code=None`` + body ``"outbound probe exec failed: ..."`` — the *probe's* ``kubectl - # exec`` failed (e.g. the agent pod was replaced by a restart and its name went stale, - # ``NotFound``). A harness/pod issue, not the pipeline — OPA was never reached. - # * ``code=503`` — the ``token-exchange`` leg failed upstream (audience refused, IdP - # unreachable), so OPA was never consulted. - # * a genuine OPA policy stall can't show as ``"error"`` at all: it surfaces as ``"deny"`` - # under deny-by-default (target side: github-tool's inbound HTTP 403 with an OPA body; - # agent side: HTTP 200 + an OPA error frame), because the generated Rego carries - # ``default allow := false``. - # The observed-vs-expected line is recomputed against *this run's* signals so a denyworld - # (or any parametrized) run diagnoses itself, not a hardcoded Policy-A probe. - observed = "; ".join(f"{sig.label()}={sig.decide(ctx)!r}(want {sig.expected!r})" for sig in signals) - # Re-resolve the pod fresh here (not the possibly-stale ``ctx["agent_pod"]``) so the raw - # line reflects the *current* live pod. Dump the raw (code, body) for the first outbound - # signal — the leg where the #139 stale-pod / 503 failures actually show up. - raw_line = "" - ob_sig = next((s for s in signals if s.kind == "outbound"), None) - if ob_sig is not None: - ob_token = mint_token( - ob_sig.subject, scn.USER_PASSWORD, keycloak_url=ctx["keycloak_url"], realm=ctx["realm"] + # Wait for bundle-service + OPA to reflect THIS run's CR (and token-exchange to settle) before + # any assertion. The ``signals`` set (this policy's ``ready_signals``, or the Policy-A default) + # is polled until every probe reaches its terminal verdict — each deterministic for its + # scenario regardless of a stale CR (which the run replaced), and each polled to a *definitive* + # allow/deny (not ``error``) so we also wait out the post-restart token-exchange 503 window. + def _ready() -> bool: + return all(sig.decide(ctx) == sig.expected for sig in signals) + + if not poll_until(_ready, timeout=BUNDLE_TIMEOUT, interval=BUNDLE_POLL_INTERVAL): + # Surface the RAW outbound (code, body) — not just the classified outcome — so a stalled + # run is self-diagnosing (issue #139). The classifier collapses three very different + # failures into ``"error"``; the raw ``(code, body)`` tells them apart in one shot: + # * ``code=None`` + body ``"outbound probe exec failed: ..."`` — the *probe's* ``kubectl + # exec`` failed (e.g. the agent pod was replaced by a restart and its name went stale, + # ``NotFound``). A harness/pod issue, not the pipeline — OPA was never reached. + # * ``code=503`` — the ``token-exchange`` leg failed upstream (audience refused, IdP + # unreachable), so OPA was never consulted. + # * a genuine OPA policy stall can't show as ``"error"`` at all: it surfaces as ``"deny"`` + # under deny-by-default (target side: github-tool's inbound HTTP 403 with an OPA body; + # agent side: HTTP 200 + an OPA error frame), because the generated Rego carries + # ``default allow := false``. + # The observed-vs-expected line is recomputed against *this run's* signals so a denyworld + # (or any parametrized) run diagnoses itself, not a hardcoded Policy-A probe. + observed = "; ".join(f"{sig.label()}={sig.decide(ctx)!r}(want {sig.expected!r})" for sig in signals) + # Re-resolve the pod fresh here (not the possibly-stale ``ctx["agent_pod"]``) so the raw + # line reflects the *current* live pod. Dump the raw (code, body) for the first outbound + # signal — the leg where the #139 stale-pod / 503 failures actually show up. + raw_line = "" + ob_sig = next((s for s in signals if s.kind == "outbound"), None) + if ob_sig is not None: + ob_token = mint_token( + ob_sig.subject, scn.USER_PASSWORD, keycloak_url=ctx["keycloak_url"], realm=ctx["realm"] + ) + ob_code, ob_body = outbound_probe( + ob_token, ob_sig.tool_bare, namespace=ctx["namespace"], agent_pod=resolve_agent_pod() + ) + raw_line = f" [raw {ob_sig.label()}: HTTP {ob_code}, body={ob_body[:300]!r}]" + raise RuntimeError( + f"live pipeline did not converge within {BUNDLE_TIMEOUT:.0f}s after onboarding " + f"{workloads} + Part B: {observed}.{raw_line} code=None + 'exec failed' body = a " + "stale/gone agent pod (harness); 503 = token-exchange never came up (OPA not reached); " + "under deny-by-default a real policy stall reads 'deny', never 'error' — see " + "k8s/opa-kind-runbook.md and issue #139." ) - ob_code, ob_body = outbound_probe( - ob_token, ob_sig.tool_bare, namespace=ctx["namespace"], agent_pod=resolve_agent_pod() + yield ctx + finally: + # Capture every AIAC CR and the combiner to the pytest host FIRST — the undeploy and the scrub + # remove or change them. Best-effort: it never raises, on the success and the failure path. + capture_aiac_crs("teardown", label=label) + # Teardown — full-to-pristine, best-effort per step (each helper tolerates already-absent objects). + for workload in reversed(workloads): # undeploy in reverse deploy order + undeploy_workload(workload) + # Explicit scrub — do not trust an unverified operator cascade. Same reset the pre-run slate runs. + _scrub_to_pristine(admin) + # Verify pristine: both clients gone AND no AuthorizationPolicy CR remains. A leaked footprint is + # a test failure, not silent drift — but do not mask a failure already propagating out of the try. + if not poll_until( + lambda: not workload_clients_present(admin) and no_authpolicies_remain(), + timeout=DEPLOY_TIMEOUT, + interval=5, + ): + msg = ( + f"teardown did not restore pristine: Keycloak client(s) still present=" + f"{workload_clients_present(admin)}, AuthorizationPolicy CR(s) remain={not no_authpolicies_remain()} " + f"in {NAMESPACE!r}. See handoff 03 teardown (full-to-pristine)." ) - raw_line = f" [raw {ob_sig.label()}: HTTP {ob_code}, body={ob_body[:300]!r}]" - raise RuntimeError( - f"live pipeline did not converge within {BUNDLE_TIMEOUT:.0f}s after onboarding " - f"{workloads} + Part B: {observed}.{raw_line} code=None + 'exec failed' body = a " - "stale/gone agent pod (harness); 503 = token-exchange never came up (OPA not reached); " - "under deny-by-default a real policy stall reads 'deny', never 'error' — see " - "k8s/opa-kind-runbook.md and issue #139." - ) - yield ctx - finally: - # Capture every AIAC CR and the combiner to the pytest host FIRST — the undeploy and the scrub - # remove or change them. Best-effort: it never raises, on the success and the failure path. - capture_aiac_crs("teardown", label=label) - # Teardown — full-to-pristine, best-effort per step (each helper tolerates already-absent objects). - for workload in reversed(workloads): # undeploy in reverse deploy order - undeploy_workload(workload) - # Explicit scrub — do not trust an unverified operator cascade. Same reset the pre-run slate runs. - _scrub_to_pristine(admin) - if grant_enabled: # best-effort, before the pristine check (which can raise) - restore_direct_access_grants(admin, TEST_REALM) - # Verify pristine: both clients gone AND no AuthorizationPolicy CR remains. A leaked footprint is - # a test failure, not silent drift — but do not mask a failure already propagating out of the try. - if not poll_until( - lambda: not workload_clients_present(admin) and no_authpolicies_remain(), timeout=DEPLOY_TIMEOUT, interval=5 - ): - msg = ( - f"teardown did not restore pristine: Keycloak client(s) still present=" - f"{workload_clients_present(admin)}, AuthorizationPolicy CR(s) remain={not no_authpolicies_remain()} " - f"in {NAMESPACE!r}. See handoff 03 teardown (full-to-pristine)." - ) - if sys.exc_info()[0] is not None: - log.error("%s [suppressed — a prior error is propagating]", msg) - else: - raise RuntimeError(msg) + if sys.exc_info()[0] is not None: + log.error("%s [suppressed — a prior error is propagating]", msg) + else: + raise RuntimeError(msg) # ====================================================================================== @@ -2457,53 +2475,60 @@ def pristine_stack(workloads: Sequence[str], *, policy_md: str = scn.POLICY_ABST never comes). Yields ``ctx = {"admin", "namespace", "keycloak_url", "realm", "side"}`` (``side`` = ``live_enforcement_side()``); the caller deploys. - Same skip gates (pipeline wiring, env, event path — before any mutation), same no-workloads slate - (a ``pre-run-slate`` CR capture, then undeploy + ``_scrub_to_pristine`` + - ``reenable_provisioned_clients`` + clients-gone poll), same realm/users provisioning (with Direct - Access Grants on the login client) + ``sub`` mapper gate, ``policy.md`` mount and image load for - ``workloads``; and the same teardown: a ``teardown`` CR capture first, then full-to-pristine (and - Direct Access Grants off again if the setup turned it on), verified (clients gone, no CR left).""" + Same skip gates (pipeline wiring, env, event path, Direct Access Grants on the login client — before + any mutation), same no-workloads slate (a ``pre-run-slate`` CR capture, then undeploy + + ``_scrub_to_pristine`` + ``reenable_provisioned_clients`` + clients-gone poll), same realm/users + provisioning + ``sub`` mapper gate, ``policy.md`` mount and image load for ``workloads``; and the + same teardown: a ``teardown`` CR capture first, then full-to-pristine, verified (clients gone, no CR + left), and last the grant off again if it was off (``direct_access_grants``).""" require_pipeline(namespace=NAMESPACE, workloads=[]) creds = require_env_or_skip("KEYCLOAK_URL", "KEYCLOAK_ADMIN_USERNAME", "KEYCLOAK_ADMIN_PASSWORD") keycloak_url = creds["KEYCLOAK_URL"] admin = connect_admin() require_event_path(admin=admin, realm=TEST_REALM) side = live_enforcement_side() - - label = _capture_label() - capture_aiac_crs("pre-run-slate", label=label) - undeploy_workload(scn.AGENT_WORKLOAD) - undeploy_workload(scn.TOOL_WORKLOAD) - _scrub_to_pristine(admin) - reenable_provisioned_clients(admin, TEST_REALM) - if not poll_until(lambda: not workload_clients_present(admin), timeout=DEPLOY_TIMEOUT, interval=5): - raise RuntimeError( - f"pre-run cleanup left Keycloak client(s) {workload_clients_present(admin)} for {NAMESPACE!r} — the " - "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." - ) - grant_enabled = provision_realm_and_users(admin, TEST_REALM) - - try: - verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) - ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) - load_workload_images(list(workloads)) - yield {"admin": admin, "namespace": NAMESPACE, "keycloak_url": keycloak_url, "realm": TEST_REALM, "side": side} - finally: - capture_aiac_crs("teardown", label=label) # FIRST: the undeploy and the scrub remove the CRs - for workload in reversed(list(workloads)): - undeploy_workload(workload) + # Before the first mutation, so a skip leaves the cluster as it was; restored in its own ``finally``. + with direct_access_grants(admin, TEST_REALM): + label = _capture_label() + capture_aiac_crs("pre-run-slate", label=label) + undeploy_workload(scn.AGENT_WORKLOAD) + undeploy_workload(scn.TOOL_WORKLOAD) _scrub_to_pristine(admin) - if grant_enabled: - restore_direct_access_grants(admin, TEST_REALM) - if not poll_until( - lambda: not workload_clients_present(admin) and no_authpolicies_remain(), timeout=DEPLOY_TIMEOUT, interval=5 - ): - msg = ( - f"teardown did not restore pristine: Keycloak client(s) still present=" - f"{workload_clients_present(admin)}, AuthorizationPolicy CR(s) remain={not no_authpolicies_remain()} " - f"in {NAMESPACE!r}." + reenable_provisioned_clients(admin, TEST_REALM) + if not poll_until(lambda: not workload_clients_present(admin), timeout=DEPLOY_TIMEOUT, interval=5): + raise RuntimeError( + f"pre-run cleanup left Keycloak client(s) {workload_clients_present(admin)} for {NAMESPACE!r} — the " + "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." ) - if sys.exc_info()[0] is not None: - log.error("%s [suppressed — a prior error is propagating]", msg) - else: - raise RuntimeError(msg) + provision_realm_and_users(admin, TEST_REALM) + verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) + + try: + ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) + load_workload_images(list(workloads)) + yield { + "admin": admin, + "namespace": NAMESPACE, + "keycloak_url": keycloak_url, + "realm": TEST_REALM, + "side": side, + } + finally: + capture_aiac_crs("teardown", label=label) # FIRST: the undeploy and the scrub remove the CRs + for workload in reversed(list(workloads)): + undeploy_workload(workload) + _scrub_to_pristine(admin) + if not poll_until( + lambda: not workload_clients_present(admin) and no_authpolicies_remain(), + timeout=DEPLOY_TIMEOUT, + interval=5, + ): + msg = ( + f"teardown did not restore pristine: Keycloak client(s) still present=" + f"{workload_clients_present(admin)}, AuthorizationPolicy CR(s) remain={not no_authpolicies_remain()} " + f"in {NAMESPACE!r}." + ) + if sys.exc_info()[0] is not None: + log.error("%s [suppressed — a prior error is propagating]", msg) + else: + raise RuntimeError(msg) diff --git a/test/unit/agent/uc/onboarding/test_uc1_login_client.py b/test/unit/agent/uc/onboarding/test_uc1_login_client.py index a0d3bba3..498dd307 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_login_client.py +++ b/test/unit/agent/uc/onboarding/test_uc1_login_client.py @@ -1,13 +1,12 @@ -"""``uc1_onboard.ensure_direct_access_grants`` turns on the password grant of the login client, and -``restore_direct_access_grants`` turns it off again — pure unit tests. +"""``uc1_onboard.direct_access_grants`` turns on the password grant of the login client for the life of +a fixture, and turns it off again at exit if it was off — pure unit tests. The harness mints user tokens through the login client ``rossoctl`` with the password grant, so the -client needs Direct Access Grants on. ``provision_realm_and_users`` turns it on, so each setup path -(``onboarded_stack``, ``pristine_stack`` and the disabled cross-service test) gets it, and each -teardown turns it off again when the setup turned it on. A grant that cannot be written is a skip with -the cause. These tests use a -``MagicMock`` admin (no Keycloak). They import the system harness like ``test_uc1_subject_scope.py`` -does. +client needs Direct Access Grants on. ``onboarded_stack``, ``pristine_stack`` and the disabled +cross-service test enter ``direct_access_grants`` before their first mutation. Its own ``finally`` +turns the grant off again, so a teardown step that raises cannot leave the grant on. A grant that +cannot be written is a skip with the cause. These tests use a ``MagicMock`` admin (no Keycloak). They +import the system harness like ``test_uc1_subject_scope.py`` does. """ from __future__ import annotations @@ -25,27 +24,61 @@ from test.system import uc1_onboard as uc1 # noqa: E402 +LOGIN_UUID = "uuid-1" + def _admin(client: dict | None) -> MagicMock: + """A ``KeycloakAdmin`` stand-in whose filtered ``clientId`` query finds ``client`` (or nothing).""" admin = MagicMock(name="KeycloakAdmin") - admin.get_clients.return_value = [{"clientId": "other", "id": "x"}] + ([] if client is None else [client]) + admin.get_client_id.side_effect = lambda client_id: ( + client["id"] if client is not None and client_id == client["clientId"] else None + ) + admin.get_client.side_effect = lambda uuid: dict(client) return admin def _login(enabled: bool) -> dict: - return {"clientId": uc1.KEYCLOAK_CLIENT_ID, "id": "uuid-1", "directAccessGrantsEnabled": enabled} + return {"clientId": uc1.KEYCLOAK_CLIENT_ID, "id": LOGIN_UUID, "directAccessGrantsEnabled": enabled} + + +def _grant_writes(admin: MagicMock) -> list[bool]: + return [call.args[1]["directAccessGrantsEnabled"] for call in admin.update_client.call_args_list] + + +# ====================================================================================== +# login_client — a filtered read, not a list of every client +# ====================================================================================== + + +def test_the_login_client_is_found_with_a_filtered_query() -> None: + admin = _admin(_login(True)) + assert uc1.login_client(admin, "rossoctl") == _login(True) + admin.change_current_realm.assert_called_with("rossoctl") + admin.get_client_id.assert_called_once_with(uc1.KEYCLOAK_CLIENT_ID) + admin.get_clients.assert_not_called() + + +def test_a_missing_login_client_is_none() -> None: + admin = _admin(None) + assert uc1.login_client(admin, "rossoctl") is None + admin.get_client.assert_not_called() + + +# ====================================================================================== +# ensure_direct_access_grants — the entry half +# ====================================================================================== def test_the_grant_is_enabled_with_a_partial_update() -> None: admin = _admin(_login(False)) - assert uc1.ensure_direct_access_grants(admin, "rossoctl") is True + assert uc1.ensure_direct_access_grants(admin, "rossoctl") == LOGIN_UUID admin.change_current_realm.assert_called_with("rossoctl") - admin.update_client.assert_called_once_with("uuid-1", {"directAccessGrantsEnabled": True}) + admin.update_client.assert_called_once_with(LOGIN_UUID, {"directAccessGrantsEnabled": True}) def test_a_grant_that_is_on_is_not_written() -> None: admin = _admin(_login(True)) - assert uc1.ensure_direct_access_grants(admin, "rossoctl") is False + assert uc1.ensure_direct_access_grants(admin, "rossoctl") is None admin.update_client.assert_not_called() @@ -59,7 +92,7 @@ def test_a_missing_login_client_skips_and_is_not_created() -> None: def test_a_refused_read_skips_with_the_cause() -> None: admin = _admin(_login(False)) - admin.get_clients.side_effect = KeycloakError(error_message="forbidden", response_code=403) + admin.get_client_id.side_effect = KeycloakError(error_message="forbidden", response_code=403) with pytest.raises(pytest.skip.Exception, match="cannot read the login client.*forbidden"): uc1.ensure_direct_access_grants(admin, "rossoctl") admin.update_client.assert_not_called() @@ -72,29 +105,94 @@ def test_a_refused_update_skips_with_the_cause() -> None: uc1.ensure_direct_access_grants(admin, "rossoctl") -def test_provision_realm_and_users_enables_the_grant() -> None: +def test_provision_realm_and_users_does_not_touch_the_login_client() -> None: admin = _admin(_login(False)) admin.create_user.side_effect = lambda payload, exist_ok=False: f"id-{payload['username']}" admin.get_user.return_value = {"email": "e", "firstName": "f", "lastName": "l"} - assert uc1.provision_realm_and_users(admin, "rossoctl") is True - admin.update_client.assert_called_once_with("uuid-1", {"directAccessGrantsEnabled": True}) + assert uc1.provision_realm_and_users(admin, "rossoctl") is None + admin.update_client.assert_not_called() + +# ====================================================================================== +# restore_direct_access_grants — the exit half, best-effort +# ====================================================================================== -def test_the_restore_turns_the_grant_off_with_a_partial_update() -> None: + +def test_the_restore_turns_the_grant_off_with_a_partial_update_and_no_read() -> None: admin = _admin(_login(True)) - uc1.restore_direct_access_grants(admin, "rossoctl") + uc1.restore_direct_access_grants(admin, "rossoctl", LOGIN_UUID) admin.change_current_realm.assert_called_with("rossoctl") - admin.update_client.assert_called_once_with("uuid-1", {"directAccessGrantsEnabled": False}) + admin.update_client.assert_called_once_with(LOGIN_UUID, {"directAccessGrantsEnabled": False}) + admin.get_client_id.assert_not_called() # the UUID comes from the enable -def test_a_refused_restore_is_logged_not_raised(caplog: pytest.LogCaptureFixture) -> None: +@pytest.mark.parametrize("error", [KeycloakError(error_message="forbidden", response_code=403), ConnectionError()]) +def test_a_refused_restore_is_logged_not_raised(caplog: pytest.LogCaptureFixture, error: Exception) -> None: admin = _admin(_login(True)) - admin.update_client.side_effect = KeycloakError(error_message="forbidden", response_code=403) - uc1.restore_direct_access_grants(admin, "rossoctl") # never masks a teardown failure + admin.update_client.side_effect = error + uc1.restore_direct_access_grants(admin, "rossoctl", LOGIN_UUID) # never masks a teardown failure assert "cannot turn Direct Access Grants off again" in caplog.text -def test_the_restore_of_a_missing_login_client_writes_nothing() -> None: +# ====================================================================================== +# direct_access_grants — the context manager: the restore is in its own ``finally`` +# ====================================================================================== + + +def test_the_context_turns_on_a_grant_that_was_off_and_turns_it_off_at_exit() -> None: + admin = _admin(_login(False)) + with uc1.direct_access_grants(admin, "rossoctl"): + assert _grant_writes(admin) == [True] + assert _grant_writes(admin) == [True, False] + + +def test_the_context_does_not_write_a_grant_that_was_on() -> None: + admin = _admin(_login(True)) + with uc1.direct_access_grants(admin, "rossoctl"): + pass + admin.update_client.assert_not_called() + + +@pytest.mark.parametrize( + "error", + [AssertionError("clear_policy_store: HTTP 500"), KeycloakError(error_message="token refused", response_code=401)], +) +def test_a_teardown_step_that_raises_still_restores_the_grant(error: Exception) -> None: + """The case of the review: a scrub step of the fixture teardown raises (for example + ``clear_policy_store`` or ``cleanup_provisioned``). The grant is still turned off, and the error + propagates unchanged.""" + admin = _admin(_login(False)) + with pytest.raises(type(error)) as raised: + with uc1.direct_access_grants(admin, "rossoctl"): + try: + pass # the fixture body + finally: + raise error # a teardown step of the body + assert raised.value is error + assert _grant_writes(admin) == [True, False] + + +def test_a_skip_in_the_body_still_restores_the_grant() -> None: + admin = _admin(_login(False)) + with pytest.raises(pytest.skip.Exception): + with uc1.direct_access_grants(admin, "rossoctl"): + pytest.skip("verify_login_subject cannot mint a token") + assert _grant_writes(admin) == [True, False] + + +def test_a_failed_restore_does_not_mask_the_body_error() -> None: + admin = _admin(_login(False)) + admin.update_client.side_effect = [None, KeycloakError(error_message="forbidden", response_code=403)] + with pytest.raises(RuntimeError, match="teardown did not restore pristine"): + with uc1.direct_access_grants(admin, "rossoctl"): + raise RuntimeError("teardown did not restore pristine") + + +def test_a_skip_at_entry_writes_nothing() -> None: admin = _admin(None) - uc1.restore_direct_access_grants(admin, "rossoctl") + body = MagicMock(name="body") + with pytest.raises(pytest.skip.Exception): + with uc1.direct_access_grants(admin, "rossoctl"): + body() + body.assert_not_called() admin.update_client.assert_not_called() diff --git a/test/unit/agent/uc/onboarding/test_uc1_provision_users.py b/test/unit/agent/uc/onboarding/test_uc1_provision_users.py index 012571c1..4697e09b 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_provision_users.py +++ b/test/unit/agent/uc/onboarding/test_uc1_provision_users.py @@ -38,9 +38,6 @@ def create_user(payload, exist_ok=False): admin.create_user.side_effect = create_user admin.get_user.side_effect = lambda user_id: {**(existing.get(user_id[3:]) or created[user_id[3:]])} - # The login client, with Direct Access Grants on (``test_uc1_login_client.py`` covers that step). - login = {"clientId": uc1.KEYCLOAK_CLIENT_ID, "id": "c", "directAccessGrantsEnabled": True} - admin.get_clients.return_value = [login] return admin diff --git a/test/unit/agent/uc/onboarding/test_uc1_subject_scope.py b/test/unit/agent/uc/onboarding/test_uc1_subject_scope.py index 2b2cb429..49c90db2 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_subject_scope.py +++ b/test/unit/agent/uc/onboarding/test_uc1_subject_scope.py @@ -294,6 +294,11 @@ def _admin( optional_map.update(optionals or {}) admin.get_client_default_client_scopes.side_effect = lambda uuid: [{"id": i} for i in default_map[uuid]] admin.get_client_optional_client_scopes.side_effect = lambda uuid: [{"id": i} for i in optional_map[uuid]] + # The filtered read of ``login_client`` (``get_client_id`` + ``get_client``), over the same client list. + admin.get_client_id.side_effect = lambda client_id: next( + (c["id"] for c in admin.get_clients.return_value if c.get("clientId") == client_id), None + ) + admin.get_client.side_effect = lambda uuid: next(c for c in admin.get_clients.return_value if c["id"] == uuid) return admin From b2af90dc259df3cc3111d1b39b7eac5f7b6aaf28 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Thu, 15 Oct 2026 15:27:26 +0300 Subject: [PATCH 06/15] fix(test): hold Direct Access Grants for the session, with a marker on the client Address the second review on #238. - The grant is held once per session (hold_direct_access_grants) and turned off at session end (release_direct_access_grants, from the new test/system/conftest.py). Before, each module turned it on and off. - The enable writes the marker attribute aiac.test.direct-access-grants-by-harness in the same partial update. A run that is killed leaves the marker, so the next run takes the grant over and turns it off. - The release reads the client again. It turns the grant off only when the marker is still there. The runbook and demo snippets remove the marker, so a grant that an operator turns on is not turned off. - Any read or write failure is a skip with the cause, not only a KeycloakError. A write that fails is undone at once if Keycloak applied it. A 404 says that the realm does not exist. - onboarded_stack and pristine_stack share one _clean_slate context manager for the pre-run slate and the teardown. - One _update_client helper for the partial client writes, also used by reenable_provisioned_clients. The hold takes a client_id. - The disabled cross-service test holds the grant after provision_realm_and_users, which creates the realm. - Runbook snippet: a response with an unexpected shape prints the raw response and stops; the PUT fails on a status other than 204. Keycloak 26.5.2 was checked on a scratch client: a partial attributes map changes only the given keys, and "" removes a key. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- demo/use-cases/onboarding/demo.md | 5 +- docs/agents/test.md | 6 +- docs/specs/PRD.md | 4 +- docs/testing/policy-pipeline.md | 4 +- docs/testing/uc1-onboarding-pipeline.md | 6 +- k8s/opa-kind-runbook.md | 40 +- test/system/conftest.py | 18 + ...test_uc1_onboard_cross_service_conflict.py | 58 +- test/system/launcher.py | 4 +- test/system/uc1_onboard.py | 707 +++++++++--------- .../uc/onboarding/test_uc1_login_client.py | 279 ++++--- 11 files changed, 662 insertions(+), 469 deletions(-) create mode 100644 test/system/conftest.py diff --git a/demo/use-cases/onboarding/demo.md b/demo/use-cases/onboarding/demo.md index 9f711600..e6f10203 100644 --- a/demo/use-cases/onboarding/demo.md +++ b/demo/use-cases/onboarding/demo.md @@ -221,7 +221,8 @@ To tear the forward down afterwards: `pkill -f 'port-forward .*keycloak-service' If the realm was freshly provisioned, the `rossoctl` client needs Direct Access Grants enabled. Without it, token minting fails with `unauthorized_client`. This is platform state, so the demo does -not set it: +not set it. The snippet also removes the attribute `aiac.test.direct-access-grants-by-harness`, so a +system-test run does not turn the grant off at its end: ```bash KC=http://keycloak.localtest.me:8080 @@ -232,7 +233,7 @@ ADMIN=$(curl -s -X POST "$KC/realms/master/protocol/openid-connect/token" \ CID=$(curl -s -H "Authorization: Bearer $ADMIN" "$KC/admin/realms/rossoctl/clients?clientId=rossoctl" \ | python3 -c 'import sys,json;print(json.load(sys.stdin)[0]["id"])') curl -s -H "Authorization: Bearer $ADMIN" "$KC/admin/realms/rossoctl/clients/$CID" \ - | python3 -c 'import sys,json;d=json.load(sys.stdin);d["directAccessGrantsEnabled"]=True;print(json.dumps(d))' \ + | python3 -c 'import sys,json;d=json.load(sys.stdin);d["directAccessGrantsEnabled"]=True;d.setdefault("attributes",{})["aiac.test.direct-access-grants-by-harness"]="";print(json.dumps(d))' \ | curl -s -o /dev/null -w "enable DAG HTTP %{http_code}\n" -X PUT -H "Authorization: Bearer $ADMIN" \ -H "Content-Type: application/json" "$KC/admin/realms/rossoctl/clients/$CID" --data-binary @- ``` diff --git a/docs/agents/test.md b/docs/agents/test.md index ffd02045..26877408 100644 --- a/docs/agents/test.md +++ b/docs/agents/test.md @@ -115,8 +115,10 @@ test both sides, run the lane, do a clean redeploy with the other side (see The platform prerequisites, wiring, verification and the setting of the side: `k8s/opa-kind-runbook.md`. The fixture enables Direct Access Grants on the login -client `rossoctl` itself and turns it off again at teardown if it was off, so the -lane needs no manual realm step. A +client `rossoctl` itself, once per session, and turns it off again at session end +if it turned it on, so the lane needs no manual realm step. A client attribute +records that the harness turned it on, so a run that is killed is undone by the +next run. A manual probe with the password grant outside the lane needs the grant set by hand: see the runbook section "Direct Access Grants on the login client". The per-loop shape: `test/system/uc1_onboard.py`. diff --git a/docs/specs/PRD.md b/docs/specs/PRD.md index 2dae8f97..eba7b399 100644 --- a/docs/specs/PRD.md +++ b/docs/specs/PRD.md @@ -819,7 +819,7 @@ The Controller reads `AIAC_ENFORCEMENT_SIDE` at start, and an unknown value stop - **The namespace pipelines.** The `authbridge-runtime-config` ConfigMap of each service namespace must have `opa` in the inbound pipeline, and also `mcp-parser` for a tool. Under agent side, the agents' namespaces also need `mcp-parser` and `opa` in the outbound pipeline. Onboarding check #2 finds a missing plugin. - **Pod restarts.** The webhook injects the sidecar only on pod CREATE. After a change to the pipeline or to `injectTools`, restart the agent pods and the tool pods (`rossoctl.io/type=agent` and `rossoctl.io/type=tool`). - **The username precondition (D31).** The CRs key users by username, and `sub` is the username. So usernames must be unique, must not change and must never be used again. Keep `editUsernameAllowed: false` and `registrationAllowed: false` in the realm (the operator's realm template sets `registrationAllowed: false`; it does not set `editUsernameAllowed`, so that value is `false` only by the Keycloak default). An admin must never create a deleted username again: the new user would get the grants of the old user until AIAC writes the CRs again. -- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`direct_access_grants`, before its first mutation) and turns it off again at exit if it was off; a manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). +- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`hold_direct_access_grants`, once per session, before its first mutation) and turns it off again at session end if it turned it on; a client attribute records that, so the next run undoes a killed run. A manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). - **The legacy token exchange (optional hardening).** AIAC does not need the Keycloak feature `token-exchange` (V1) to be off: the AuthBridge requests go to the standard exchange (V2), and a request that V1 handles still gets `sub` = the username for an onboarded target (§6). Keycloak recommends to turn V1 off, because then a request that V2 declines fails with `400` ("Standard token exchange is not enabled for the requested client") and does not silently go to a different engine. `KC_FEATURES` is in the Keycloak deployment of the rossoctl platform, not in this repo. If V1 stays on, AIAC needs no change. Detail: [`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md). @@ -874,7 +874,7 @@ Require a live rossoctl/Kind cluster with the AuthBridge OPA pipeline wired in ( | `LLM_BASE_URL` / `LLM_MODEL` / `LLM_API_KEY` | LLM endpoint for onboarding | | `AIAC_TEST_REALM` | Optional: realm of the live stack (default from the scenario) | -System tests onboard through the in-cluster Controller and assert the deployed OPA plugin's allow/deny on real HTTP requests through AuthBridge. They skip cleanly when the cluster or the env is missing. Under target side, github-tool's own inbound OPA decides the tool calls, and the agent's outbound is a pass-through. A service that has no CR is denied, and a quarantine deletes the CR. The demo passes under each enforcement side; each side is a separate deploy (a clean redeploy changes the side). After each workload converges, `require_subject_scope` checks that its client links `aiac-username-sub` as a default scope (D31); a missing link is a failure, not a skip. `require_subject_scope` also checks that the login client `rossoctl` links the scope as a default scope (a missing login client, a missing link or an optional link only is a failure). Rung 2 also checks the scope and the links of `rossoctl`, `github-agent` and `github-tool`, that `rossoctl` has no client mapper that writes `sub`, and that the `rossoctl` login token of `dev-user` has `sub` = `dev-user` (`test_subject_scope_linked`), and the `sub` of an exchanged token (`test_exchanged_token_subject_is_username`: the harness exchanges as the agent client with its client secret, the identity that `k8s/opa-kind-enable.sh` gives AuthBridge; it skips only if the agent client uses another client authenticator). `verify_login_subject` checks the login token through `rossoctl`: the fixture first enables Direct Access Grants on `rossoctl` and skips with the cause when it cannot (a missing `rossoctl`, or an admin that cannot manage clients); it skips when it still cannot mint the token, and fails when `sub` is not the username, because the link is an AIAC step. Rung 5 checks that the rollback keeps the scope (`test_rollback_keeps_the_subject_scope`). +System tests onboard through the in-cluster Controller and assert the deployed OPA plugin's allow/deny on real HTTP requests through AuthBridge. They skip cleanly when the cluster or the env is missing. Under target side, github-tool's own inbound OPA decides the tool calls, and the agent's outbound is a pass-through. A service that has no CR is denied, and a quarantine deletes the CR. The demo passes under each enforcement side; each side is a separate deploy (a clean redeploy changes the side). After each workload converges, `require_subject_scope` checks that its client links `aiac-username-sub` as a default scope (D31); a missing link is a failure, not a skip. `require_subject_scope` also checks that the login client `rossoctl` links the scope as a default scope (a missing login client, a missing link or an optional link only is a failure). Rung 2 also checks the scope and the links of `rossoctl`, `github-agent` and `github-tool`, that `rossoctl` has no client mapper that writes `sub`, and that the `rossoctl` login token of `dev-user` has `sub` = `dev-user` (`test_subject_scope_linked`), and the `sub` of an exchanged token (`test_exchanged_token_subject_is_username`: the harness exchanges as the agent client with its client secret, the identity that `k8s/opa-kind-enable.sh` gives AuthBridge; it skips only if the agent client uses another client authenticator). `verify_login_subject` checks the login token through `rossoctl`: the fixture first enables Direct Access Grants on `rossoctl` and skips with the cause when it cannot (a missing realm or `rossoctl`, or an admin that cannot manage clients); it skips when it still cannot mint the token, and fails when `sub` is not the username, because the link is an AIAC step. Rung 5 checks that the rollback keeps the scope (`test_rollback_keeps_the_subject_scope`). Selection is marker-only — no path argument. `integration` (several units in-process, no cluster), `system`, `llm` (real LLM, no cluster) and `eval` are opt-in markers: diff --git a/docs/testing/policy-pipeline.md b/docs/testing/policy-pipeline.md index 724240bc..95089b4f 100644 --- a/docs/testing/policy-pipeline.md +++ b/docs/testing/policy-pipeline.md @@ -94,8 +94,8 @@ over the fully onboarded stack. and checks that its `sub` is the username, and `ensure_agent_policy` mounts the single abstract `policy.md` on the Controller pod. `verify_login_subject` skips when it cannot mint the token (the fixture enables Direct Access Grants before the slate, and skips with the cause if - the login client is absent or the admin cannot manage clients; the teardown turns the grant off - again if it was off), and fails when `sub` is not the username. The `sub` comes + the realm or the login client is absent or the admin cannot manage clients; the session end turns + the grant off again if the harness turned it on), and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, which the IdP Configuration Service links to `rossoctl` at its startup and at each onboarding. That link is an AIAC step, not a prerequisite, so a wrong `sub` is a failure. The exchanged token gets the same mapping from the same scope, diff --git a/docs/testing/uc1-onboarding-pipeline.md b/docs/testing/uc1-onboarding-pipeline.md index 98912d23..02cbb071 100644 --- a/docs/testing/uc1-onboarding-pipeline.md +++ b/docs/testing/uc1-onboarding-pipeline.md @@ -225,8 +225,10 @@ cleanly** when any of it is absent (they never stand it up, and never false-pass reads the realm role universe when the event fires `onboard_and_apply`); idempotent; left in place. `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its `sub` is the username. Before the pre-run slate, the fixture enables Direct Access Grants on - `rossoctl` (the context manager `direct_access_grants`; at exit its own `finally` turns it off again - if it was off) and skips with the cause if `rossoctl` is absent or the admin cannot manage clients. It skips when it + `rossoctl` (`hold_direct_access_grants`, once per session; the session end turns it off again if the + harness turned it on, and the client attribute `aiac.test.direct-access-grants-by-harness` records + that, so the next run undoes a killed run) and skips with the cause if the realm or `rossoctl` is + absent or the admin cannot manage clients. It skips when it still cannot mint the token, and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, the one source of the rule (D31). The IdP Configuration Service links it to `rossoctl` at its startup and at each onboarding, so the link is an AIAC step, not a diff --git a/k8s/opa-kind-runbook.md b/k8s/opa-kind-runbook.md index d5262071..81b477f5 100644 --- a/k8s/opa-kind-runbook.md +++ b/k8s/opa-kind-runbook.md @@ -226,28 +226,46 @@ such pods, it skips this check. A manual probe that mints a user token with the password grant through the login client `rossoctl` needs Direct Access Grants on that client. In a fresh realm it is off, and the mint fails with `unauthorized_client`. The system -tests do not need this step: their fixture enables the grant itself -(`direct_access_grants` in `test/system/uc1_onboard.py`) and turns it off -again at teardown if it was off. To enable it by hand (a partial update: only -this field changes), source the repo-root `.env` first. The snippet uses the -same variables as the harness: +tests do not need this step: their fixture enables the grant itself, once per +session (`hold_direct_access_grants` in `test/system/uc1_onboard.py`), and +turns it off again at session end if it turned it on. The client attribute +`aiac.test.direct-access-grants-by-harness` records that the harness turned it +on, so the next run undoes a run that was killed. + +To enable it by hand, source the repo-root `.env` first. The snippet uses the +same variables as the harness. It is a partial update: only the grant changes, +and it removes the harness marker. So a system run that is in progress, or a +later one, does not turn your grant off: ```bash ( set -e # a subshell: a failed check stops the snippet, not your shell set -a; . .env; set +a KC=${KEYCLOAK_URL:?}; REALM=${AIAC_TEST_REALM:-rossoctl}; ADMIN_REALM=${KEYCLOAK_ADMIN_REALM:-master} +# Print the value at a key path (for example "0 id") of an admin API response; when the response has +# another shape, print "" and show the raw response on stderr. +pick() { python3 -c 'import sys,json +raw=sys.stdin.read() +try: + v=json.loads(raw) + for k in sys.argv[1:]: v=v[int(k)] if k.isdigit() else v[k] +except Exception: + v="" +print(v or "") +v or print("unexpected admin API response:", raw[:300], file=sys.stderr)' "$@"; } ADMIN=$(curl -s -X POST "$KC/realms/$ADMIN_REALM/protocol/openid-connect/token" \ -d client_id=admin-cli -d grant_type=password \ --data-urlencode "username=${KEYCLOAK_ADMIN_USERNAME:?}" \ - --data-urlencode "password=${KEYCLOAK_ADMIN_PASSWORD:?}" \ - | python3 -c 'import sys,json;print(json.load(sys.stdin).get("access_token",""))') + --data-urlencode "password=${KEYCLOAK_ADMIN_PASSWORD:?}" | pick access_token) [ -n "$ADMIN" ] || { echo "admin login at realm $ADMIN_REALM failed: check KEYCLOAK_ADMIN_*"; exit 1; } CID=$(curl -s -H "Authorization: Bearer $ADMIN" "$KC/admin/realms/$REALM/clients?clientId=rossoctl" \ - | python3 -c 'import sys,json;c=json.load(sys.stdin);print(c[0]["id"] if c else "")') -[ -n "$CID" ] || { echo "login client rossoctl not found in realm $REALM"; exit 1; } -curl -s -o /dev/null -w "enable DAG HTTP %{http_code}\n" -X PUT \ + | pick 0 id) +[ -n "$CID" ] || { echo "login client rossoctl not found in realm $REALM (or the admin cannot view clients)"; exit 1; } +CODE=$(curl -s -o /dev/null -w "%{http_code}" -X PUT \ -H "Authorization: Bearer $ADMIN" -H "Content-Type: application/json" \ - "$KC/admin/realms/$REALM/clients/$CID" -d '{"directAccessGrantsEnabled": true}' + "$KC/admin/realms/$REALM/clients/$CID" \ + -d '{"directAccessGrantsEnabled": true, "attributes": {"aiac.test.direct-access-grants-by-harness": ""}}') +[ "$CODE" = 204 ] || { echo "enable DAG failed: HTTP $CODE"; exit 1; } +echo "enable DAG HTTP 204" ) ``` diff --git a/test/system/conftest.py b/test/system/conftest.py new file mode 100644 index 00000000..15e8c412 --- /dev/null +++ b/test/system/conftest.py @@ -0,0 +1,18 @@ +"""Session hooks of the system tree.""" + +from __future__ import annotations + +import sys + +import pytest + + +@pytest.fixture(scope="session", autouse=True) +def _release_direct_access_grants(): + """At the end of the session, turn Direct Access Grants off again on each login client that the + harness turned it on for (``uc1_onboard.hold_direct_access_grants``). Session scope: the grant is + turned on once, by the first stack of the session, and off once. A no-op when no stack ran.""" + yield + uc1 = sys.modules.get("test.system.uc1_onboard") # not imported = no stack ran = nothing to release + if uc1 is not None: + uc1.release_direct_access_grants() diff --git a/test/system/disabled/test_uc1_onboard_cross_service_conflict.py b/test/system/disabled/test_uc1_onboard_cross_service_conflict.py index e00f3075..b08c4c62 100644 --- a/test/system/disabled/test_uc1_onboard_cross_service_conflict.py +++ b/test/system/disabled/test_uc1_onboard_cross_service_conflict.py @@ -164,35 +164,35 @@ def test_cross_service_conflict_is_surfaced_as_422_conflict_report() -> None: keycloak_url = creds["KEYCLOAK_URL"] admin = uc1.connect_admin() - # The password grant on the login client, before the first mutation: a skip at entry changes nothing, - # and its own ``finally`` turns the grant off again if it was off, whatever cleanup step raises. - with uc1.direct_access_grants(admin, TEST_REALM): - # Open the cleanup guard BEFORE the first shared-state mutation: the clean-slate steps below - # (Agent CR delete, Keycloak cleanup, Policy Store clear, provisioning) already mutate shared - # cluster state, so a failure mid-setup must still hit the ``finally`` teardown. - try: - uc1.delete_workload_crs() # clean policy slate (drop any prior run's CRs) - uc1.cleanup_provisioned(admin, TEST_REALM) # clean slate (Keycloak) - uc1.clear_policy_store() # clean slate ONCE — NOT between phases (the agent must see the tool's rule) - uc1.provision_realm_and_users(admin, TEST_REALM) # PRB reads the role universe - uc1.verify_login_subject( - keycloak_url=keycloak_url, realm=TEST_REALM, user="test-user", password=scn.USER_PASSWORD - ) - - # Phase 1 — tool onboarding under the exclusivity policy: clean (200), persists the tool-side - # DENY(tester -> github-tool.source-*) on SPM(github-tool). - _onboard_via_fresh_controller(POLICY_TOOL_EXCLUSIVE, scn.TOOL_WORKLOAD, expect_conflict=False) - - # Phase 2 — agent onboarding under the source-granting policy: the outbound subject gate's fresh - # ALLOW(tester -> github-tool.source-*) collides with the tool's already-applied DENY on the same - # (role, scope). The #2504 store read surfaces it -> 422 + ConflictReport. - report = _onboard_via_fresh_controller(POLICY_AGENT_GRANTS_SOURCE, scn.AGENT_WORKLOAD, expect_conflict=True) - finally: - uc1.delete_workload_crs() # after — drop any CR (there should be none on the raising path) - uc1.cleanup_provisioned(admin, TEST_REALM) # restore the pre-run Keycloak state - # Phase 1 persisted a tool-side deny; clear it so later integration tests don't read stale - # inbound rules and become order-dependent. - uc1.clear_policy_store() + # Open the cleanup guard BEFORE the first shared-state mutation: the clean-slate steps below + # (Agent CR delete, Keycloak cleanup, Policy Store clear, provisioning) already mutate shared + # cluster state, so a failure mid-setup must still hit the ``finally`` teardown. + try: + uc1.delete_workload_crs() # clean policy slate (drop any prior run's CRs) + uc1.cleanup_provisioned(admin, TEST_REALM) # clean slate (Keycloak) + uc1.clear_policy_store() # clean slate ONCE — NOT between phases (the agent must see the tool's rule) + uc1.provision_realm_and_users(admin, TEST_REALM) # PRB reads the role universe + # After the provisioning: it creates the realm, so a missing login client is the true skip cause. + # Turned off again at session end if the harness turned it on. + uc1.hold_direct_access_grants(admin, TEST_REALM) + uc1.verify_login_subject( + keycloak_url=keycloak_url, realm=TEST_REALM, user="test-user", password=scn.USER_PASSWORD + ) + + # Phase 1 — tool onboarding under the exclusivity policy: clean (200), persists the tool-side + # DENY(tester -> github-tool.source-*) on SPM(github-tool). + _onboard_via_fresh_controller(POLICY_TOOL_EXCLUSIVE, scn.TOOL_WORKLOAD, expect_conflict=False) + + # Phase 2 — agent onboarding under the source-granting policy: the outbound subject gate's fresh + # ALLOW(tester -> github-tool.source-*) collides with the tool's already-applied DENY on the same + # (role, scope). The #2504 store read surfaces it -> 422 + ConflictReport. + report = _onboard_via_fresh_controller(POLICY_AGENT_GRANTS_SOURCE, scn.AGENT_WORKLOAD, expect_conflict=True) + finally: + uc1.delete_workload_crs() # after — drop any CR (there should be none on the raising path) + uc1.cleanup_provisioned(admin, TEST_REALM) # restore the pre-run Keycloak state + # Phase 1 persisted a tool-side deny; clear it so later integration tests don't read stale + # inbound rules and become order-dependent. + uc1.clear_policy_store() # The 422 body is the shared ConflictReport shape (same as the deterministic routes test), carrying # at least one cross-service conflict with a real role + scope. Structural (not id-pinned) so it is diff --git a/test/system/launcher.py b/test/system/launcher.py index e6946f07..61b61841 100644 --- a/test/system/launcher.py +++ b/test/system/launcher.py @@ -1033,8 +1033,8 @@ def verify_login_subject( The live loop keys OPA decisions on ``input.identity.subject`` (the token ``sub``). To mint the token, Direct Access Grants must be enabled on ``client_id``. The fixture enables it before this call - (``uc1_onboard.direct_access_grants`` -> ``ensure_direct_access_grants``, which itself skips with - the cause on a missing login client or a refused read or write), so a mint failure here means the + (``uc1_onboard.hold_direct_access_grants``, which itself skips with the cause on a missing realm or + login client or a failed read or write), so a mint failure here means the user cannot log in, and it is a skip. The ``sub`` = username comes from the client scope ``aiac-username-sub``, which the IdP Configuration Service links as a default scope to the login client (``client_id``, by default ``rossoctl``) at its startup and at each onboarding (D31). That is an AIAC step, not a prerequisite, so a wrong ``sub`` fails. diff --git a/test/system/uc1_onboard.py b/test/system/uc1_onboard.py index 5887543d..7e1c3068 100644 --- a/test/system/uc1_onboard.py +++ b/test/system/uc1_onboard.py @@ -32,8 +32,9 @@ teardown step, on the success and the failure path, best-effort (it never raises) → teardown full-to-pristine (undeploy + delete all registrations + delete CRs, verified); the realm users and roles stay (leave-in-place) - → Direct Access Grants on the login client: the fixture turns it on before the pre-run slate and - turns it off again at the end if it was off (``direct_access_grants``) + → Direct Access Grants on the login client: the first stack of the session turns it on before the + pre-run slate, and the session end turns it off again if the harness turned it on + (``hold_direct_access_grants``, with a marker on the client, so a killed run is undone later) The only thing that differs between rungs is *which workloads are onboarded and in what order*, so all the machinery lives here and each ``test_uc1_onboard_*.py`` supplies just its own oracle @@ -42,8 +43,8 @@ This module owns: * **Config** (env, spec § Configuration) — single stack, no variants. -* **Keycloak** — ``connect_admin`` / ``direct_access_grants`` (the password grant on the login client, - for the life of the fixture) / ``provision_realm_and_users`` (the fixture UC-1 does *not* do) / +* **Keycloak** — ``connect_admin`` / ``hold_direct_access_grants`` (the password grant on the login + client, for the session) / ``provision_realm_and_users`` (the fixture UC-1 does *not* do) / ``cleanup_provisioned`` / ``delete_workload_registrations`` (teardown). * **CRs** — ``delete_workload_crs`` (the agent's and the tool's CR) / ``sweep_authpolicies`` (every other CR in the namespace) / ``delete_authpolicy`` (one CR, raising). @@ -335,7 +336,7 @@ def provision_realm_and_users(admin, realm: str) -> None: each role's ``actorIds`` (member usernames) — the PCE needs them to build the ``subject_roles`` map the inbound/outbound gates key on. Never deletes/recreates — reruns converge (spec: shared realm, leave-in-place: the users and the roles stay after the run). The password grant on the login - client is not set here: see ``direct_access_grants``.""" + client is not set here: see ``hold_direct_access_grants``.""" from keycloak.exceptions import KeycloakError try: @@ -425,7 +426,7 @@ def reenable_provisioned_clients(admin, realm: str) -> None: for client in admin.get_clients(): if client.get("name") in demo_client_names and not client.get("enabled", True): try: - admin.update_client(client["id"], {"enabled": True}) + _update_client(admin, realm, client["id"], {"enabled": True}) log.info("cleanup: re-enabled disabled client %r left by a prior run", client.get("name")) except KeycloakError as exc: log.warning("cleanup: re-enable client %r failed: %s", client.get("name"), exc) @@ -1032,94 +1033,155 @@ def subject_scope_problems( return problems -def login_client(admin, realm: str = TEST_REALM) -> dict | None: - """The Keycloak client representation of the login client ``rossoctl`` (``KEYCLOAK_CLIENT_ID``, the - client the harness mints user tokens through) in ``realm``; ``None`` if absent. A query filtered by - ``clientId`` finds its UUID, so the call does not list every client of the realm.""" +def login_client(admin, realm: str = TEST_REALM, client_id: str = KEYCLOAK_CLIENT_ID) -> dict | None: + """The Keycloak client representation of the login client ``client_id`` (default ``rossoctl``, + ``KEYCLOAK_CLIENT_ID``: the client the harness mints user tokens through) in ``realm``; ``None`` if + absent. A query filtered by ``clientId`` finds its UUID, so the call does not list every client of the + realm.""" admin.change_current_realm(realm) - client_uuid = admin.get_client_id(KEYCLOAK_CLIENT_ID) + client_uuid = admin.get_client_id(client_id) return admin.get_client(client_uuid) if client_uuid else None -def ensure_direct_access_grants(admin, realm: str) -> str | None: - """Turn on Direct Access Grants on the login client (``KEYCLOAK_CLIENT_ID``) of ``realm``, so that - ``mint_token`` can use the password grant. Use it through ``direct_access_grants``, which turns the - grant off again. +def _update_client(admin, realm: str, client_uuid: str, fields: dict) -> None: + """A partial update of one Keycloak client of ``realm``: only ``fields`` change. In ``attributes``, + only the given keys change, and ``""`` removes a key (Keycloak 26). So the write never puts a stale + copy of the whole client over a change that the IdP Configuration Service made at the same time (for + example its D31 scope link). The caller handles the errors.""" + admin.change_current_realm(realm) + admin.update_client(client_uuid, fields) + + +# Direct Access Grants on the login client. The harness mints user tokens with the password grant, so the +# grant must be on. When the harness turns it on, it writes this client attribute in the same update. The +# marker is the record that the harness owns the grant, and it is in Keycloak, not in process memory: a +# run that is killed before its session teardown leaves the marker, and the next run takes the grant over +# and turns it off at its end. A grant that is on with no marker belongs to the realm (or to an operator: +# the runbook snippet removes the marker), and the harness never changes it. +DAG_MARKER_ATTRIBUTE = "aiac.test.direct-access-grants-by-harness" + +# The login clients whose grant the harness holds in this session: ``(realm, clientId)`` -> ``(admin, +# client UUID)``. ``release_direct_access_grants`` empties it at session end (``test/system/conftest.py``). +_dag_holds: dict[tuple[str, str], tuple[object, str]] = {} + + +def _login_client_read_skip(realm: str, client_id: str, exc: Exception) -> str: + """The skip message for a read of the login client that failed with ``exc``.""" + if getattr(exc, "response_code", None) == 404: + return ( + f"the realm {realm!r} does not exist, so it has no login client {client_id!r} to enable Direct " + f"Access Grants on ({exc}). The realm owns its login client; the harness does not create it." + ) + return ( + f"cannot read the login client {client_id!r} in realm {realm!r} to enable Direct Access Grants: " + f"{type(exc).__name__}: {exc}. Make sure that the admin user has the view-clients and manage-clients roles." + ) - Idempotent: a client that has the grant already is not written. The write is a partial update (only - ``directAccessGrantsEnabled``), so it does not write back a stale copy of the client over a change - that the IdP Configuration Service made (for example its D31 scope link). Returns the client UUID - when this call turned the grant on, else ``None``. - ``pytest.skip`` (with the cause) when the login client does not exist (the realm owns its login - client, so nothing is created) or when the admin API refuses the read or the update (for example the - admin cannot manage clients). Both are prerequisite gaps, not AIAC failures. A refused update writes - nothing, so each skip leaves the realm as it was.""" +def hold_direct_access_grants(admin, realm: str, client_id: str = KEYCLOAK_CLIENT_ID) -> None: + """Make sure that Direct Access Grants is on for the login client ``client_id`` of ``realm`` until the + end of the pytest session, so that ``mint_token`` can use the password grant. + + * The first call of the session reads the client. Later calls for the same client do nothing. + * A grant that is on with no marker (``DAG_MARKER_ATTRIBUTE``) is the realm's own setting: it is not + written, and it is not turned off later. + * A grant that is off is turned on, with the marker, in one partial update (``_update_client``). + * A marker from an earlier run that did not finish: this session takes the grant over. + + In the last two cases ``release_direct_access_grants`` turns the grant off at session end. + + ``pytest.skip`` (with the cause) when the realm or the login client does not exist (nothing is + created) or when the read or the update fails, for any reason (refused by the admin API, a transport + error, an unexpected response). These are prerequisite gaps, not AIAC failures. An update that fails + is undone at once if the server applied it (``_release_one`` reads the client again), so a skip leaves + the realm as it was.""" import pytest - from keycloak.exceptions import KeycloakError + key = (realm, client_id) + if key in _dag_holds: + return try: - client = login_client(admin, realm) - except KeycloakError as exc: - pytest.skip( - f"cannot read the login client {KEYCLOAK_CLIENT_ID!r} in realm {realm!r} to enable Direct Access " - f"Grants: {exc}. Make sure that the realm exists and that the admin user has the view-clients and " - "manage-clients roles." - ) + client = login_client(admin, realm, client_id) + except Exception as exc: # noqa: BLE001 — any read failure is a prerequisite gap: skip with the cause + pytest.skip(_login_client_read_skip(realm, client_id, exc)) if client is None: pytest.skip( - f"the login client {KEYCLOAK_CLIENT_ID!r} does not exist in realm {realm!r}, so the harness cannot " - "mint a user token through it (the realm owns its login client; the harness does not create it)" + f"the login client {client_id!r} does not exist in realm {realm!r}, so the harness cannot mint a " + "user token through it (the realm owns its login client; the harness does not create it)" ) - if client.get("directAccessGrantsEnabled"): - return None + marked = (client.get("attributes") or {}).get(DAG_MARKER_ATTRIBUTE) == "true" + if client.get("directAccessGrantsEnabled") and not marked: + return + # Record the hold before the write: a write that raises after the server applied it is still undone. + _dag_holds[key] = (admin, client["id"]) + if marked: + log.warning( + "an earlier run turned on Direct Access Grants on the login client %r in realm %r and did not " + "turn it off: this session takes it over and turns it off at its end", + client_id, + realm, + ) + return try: - admin.update_client(client["id"], {"directAccessGrantsEnabled": True}) - except KeycloakError as exc: + _update_client( + admin, + realm, + client["id"], + {"directAccessGrantsEnabled": True, "attributes": {DAG_MARKER_ATTRIBUTE: "true"}}, + ) + except Exception as exc: # noqa: BLE001 — a refused write or a lost response: undo if applied, then skip + _release_one(key) pytest.skip( - f"cannot enable Direct Access Grants on the login client {KEYCLOAK_CLIENT_ID!r} in realm {realm!r}: " - f"{exc}. Give the admin user the manage-clients role, or enable the grant by hand " + f"cannot enable Direct Access Grants on the login client {client_id!r} in realm {realm!r}: " + f"{type(exc).__name__}: {exc}. " + "Give the admin user the manage-clients role, or enable the grant by hand " "(k8s/opa-kind-runbook.md, 'Direct Access Grants on the login client')." ) log.warning( - "enabled Direct Access Grants on the login client %r in realm %r (turned off again at teardown)", - KEYCLOAK_CLIENT_ID, + "enabled Direct Access Grants on the login client %r in realm %r (turned off again at session end)", + client_id, realm, ) - return client["id"] -def restore_direct_access_grants(admin, realm: str, client_uuid: str) -> None: - """Turn Direct Access Grants off again on the login client ``client_uuid`` of ``realm`` (the UUID that - ``ensure_direct_access_grants`` returned): a partial update, like the enable. Best-effort: any error - is logged, never raised, so it does not mask a teardown failure that is propagating.""" +def _release_one(key: tuple[str, str]) -> None: + """Turn the grant of one held login client off again, and remove the marker, if the marker is still + on the client. The client is read again first: when the marker is gone, someone else took the grant + over during the run (the runbook snippet removes the marker), and the grant is not changed. + Best-effort: any error is logged, never raised, so it does not mask a failure that is propagating. + The marker stays when the write fails, so the next run turns the grant off.""" + admin, client_uuid = _dag_holds.pop(key) + realm, client_id = key try: admin.change_current_realm(realm) - admin.update_client(client_uuid, {"directAccessGrantsEnabled": False}) + client = admin.get_client(client_uuid) + if (client.get("attributes") or {}).get(DAG_MARKER_ATTRIBUTE) != "true": + log.info( + "Direct Access Grants on the login client %r in realm %r has no harness marker now: not changed", + client_id, + realm, + ) + return + _update_client( + admin, realm, client_uuid, {"directAccessGrantsEnabled": False, "attributes": {DAG_MARKER_ATTRIBUTE: ""}} + ) except Exception as exc: # noqa: BLE001 — best-effort teardown: log, never mask the real failure log.error( - "cannot turn Direct Access Grants off again on the login client %r in realm %r: %s", - KEYCLOAK_CLIENT_ID, + "cannot turn Direct Access Grants off again on the login client %r in realm %r: %s. The marker " + "%r stays, so the next run turns it off.", + client_id, realm, exc, + DAG_MARKER_ATTRIBUTE, ) -@contextmanager -def direct_access_grants(admin, realm: str) -> Iterator[None]: - """Direct Access Grants on the login client of ``realm`` for the ``with`` body. - - At entry: ``ensure_direct_access_grants`` (turn it on, or skip with the cause). At exit: if the entry - turned it on, ``restore_direct_access_grants`` turns it off again, so a realm that had it off stays - as it was. The restore is in this context manager's own ``finally``: it runs when the body raises, - skips or returns, and also when a teardown step of the body raises. Enter it before the first - mutation of shared cluster state, so a skip at entry changes nothing.""" - client_uuid = ensure_direct_access_grants(admin, realm) - try: - yield - finally: - if client_uuid is not None: - restore_direct_access_grants(admin, realm, client_uuid) +def release_direct_access_grants() -> None: + """At the end of the pytest session (``test/system/conftest.py``): turn Direct Access Grants off again + on each login client that ``hold_direct_access_grants`` holds (``_release_one``). A realm that had the + grant on is not changed. A no-op when no stack ran.""" + for key in list(_dag_holds): + _release_one(key) def _workload_list(workloads: str | Sequence[str]) -> list[str]: @@ -1418,6 +1480,79 @@ def _scrub_to_pristine(admin) -> None: clear_policy_store() # Policy Store SPMs (PV survives redeploys) +@contextmanager +def _clean_slate(admin, workloads: Sequence[str], *, keycloak_url: str) -> Iterator[None]: + """The pre-run slate and the teardown that ``onboarded_stack`` and ``pristine_stack`` share. The + caller runs the skip gates first; the ``with`` body onboards (or fails to onboard) ``workloads``. + + At entry: Direct Access Grants on the login client (``hold_direct_access_grants``, before the first + mutation, so a skip there leaves the cluster as it was; it is turned off at session end), a + ``pre-run-slate`` CR capture, the no-workloads slate (undeploy both workloads, + ``_scrub_to_pristine``, ``reenable_provisioned_clients``, poll the clients gone), the realm users and + roles, and the ``sub`` check of a login token (``verify_login_subject``). At exit, also when the body + raises or skips: a ``teardown`` CR capture first, then full-to-pristine (undeploy ``workloads`` in + reverse order, ``_scrub_to_pristine``), verified (clients gone, no CR left). The users and the roles + are shared realm state and stay.""" + # Direct Access Grants on the login client (the password grant of ``mint_token`` needs it). + hold_direct_access_grants(admin, TEST_REALM) + + # Pre-run: start from a NO-WORKLOADS slate. The trigger is event-driven — deploying a workload only + # re-fires ``CLIENT_CREATED`` if there is no client for it yet, so a leftover workload + its Keycloak + # client from a prior run would silently swallow the event and onboarding would never trigger. Remove + # both workloads and every registration first (best-effort, tolerant of already-absent), then poll the + # clients actually gone before deploying. Capture the CRs first: a prior run's leftovers are evidence. + label = _capture_label() # the rung, read once here (a session fixture is torn down after the session) + capture_aiac_crs("pre-run-slate", label=label) + undeploy_workload(scn.AGENT_WORKLOAD) + undeploy_workload(scn.TOOL_WORKLOAD) + _scrub_to_pristine( + admin + ) # clients + *-aud scopes + workload CRs + stray AuthorizationPolicy CRs + roles/scopes + store + reenable_provisioned_clients( + admin, TEST_REALM + ) # undo any prior run's failed-service disable (a no-op once scrubbed) + if not poll_until(lambda: not workload_clients_present(admin), timeout=DEPLOY_TIMEOUT, interval=5): + raise RuntimeError( + f"pre-run cleanup left Keycloak client(s) {workload_clients_present(admin)} for {NAMESPACE!r} — the " + "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." + ) + + provision_realm_and_users(admin, TEST_REALM) # BEFORE deploying (PRB reads the role universe when the event fires) + # Skip if a token can't be minted (the grant is on: ``hold_direct_access_grants`` above). The login token's + # ``sub`` comes from ``aiac-username-sub``, which the IdP Configuration Service links to the login client + # at startup (D31): fail if it isn't the username. ``require_subject_scope`` checks the links again after + # each onboarding. + verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) + + try: + yield + finally: + # Capture every AIAC CR and the combiner to the pytest host FIRST — the undeploy and the scrub + # remove or change them. Best-effort: it never raises, on the success and the failure path. + capture_aiac_crs("teardown", label=label) + # Teardown — full-to-pristine, best-effort per step (each helper tolerates already-absent objects). + for workload in reversed(list(workloads)): # undeploy in reverse deploy order + undeploy_workload(workload) + # Explicit scrub — do not trust an unverified operator cascade. Same reset the pre-run slate runs. + _scrub_to_pristine(admin) + # Verify pristine: both clients gone AND no AuthorizationPolicy CR remains. A leaked footprint is + # a test failure, not silent drift — but do not mask a failure already propagating out of the body. + if not poll_until( + lambda: not workload_clients_present(admin) and no_authpolicies_remain(), + timeout=DEPLOY_TIMEOUT, + interval=5, + ): + msg = ( + f"teardown did not restore pristine: Keycloak client(s) still present=" + f"{workload_clients_present(admin)}, AuthorizationPolicy CR(s) remain={not no_authpolicies_remain()} " + f"in {NAMESPACE!r}. See handoff 03 teardown (full-to-pristine)." + ) + if sys.exc_info()[0] is not None: + log.error("%s [suppressed — a prior error is propagating]", msg) + else: + raise RuntimeError(msg) + + @contextmanager def onboarded_stack( workloads: list[str], @@ -1449,8 +1584,9 @@ def onboarded_stack( (``capture_aiac_crs``, phase ``teardown``; best-effort, it never masks a failure), then is **full-to-pristine** — the workloads and every registration are removed and the removal is verified. The users and the roles of ``provision_realm_and_users`` are shared realm state and stay. - Direct Access Grants on the login client is on for the whole flow (``direct_access_grants``, entered - after the skip gates and before the pre-run slate) and is turned off again at the end if it was off. + The slate and the teardown are ``_clean_slate``, shared with ``pristine_stack``. Direct Access Grants + on the login client is on for the whole flow (``hold_direct_access_grants``, after the skip gates and + before the pre-run slate) and is turned off again at session end if the harness turned it on. The capture directories are named for the test module that started the stack. The workload order is the rung's identity — e.g. rung 2 passes ``[agent, tool]`` so tool onboarding completes the tool check (under target side it writes github-tool's own CR with both gates; under @@ -1485,229 +1621,171 @@ def onboarded_stack( require_event_path(admin=admin, realm=TEST_REALM) side = live_enforcement_side() # read-only; the suite runs under the live side - # Direct Access Grants on the login client (the password grant of ``mint_token`` needs it). Enter it - # before the first mutation, so a skip (missing client, refused write) leaves the cluster as it was. - # Its own ``finally`` turns the grant off again if it was off, whatever teardown step below raises. - with direct_access_grants(admin, TEST_REALM): - # Pre-run: start from a NO-WORKLOADS slate. The trigger is event-driven — deploying a workload only - # re-fires ``CLIENT_CREATED`` if there is no client for it yet, so a leftover workload + its Keycloak - # client from a prior run would silently swallow the event and onboarding would never trigger. Remove - # both workloads and every registration first (best-effort, tolerant of already-absent), then poll the - # clients actually gone before deploying. Capture the CRs first: a prior run's leftovers are evidence. - label = _capture_label() # the rung, read once here (a session fixture is torn down after the session) - capture_aiac_crs("pre-run-slate", label=label) - undeploy_workload(scn.AGENT_WORKLOAD) - undeploy_workload(scn.TOOL_WORKLOAD) - _scrub_to_pristine( - admin - ) # clients + *-aud scopes + workload CRs + stray AuthorizationPolicy CRs + roles/scopes + store - reenable_provisioned_clients( - admin, TEST_REALM - ) # undo any prior run's failed-service disable (a no-op once scrubbed) - if not poll_until(lambda: not workload_clients_present(admin), timeout=DEPLOY_TIMEOUT, interval=5): - raise RuntimeError( - f"pre-run cleanup left Keycloak client(s) {workload_clients_present(admin)} for {NAMESPACE!r} — the " - "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." - ) - - # BEFORE deploying (PRB reads the role universe when the event fires). - provision_realm_and_users(admin, TEST_REALM) - # Skip if a token can't be minted (the grant is on: ``direct_access_grants`` above). The login token's - # ``sub`` comes from ``aiac-username-sub``, which the IdP Configuration Service links to the login - # client at startup (D31): fail if it isn't the username. ``require_subject_scope`` below checks the - # links again after each onboarding. - verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) - - tool_onboarded = scn.TOOL_WORKLOAD in workloads - signals = list(ready_signals) if ready_signals is not None else _default_ready_signals(tool_onboarded) - try: - # Mount this run's policy.md BEFORE deploying. - ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) - - # Minimal probe ctx for the per-workload agent convergence gate: the inbound leg reaches the - # agent through its pod-agnostic Service, so no ``agent_pod`` is needed yet (it is resolved after - # Part B for the outbound leg). The full ``ctx`` is assembled below once ``agent_pod`` is known. - probe_ctx = {"admin": admin, "namespace": NAMESPACE, "keycloak_url": keycloak_url, "realm": TEST_REALM} - - # Fulfill the images precondition BEFORE any deploy: build-if-absent + ``kind load`` the demo - # image(s) for exactly the workloads this rung deploys. Loading is order-independent, so it is - # done once here rather than per-iteration; the loop below then only ``kubectl apply``s. - load_workload_images(workloads) - - # Deploy in the rung's ``workloads`` order, one at a time, converging before the next — each - # ``deploy_workload`` fires the event-driven trigger (deploy -> operator -> CLIENT_CREATED -> - # SPI -> NATS -> consumer). The order is the rung's ordering proof. - for workload in workloads: - deployed_at = datetime.now(timezone.utc) # the log window of the race hint (handoff 20) - deploy_workload(workload) - if not wait_for_registration(admin, workload): + tool_onboarded = scn.TOOL_WORKLOAD in workloads + signals = list(ready_signals) if ready_signals is not None else _default_ready_signals(tool_onboarded) + with _clean_slate(admin, workloads, keycloak_url=keycloak_url): + # Mount this run's policy.md BEFORE deploying. + ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) + + # Minimal probe ctx for the per-workload agent convergence gate: the inbound leg reaches the + # agent through its pod-agnostic Service, so no ``agent_pod`` is needed yet (it is resolved after + # Part B for the outbound leg). The full ``ctx`` is assembled below once ``agent_pod`` is known. + probe_ctx = {"admin": admin, "namespace": NAMESPACE, "keycloak_url": keycloak_url, "realm": TEST_REALM} + + # Fulfill the images precondition BEFORE any deploy: build-if-absent + ``kind load`` the demo + # image(s) for exactly the workloads this rung deploys. Loading is order-independent, so it is + # done once here rather than per-iteration; the loop below then only ``kubectl apply``s. + load_workload_images(workloads) + + # Deploy in the rung's ``workloads`` order, one at a time, converging before the next — each + # ``deploy_workload`` fires the event-driven trigger (deploy -> operator -> CLIENT_CREATED -> + # SPI -> NATS -> consumer). The order is the rung's ordering proof. + for workload in workloads: + deployed_at = datetime.now(timezone.utc) # the log window of the race hint (handoff 20) + deploy_workload(workload) + if not wait_for_registration(admin, workload): + raise RuntimeError( + f"operator did not register Keycloak client {NAMESPACE}/{workload!r} within " + f"{DEPLOY_TIMEOUT:.0f}s of deploying it — is the event path wired (NATS broker + " + "aiac-event-listener SPI)? (The rollout already passed, so the image is loaded.) " + "See docs/testing/uc1-onboarding-pipeline.md Preconditions." + ) + if workload == scn.AGENT_WORKLOAD: + # Option A′ — one representative ENFORCED decision proves OPA loaded the agent's bundle. + # A bare AuthorizationPolicy CR proves only that the operator reconciled, not that the + # bundle is in force; a live inbound dev-user=allow through AuthBridge->OPA is positive + # proof. dev-user inbound is deterministically allow on every rung. The changed combiner + # denies the agent pod until its own CR is loaded (D20), so the allow cannot come from a + # missing CR. + gate = ReadySignal("inbound", "dev-user", "allow") + if not poll_until( + lambda g=gate: g.decide(probe_ctx) == g.expected, + timeout=BUNDLE_TIMEOUT, + interval=BUNDLE_POLL_INTERVAL, + ): + # The hint says when the Controller log shows the event-before-commit race (handoff 20): + # then the onboarding had not run yet, and OPA had no CR of the agent to load. raise RuntimeError( - f"operator did not register Keycloak client {NAMESPACE}/{workload!r} within " - f"{DEPLOY_TIMEOUT:.0f}s of deploying it — is the event path wired (NATS broker + " - "aiac-event-listener SPI)? (The rollout already passed, so the image is loaded.) " - "See docs/testing/uc1-onboarding-pipeline.md Preconditions." - ) - if workload == scn.AGENT_WORKLOAD: - # Option A′ — one representative ENFORCED decision proves OPA loaded the agent's bundle. - # A bare AuthorizationPolicy CR proves only that the operator reconciled, not that the - # bundle is in force; a live inbound dev-user=allow through AuthBridge->OPA is positive - # proof. dev-user inbound is deterministically allow on every rung. The changed combiner - # denies the agent pod until its own CR is loaded (D20), so the allow cannot come from a - # missing CR. - gate = ReadySignal("inbound", "dev-user", "allow") - if not poll_until( - lambda g=gate: g.decide(probe_ctx) == g.expected, - timeout=BUNDLE_TIMEOUT, - interval=BUNDLE_POLL_INTERVAL, - ): - # The hint says when the Controller log shows the event-before-commit race (handoff 20): - # then the onboarding had not run yet, and OPA had no CR of the agent to load. - raise RuntimeError( - append_hint( - f"agent did not converge after deploy: {gate.label()}={gate.decide(probe_ctx)!r} " - f"(want {gate.expected!r}) within {BUNDLE_TIMEOUT:.0f}s — OPA had not loaded the " - "agent's bundle. See k8s/opa-kind-runbook.md and issue #139.", - controller_commit_race_hint(deployed_at, admin=admin, workload=workload), - ) + append_hint( + f"agent did not converge after deploy: {gate.label()}={gate.decide(probe_ctx)!r} " + f"(want {gate.expected!r}) within {BUNDLE_TIMEOUT:.0f}s — OPA had not loaded the " + "agent's bundle. See k8s/opa-kind-runbook.md and issue #139.", + controller_commit_race_hint(deployed_at, admin=admin, workload=workload), ) - else: - # Tool — it has its own CR (D20): under target side its inbound decides the tool calls, - # under agent side it is a pass-through CR. A live decision on its inbound needs the - # outbound leg (Part B, below), so the per-workload gate reads state instead: the scopes - # provisioned, SPM(github-tool) stored (the PRB ran, not only the bootstrap), and the CR - # present with both request packages (``tool_onboarding_state``). - seen = {"state": {}} - - def _tool_converged() -> bool: - seen["state"] = tool_onboarding_state(admin) - return all(seen["state"].values()) - - if not poll_until(_tool_converged, timeout=ONBOARD_TIMEOUT, interval=5): - raise RuntimeError( - append_hint( - f"tool did not converge after deploy within {ONBOARD_TIMEOUT:.0f}s: {seen['state']} " - f"(scopes = all of {sorted(scn.TOOL_SCOPES)} provisioned; spm = " - f"SPM({scn.TOOL_WORKLOAD}) stored; cr = its AuthorizationPolicy CR with both request " - "packages). A failed onboarding check (D30: the tool pod has no authbridge-proxy " - "sidecar — injectTools; no opa / mcp-parser in the namespace inbound pipeline; an " - "httpGet probe) leaves no CR — see the Controller log and k8s/opa-kind-enable.sh.", - controller_commit_race_hint(deployed_at, admin=admin, workload=workload), - ) + ) + else: + # Tool — it has its own CR (D20): under target side its inbound decides the tool calls, + # under agent side it is a pass-through CR. A live decision on its inbound needs the + # outbound leg (Part B, below), so the per-workload gate reads state instead: the scopes + # provisioned, SPM(github-tool) stored (the PRB ran, not only the bootstrap), and the CR + # present with both request packages (``tool_onboarding_state``). + seen = {"state": {}} + + def _tool_converged() -> bool: + seen["state"] = tool_onboarding_state(admin) + return all(seen["state"].values()) + + if not poll_until(_tool_converged, timeout=ONBOARD_TIMEOUT, interval=5): + raise RuntimeError( + append_hint( + f"tool did not converge after deploy within {ONBOARD_TIMEOUT:.0f}s: {seen['state']} " + f"(scopes = all of {sorted(scn.TOOL_SCOPES)} provisioned; spm = " + f"SPM({scn.TOOL_WORKLOAD}) stored; cr = its AuthorizationPolicy CR with both request " + "packages). A failed onboarding check (D30: the tool pod has no authbridge-proxy " + "sidecar — injectTools; no opa / mcp-parser in the namespace inbound pipeline; an " + "httpGet probe) leaves no CR — see the Controller log and k8s/opa-kind-enable.sh.", + controller_commit_race_hint(deployed_at, admin=admin, workload=workload), ) + ) - # The workload converged, so its Provision ran and linked the subject scope (D31). Fail fast - # (not skip) when it did not: a missing link is AIAC's own fault, and without it the exchanged - # token keeps sub = the user ID, so every tool call would be denied later with no clear cause. - require_subject_scope(admin, workload) - - # Part B — enable the outbound token-exchange leg so the tool check is actually reached (under - # target side github-tool's inbound OPA, which accepts only a token with its audience; under - # agent side the agent's outbound OPA). Done after onboarding so the restarted agent (and its - # OPA sidecar) picks up both the new route and, on its next poll, the recomposed bundle. - # - # Prepared ONLY when this run actually probes outbound (``has_outbound``). The agent-only rung - # (rung 1) has no tool onboarded, so there is no real ``agent -> tool`` call to make and no - # outbound signal in its set — its outbound leg has no real-life counterpart, and token-exchange - # would short-circuit before OPA anyway (no ``github-tool`` audience grant, since the operator - # creates ``agent-{ns}-github-tool-aud`` only on TOOL deploy). So on a tool-less rung Part B is - # skipped entirely: no outbound route, no restart. Its inbound gate already converged in the - # per-workload loop above, and skipping the restart means rung 1 asserts the exact bundle the - # live event-driven onboard produced (no restart papering over a bad compose). - # - # When there IS an outbound leg: add the route so the forward proxy intercepts the github-tool - # call and runs token-exchange, grant the agent client the ``*-aud`` scope (only ever present - # once the tool is deployed) so the exchange succeeds and OPA is reached, then restart the agent - # so it reloads the route and re-fetches the recomposed bundle. - has_outbound = any(sig.kind == "outbound" for sig in signals) - if has_outbound: - ensure_github_tool_route(NAMESPACE) - if tool_onboarded: - grant_exchange_scope(admin) - restart_agent(NAMESPACE) - # Resolved once for the ctx contract, but the live outbound path re-resolves per probe - # (``resolve_agent_pod``) so a pod replaced after this point can't poison the whole run (#139). - agent_pod = resolve_agent_pod() - - ctx = { - "admin": admin, - "namespace": NAMESPACE, - "agent_pod": agent_pod, - "keycloak_url": keycloak_url, - "realm": TEST_REALM, - "tool_onboarded": tool_onboarded, - "side": side, - } + # The workload converged, so its Provision ran and linked the subject scope (D31). Fail fast + # (not skip) when it did not: a missing link is AIAC's own fault, and without it the exchanged + # token keeps sub = the user ID, so every tool call would be denied later with no clear cause. + require_subject_scope(admin, workload) + + # Part B — enable the outbound token-exchange leg so the tool check is actually reached (under + # target side github-tool's inbound OPA, which accepts only a token with its audience; under + # agent side the agent's outbound OPA). Done after onboarding so the restarted agent (and its + # OPA sidecar) picks up both the new route and, on its next poll, the recomposed bundle. + # + # Prepared ONLY when this run actually probes outbound (``has_outbound``). The agent-only rung + # (rung 1) has no tool onboarded, so there is no real ``agent -> tool`` call to make and no + # outbound signal in its set — its outbound leg has no real-life counterpart, and token-exchange + # would short-circuit before OPA anyway (no ``github-tool`` audience grant, since the operator + # creates ``agent-{ns}-github-tool-aud`` only on TOOL deploy). So on a tool-less rung Part B is + # skipped entirely: no outbound route, no restart. Its inbound gate already converged in the + # per-workload loop above, and skipping the restart means rung 1 asserts the exact bundle the + # live event-driven onboard produced (no restart papering over a bad compose). + # + # When there IS an outbound leg: add the route so the forward proxy intercepts the github-tool + # call and runs token-exchange, grant the agent client the ``*-aud`` scope (only ever present + # once the tool is deployed) so the exchange succeeds and OPA is reached, then restart the agent + # so it reloads the route and re-fetches the recomposed bundle. + has_outbound = any(sig.kind == "outbound" for sig in signals) + if has_outbound: + ensure_github_tool_route(NAMESPACE) + if tool_onboarded: + grant_exchange_scope(admin) + restart_agent(NAMESPACE) + # Resolved once for the ctx contract, but the live outbound path re-resolves per probe + # (``resolve_agent_pod``) so a pod replaced after this point can't poison the whole run (#139). + agent_pod = resolve_agent_pod() + + ctx = { + "admin": admin, + "namespace": NAMESPACE, + "agent_pod": agent_pod, + "keycloak_url": keycloak_url, + "realm": TEST_REALM, + "tool_onboarded": tool_onboarded, + "side": side, + } - # Wait for bundle-service + OPA to reflect THIS run's CR (and token-exchange to settle) before - # any assertion. The ``signals`` set (this policy's ``ready_signals``, or the Policy-A default) - # is polled until every probe reaches its terminal verdict — each deterministic for its - # scenario regardless of a stale CR (which the run replaced), and each polled to a *definitive* - # allow/deny (not ``error``) so we also wait out the post-restart token-exchange 503 window. - def _ready() -> bool: - return all(sig.decide(ctx) == sig.expected for sig in signals) - - if not poll_until(_ready, timeout=BUNDLE_TIMEOUT, interval=BUNDLE_POLL_INTERVAL): - # Surface the RAW outbound (code, body) — not just the classified outcome — so a stalled - # run is self-diagnosing (issue #139). The classifier collapses three very different - # failures into ``"error"``; the raw ``(code, body)`` tells them apart in one shot: - # * ``code=None`` + body ``"outbound probe exec failed: ..."`` — the *probe's* ``kubectl - # exec`` failed (e.g. the agent pod was replaced by a restart and its name went stale, - # ``NotFound``). A harness/pod issue, not the pipeline — OPA was never reached. - # * ``code=503`` — the ``token-exchange`` leg failed upstream (audience refused, IdP - # unreachable), so OPA was never consulted. - # * a genuine OPA policy stall can't show as ``"error"`` at all: it surfaces as ``"deny"`` - # under deny-by-default (target side: github-tool's inbound HTTP 403 with an OPA body; - # agent side: HTTP 200 + an OPA error frame), because the generated Rego carries - # ``default allow := false``. - # The observed-vs-expected line is recomputed against *this run's* signals so a denyworld - # (or any parametrized) run diagnoses itself, not a hardcoded Policy-A probe. - observed = "; ".join(f"{sig.label()}={sig.decide(ctx)!r}(want {sig.expected!r})" for sig in signals) - # Re-resolve the pod fresh here (not the possibly-stale ``ctx["agent_pod"]``) so the raw - # line reflects the *current* live pod. Dump the raw (code, body) for the first outbound - # signal — the leg where the #139 stale-pod / 503 failures actually show up. - raw_line = "" - ob_sig = next((s for s in signals if s.kind == "outbound"), None) - if ob_sig is not None: - ob_token = mint_token( - ob_sig.subject, scn.USER_PASSWORD, keycloak_url=ctx["keycloak_url"], realm=ctx["realm"] - ) - ob_code, ob_body = outbound_probe( - ob_token, ob_sig.tool_bare, namespace=ctx["namespace"], agent_pod=resolve_agent_pod() - ) - raw_line = f" [raw {ob_sig.label()}: HTTP {ob_code}, body={ob_body[:300]!r}]" - raise RuntimeError( - f"live pipeline did not converge within {BUNDLE_TIMEOUT:.0f}s after onboarding " - f"{workloads} + Part B: {observed}.{raw_line} code=None + 'exec failed' body = a " - "stale/gone agent pod (harness); 503 = token-exchange never came up (OPA not reached); " - "under deny-by-default a real policy stall reads 'deny', never 'error' — see " - "k8s/opa-kind-runbook.md and issue #139." + # Wait for bundle-service + OPA to reflect THIS run's CR (and token-exchange to settle) before + # any assertion. The ``signals`` set (this policy's ``ready_signals``, or the Policy-A default) + # is polled until every probe reaches its terminal verdict — each deterministic for its + # scenario regardless of a stale CR (which the run replaced), and each polled to a *definitive* + # allow/deny (not ``error``) so we also wait out the post-restart token-exchange 503 window. + def _ready() -> bool: + return all(sig.decide(ctx) == sig.expected for sig in signals) + + if not poll_until(_ready, timeout=BUNDLE_TIMEOUT, interval=BUNDLE_POLL_INTERVAL): + # Surface the RAW outbound (code, body) — not just the classified outcome — so a stalled + # run is self-diagnosing (issue #139). The classifier collapses three very different + # failures into ``"error"``; the raw ``(code, body)`` tells them apart in one shot: + # * ``code=None`` + body ``"outbound probe exec failed: ..."`` — the *probe's* ``kubectl + # exec`` failed (e.g. the agent pod was replaced by a restart and its name went stale, + # ``NotFound``). A harness/pod issue, not the pipeline — OPA was never reached. + # * ``code=503`` — the ``token-exchange`` leg failed upstream (audience refused, IdP + # unreachable), so OPA was never consulted. + # * a genuine OPA policy stall can't show as ``"error"`` at all: it surfaces as ``"deny"`` + # under deny-by-default (target side: github-tool's inbound HTTP 403 with an OPA body; + # agent side: HTTP 200 + an OPA error frame), because the generated Rego carries + # ``default allow := false``. + # The observed-vs-expected line is recomputed against *this run's* signals so a denyworld + # (or any parametrized) run diagnoses itself, not a hardcoded Policy-A probe. + observed = "; ".join(f"{sig.label()}={sig.decide(ctx)!r}(want {sig.expected!r})" for sig in signals) + # Re-resolve the pod fresh here (not the possibly-stale ``ctx["agent_pod"]``) so the raw + # line reflects the *current* live pod. Dump the raw (code, body) for the first outbound + # signal — the leg where the #139 stale-pod / 503 failures actually show up. + raw_line = "" + ob_sig = next((s for s in signals if s.kind == "outbound"), None) + if ob_sig is not None: + ob_token = mint_token( + ob_sig.subject, scn.USER_PASSWORD, keycloak_url=ctx["keycloak_url"], realm=ctx["realm"] ) - yield ctx - finally: - # Capture every AIAC CR and the combiner to the pytest host FIRST — the undeploy and the scrub - # remove or change them. Best-effort: it never raises, on the success and the failure path. - capture_aiac_crs("teardown", label=label) - # Teardown — full-to-pristine, best-effort per step (each helper tolerates already-absent objects). - for workload in reversed(workloads): # undeploy in reverse deploy order - undeploy_workload(workload) - # Explicit scrub — do not trust an unverified operator cascade. Same reset the pre-run slate runs. - _scrub_to_pristine(admin) - # Verify pristine: both clients gone AND no AuthorizationPolicy CR remains. A leaked footprint is - # a test failure, not silent drift — but do not mask a failure already propagating out of the try. - if not poll_until( - lambda: not workload_clients_present(admin) and no_authpolicies_remain(), - timeout=DEPLOY_TIMEOUT, - interval=5, - ): - msg = ( - f"teardown did not restore pristine: Keycloak client(s) still present=" - f"{workload_clients_present(admin)}, AuthorizationPolicy CR(s) remain={not no_authpolicies_remain()} " - f"in {NAMESPACE!r}. See handoff 03 teardown (full-to-pristine)." + ob_code, ob_body = outbound_probe( + ob_token, ob_sig.tool_bare, namespace=ctx["namespace"], agent_pod=resolve_agent_pod() ) - if sys.exc_info()[0] is not None: - log.error("%s [suppressed — a prior error is propagating]", msg) - else: - raise RuntimeError(msg) + raw_line = f" [raw {ob_sig.label()}: HTTP {ob_code}, body={ob_body[:300]!r}]" + raise RuntimeError( + f"live pipeline did not converge within {BUNDLE_TIMEOUT:.0f}s after onboarding " + f"{workloads} + Part B: {observed}.{raw_line} code=None + 'exec failed' body = a " + "stale/gone agent pod (harness); 503 = token-exchange never came up (OPA not reached); " + "under deny-by-default a real policy stall reads 'deny', never 'error' — see " + "k8s/opa-kind-runbook.md and issue #139." + ) + yield ctx # ====================================================================================== @@ -2475,60 +2553,23 @@ def pristine_stack(workloads: Sequence[str], *, policy_md: str = scn.POLICY_ABST never comes). Yields ``ctx = {"admin", "namespace", "keycloak_url", "realm", "side"}`` (``side`` = ``live_enforcement_side()``); the caller deploys. - Same skip gates (pipeline wiring, env, event path, Direct Access Grants on the login client — before - any mutation), same no-workloads slate (a ``pre-run-slate`` CR capture, then undeploy + - ``_scrub_to_pristine`` + ``reenable_provisioned_clients`` + clients-gone poll), same realm/users - provisioning + ``sub`` mapper gate, ``policy.md`` mount and image load for ``workloads``; and the - same teardown: a ``teardown`` CR capture first, then full-to-pristine, verified (clients gone, no CR - left), and last the grant off again if it was off (``direct_access_grants``).""" + Same skip gates (pipeline wiring, env, event path), the same slate and teardown (``_clean_slate``: + Direct Access Grants on the login client, the no-workloads slate, realm users and roles, the ``sub`` + check; at exit full-to-pristine, verified), then the ``policy.md`` mount and the image load for + ``workloads``.""" require_pipeline(namespace=NAMESPACE, workloads=[]) creds = require_env_or_skip("KEYCLOAK_URL", "KEYCLOAK_ADMIN_USERNAME", "KEYCLOAK_ADMIN_PASSWORD") keycloak_url = creds["KEYCLOAK_URL"] admin = connect_admin() require_event_path(admin=admin, realm=TEST_REALM) side = live_enforcement_side() - # Before the first mutation, so a skip leaves the cluster as it was; restored in its own ``finally``. - with direct_access_grants(admin, TEST_REALM): - label = _capture_label() - capture_aiac_crs("pre-run-slate", label=label) - undeploy_workload(scn.AGENT_WORKLOAD) - undeploy_workload(scn.TOOL_WORKLOAD) - _scrub_to_pristine(admin) - reenable_provisioned_clients(admin, TEST_REALM) - if not poll_until(lambda: not workload_clients_present(admin), timeout=DEPLOY_TIMEOUT, interval=5): - raise RuntimeError( - f"pre-run cleanup left Keycloak client(s) {workload_clients_present(admin)} for {NAMESPACE!r} — the " - "fresh deploy would not re-fire CLIENT_CREATED, so event-driven onboarding would never trigger." - ) - provision_realm_and_users(admin, TEST_REALM) - verify_login_subject(keycloak_url=keycloak_url, realm=TEST_REALM, user="dev-user", password=scn.USER_PASSWORD) - - try: - ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) - load_workload_images(list(workloads)) - yield { - "admin": admin, - "namespace": NAMESPACE, - "keycloak_url": keycloak_url, - "realm": TEST_REALM, - "side": side, - } - finally: - capture_aiac_crs("teardown", label=label) # FIRST: the undeploy and the scrub remove the CRs - for workload in reversed(list(workloads)): - undeploy_workload(workload) - _scrub_to_pristine(admin) - if not poll_until( - lambda: not workload_clients_present(admin) and no_authpolicies_remain(), - timeout=DEPLOY_TIMEOUT, - interval=5, - ): - msg = ( - f"teardown did not restore pristine: Keycloak client(s) still present=" - f"{workload_clients_present(admin)}, AuthorizationPolicy CR(s) remain={not no_authpolicies_remain()} " - f"in {NAMESPACE!r}." - ) - if sys.exc_info()[0] is not None: - log.error("%s [suppressed — a prior error is propagating]", msg) - else: - raise RuntimeError(msg) + with _clean_slate(admin, workloads, keycloak_url=keycloak_url): + ensure_agent_policy(CONTROLLER_NAMESPACE, policy_md=policy_md) + load_workload_images(list(workloads)) + yield { + "admin": admin, + "namespace": NAMESPACE, + "keycloak_url": keycloak_url, + "realm": TEST_REALM, + "side": side, + } diff --git a/test/unit/agent/uc/onboarding/test_uc1_login_client.py b/test/unit/agent/uc/onboarding/test_uc1_login_client.py index 498dd307..2d2bdf67 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_login_client.py +++ b/test/unit/agent/uc/onboarding/test_uc1_login_client.py @@ -1,12 +1,15 @@ -"""``uc1_onboard.direct_access_grants`` turns on the password grant of the login client for the life of -a fixture, and turns it off again at exit if it was off — pure unit tests. +"""``uc1_onboard.hold_direct_access_grants`` turns on the password grant of the login client for the +session, and ``release_direct_access_grants`` turns it off again at session end if the harness turned it +on — pure unit tests. The harness mints user tokens through the login client ``rossoctl`` with the password grant, so the -client needs Direct Access Grants on. ``onboarded_stack``, ``pristine_stack`` and the disabled -cross-service test enter ``direct_access_grants`` before their first mutation. Its own ``finally`` -turns the grant off again, so a teardown step that raises cannot leave the grant on. A grant that -cannot be written is a skip with the cause. These tests use a ``MagicMock`` admin (no Keycloak). They -import the system harness like ``test_uc1_subject_scope.py`` does. +client needs Direct Access Grants on. ``_clean_slate`` (shared by ``onboarded_stack`` and +``pristine_stack``) holds the grant before its first mutation, and ``test/system/conftest.py`` releases +it at session end. The harness writes a marker attribute with the grant, so a run that is killed before +its release is undone by the next run, and a grant that someone else turned on is never turned off. A +grant that cannot be read or written is a skip with the cause. These tests use a stateful fake admin (no +Keycloak) that applies the partial-update rules of Keycloak 26. They import the system harness like +``test_uc1_subject_scope.py`` does. """ from __future__ import annotations @@ -25,22 +28,47 @@ from test.system import uc1_onboard as uc1 # noqa: E402 LOGIN_UUID = "uuid-1" +MARKER = uc1.DAG_MARKER_ATTRIBUTE + + +@pytest.fixture(autouse=True) +def _no_holds(monkeypatch: pytest.MonkeyPatch) -> None: + """Each test starts a fresh session: no login client is held.""" + monkeypatch.setattr(uc1, "_dag_holds", {}) + + +def _login(enabled: bool, *, marked: bool = False, client_id: str = uc1.KEYCLOAK_CLIENT_ID) -> dict: + attributes = {"keep.me": "yes", **({MARKER: "true"} if marked else {})} + return {"clientId": client_id, "id": LOGIN_UUID, "directAccessGrantsEnabled": enabled, "attributes": attributes} def _admin(client: dict | None) -> MagicMock: - """A ``KeycloakAdmin`` stand-in whose filtered ``clientId`` query finds ``client`` (or nothing).""" + """A ``KeycloakAdmin`` stand-in over one stored client. Its filtered ``clientId`` query finds the + client (or nothing), and ``update_client`` applies a partial update as Keycloak 26 does: only the + given fields change; in ``attributes`` only the given keys change, and ``""`` removes a key.""" admin = MagicMock(name="KeycloakAdmin") + admin.stored = client + + def _update(uuid: str, fields: dict) -> None: + stored = admin.stored + for key, value in fields.items(): + if key != "attributes": + stored[key] = value + continue + for name, attr in value.items(): + if attr == "": + stored["attributes"].pop(name, None) + else: + stored["attributes"][name] = attr + admin.get_client_id.side_effect = lambda client_id: ( - client["id"] if client is not None and client_id == client["clientId"] else None + admin.stored["id"] if admin.stored is not None and client_id == admin.stored["clientId"] else None ) - admin.get_client.side_effect = lambda uuid: dict(client) + admin.get_client.side_effect = lambda uuid: {**admin.stored, "attributes": dict(admin.stored["attributes"])} + admin.update_client.side_effect = _update return admin -def _login(enabled: bool) -> dict: - return {"clientId": uc1.KEYCLOAK_CLIENT_ID, "id": LOGIN_UUID, "directAccessGrantsEnabled": enabled} - - def _grant_writes(admin: MagicMock) -> list[bool]: return [call.args[1]["directAccessGrantsEnabled"] for call in admin.update_client.call_args_list] @@ -65,44 +93,107 @@ def test_a_missing_login_client_is_none() -> None: # ====================================================================================== -# ensure_direct_access_grants — the entry half +# hold_direct_access_grants — the entry half, once per session # ====================================================================================== -def test_the_grant_is_enabled_with_a_partial_update() -> None: +def test_the_grant_is_enabled_with_the_marker_in_one_partial_update() -> None: admin = _admin(_login(False)) - assert uc1.ensure_direct_access_grants(admin, "rossoctl") == LOGIN_UUID + uc1.hold_direct_access_grants(admin, "rossoctl") admin.change_current_realm.assert_called_with("rossoctl") - admin.update_client.assert_called_once_with(LOGIN_UUID, {"directAccessGrantsEnabled": True}) + admin.update_client.assert_called_once_with( + LOGIN_UUID, {"directAccessGrantsEnabled": True, "attributes": {MARKER: "true"}} + ) + assert admin.stored["attributes"]["keep.me"] == "yes" # other attributes are not written -def test_a_grant_that_is_on_is_not_written() -> None: +def test_a_grant_that_is_on_with_no_marker_is_not_written_and_not_released() -> None: admin = _admin(_login(True)) - assert uc1.ensure_direct_access_grants(admin, "rossoctl") is None + uc1.hold_direct_access_grants(admin, "rossoctl") + uc1.release_direct_access_grants() admin.update_client.assert_not_called() +def test_the_grant_is_read_and_written_once_per_session() -> None: + admin = _admin(_login(False)) + for _ in range(3): # three stacks (three test modules) in one session + uc1.hold_direct_access_grants(admin, "rossoctl") + uc1.release_direct_access_grants() + assert admin.get_client_id.call_count == 1 + assert _grant_writes(admin) == [True, False] + + +def test_a_marker_left_by_a_killed_run_is_taken_over_and_released() -> None: + """The case of the review: an earlier run turned the grant on and was killed before its release. The + marker is still on the client, so this session turns the grant off at its end.""" + admin = _admin(_login(True, marked=True)) + uc1.hold_direct_access_grants(admin, "rossoctl") + admin.update_client.assert_not_called() # already on: nothing to write at entry + uc1.release_direct_access_grants() + assert admin.stored["directAccessGrantsEnabled"] is False + assert MARKER not in admin.stored["attributes"] + + def test_a_missing_login_client_skips_and_is_not_created() -> None: admin = _admin(None) with pytest.raises(pytest.skip.Exception, match=uc1.KEYCLOAK_CLIENT_ID): - uc1.ensure_direct_access_grants(admin, "rossoctl") + uc1.hold_direct_access_grants(admin, "rossoctl") admin.update_client.assert_not_called() admin.create_client.assert_not_called() -def test_a_refused_read_skips_with_the_cause() -> None: +def test_a_missing_realm_skips_with_that_cause() -> None: + admin = _admin(_login(False)) + admin.get_client_id.side_effect = KeycloakError(error_message="Realm not found.", response_code=404) + with pytest.raises(pytest.skip.Exception, match="realm 'rossoctl' does not exist"): + uc1.hold_direct_access_grants(admin, "rossoctl") + admin.update_client.assert_not_called() + + +@pytest.mark.parametrize( + "error", + [KeycloakError(error_message="forbidden", response_code=403), ConnectionError("reset"), TypeError("not a list")], +) +def test_a_failed_read_skips_with_the_cause(error: Exception) -> None: admin = _admin(_login(False)) - admin.get_client_id.side_effect = KeycloakError(error_message="forbidden", response_code=403) - with pytest.raises(pytest.skip.Exception, match="cannot read the login client.*forbidden"): - uc1.ensure_direct_access_grants(admin, "rossoctl") + admin.get_client_id.side_effect = error + with pytest.raises(pytest.skip.Exception, match=f"cannot read the login client.*{error}"): + uc1.hold_direct_access_grants(admin, "rossoctl") admin.update_client.assert_not_called() -def test_a_refused_update_skips_with_the_cause() -> None: +def test_a_refused_update_skips_with_the_cause_and_writes_nothing_more() -> None: admin = _admin(_login(False)) admin.update_client.side_effect = KeycloakError(error_message="forbidden", response_code=403) with pytest.raises(pytest.skip.Exception, match="cannot enable Direct Access Grants.*forbidden"): - uc1.ensure_direct_access_grants(admin, "rossoctl") + uc1.hold_direct_access_grants(admin, "rossoctl") + assert admin.update_client.call_count == 1 # the read again finds no marker: nothing to undo + assert uc1._dag_holds == {} + + +def test_an_update_that_was_applied_but_lost_its_response_is_undone_before_the_skip() -> None: + """The case of the review: the PUT is applied, but the response times out or the connection resets.""" + admin = _admin(_login(False)) + apply = admin.update_client.side_effect + + def _applied_then_reset(uuid: str, fields: dict) -> None: + apply(uuid, fields) + if admin.update_client.call_count == 1: + raise ConnectionError("connection reset") + + admin.update_client.side_effect = _applied_then_reset + with pytest.raises(pytest.skip.Exception, match="cannot enable Direct Access Grants.*connection reset"): + uc1.hold_direct_access_grants(admin, "rossoctl") + assert admin.stored["directAccessGrantsEnabled"] is False + assert MARKER not in admin.stored["attributes"] + assert uc1._dag_holds == {} + + +def test_another_login_client_can_be_held() -> None: + admin = _admin(_login(False, client_id="other-login")) + uc1.hold_direct_access_grants(admin, "rossoctl", client_id="other-login") + admin.get_client_id.assert_called_once_with("other-login") + assert _grant_writes(admin) == [True] def test_provision_realm_and_users_does_not_touch_the_login_client() -> None: @@ -114,85 +205,105 @@ def test_provision_realm_and_users_does_not_touch_the_login_client() -> None: # ====================================================================================== -# restore_direct_access_grants — the exit half, best-effort +# release_direct_access_grants — the exit half at session end, best-effort # ====================================================================================== -def test_the_restore_turns_the_grant_off_with_a_partial_update_and_no_read() -> None: - admin = _admin(_login(True)) - uc1.restore_direct_access_grants(admin, "rossoctl", LOGIN_UUID) - admin.change_current_realm.assert_called_with("rossoctl") - admin.update_client.assert_called_once_with(LOGIN_UUID, {"directAccessGrantsEnabled": False}) - admin.get_client_id.assert_not_called() # the UUID comes from the enable +def test_the_release_reads_again_and_turns_the_grant_and_the_marker_off() -> None: + admin = _admin(_login(False)) + uc1.hold_direct_access_grants(admin, "rossoctl") + admin.get_client.reset_mock() + uc1.release_direct_access_grants() + admin.get_client.assert_called_once_with(LOGIN_UUID) # the UUID comes from the hold + assert admin.update_client.call_args.args == ( + LOGIN_UUID, + {"directAccessGrantsEnabled": False, "attributes": {MARKER: ""}}, + ) + assert admin.stored == _login(False) + assert uc1._dag_holds == {} + + +def test_a_grant_that_someone_took_over_during_the_run_is_not_turned_off() -> None: + """The case of the review: during the run an operator enables the grant on purpose (the runbook + snippet), which removes the marker. The release reads the client again and leaves the grant on.""" + admin = _admin(_login(False)) + uc1.hold_direct_access_grants(admin, "rossoctl") + admin.update_client(LOGIN_UUID, {"directAccessGrantsEnabled": True, "attributes": {MARKER: ""}}) # the operator + admin.update_client.reset_mock() + uc1.release_direct_access_grants() + admin.update_client.assert_not_called() + assert admin.stored["directAccessGrantsEnabled"] is True @pytest.mark.parametrize("error", [KeycloakError(error_message="forbidden", response_code=403), ConnectionError()]) -def test_a_refused_restore_is_logged_not_raised(caplog: pytest.LogCaptureFixture, error: Exception) -> None: - admin = _admin(_login(True)) +def test_a_failed_release_is_logged_not_raised(caplog: pytest.LogCaptureFixture, error: Exception) -> None: + admin = _admin(_login(False)) + uc1.hold_direct_access_grants(admin, "rossoctl") admin.update_client.side_effect = error - uc1.restore_direct_access_grants(admin, "rossoctl", LOGIN_UUID) # never masks a teardown failure + uc1.release_direct_access_grants() # never masks a teardown failure assert "cannot turn Direct Access Grants off again" in caplog.text + assert uc1._dag_holds == {} -# ====================================================================================== -# direct_access_grants — the context manager: the restore is in its own ``finally`` -# ====================================================================================== +def test_a_release_with_nothing_held_does_nothing() -> None: + uc1.release_direct_access_grants() -def test_the_context_turns_on_a_grant_that_was_off_and_turns_it_off_at_exit() -> None: - admin = _admin(_login(False)) - with uc1.direct_access_grants(admin, "rossoctl"): - assert _grant_writes(admin) == [True] - assert _grant_writes(admin) == [True, False] - +# ====================================================================================== +# _clean_slate — the slate and teardown that onboarded_stack and pristine_stack share +# ====================================================================================== -def test_the_context_does_not_write_a_grant_that_was_on() -> None: - admin = _admin(_login(True)) - with uc1.direct_access_grants(admin, "rossoctl"): - pass - admin.update_client.assert_not_called() +_SLATE_STEPS = ( + "hold_direct_access_grants", + "capture_aiac_crs", + "undeploy_workload", + "_scrub_to_pristine", + "reenable_provisioned_clients", + "provision_realm_and_users", + "verify_login_subject", +) -@pytest.mark.parametrize( - "error", - [AssertionError("clear_policy_store: HTTP 500"), KeycloakError(error_message="token refused", response_code=401)], -) -def test_a_teardown_step_that_raises_still_restores_the_grant(error: Exception) -> None: - """The case of the review: a scrub step of the fixture teardown raises (for example - ``clear_policy_store`` or ``cleanup_provisioned``). The grant is still turned off, and the error - propagates unchanged.""" - admin = _admin(_login(False)) - with pytest.raises(type(error)) as raised: - with uc1.direct_access_grants(admin, "rossoctl"): - try: - pass # the fixture body - finally: - raise error # a teardown step of the body - assert raised.value is error - assert _grant_writes(admin) == [True, False] +@pytest.fixture +def steps(monkeypatch: pytest.MonkeyPatch) -> MagicMock: + """Every step of ``_clean_slate`` replaced by a recorder; the polls succeed at once.""" + recorder = MagicMock(name="steps") + for name in _SLATE_STEPS: + monkeypatch.setattr(uc1, name, getattr(recorder, name)) + monkeypatch.setattr(uc1, "poll_until", lambda *a, **k: True) + monkeypatch.setattr(uc1, "workload_clients_present", lambda admin: set()) + monkeypatch.setattr(uc1, "no_authpolicies_remain", lambda: True) + monkeypatch.setattr(uc1, "_capture_label", lambda: "rung") + return recorder -def test_a_skip_in_the_body_still_restores_the_grant() -> None: - admin = _admin(_login(False)) - with pytest.raises(pytest.skip.Exception): - with uc1.direct_access_grants(admin, "rossoctl"): - pytest.skip("verify_login_subject cannot mint a token") - assert _grant_writes(admin) == [True, False] +def _step_names(recorder: MagicMock) -> list[str]: + return [call[0] for call in recorder.mock_calls] -def test_a_failed_restore_does_not_mask_the_body_error() -> None: - admin = _admin(_login(False)) - admin.update_client.side_effect = [None, KeycloakError(error_message="forbidden", response_code=403)] - with pytest.raises(RuntimeError, match="teardown did not restore pristine"): - with uc1.direct_access_grants(admin, "rossoctl"): - raise RuntimeError("teardown did not restore pristine") +def test_the_slate_holds_the_grant_before_its_first_mutation(steps: MagicMock) -> None: + with uc1._clean_slate(MagicMock(), ["github-agent"], keycloak_url="http://kc"): + assert _step_names(steps)[0] == "hold_direct_access_grants" + assert _step_names(steps)[-1] == "verify_login_subject" + assert _step_names(steps)[-2:] == ["undeploy_workload", "_scrub_to_pristine"] -def test_a_skip_at_entry_writes_nothing() -> None: - admin = _admin(None) +def test_a_skip_at_the_hold_mutates_nothing(steps: MagicMock) -> None: + steps.hold_direct_access_grants.side_effect = pytest.skip.Exception("no login client") body = MagicMock(name="body") with pytest.raises(pytest.skip.Exception): - with uc1.direct_access_grants(admin, "rossoctl"): + with uc1._clean_slate(MagicMock(), ["github-agent"], keycloak_url="http://kc"): body() body.assert_not_called() - admin.update_client.assert_not_called() + assert _step_names(steps) == ["hold_direct_access_grants"] + + +@pytest.mark.parametrize("error", [RuntimeError("did not converge"), pytest.skip.Exception("no token")]) +def test_the_body_failing_still_runs_the_teardown(steps: MagicMock, error: BaseException) -> None: + with pytest.raises(type(error)): + with uc1._clean_slate(MagicMock(), ["github-agent", "github-tool"], keycloak_url="http://kc"): + steps.reset_mock() + raise error + assert steps.mock_calls[0].args == ("teardown",) # the CR capture comes first + assert [c.args[0] for c in steps.undeploy_workload.call_args_list] == ["github-tool", "github-agent"] + steps._scrub_to_pristine.assert_called_once() From 519ca7f3e02524d4bb7aedf12420c49bd9691ba1 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Thu, 15 Oct 2026 15:46:29 +0300 Subject: [PATCH 07/15] fix(agent): name a refused LLM key as LLMCredentialsError A refused LLM API key (HTTP 401/403) was logged as UnparseableLLMResponseError ("schema-invalid response"), which hid the cause. raise_sanitized now maps 401/403 to the new LLMCredentialsError (PRB subclass in graph.py). It stays permanent (DLQ on the first delivery) and still triggers the UC1 rollback. It gets its own logger (aiac.onboarding.llm_credentials) and a sanitized 502 route body. upstream._status_code becomes the public http_status. The failure_rollback system test now also requires the PRB request to the injected URL and its 404 in the Controller log, so phase 1 proves that the injection caused the failure. Docs: deploy guide step 3b gets "Check or change the API key" (the Controller reads the key only from aiac-agent-secret, not from .env). The system-lane preconditions add a valid in-cluster LLM key. The specs list the new error. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- docs/agents/test.md | 6 ++++ docs/specs/components/aiac-agent.md | 5 +-- .../aiac-agent/policy-rules-builder.md | 14 +++++--- .../aiac-agent/uc1-service-onboarding.md | 4 +-- docs/specs/components/event-broker.md | 2 +- docs/testing/uc1-onboarding-pipeline.md | 6 ++-- k8s/aiac-deployment-guide.md | 31 ++++++++++++++-- src/aiac/agent/controller/routes.py | 8 +++++ src/aiac/agent/eventbus/consumer.py | 16 +++++---- src/aiac/agent/llm.py | 20 +++++++++-- src/aiac/agent/policy_rules_builder/graph.py | 24 ++++++++++--- src/aiac/agent/shared/error_logging.py | 2 ++ src/aiac/agent/uc/onboarding/orchestrator.py | 8 +++-- src/aiac/shared/upstream.py | 4 +-- .../test_uc1_onboard_failure_rollback.py | 19 +++++++++- test/unit/agent/controller/test_routes.py | 14 ++++++++ test/unit/agent/eventbus/test_consumer.py | 3 ++ .../unit/agent/policy_digester/test_digest.py | 17 ++++++++- .../agent/policy_rules_builder/test_graph.py | 18 ++++++++++ test/unit/agent/shared/test_error_logging.py | 14 ++++++++ test/unit/agent/test_llm.py | 36 +++++++++++++++++++ .../agent/uc/onboarding/test_orchestrator.py | 6 ++-- 22 files changed, 240 insertions(+), 37 deletions(-) diff --git a/docs/agents/test.md b/docs/agents/test.md index 26877408..c1cbdd74 100644 --- a/docs/agents/test.md +++ b/docs/agents/test.md @@ -100,6 +100,12 @@ Preconditions: - `k8s/opa-kind-enable.sh` does both steps (run it one time). - The event path wired: the NATS broker and the `aiac-event-listener` Keycloak SPI (see `docs/testing/uc1-onboarding-pipeline.md`, "Preconditions"). +- An LLM key that the endpoint accepts, in the Controller's Secret + `aiac-system/aiac-agent-secret`. The in-cluster Controller does not read + `.env`. If the endpoint refuses the key, each onboarding rolls back, and the + fixtures time out with "agent did not converge" or "tool did not converge". + The Controller log then shows `LLMCredentialsError`. To check or change the + key: `k8s/aiac-deployment-guide.md`, step 3b, "Check or change the API key". Do **not** deploy the demo `github-agent` / `github-tool` yourself. The session fixture (`onboarded_stack` in `test/system/uc1_onboard.py`) builds and loads the diff --git a/docs/specs/components/aiac-agent.md b/docs/specs/components/aiac-agent.md index 53e59fe5..558c2566 100644 --- a/docs/specs/components/aiac-agent.md +++ b/docs/specs/components/aiac-agent.md @@ -107,7 +107,7 @@ The consumer **awaits** the internal handler before it issues the NATS acknowled On handler failure, the consumer classifies the exception by **type**, not by HTTP status code: -- **Permanent** — `PolicyConflictError`, `PolicyContradictionError`, `PolicyRulesBuilderError`, `UnparseableLLMResponseError`, and `EnforcementPreconditionError` (a failed UC1 precondition check, D30). The consumer calls `term()` and routes the message to `aiac.apply.dlq` **immediately**. There is no redelivery, because the same input cannot succeed on a retry. A failed precondition check needs a fix in the cluster first. After the fix, the operator starts the onboarding again (see [`uc1-service-onboarding.md` → Precondition checks](aiac-agent/uc1-service-onboarding.md#precondition-checks-d30)). +- **Permanent** — `PolicyConflictError`, `PolicyContradictionError`, `PolicyRulesBuilderError`, `UnparseableLLMResponseError`, `LLMCredentialsError` (the LLM endpoint refused the API key, HTTP 401 / 403), and `EnforcementPreconditionError` (a failed UC1 precondition check, D30). The consumer calls `term()` and routes the message to `aiac.apply.dlq` **immediately**. There is no redelivery, because the same input cannot succeed on a retry. A refused LLM key and a failed precondition check need a fix in the cluster first. After the fix, the operator starts the onboarding again (see [`uc1-service-onboarding.md` → Precondition checks](aiac-agent/uc1-service-onboarding.md#precondition-checks-d30)). - **Retryable** — `LLMAccessError`, plus any genuinely unknown or transient error. The consumer does **not** ack. NATS redelivers after `AckWait`, up to `MAX_DELIVER` (5) deliveries, then routes the message to `aiac.apply.dlq`. A failed `rerender_role` (a role-membership subject; it re-raises each IdP, Policy Store or PDP error) is in this class. A redelivery is safe, because each run reads the current holders. - **Retryable, not visible yet** — `ServiceNotVisibleError` (UC1 only). The IdP still answers `404` for the new service after the Orchestrator's bounded wait on the first read, or after the one read of a redelivery (the event-before-commit race; see [`uc1-service-onboarding.md` → The first read waits for a new client](aiac-agent/uc1-service-onboarding.md#the-first-read-waits-for-a-new-client-d33)). The client can become visible some seconds later, so the consumer does not wait for `AckWait`: - Below `MAX_DELIVER`, the consumer calls `nak(delay=…)` with the delay `AIAC_NOT_VISIBLE_NAK_DELAY_SECONDS` (default `30` s; see [Configuration](#configuration) below). NATS redelivers the message after the delay, not after `AckWait` (600 s). The consumer does not ack the message, and it does not call `term()`. @@ -314,12 +314,13 @@ The PRB raises a typed exception hierarchy (specified in [`aiac-agent/policy-rul | `PolicyRulesBuilderError` | PRB `_audit`, after `MAX_AUDIT_RETRIES` | `422` | | `LLMAccessError` | PRB `_structured_call`, transient retries exhausted | `502` | | `UnparseableLLMResponseError` | PRB `_structured_call`, reachable but unparseable | `502` | +| `LLMCredentialsError` | PRB `_structured_call`, the endpoint refused the credentials (HTTP 401 / 403) | `502` | | `PolicyContradictionError` | PRB `_audit`, genuine contradiction | `422` | | `PolicyConflictError` (carries a `ConflictReport`) | `ServicePolicyBuilder.build` (UC1) | `422` | | `EnforcementPreconditionError` (carries `failures: list[str]`) | UC1 Orchestrator, the precondition checks (D30), before Provision | `409` | | `ServiceNotVisibleError` (an `HTTPException`) | UC1 Orchestrator, the first `get_service`, when the IdP still answers `404` after the client wait (D33) | `502` | -The base class `PolicyRulesBuilderBaseError` is a `500` safety net: any unforeseen PRB error still returns a defined status, not an untyped `500`. Both `LLMAccessError` and `UnparseableLLMResponseError` map to `502`, but they differ on the async path (see [Async failure classification](#async-failure-classification)). The HTTP status is decoupled from the async retry class. +The base class `PolicyRulesBuilderBaseError` is a `500` safety net: any unforeseen PRB error still returns a defined status, not an untyped `500`. `LLMAccessError`, `UnparseableLLMResponseError` and `LLMCredentialsError` all map to `502`, but they differ on the async path (see [Async failure classification](#async-failure-classification)). The HTTP status is decoupled from the async retry class. ### Sanitized body vs. full log diff --git a/docs/specs/components/aiac-agent/policy-rules-builder.md b/docs/specs/components/aiac-agent/policy-rules-builder.md index d41e5e34..9bb7d52d 100644 --- a/docs/specs/components/aiac-agent/policy-rules-builder.md +++ b/docs/specs/components/aiac-agent/policy-rules-builder.md @@ -78,12 +78,13 @@ The async **retry class** is a separate axis — see the decoupling note below t | `PolicyRulesBuilderError` | `_audit`, after `MAX_AUDIT_RETRIES` (3) ordinary rejections | 422 | permanent | The audit budget is exhausted | | `LLMAccessError` | `_structured_call`, when **transient** failures (5xx / connect / timeout) are **exhausted** | 502 | **retryable** | Wraps the original error; message sanitized | | `UnparseableLLMResponseError` | `_structured_call`, when the LLM is **reachable but the response is unparseable / schema-invalid** | 502 | permanent | A non-transient parse or validation error; message sanitized | +| `LLMCredentialsError` | `_structured_call`, when the LLM endpoint **refuses the credentials** (HTTP 401 / 403, for example a revoked API key) | 502 | permanent | Not retried; the key must be fixed first. Message sanitized | | `PolicyContradictionError` | `_audit`, on a **genuine** intra-focal grant/deny contradiction | 422 | permanent | Carries the focal entity + all contradictions; short-circuits past retry. UC1 aggregates it — see **Contradiction contract** | | `PolicyConflictError` | `ServicePolicyBuilder.build` (UC1, `conflict_detection.py`) — **not** the graph | 422 | permanent | Carries a `ConflictReport`; aggregates per-focal contradictions across a UC1 run | -**HTTP status is decoupled from the async retry class.** `LLMAccessError` and -`UnparseableLLMResponseError` both map to **502**, but only `LLMAccessError` is **retryable** — the -unparseable case is **permanent**. A caller must read the async class, not the HTTP code, to decide +**HTTP status is decoupled from the async retry class.** `LLMAccessError`, +`UnparseableLLMResponseError` and `LLMCredentialsError` all map to **502**, but only `LLMAccessError` is +**retryable** — the unparseable and the refused-credentials cases are **permanent**. A caller must read the async class, not the HTTP code, to decide whether to retry. --- @@ -293,12 +294,15 @@ distinct: classifies which failures retry (5xx / connect / timeout). `LLM_REQUEST_TIMEOUT` (default `120`) bounds each request. The Phase-1 file read does **not** retry; it raises directly. -`_structured_call` has two raise outcomes: +`_structured_call` has three raise outcomes: - **Transient failures exhausted** → raise `LLMAccessError` (502, **retryable**). It wraps the original error and sanitizes the message. +- **The endpoint refuses the credentials** (HTTP 401 / 403) → raise `LLMCredentialsError` (502, + **permanent**), sanitized. It is not retried. - **Reachable but the response is unparseable / schema-invalid** (a non-transient parse or - validation error) → raise `UnparseableLLMResponseError` (502, **permanent**), sanitized. + validation error, or any other non-transient error) → raise `UnparseableLLMResponseError` (502, + **permanent**), sanitized. See **Exceptions** for the full taxonomy and the HTTP-vs-retry-class decoupling. diff --git a/docs/specs/components/aiac-agent/uc1-service-onboarding.md b/docs/specs/components/aiac-agent/uc1-service-onboarding.md index 4945f06a..880dc722 100644 --- a/docs/specs/components/aiac-agent/uc1-service-onboarding.md +++ b/docs/specs/components/aiac-agent/uc1-service-onboarding.md @@ -156,7 +156,7 @@ The bootstrap CR takes effect at the next poll of the tool's OPA plugin (10 s mi ## Failure & Rollback -The Orchestrator wraps the `provision → ServicePolicyBuilder.build` pipeline in a compensating rollback. On any of `PolicyConflictError`, `PolicyRulesBuilderError`, `LLMAccessError`, or `UnparseableLLMResponseError`, the Orchestrator does these steps in this order. A failed precondition check (`EnforcementPreconditionError`) is not one of these errors: it causes no rollback and no quarantine (see [Precondition checks](#precondition-checks-d30)). The trigger is the same for agents and tools, and it fires on the first failure (also for the retryable `LLMAccessError`). +The Orchestrator wraps the `provision → ServicePolicyBuilder.build` pipeline in a compensating rollback. On any of `PolicyConflictError`, `PolicyRulesBuilderError`, `LLMAccessError`, `UnparseableLLMResponseError`, or `LLMCredentialsError`, the Orchestrator does these steps in this order. A failed precondition check (`EnforcementPreconditionError`) is not one of these errors: it causes no rollback and no quarantine (see [Precondition checks](#precondition-checks-d30)). The trigger is the same for agents and tools, and it fires on the first failure (also for the retryable `LLMAccessError`). 1. It rolls back what Service Provision created (`_rollback`) and logs the rollback actions (info). The last rollback step disables the client. 2. It calls the PCE `quarantine(client_id, created_roles)` (the clientId resolved in step 0, not the UUID) (see [`../policy-computation-engine.md` → Quarantine (failed onboarding)](../policy-computation-engine.md#quarantine-failed-onboarding)). The quarantine deletes the SPM of the service and removes its roles from the other SPMs (also the created roles, which the rollback deleted from the IdP, so the catalog no longer lists them), but not a role that another service in the catalog also holds (a shared role, [D32](../../PRD.md#key-architectural-decisions)). The edges of a shared role stay, and the PCE renders again the CRs of the SPMs that have them, without the service (a disabled service is not a holder; these SPMs are not written). The quarantine also deletes the CR of the service (`delete_service_cr`, for an agent and for a tool; for a tool, also a bootstrap CR), and deploys the CRs of the affected services of the current side. The service leaves the managed set (D21). The global combiner denies a pod that has no client CR, so the service is then denied (D20). The delete takes effect at the next poll of the OPA plugin of the service (up to 120 s). If `quarantine` raises, `SPM(X)` stays in the store (the PCE changes the store last), so the resync at the next Controller start quarantines the service again. @@ -385,7 +385,7 @@ The role-focal pass of an own role also gets the other owner's copy of a shared - **user roles** — realm roles linked to at least one subject (via `subjects[*].roles`, composite-expanded through `flatten_role`) and not owned by any service (`role.id` not in the union of every service's role ids, disabled services included). These carry `kind=User`. - **other scopes** — `aiac.managed` scopes with `serviceId != focus.serviceId`, from every other **enabled** service (a disabled service is skipped). A shared scope ([D32](../../PRD.md#key-architectural-decisions)) has one copy for each owner, so it is in this list one time for each other owner (the copy of the focus service is not in it). The PRB prompt lists each scope id one time, and the rule assembly gives one rule for each copy, so the SPM of each owner gets the rule. 5. **Flatten candidate roles to their closure** before any PRB call, via the shared `flatten_role` helper (see [Composite role flattening](#composite-role-flattening)): union of other-agent roles + user roles, deduplicated by `role.id` (`candidate_roles`); on the agent path, also expand each of the focus service's own roles. A shared role has one copy for each service that holds it, each with `actorIds = [that service]`. The dedup merges the holders of all the copies (`RoleHolders`, D32), so the candidate role carries every holder, not only the holder of the first copy. The focus service counts as live, so it is one of these holders when it holds the role. The kind of a candidate comes from the catalog ownership: a role id that a service holds (in `get_services()`, also a disabled service or the focus) is `kind=Agent`, also when the flatten gives it as a composite child, and any other role is `kind=User`. Each candidate is the full copy of its role (the service copy, or the `GET /roles` copy that a subject holds directly), so a user role takes its members from that copy, not from a composite child copy. On the agent path, each own role is the focus's copy, and each child in its tree is the full copy of its role, with the same kind rule. So the role-focal pass (`build_role_rules` once for each role in the closure) and the diagnostic survey give the correct kind to a child of a composite own role. -6. Call PRB and merge. Wrap each PRB call; catch `PolicyContradictionError`, accumulate `(focal, contradictions)`, and **continue** the fan-out (accumulate-and-merge). A hard failure — `PolicyRulesBuilderError`, `LLMAccessError`, or `UnparseableLLMResponseError` — aborts the fan-out immediately and propagates, so the Orchestrator can roll back (see [Failure & Rollback](#failure--rollback)). The PRB raise semantics are specified in [`policy-rules-builder.md`](policy-rules-builder.md) (handoff 03). +6. Call PRB and merge. Wrap each PRB call; catch `PolicyContradictionError`, accumulate `(focal, contradictions)`, and **continue** the fan-out (accumulate-and-merge). A hard failure — `PolicyRulesBuilderError`, `LLMAccessError`, `UnparseableLLMResponseError`, or `LLMCredentialsError` — aborts the fan-out immediately and propagates, so the Orchestrator can roll back (see [Failure & Rollback](#failure--rollback)). The PRB raise semantics are specified in [`policy-rules-builder.md`](policy-rules-builder.md) (handoff 03). - **`service_type = tool`:** call `build_scope_rules(candidate_roles, scope)` for each of the focus service's own scopes. Merge results into a single `list[PolicyRule]`. - **`service_type = agent`:** call `build_scope_rules(candidate_roles, scope)` for each own scope; for each of the focus service's own roles, call `build_role_rules(r, other_scopes)` **once per role `r` in that role's closure**. Merge all results into a single `list[PolicyRule]`. 7. Merge contradictions into one report. After the fan-out, run `detect_conflicts` and **union** its deterministic conflicts with the accumulated auditor contradictions into a single `ConflictReport` (a withheld focal appears as a Conflict row). If the report has any conflicts, run a best-effort `enrich_report`, then raise `PolicyConflictError(report)` — which the Controller maps to `422`. If the report has no conflicts, continue. diff --git a/docs/specs/components/event-broker.md b/docs/specs/components/event-broker.md index a41ebeb1..af50503c 100644 --- a/docs/specs/components/event-broker.md +++ b/docs/specs/components/event-broker.md @@ -20,7 +20,7 @@ The Event Broker is a single-node NATS JetStream instance. It owns no business l | Consumer filter subjects | `aiac.apply.service.*`, `aiac.apply.role.*`, `aiac.apply.role-members.*`, `aiac.apply.policy.build` (`CONSUMER_FILTER_SUBJECTS`, `stream.py`). Not the DLQ subject, so the consumer never gets a dead-lettered message again. At each start, the Agent updates an existing durable consumer to this config (see [Consumer config at start](#consumer-config-at-start)) | | Authentication | None — ClusterIP network isolation is the access control mechanism | | Dead-letter subject | `aiac.apply.dlq` | -| Max delivery attempts | 5 — after the 5th failed delivery, the Agent consumer republishes the message to `aiac.apply.dlq` and calls `term()`. A permanent error (`PolicyConflictError`, `PolicyContradictionError`, `PolicyRulesBuilderError`, `UnparseableLLMResponseError`, `EnforcementPreconditionError`) goes to the DLQ on the first delivery. A nak with a delay (see [Delivery Guarantees](#delivery-guarantees)) also uses one of the 5 deliveries | +| Max delivery attempts | 5 — after the 5th failed delivery, the Agent consumer republishes the message to `aiac.apply.dlq` and calls `term()`. A permanent error (`PolicyConflictError`, `PolicyContradictionError`, `PolicyRulesBuilderError`, `UnparseableLLMResponseError`, `LLMCredentialsError`, `EnforcementPreconditionError`) goes to the DLQ on the first delivery. A nak with a delay (see [Delivery Guarantees](#delivery-guarantees)) also uses one of the 5 deliveries | | Ack wait | `600` s (`ACK_WAIT_SECONDS`, sized for long LLM onboardings) | --- diff --git a/docs/testing/uc1-onboarding-pipeline.md b/docs/testing/uc1-onboarding-pipeline.md index 02cbb071..8845ecb4 100644 --- a/docs/testing/uc1-onboarding-pipeline.md +++ b/docs/testing/uc1-onboarding-pipeline.md @@ -485,8 +485,10 @@ changed combiner denies a pod that has no client CR, so a deleted CR means deny. scope of the disabled agent client (`test_rollback_keeps_the_subject_scope`). Provision keeps the link out of the created-manifest, so the rollback does not delete it (D31); - the agent has **no SPM** in the policy store; - - the Controller log shows the injected error (`UnparseableLLMResponseError`) and the move of the event - to the dead-letter subject `aiac.apply.dlq`. + - the Controller log shows the PRB's chat-completions request to the injected URL and its `404`, the + injected error (`UnparseableLLMResponseError`) and the move of the event to the dead-letter subject + `aiac.apply.dlq`. The `404` line proves that the injection caused the failure: other LLM faults + (for example a schema-invalid response) raise the same error type. - **Failed tool:** - the tool's CR, which its phase-3 onboarding wrote, is **deleted** (phase 3 records the CR as present, so "no CR" is a real transition), and the tool has **no SPM**; diff --git a/k8s/aiac-deployment-guide.md b/k8s/aiac-deployment-guide.md index be303849..989268a0 100644 --- a/k8s/aiac-deployment-guide.md +++ b/k8s/aiac-deployment-guide.md @@ -118,6 +118,33 @@ kubectl patch configmap aiac-agent-config -n aiac-system --type merge \ Both are read by the Agent at startup, so a change to either takes effect on the next (re)start: `kubectl rollout restart deployment/aiac-agent -n aiac-system`. +### Check or change the API key + +The Agent reads the key only from the Secret `aiac-agent-secret`. It does not read `.env`. Thus a new +key in `.env` (for example after a rotation) does not reach the Agent until you update the Secret. + +Check that the endpoint accepts the key. HTTP `200` means that it accepts the key: + +```bash +set -a; . .env; set +a +curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $LLM_API_KEY" "$LLM_BASE_URL/models" +``` + +Change the key, then restart the Agent: + +```bash +kubectl create secret generic aiac-agent-secret -n aiac-system \ + --from-literal=LLM_API_KEY="$LLM_API_KEY" --dry-run=client -o yaml | kubectl apply -f - +kubectl rollout restart deployment/aiac-agent -n aiac-system +kubectl rollout status deployment/aiac-agent -n aiac-system +``` + +If the endpoint refuses the key (HTTP `401` or `403`), each onboarding fails. The Agent logs +`LLMCredentialsError` (logger `aiac.onboarding.llm_credentials`), rolls the onboarding back, +quarantines the service and moves the event to `aiac.apply.dlq`. The service has no CR, so the +combiner denies it (D20). After you fix the key, onboard each failed service again (see +[UC1 → Failure & Rollback](../docs/specs/components/aiac-agent/uc1-service-onboarding.md#failure--rollback)). + ## 4 — Configure the environment Edit the `aiac-pdp-config` ConfigMap in `pdp-interface-deployment.yaml` to match your environment: @@ -353,8 +380,8 @@ kubectl rollout status deployment/aiac-agent -n aiac-system side, see `docs/agents/deploy.md`, `ac-side`. - `.env` must have `LLM_BASE_URL` and `LLM_MODEL`. If one is missing, the `:?` check stops the block before the apply. Then stop and ask. Do not leave the placeholder in the ConfigMap. -- The Secret `aiac-agent-secret` (`LLM_API_KEY`) is not changed by this flow. To change it, see - step 3b. +- The Secret `aiac-agent-secret` (`LLM_API_KEY`) is not changed by this flow, also when the key in + `.env` changed. To check or change the key, see step 3b, "Check or change the API key". --- diff --git a/src/aiac/agent/controller/routes.py b/src/aiac/agent/controller/routes.py index d868ff22..cc74b338 100644 --- a/src/aiac/agent/controller/routes.py +++ b/src/aiac/agent/controller/routes.py @@ -36,6 +36,7 @@ ) from aiac.agent.policy_rules_builder.graph import ( LLMAccessError, + LLMCredentialsError, PolicyContradictionError, PolicyRulesBuilderBaseError, PolicyRulesBuilderError, @@ -92,6 +93,13 @@ def _unparseable_llm_response_error(_request: Request, exc: UnparseableLLMRespon return _sanitized(exc, 502, "The policy language model returned an unusable response.") +# The LLM endpoint refused the Controller's credentials (HTTP 401 / 403, e.g. a revoked API key): +# an upstream fault the Controller cannot recover from until the key is fixed, so 502. +@app.exception_handler(LLMCredentialsError) +def _llm_credentials_error(_request: Request, exc: LLMCredentialsError) -> JSONResponse: + return _sanitized(exc, 502, "The policy language model endpoint rejected the Controller's credentials.") + + # The two grant/deny conflict mechanisms both surface as a 422 whose body IS a structured # ``ConflictReport`` (settled design Q15 — one report shape at the boundary): # * ``PolicyConflictError`` (cross-pass structural detector, already enriched with verbatim quotes diff --git a/src/aiac/agent/eventbus/consumer.py b/src/aiac/agent/eventbus/consumer.py index 0c127154..43d3dbb3 100644 --- a/src/aiac/agent/eventbus/consumer.py +++ b/src/aiac/agent/eventbus/consumer.py @@ -7,10 +7,10 @@ (``aiac.apply.role-members.{role-id}``, D32) calls the PCE ``rerender_role`` instead, as the ``POST /apply/role-members/{role_id}`` route does: no use-case handler (no PRB run) and no ``compute_and_apply``. Before it subscribes, the start updates an existing durable consumer to the -config of the code (``ensure_consumer``). On a **permanent** failure (see -``_PERMANENT_ERRORS``: a policy conflict or contradiction, a PRB fault, an unparseable LLM -response, or a failed UC1 precondition check) the message is republished to the DLQ subject and -terminated at the FIRST delivery. On any other failure the message is left unacked (NATS +config of the code (``ensure_consumer``). On a **permanent** failure (see ``_PERMANENT_ERRORS``: a +policy conflict or contradiction, a PRB fault, an unparseable LLM response, LLM credentials the +endpoint refused, or a failed UC1 precondition check) the message is republished to the DLQ subject +and terminated at the FIRST delivery. On any other failure the message is left unacked (NATS redelivers after ``ACK_WAIT_SECONDS``) until ``num_delivered`` reaches ``MAX_DELIVER``, at which point it is republished to the DLQ subject and terminated (stops redelivery on this consumer). One retryable failure is nak'd instead of left unacked: ``ServiceNotVisibleError`` (the IdP does not show @@ -48,6 +48,7 @@ ) from aiac.agent.policy_rules_builder.conflict_detection import PolicyConflictError from aiac.agent.policy_rules_builder.graph import ( + LLMCredentialsError, PolicyContradictionError, PolicyRulesBuilderError, UnparseableLLMResponseError, @@ -76,9 +77,9 @@ _POLICY_BUILD_SUBJECT = "aiac.apply.policy.build" # PERMANENT failures: redelivery can never make them succeed (a real policy conflict / -# contradiction, an unparseable LLM response, a builder fault, or a failed UC1 precondition -# check — that one needs a fix in the cluster first), so they are DLQ'd + term()ed on the -# FIRST delivery. Everything else — LLMAccessError (a transient LLM +# contradiction, an unparseable LLM response, a builder fault, LLM credentials the endpoint refused, +# or a failed UC1 precondition check — those two need a fix in the cluster first), so they are +# DLQ'd + term()ed on the FIRST delivery. Everything else — LLMAccessError (a transient LLM # outage that may clear) and genuinely unknown/transient errors — is RETRYABLE: left # unacked to redeliver until MAX_DELIVER, then DLQ'd (ServiceNotVisibleError is nak'd with a # short delay instead; see _NOT_VISIBLE_NAK_DELAY_ENV below). LLMAccessError is a sibling of the @@ -89,6 +90,7 @@ PolicyContradictionError, PolicyRulesBuilderError, UnparseableLLMResponseError, + LLMCredentialsError, EnforcementPreconditionError, ) diff --git a/src/aiac/agent/llm.py b/src/aiac/agent/llm.py index 18c87e92..ea686f33 100644 --- a/src/aiac/agent/llm.py +++ b/src/aiac/agent/llm.py @@ -25,7 +25,7 @@ from pydantic import SecretStr from tenacity import Retrying, retry_if_exception, stop_after_attempt, wait_exponential -from aiac.shared.upstream import is_transient +from aiac.shared.upstream import http_status, is_transient _DEFAULT_LLM_REQUEST_TIMEOUT = 120.0 _DEFAULT_LLM_MAX_RETRIES = 3 @@ -51,6 +51,16 @@ class UnparseableLLMResponseError(LLMError): tell the two apart.""" +class LLMCredentialsError(LLMError): + """The LLM endpoint refused the request's credentials (HTTP 401: the API key is not valid) or + their permissions (HTTP 403). Not transient, so not retried, and permanent until the key is + fixed. Distinct from ``UnparseableLLMResponseError`` so the log names the real cause.""" + + +# The HTTP statuses that mean the LLM endpoint refused the credentials, not the response. +_CREDENTIALS_STATUSES = frozenset({401, 403}) + + @dataclass(frozen=True) class LLMSettings: """A single LLM consumer's fully-resolved parameter set — client params (endpoint, model, key, @@ -148,13 +158,17 @@ def raise_sanitized( *, access_error: type[LLMError] = LLMAccessError, unparseable_error: type[LLMError] = UnparseableLLMResponseError, + credentials_error: type[LLMError] = LLMCredentialsError, ) -> NoReturn: """Re-raise ``err`` as a typed, sanitized LLM error. A still-transient error means the retry - budget was exhausted against an unreachable LLM -> ``access_error``; anything else means the LLM - was reachable but its response was unusable -> ``unparseable_error``. The messages are STATIC + budget was exhausted against an unreachable LLM -> ``access_error``; an HTTP 401 / 403 means the + LLM refused the credentials -> ``credentials_error``; anything else means the LLM was reachable + but its response was unusable -> ``unparseable_error``. The messages are STATIC (never ``str(err)``) so an endpoint / host / API key embedded in the raw error cannot leak; the raw error stays reachable via ``__cause__``. Callers pass their own subclasses to specialise the error vocabulary (the PRB folds these into ``PolicyRulesBuilderBaseError``).""" if is_transient(err): raise access_error("LLM endpoint unreachable after exhausting transport retries") from err + if http_status(err) in _CREDENTIALS_STATUSES: + raise credentials_error("LLM endpoint rejected the credentials (HTTP 401/403)") from err raise unparseable_error("LLM returned an unparseable or schema-invalid response") from err diff --git a/src/aiac/agent/policy_rules_builder/graph.py b/src/aiac/agent/policy_rules_builder/graph.py index 86cd8be2..d2ff59c8 100644 --- a/src/aiac/agent/policy_rules_builder/graph.py +++ b/src/aiac/agent/policy_rules_builder/graph.py @@ -15,6 +15,7 @@ from pydantic import BaseModel from aiac.agent.llm import LLMAccessError as _LLMAccessError +from aiac.agent.llm import LLMCredentialsError as _LLMCredentialsError from aiac.agent.llm import UnparseableLLMResponseError as _UnparseableLLMResponseError from aiac.agent.llm import build_llm, call_with_retry, load_llm_settings, raise_sanitized from aiac.idp.configuration.models import Role, Scope @@ -86,6 +87,15 @@ class UnparseableLLMResponseError(_UnparseableLLMResponseError, PolicyRulesBuild ``PolicyRulesBuilderBaseError`` hierarchy (see ``LLMAccessError``).""" +class LLMCredentialsError(_LLMCredentialsError, PolicyRulesBuilderBaseError): + """Raised by ``_structured_call`` when the LLM endpoint refuses the credentials (HTTP 401 / 403: + for example a revoked API key). Not retried. Distinct from ``UnparseableLLMResponseError`` so the + log names the real cause. Sanitized the same way; original error chained on ``__cause__``. + + Specialises the shared ``aiac.agent.llm.LLMCredentialsError`` into the PRB's + ``PolicyRulesBuilderBaseError`` hierarchy (see ``LLMAccessError``).""" + + class PolicyContradictionError(PolicyRulesBuilderBaseError): """Raised when the policy GENUINELY both grants and prohibits the same (focal, candidate) pair (a direct conflict or a coarse-scope granularity mismatch). Carries the focal entity and ALL @@ -146,10 +156,16 @@ def _structured_call(schema: type[T], messages: list[BaseMessage]) -> T: return cast(T, call_with_retry(runnable, messages, settings=settings)) except Exception as err: # raise_sanitized classifies the ORIGINAL error exactly as the retry loop did (transient - # exhaustion -> LLMAccessError; reachable-but-unparseable -> UnparseableLLMResponseError), - # with a STATIC endpoint-free message and the raw error chained on __cause__. The PRB - # subclasses are passed so the raised type stays inside PolicyRulesBuilderBaseError. - raise_sanitized(err, access_error=LLMAccessError, unparseable_error=UnparseableLLMResponseError) + # exhaustion -> LLMAccessError; refused credentials, HTTP 401/403 -> LLMCredentialsError; + # reachable-but-unparseable -> UnparseableLLMResponseError), with a STATIC endpoint-free + # message and the raw error chained on __cause__. The PRB subclasses are passed so the + # raised type stays inside PolicyRulesBuilderBaseError. + raise_sanitized( + err, + access_error=LLMAccessError, + unparseable_error=UnparseableLLMResponseError, + credentials_error=LLMCredentialsError, + ) # shared node helpers (typed against _PRBWorking; direction specifics passed as kwargs) diff --git a/src/aiac/agent/shared/error_logging.py b/src/aiac/agent/shared/error_logging.py index e4900e0e..1ebaee89 100644 --- a/src/aiac/agent/shared/error_logging.py +++ b/src/aiac/agent/shared/error_logging.py @@ -20,6 +20,7 @@ from aiac.agent.policy_rules_builder.conflict_detection import PolicyConflictError from aiac.agent.policy_rules_builder.graph import ( LLMAccessError, + LLMCredentialsError, PolicyContradictionError, PolicyRulesBuilderError, UnparseableLLMResponseError, @@ -32,6 +33,7 @@ _LOGGER_BY_TYPE: dict[type[BaseException], str] = { LLMAccessError: "aiac.onboarding.llm_access", UnparseableLLMResponseError: "aiac.onboarding.llm_response", + LLMCredentialsError: "aiac.onboarding.llm_credentials", PolicyContradictionError: "aiac.onboarding.contradiction", PolicyConflictError: "aiac.onboarding.contradiction", PolicyRulesBuilderError: "aiac.onboarding.builder", diff --git a/src/aiac/agent/uc/onboarding/orchestrator.py b/src/aiac/agent/uc/onboarding/orchestrator.py index a8c88e86..a106e92f 100644 --- a/src/aiac/agent/uc/onboarding/orchestrator.py +++ b/src/aiac/agent/uc/onboarding/orchestrator.py @@ -50,6 +50,7 @@ from aiac.agent.policy_rules_builder.conflict_detection import PolicyConflictError from aiac.agent.policy_rules_builder.graph import ( LLMAccessError, + LLMCredentialsError, PolicyRulesBuilderError, UnparseableLLMResponseError, ) @@ -67,9 +68,9 @@ logger = logging.getLogger(__name__) # The build failures that trigger the UC1 compensating rollback. A conflict finding -# (``PolicyConflictError``) and the three hard PRB faults (``PolicyRulesBuilderError``, -# ``LLMAccessError``, ``UnparseableLLMResponseError``) all leave a provisioned-but-unusable -# service, so each rolls back what Provision created. Any other exception (e.g. an +# (``PolicyConflictError``) and the four hard PRB faults (``PolicyRulesBuilderError``, +# ``LLMAccessError``, ``UnparseableLLMResponseError``, ``LLMCredentialsError``) all leave a +# provisioned-but-unusable service, so each rolls back what Provision created. Any other exception (e.g. an # ``HTTPException`` from IdP focus resolution) propagates untouched — no teardown. # ``EnforcementPreconditionError`` is deliberately NOT here (checkpoint O2): the checks run before # Provision, so nothing exists that needs compensation. A first onboarding then has no CR, so D20 @@ -79,6 +80,7 @@ PolicyRulesBuilderError, LLMAccessError, UnparseableLLMResponseError, + LLMCredentialsError, ) diff --git a/src/aiac/shared/upstream.py b/src/aiac/shared/upstream.py index ae7ac8a4..a5f96799 100644 --- a/src/aiac/shared/upstream.py +++ b/src/aiac/shared/upstream.py @@ -35,7 +35,7 @@ def max_retries() -> int: return value if value > 0 else _DEFAULT_MAX_RETRIES -def _status_code(exc: BaseException) -> int | None: +def http_status(exc: BaseException) -> int | None: """Best-effort HTTP status of an exception: ``requests`` HTTPError carries it on ``.response.status_code``; a Kubernetes ``ApiException`` and the IdP library's ``IdPHTTPError`` carry it on ``.status``. Only an ``int`` or a numeric ``str`` is a status: any other value (also @@ -79,7 +79,7 @@ def is_transient(exc: BaseException) -> bool: # permanent, dead-lettering an onboard on a transient LLM blip. if any(base.__name__ in transient_names for base in type(exc).__mro__): return True - status = _status_code(exc) + status = http_status(exc) return status is not None and status >= 500 diff --git a/test/system/test_uc1_onboard_failure_rollback.py b/test/system/test_uc1_onboard_failure_rollback.py index 2c79f437..183d7506 100644 --- a/test/system/test_uc1_onboard_failure_rollback.py +++ b/test/system/test_uc1_onboard_failure_rollback.py @@ -54,6 +54,10 @@ the onboarding after the seam is restored, racing the lift (and the teardown). The 404 is also independent of the real LLM provider (no bad-key / bad-model behaviour to depend on). The rest of the path is the same: both errors are in ``_ROLLBACK_ERRORS``, so the rollback and quarantine are identical. +The error type alone does not prove that the injection caused the failure: every LLM fault that is not +transient and not a refused credential (a schema-invalid response, another 4xx) also raises +``UnparseableLLMResponseError``. So the log check also asks for the PRB's chat-completions request to +``UNUSABLE_LLM_BASE_URL`` and its 404 (``_injected_request_line``). **Re-onboarding trigger — the NATS subject.** ``publish_service_event`` publishes ``aiac.apply.service.`` (the subject + payload the ``aiac-event-listener`` SPI emits on @@ -119,6 +123,13 @@ DLQ_SUBJECT = "aiac.apply.dlq" +def _injected_request_line() -> str: + """The Controller's httpx log line for the PRB call that the injection answers: the chat-completions + request to ``uc1.UNUSABLE_LLM_BASE_URL``, answered 404. Up to the status code, so the reason phrase + does not matter.""" + return f'HTTP Request: POST {uc1.UNUSABLE_LLM_BASE_URL.rstrip("/")}/chat/completions "HTTP/1.1 404' + + def _dlq_line(uuid: str) -> str: """The consumer's dead-letter log line for ``uuid`` (``consumer._dispatch``), up to its cause.""" return f"moved aiac.apply.service.{uuid} to {DLQ_SUBJECT} (permanent failure" @@ -411,8 +422,14 @@ def test_failed_service_has_no_spm(run: dict, key: str) -> None: @pytest.mark.parametrize("key", ["agent_failed", "tool_failed"]) def test_failure_logged_and_dead_lettered(run: dict, key: str) -> None: """The Controller logs the build failure (its type) and moves the event to the DLQ as a - **permanent** failure on the first delivery — the injection's error, never a redelivery loop.""" + **permanent** failure on the first delivery — the injection's error, never a redelivery loop. The + PRB's request to the injected URL and its 404 prove that the injection caused the failure, not + another LLM fault of the same type (module docstring).""" obs = _failed(run, key) + assert _injected_request_line() in obs["logs"], ( + f"{key}: no {_injected_request_line()!r} in the Controller log — the build failed, but not on the " + "injected 404. Did the injection reach the Controller?" + ) assert INJECTED_ERROR in obs["logs"], f"{key}: no {INJECTED_ERROR!r} in the Controller log" assert _dlq_line(obs["uuid"]) in obs["logs"], f"{key}: no dead-letter line {_dlq_line(obs['uuid'])!r}" diff --git a/test/unit/agent/controller/test_routes.py b/test/unit/agent/controller/test_routes.py index 7a61eeb9..46e67a50 100644 --- a/test/unit/agent/controller/test_routes.py +++ b/test/unit/agent/controller/test_routes.py @@ -19,6 +19,7 @@ from aiac.agent.policy_rules_builder.graph import ( Contradiction, LLMAccessError, + LLMCredentialsError, PolicyContradictionError, PolicyRulesBuilderBaseError, PolicyRulesBuilderError, @@ -421,6 +422,19 @@ def test_unparseable_llm_response_error_surfaces_502_with_sanitized_body_and_no_ pce.assert_not_called() +def test_llm_credentials_error_surfaces_502_with_sanitized_body_and_no_leak(): + # The LLM endpoint refused the Controller's API key: an upstream fault (502), sanitized. + err = LLMCredentialsError(_POISON) + with ( + patch("aiac.agent.controller.routes.onboard_and_apply", side_effect=err), + patch("aiac.agent.controller.routes.compute_and_apply") as pce, + ): + resp = client.post("/apply/service/svc-llm") + + _assert_sanitized(resp, 502) + pce.assert_not_called() + + def test_policy_rules_builder_error_body_is_sanitized_to_detail_without_leak(): # PolicyRulesBuilderError stays 422 (policy-input problem) but its body is now a leak-free # summary, not str(exc) — a poisoned message must not escape to the client. diff --git a/test/unit/agent/eventbus/test_consumer.py b/test/unit/agent/eventbus/test_consumer.py index 749d8e66..fa4fa948 100644 --- a/test/unit/agent/eventbus/test_consumer.py +++ b/test/unit/agent/eventbus/test_consumer.py @@ -36,6 +36,7 @@ from aiac.agent.policy_rules_builder.diagnostic_models import ConflictReport, ConflictStatus from aiac.agent.policy_rules_builder.graph import ( LLMAccessError, + LLMCredentialsError, PolicyContradictionError, PolicyRulesBuilderError, UnparseableLLMResponseError, @@ -353,6 +354,8 @@ def test_dispatch_routes_to_dlq_after_max_deliver(): PolicyContradictionError("scope:read", []), PolicyRulesBuilderError("builder blew up"), UnparseableLLMResponseError("cannot parse"), + # A refused LLM key needs a new key in the cluster first: no redelivery helps. + LLMCredentialsError("key refused"), # A failed onboarding precondition check (D30) needs a fix in the cluster first: no retry helps. EnforcementPreconditionError(["#1 sidecar: no authbridge-proxy container"]), ] diff --git a/test/unit/agent/policy_digester/test_digest.py b/test/unit/agent/policy_digester/test_digest.py index ba46837f..f7ee8447 100644 --- a/test/unit/agent/policy_digester/test_digest.py +++ b/test/unit/agent/policy_digester/test_digest.py @@ -10,10 +10,12 @@ from unittest.mock import MagicMock, patch +import httpx +import openai import pytest from langchain_core.messages import HumanMessage, SystemMessage -from aiac.agent.llm import LLMAccessError, UnparseableLLMResponseError +from aiac.agent.llm import LLMAccessError, LLMCredentialsError, UnparseableLLMResponseError from aiac.agent.policy_digester import digest_policy _SRC = "Developers may read the source repository. Testers may not write issues." @@ -85,3 +87,16 @@ def test_digest_call_sanitizes_permanent_as_unparseable_error() -> None: ): with pytest.raises(UnparseableLLMResponseError): digest_policy(_SRC) + + +def test_digest_call_sanitizes_rejected_key_as_credentials_error() -> None: + response = httpx.Response(401, request=httpx.Request("POST", "http://llm.example/v1/chat/completions")) + with ( + patch("aiac.agent.policy_digester.digest.build_llm", return_value=MagicMock()), + patch( + "aiac.agent.policy_digester.digest.call_with_retry", + side_effect=openai.AuthenticationError("bad key", response=response, body=None), + ), + ): + with pytest.raises(LLMCredentialsError): + digest_policy(_SRC) diff --git a/test/unit/agent/policy_rules_builder/test_graph.py b/test/unit/agent/policy_rules_builder/test_graph.py index 0ecb1108..b0f86c3e 100644 --- a/test/unit/agent/policy_rules_builder/test_graph.py +++ b/test/unit/agent/policy_rules_builder/test_graph.py @@ -9,6 +9,8 @@ from contextlib import ExitStack from unittest.mock import MagicMock, patch +import httpx +import openai import pytest from openai import APIConnectionError, APITimeoutError @@ -16,6 +18,7 @@ AuditVerdict, Contradiction, LLMAccessError, + LLMCredentialsError, PolicyContradictionError, PolicyRulesBuilderBaseError, PolicyRulesBuilderError, @@ -352,6 +355,21 @@ def test_parse_error_raises_sanitized_unparseable_response_error(monkeypatch): assert secret not in message # message is sanitized -- no endpoint/host/key leak +def test_rejected_api_key_raises_sanitized_credentials_error_without_retry(monkeypatch): + # The LLM proxy answers 401 (a refused API key): not transient, so not retried, and typed as + # LLMCredentialsError -- not UnparseableLLMResponseError, which hid the cause in the logs. + response = httpx.Response(401, request=httpx.Request("POST", _FAKE_ENDPOINT)) + leaky = openai.AuthenticationError(f"Invalid proxy server token key={_FAKE_KEY}", response=response, body=None) + raised = _run_with_failing_invoke(leaky, monkeypatch) + + assert type(raised) is LLMCredentialsError + assert isinstance(raised, PolicyRulesBuilderBaseError) # shares the PRB base hierarchy + assert raised.__cause__ is leaky + message = str(raised) + for secret in _LEAK_SUBSTRINGS: + assert secret not in message + + # =========================================================================== # # #123 — deny extraction from natural-language policy text. # # The proposer now returns explicit prohibitions (denied_* name lists) and an # diff --git a/test/unit/agent/shared/test_error_logging.py b/test/unit/agent/shared/test_error_logging.py index 7fac5b5f..3d8f3825 100644 --- a/test/unit/agent/shared/test_error_logging.py +++ b/test/unit/agent/shared/test_error_logging.py @@ -13,6 +13,7 @@ from aiac.agent.policy_rules_builder.graph import ( Contradiction, LLMAccessError, + LLMCredentialsError, PolicyContradictionError, PolicyRulesBuilderError, UnparseableLLMResponseError, @@ -63,6 +64,19 @@ def test_unparseable_llm_response_error_routes_to_llm_response_logger(caplog): assert records[0].levelno == logging.ERROR +def test_llm_credentials_error_routes_to_llm_credentials_logger(caplog): + with caplog.at_level(logging.DEBUG): + try: + raise LLMCredentialsError("LLM endpoint rejected the credentials") + except LLMCredentialsError as exc: + log_by_type(exc) + + records = _onboarding_records(caplog) + assert len(records) == 1 + assert records[0].name == "aiac.onboarding.llm_credentials" + assert records[0].levelno == logging.ERROR + + def test_policy_contradiction_error_routes_to_contradiction_logger(caplog): contradictions = [Contradiction(candidate_name="reader", description="grant vs deny")] with caplog.at_level(logging.DEBUG): diff --git a/test/unit/agent/test_llm.py b/test/unit/agent/test_llm.py index d7acc04f..42794901 100644 --- a/test/unit/agent/test_llm.py +++ b/test/unit/agent/test_llm.py @@ -14,10 +14,13 @@ from dataclasses import replace from unittest.mock import MagicMock, patch +import httpx +import openai import pytest from aiac.agent.llm import ( LLMAccessError, + LLMCredentialsError, LLMSettings, UnparseableLLMResponseError, build_llm, @@ -184,3 +187,36 @@ class MyAccess(LLMAccessError): ... with pytest.raises(MyAccess): raise_sanitized(TimeoutError("x"), access_error=MyAccess) + + +def _openai_status_error(cls: type[openai.APIStatusError], status: int) -> openai.APIStatusError: + response = httpx.Response(status, request=httpx.Request("POST", "http://llm.example/v1/chat/completions")) + return cls(_POISON, response=response, body=None) + + +@pytest.mark.parametrize( + ("cls", "status"), + [(openai.AuthenticationError, 401), (openai.PermissionDeniedError, 403)], + ids=["401", "403"], +) +def test_raise_sanitized_rejected_credentials_give_credentials_error(cls, status) -> None: + # The LLM proxy refuses the API key (401) or its permissions (403): not an unparseable response. + cause = _openai_status_error(cls, status) + with pytest.raises(LLMCredentialsError) as ei: + raise_sanitized(cause) + assert not isinstance(ei.value, UnparseableLLMResponseError) + assert _POISON not in str(ei.value) + assert ei.value.__cause__ is cause + + +def test_raise_sanitized_other_4xx_stays_unparseable_error() -> None: + # A 404 (e.g. a wrong base URL path) keeps its old classification. + with pytest.raises(UnparseableLLMResponseError): + raise_sanitized(_openai_status_error(openai.NotFoundError, 404)) + + +def test_raise_sanitized_uses_caller_supplied_credentials_error_type() -> None: + class MyCredentials(LLMCredentialsError): ... + + with pytest.raises(MyCredentials): + raise_sanitized(_openai_status_error(openai.AuthenticationError, 401), credentials_error=MyCredentials) diff --git a/test/unit/agent/uc/onboarding/test_orchestrator.py b/test/unit/agent/uc/onboarding/test_orchestrator.py index 5e4397de..2816f8bd 100644 --- a/test/unit/agent/uc/onboarding/test_orchestrator.py +++ b/test/unit/agent/uc/onboarding/test_orchestrator.py @@ -36,6 +36,7 @@ from aiac.agent.policy_rules_builder.diagnostic_models import ConflictReport from aiac.agent.policy_rules_builder.graph import ( LLMAccessError, + LLMCredentialsError, PolicyRulesBuilderError, UnparseableLLMResponseError, ) @@ -134,9 +135,9 @@ def _config_returning(service): def _rollback_errors(): - """The four typed build failures that trigger the UC1 compensating rollback. + """The five typed build failures that trigger the UC1 compensating rollback. - Instances (not classes): PolicyConflictError needs a ConflictReport, so all four are + Instances (not classes): PolicyConflictError needs a ConflictReport, so all five are built here and passed as `build` side effects. `from_survey([], [], 0)` is the minimal valid report (see test_error_logging.py).""" return [ @@ -144,6 +145,7 @@ def _rollback_errors(): PolicyRulesBuilderError("auditor rejected after retries"), LLMAccessError("LLM endpoint unreachable"), UnparseableLLMResponseError("schema validation failed"), + LLMCredentialsError("LLM endpoint rejected the credentials"), ] From 942730b57acfa93459c5f132d1417048515dcd70 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Thu, 15 Oct 2026 17:41:57 +0300 Subject: [PATCH 08/15] docs(deploy): make the deployment guide the single source, with no demo dependency - The clean redeploy in opa-kind-runbook.md no longer calls the demo's teardown.sh / enable.sh or reads team1/openai-secret. It deletes and installs AIAC with the AIAC manifests, as guide steps 3-5 say, and takes all configuration from .env. It checks the LLM endpoint and key before any change. - The agent rebuild block in aiac-deployment-guide.md now reloads the LLM endpoint, model and key from .env as one set (the key goes into aiac-agent-secret), after it checks that the endpoint accepts the key. Before, a swap of the LLM in .env moved only the endpoint, which gave HTTP 401 on each onboarding. - The OPA and Keycloak SPI platform preconditions move from docs/agents/deploy.md to the guide ("Platform preconditions"). deploy.md and test.md point to that section instead of copying it. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- docs/agents/deploy.md | 53 ++++---------------- docs/agents/test.md | 12 ++--- k8s/aiac-deployment-guide.md | 93 +++++++++++++++++++++++++++++++----- k8s/opa-kind-runbook.md | 70 ++++++++++++++++++--------- 4 files changed, 140 insertions(+), 88 deletions(-) diff --git a/docs/agents/deploy.md b/docs/agents/deploy.md index cfc57a59..b49f5e64 100644 --- a/docs/agents/deploy.md +++ b/docs/agents/deploy.md @@ -1,48 +1,11 @@ -# Deploy parameters and preconditions - -Names the deploy-time parameters and preconditions of this repo for the `deploy` -skill. The deploy flow is in `k8s/aiac-deployment-guide.md`. The commands for each -parameter are in the linked doc — this file copies none of them. - -## Precondition: OPA on the Kind platform - -AIAC needs OPA in the AuthBridge pipeline, `bundle-service`, and the changed -global combiner (D20). Without them the AIAC Controller does not start. - -| | | -|---|---| -| Applies to | Every deploy of an AIAC unit in `aiac-system`, and a parameter-only deploy | -| Check | `./k8s/opa-kind-verify.sh` (read-only, exits non-zero on the first failed check) | -| Install | `OPERATOR_DIR=… ROSSOCTL_DIR=… CORTEX_DIR=… ./k8s/opa-kind-enable.sh`, from the repo root. Details: `k8s/opa-kind-runbook.md` → Enable | -| Install warning | Two image builds (a few minutes). It runs `helm upgrade` on the rossoctl chart and restarts the agent and tool pods in `team1`. | -| Inputs | Sibling clones `operator`, `rossoctl`, `cortex`. Defaults are `../operator`, `../rossoctl`, `../cortex`, relative to the repo root. If one is missing, stop and ask for its path. | -| Re-check | After any `kubectl apply` of a platform manifest or an operator-chart upgrade. The check also fails when the stock combiner is back. | - -Enable runs the verify script as its last step, so after an install the check -has already passed once. - -`k8s/opa-kind-enable.sh` and `k8s/opa-kind-restore.sh` run `helm upgrade` on the -rossoctl chart. This can put the stock Keycloak image back. After either script, -do the Keycloak SPI check below. - -## Precondition: Keycloak event-listener SPI - -AIAC learns of a new Keycloak client from the `aiac-event-listener` SPI, which -publishes to the NATS broker. Without it no registration event reaches the -Controller, and the onboarding path (and the system lane) does not work. - -| | | -|---|---| -| Applies to | Every deploy of an AIAC unit in `aiac-system`, and a parameter-only deploy | -| Check | `./k8s/keycloak-spi-verify.sh` (read-only, exits non-zero on the first failed check) | -| Install | `./k8s/keycloak-spi-install.sh`, from the repo root | -| Install warning | One image build (maven container, then a derived Keycloak image). It restarts the Keycloak StatefulSet once, so Keycloak is down for a short time. On the realm it adds `aiac-event-listener` to the event listeners and turns on admin events. The other event settings stay. | -| Inputs | The OPA precondition above first (the platform must be on Kind). A broker in `aiac-system` is not needed to install, but the listener drops events until `aiac-event-broker` runs. Both scripts read `KEYCLOAK_ADMIN_USERNAME` / `KEYCLOAK_ADMIN_PASSWORD` from the environment, else from `.env`, else from secret `aiac-system/keycloak-admin-secret`, else use `admin`. | -| Re-install | The install records the stock Keycloak image in the StatefulSet annotation `aiac.rossoctl.dev/keycloak-base-image`, so a re-run derives from the same image. If the live image is the derived one and the annotation is missing (an install by an older script), the install stops: set `KEYCLOAK_IMAGE_BASE` to the stock image. | -| Re-check | After any change of the Keycloak StatefulSet. It can put the stock Keycloak image back. | - -Install runs the verify script as its last step, so after an install the check -has already passed once. Do the OPA precondition first. +# Deploy parameters + +Names the deploy-time parameters of this repo for the `deploy` skill. The deploy +flow and its platform preconditions (OPA on the Kind platform, the Keycloak +event-listener SPI) are in `k8s/aiac-deployment-guide.md` → Platform +preconditions. Check them before every deploy, also a parameter-only deploy. The +commands for each parameter are in the linked doc — this file copies none of +them. ## `ac-side` — access-control scheme (enforcement side) diff --git a/docs/agents/test.md b/docs/agents/test.md index c1cbdd74..73a2d488 100644 --- a/docs/agents/test.md +++ b/docs/agents/test.md @@ -94,12 +94,10 @@ assert the allow/deny of the **deployed OPA plugin**. They do not use Preconditions: -- A rossoctl/Kind cluster with the AuthBridge OPA pipeline wired into both legs. -- The AIAC global combiner. It denies a pod that has no `AuthorizationPolicy` - CR, and the Controller does not start without it. -- `k8s/opa-kind-enable.sh` does both steps (run it one time). -- The event path wired: the NATS broker and the `aiac-event-listener` Keycloak - SPI (see `docs/testing/uc1-onboarding-pipeline.md`, "Preconditions"). +- AIAC deployed on a rossoctl/Kind cluster that meets the platform + preconditions in `k8s/aiac-deployment-guide.md` → Platform preconditions: OPA + on the Kind platform (with the AIAC global combiner, which denies a pod that + has no `AuthorizationPolicy` CR) and the Keycloak event-listener SPI. - An LLM key that the endpoint accepts, in the Controller's Secret `aiac-system/aiac-agent-secret`. The in-cluster Controller does not read `.env`. If the endpoint refuses the key, each onboarding rolls back, and the @@ -130,7 +128,7 @@ see the runbook section "Direct Access Grants on the login client". The per-loop shape: `test/system/uc1_onboard.py`. ```bash -k8s/opa-kind-enable.sh # one time: wire the OPA plugin and the AIAC combiner into Kind +k8s/opa-kind-verify.sh && k8s/keycloak-spi-verify.sh # the platform preconditions set -a; . .env; set +a .venv/bin/pytest -m system ``` diff --git a/k8s/aiac-deployment-guide.md b/k8s/aiac-deployment-guide.md index 989268a0..610bd774 100644 --- a/k8s/aiac-deployment-guide.md +++ b/k8s/aiac-deployment-guide.md @@ -17,6 +17,49 @@ This guide covers the full AIAC deployment in the `aiac-system` namespace. - Keycloak reachable from within the cluster (default: `http://keycloak-service.keycloak.svc:8080`). - For local Kind clusters: `kind` CLI and `docker`. +## Platform preconditions + +Two platform preconditions apply to every deploy of an AIAC unit in `aiac-system`, also to a +deploy that changes only a parameter. Check both before you deploy, in this order. If a check +fails, install that precondition, then deploy. + +### OPA on the Kind platform + +AIAC needs OPA in the AuthBridge pipeline, `bundle-service`, and the changed global combiner +(D20). Without them the AIAC Controller does not start. + +| | | +|---|---| +| Check | `./k8s/opa-kind-verify.sh` (read-only, exits non-zero on the first failed check) | +| Install | `OPERATOR_DIR=… ROSSOCTL_DIR=… CORTEX_DIR=… ./k8s/opa-kind-enable.sh`, from the repo root. Details: [`opa-kind-runbook.md` → Enable](opa-kind-runbook.md#enable) | +| Install warning | Two image builds (a few minutes). It runs `helm upgrade` on the rossoctl chart and restarts the agent and tool pods in `team1`. | +| Inputs | Sibling clones `operator`, `rossoctl`, `cortex`. Defaults are `../operator`, `../rossoctl`, `../cortex`, relative to the repo root. If one is missing, stop and ask for its path. | +| Re-check | After any `kubectl apply` of a platform manifest or an operator-chart upgrade. The check also fails when the stock combiner is back. | + +The install runs the verify script as its last step, so after an install the check has already +passed once. + +`k8s/opa-kind-enable.sh` and `k8s/opa-kind-restore.sh` run `helm upgrade` on the rossoctl chart. +This can put the stock Keycloak image back. After either script, do the Keycloak SPI check below. + +### Keycloak event-listener SPI + +AIAC learns of a new Keycloak client from the `aiac-event-listener` SPI, which publishes to the +NATS broker. Without it no registration event reaches the Controller, and the onboarding path +(and the system test lane) does not work. + +| | | +|---|---| +| Check | `./k8s/keycloak-spi-verify.sh` (read-only, exits non-zero on the first failed check) | +| Install | `./k8s/keycloak-spi-install.sh`, from the repo root | +| Install warning | One image build (maven container, then a derived Keycloak image). It restarts the Keycloak StatefulSet once, so Keycloak is down for a short time. On the realm it adds `aiac-event-listener` to the event listeners and turns on admin events. The other event settings stay. | +| Inputs | The OPA precondition above first (the platform must be on Kind). A broker in `aiac-system` is not needed to install, but the listener drops events until `aiac-event-broker` runs. Both scripts read `KEYCLOAK_ADMIN_USERNAME` / `KEYCLOAK_ADMIN_PASSWORD` from the environment, else from `.env`, else from secret `aiac-system/keycloak-admin-secret`, else use `admin`. | +| Re-install | The install records the stock Keycloak image in the StatefulSet annotation `aiac.rossoctl.dev/keycloak-base-image`, so a re-run derives from the same image. If the live image is the derived one and the annotation is missing (an install by an older script), the install stops: set `KEYCLOAK_IMAGE_BASE` to the stock image. | +| Re-check | After any change of the Keycloak StatefulSet. It can put the stock Keycloak image back. | + +The install runs the verify script as its last step, so after an install the check has already +passed once. + ## 1 — Build the images Run from the repo root: @@ -122,6 +165,9 @@ Both are read by the Agent at startup, so a change to either takes effect on the The Agent reads the key only from the Secret `aiac-agent-secret`. It does not read `.env`. Thus a new key in `.env` (for example after a rotation) does not reach the Agent until you update the Secret. +The rebuild flow ("Rebuilding the AIAC Agent: always reload its LLM configuration") updates it +from `.env` together with the endpoint and the model. Use the commands below to change only the +key, without a rebuild. Check that the endpoint accepts the key. HTTP `200` means that it accepts the key: @@ -340,22 +386,33 @@ kind load docker-image localhost/aiac-pdp-config:local --name kubectl rollout restart deployment/aiac-interface -n aiac-system ``` -### Rebuilding the AIAC Agent: always reload its ConfigMap +### Rebuilding the AIAC Agent: always reload its LLM configuration -Every time you rebuild `localhost/aiac-agent:local`, also reload the `aiac-agent-config` -ConfigMap. Do not skip it, also when only code changed. It makes sure the Agent runs with the -LLM endpoint and model of the repo-root `.env`, and not with an older value in the cluster. +Every time you rebuild `localhost/aiac-agent:local`, also reload the LLM configuration: the +endpoint and the model into the `aiac-agent-config` ConfigMap, and the key into the +`aiac-agent-secret` Secret. Do not skip it, also when only code changed. It makes sure the Agent +runs with the LLM endpoint, model **and key** of the repo-root `.env`, as one set. If you swap the +LLM in `.env` (another endpoint with its own key), the next rebuild moves all three together. An +endpoint from `.env` with an older key in the cluster gives HTTP `401` on each onboarding. ```bash docker build -f src/aiac/agent/controller/Dockerfile -t localhost/aiac-agent:local src/ kind load docker-image localhost/aiac-agent:local --name -# Reload the ConfigMap: the apply writes the manifest value of each key (the LLM keys are -# placeholders). The patch puts every live key back and then the real LLM values from .env -# (step 3b). A subshell: a failed check or step stops the rest of the block, not your shell. +# Reload the LLM configuration: the apply writes the manifest value of each ConfigMap key (the LLM +# keys are placeholders). The patch puts every live key back and then the real LLM values from +# .env (step 3b). The Secret gets the key from the same .env. A subshell: a failed check or step +# stops the rest of the block, not your shell. ( set -eo pipefail set -a; . .env; set +a -: "${LLM_BASE_URL:?missing in .env}" "${LLM_MODEL:?missing in .env}" +: "${LLM_BASE_URL:?missing in .env}" "${LLM_MODEL:?missing in .env}" "${LLM_API_KEY:?missing in .env}" +# Check the .env set BEFORE any change: the endpoint must accept the key. If not, the block stops +# and the running Agent keeps its configuration. +RESP=$(curl -s -w '\n%{http_code}' -H "Authorization: Bearer $LLM_API_KEY" "$LLM_BASE_URL/models" || true) +CODE=${RESP##*$'\n'} +[ "$CODE" = 200 ] || { echo "ERROR: $LLM_BASE_URL/models answered HTTP $CODE to the .env key. Fix .env first." >&2; exit 1; } +printf '%s' "${RESP%$'\n'*}" | jq -e --arg m "$LLM_MODEL" 'any(.data[]?; .id == $m)' >/dev/null \ + || echo "WARNING: $LLM_MODEL is not in the model list of $LLM_BASE_URL. Check LLM_MODEL in .env." >&2 # Read ALL live keys BEFORE the apply: the apply resets each key to its manifest value. LIVE=$(kubectl get configmap aiac-agent-config -n aiac-system -o json | jq -c '.data // {}') kubectl apply -f k8s/agent-deployment.yaml @@ -364,8 +421,11 @@ kubectl apply -f k8s/agent-deployment.yaml kubectl patch configmap aiac-agent-config -n aiac-system --type merge -p "$(jq -n \ --argjson live "$LIVE" --arg url "$LLM_BASE_URL" --arg model "$LLM_MODEL" \ '{data: ($live + {LLM_BASE_URL: $url, LLM_MODEL: $model})}')" +# The key, from the same .env. The Secret holds only LLM_API_KEY, so a full replace is safe. +kubectl create secret generic aiac-agent-secret -n aiac-system \ + --from-literal=LLM_API_KEY="$LLM_API_KEY" --dry-run=client -o yaml | kubectl apply -f - -# Restart AFTER the patch. The Agent reads the ConfigMap at start only. +# Restart AFTER the patch and the Secret. The Agent reads both at start only. kubectl rollout restart deployment/aiac-agent -n aiac-system kubectl rollout status deployment/aiac-agent -n aiac-system ) @@ -378,10 +438,17 @@ kubectl rollout status deployment/aiac-agent -n aiac-system `LLM_RETRY_BACKOFF_MAX`) keeps its live value. Only a key that is new in the manifest gets its manifest value. To set a key back to its manifest default, patch it by hand. To change the side, see `docs/agents/deploy.md`, `ac-side`. -- `.env` must have `LLM_BASE_URL` and `LLM_MODEL`. If one is missing, the `:?` check stops the - block before the apply. Then stop and ask. Do not leave the placeholder in the ConfigMap. -- The Secret `aiac-agent-secret` (`LLM_API_KEY`) is not changed by this flow, also when the key in - `.env` changed. To check or change the key, see step 3b, "Check or change the API key". +- `.env` must have `LLM_BASE_URL`, `LLM_MODEL` and `LLM_API_KEY`. If one is missing, the `:?` + check stops the block before the apply. Then stop and ask. Do not leave the placeholder in the + ConfigMap. +- The block checks the `.env` set before it changes anything: `GET $LLM_BASE_URL/models` with the + `.env` key must answer HTTP `200`. Any other answer (`401`: the endpoint does not know the key; + `000`: the endpoint is unreachable) stops the block, and the cluster keeps its configuration. + If the model list does not contain `LLM_MODEL`, the block only warns, because an endpoint can + serve a model that it does not list. +- The AIAC procedures write the key only from `.env`: this block, step 3b and the clean redeploy + in `opa-kind-runbook.md`. A script that writes `aiac-agent-secret` from another source can put + back a key that does not match the `.env` endpoint. Run this block after such a script. --- diff --git a/k8s/opa-kind-runbook.md b/k8s/opa-kind-runbook.md index 81b477f5..478c144b 100644 --- a/k8s/opa-kind-runbook.md +++ b/k8s/opa-kind-runbook.md @@ -326,46 +326,70 @@ no service is managed: onboard each service again. > For example, a pass-through outbound on `github-agent` and a pass-through > inbound on `github-tool` check nothing. -The clean redeploy (Kind; `enable.sh` automates -[`aiac-deployment-guide.md`](aiac-deployment-guide.md)). +The clean redeploy (Kind). It deletes AIAC and installs it again with the AIAC +manifests, as [`aiac-deployment-guide.md`](aiac-deployment-guide.md) steps 3–5 +say. It does not touch Keycloak, so the `aiac-event-listener` SPI stays. -Precondition: the LLM key Secret `team1/openai-secret` (data key `apikey`) -exists. `enable.sh` reads it, and stops if it is missing. The teardown does not -delete it. To make it: -`kubectl create secret generic openai-secret -n team1 --from-literal=apikey="$LLM_API_KEY"` -(see [`demo.md` → Two secrets to create first](../demo/use-cases/onboarding/demo.md#two-secrets-to-create-first)). +Preconditions: + +- Run it from the repo root. The repo-root `.env` has `KEYCLOAK_ADMIN_USERNAME`, + `KEYCLOAK_ADMIN_PASSWORD`, `LLM_BASE_URL`, `LLM_MODEL` and `LLM_API_KEY`. All + AIAC configuration comes from `.env`. The LLM endpoint, model and key come + from it as one set. +- The four AIAC images are in Kind. A namespace delete does not remove them. If + an image changed or is not loaded, do guide steps 1–2 (build and load) first. ```bash +# A subshell: a failed check or step stops the rest of the block, not your shell. +( set -eo pipefail SIDE=agent-side # or target-side +set -a; . .env; set +a +: "${KEYCLOAK_ADMIN_USERNAME:?missing in .env}" "${KEYCLOAK_ADMIN_PASSWORD:?missing in .env}" +: "${LLM_BASE_URL:?missing in .env}" "${LLM_MODEL:?missing in .env}" "${LLM_API_KEY:?missing in .env}" + +# 0. Check the LLM set of .env BEFORE any change: the endpoint must accept the +# key (guide, "Rebuilding the AIAC Agent: always reload its LLM configuration"). +CODE=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $LLM_API_KEY" "$LLM_BASE_URL/models" || true) +[ "$CODE" = 200 ] || { echo "ERROR: $LLM_BASE_URL/models answered HTTP $CODE to the .env key. Fix .env first." >&2; exit 1; } # 1. Delete every AIAC CR. The combiner (D20) then denies each managed pod # until it is onboarded again. kubectl delete authorizationpolicy -A -l app.kubernetes.io/managed-by=aiac-pdp-policy-writer # 2. Delete AIAC: the aiac-system namespace, with the Policy Model Store PVC -# and the keycloak-admin-secret. -demo/use-cases/onboarding/teardown.sh --aiac-only --yes +# and every AIAC Secret. +kubectl delete namespace aiac-system --ignore-not-found --timeout=300s -# 3. Make the namespace and the Keycloak admin Secret again. `enable.sh` does -# not make the Secret, and without it aiac-interface stays in -# ContainerCreating. -set -a; . .env; set +a +# 3. The namespace and the two Secrets (guide steps 3 and 3b). Without +# keycloak-admin-secret, aiac-interface stays in ContainerCreating. kubectl create namespace aiac-system kubectl create secret generic keycloak-admin-secret -n aiac-system \ --from-literal=KEYCLOAK_ADMIN_USERNAME="$KEYCLOAK_ADMIN_USERNAME" \ --from-literal=KEYCLOAK_ADMIN_PASSWORD="$KEYCLOAK_ADMIN_PASSWORD" - -# 4. Deploy AIAC again: the stack, then the broker. The Controller init -# container waits for the broker. -demo/use-cases/onboarding/enable.sh --stack-only -demo/use-cases/onboarding/enable.sh --broker-only - -# 5. Set the side. `k8s/agent-deployment.yaml` sets `target-side`, so set the -# side after step 4. The store is empty, so the Controller start writes no CR. -kubectl patch configmap aiac-agent-config -n aiac-system --type merge \ - -p "{\"data\":{\"AIAC_ENFORCEMENT_SIDE\":\"${SIDE}\"}}" +kubectl create secret generic aiac-agent-secret -n aiac-system \ + --from-literal=LLM_API_KEY="$LLM_API_KEY" + +# 4. Deploy, in the order of guide step 5. The Controller init container +# waits for the broker. +kubectl apply -f k8s/pdp-interface-deployment.yaml +kubectl apply -f k8s/event-broker-deployment.yaml +kubectl apply -f k8s/policy-model-store-statefulset.yaml +kubectl apply -f k8s/agent-deployment.yaml + +# 5. The LLM endpoint and model (guide step 3b) and the side. The manifest has +# LLM placeholders and `target-side`, so patch them after step 4, then +# restart. The store is empty, so the Controller start writes no CR. +kubectl patch configmap aiac-agent-config -n aiac-system --type merge -p "$(jq -n \ + --arg url "$LLM_BASE_URL" --arg model "$LLM_MODEL" --arg side "$SIDE" \ + '{data: {LLM_BASE_URL: $url, LLM_MODEL: $model, AIAC_ENFORCEMENT_SIDE: $side}}')" kubectl rollout restart deployment/aiac-agent -n aiac-system + +# 6. Wait for all of AIAC (guide step 5). +kubectl wait deployment/aiac-interface -n aiac-system --for=condition=Available --timeout=180s +kubectl wait deployment/aiac-event-broker -n aiac-system --for=condition=Available --timeout=180s +kubectl wait statefulset/aiac-policy-model-store -n aiac-system --for=jsonpath='{.status.readyReplicas}'=1 --timeout=180s kubectl rollout status deployment/aiac-agent -n aiac-system --timeout=300s +) ``` At start, the Controller checks the combiner (check #4, D30). Under agent side, From 47c184e0ff90abfeba5ce10b42ab04fee86a9208 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Thu, 15 Oct 2026 18:25:50 +0300 Subject: [PATCH 09/15] fix(test): turn the grant on again when the marker stays but the grant is off When the login client has the harness marker but Direct Access Grants is off (an operator turned it off, a realm re-import, or a partial release), hold_direct_access_grants now writes the grant on. It keeps the marker and records the hold, so the session release turns the grant off again. Before, the hold did no write, the password grant stayed off, and every stack skipped. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- test/system/uc1_onboard.py | 8 ++++++-- .../uc/onboarding/test_uc1_login_client.py | 17 +++++++++++++++++ 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/test/system/uc1_onboard.py b/test/system/uc1_onboard.py index 7e1c3068..61374966 100644 --- a/test/system/uc1_onboard.py +++ b/test/system/uc1_onboard.py @@ -1086,7 +1086,8 @@ def hold_direct_access_grants(admin, realm: str, client_id: str = KEYCLOAK_CLIEN * A grant that is on with no marker (``DAG_MARKER_ATTRIBUTE``) is the realm's own setting: it is not written, and it is not turned off later. * A grant that is off is turned on, with the marker, in one partial update (``_update_client``). - * A marker from an earlier run that did not finish: this session takes the grant over. + * A marker from an earlier run that did not finish: this session takes the grant over. If the grant + is off with the marker on, it is turned on again (the marker stays). In the last two cases ``release_direct_access_grants`` turns the grant off at session end. @@ -1121,7 +1122,10 @@ def hold_direct_access_grants(admin, realm: str, client_id: str = KEYCLOAK_CLIEN client_id, realm, ) - return + if client.get("directAccessGrantsEnabled"): + return + # The marker stays but the grant is off (an operator turned it off, a realm re-import, or a + # release that wrote only part of its update): turn the grant on again below. try: _update_client( admin, diff --git a/test/unit/agent/uc/onboarding/test_uc1_login_client.py b/test/unit/agent/uc/onboarding/test_uc1_login_client.py index 2d2bdf67..aaeabebb 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_login_client.py +++ b/test/unit/agent/uc/onboarding/test_uc1_login_client.py @@ -134,6 +134,23 @@ def test_a_marker_left_by_a_killed_run_is_taken_over_and_released() -> None: assert MARKER not in admin.stored["attributes"] +def test_a_marker_with_the_grant_off_turns_the_grant_on_and_is_released() -> None: + """The case of the review: the marker is on the client but the grant is off (an operator turned it + off, a realm re-import, or a partial release). The session turns the grant on again, keeps the marker, + and turns the grant off at its end.""" + admin = _admin(_login(False, marked=True)) + uc1.hold_direct_access_grants(admin, "rossoctl") + admin.update_client.assert_called_once_with( + LOGIN_UUID, {"directAccessGrantsEnabled": True, "attributes": {MARKER: "true"}} + ) + assert admin.stored["directAccessGrantsEnabled"] is True + assert admin.stored["attributes"][MARKER] == "true" + assert ("rossoctl", uc1.KEYCLOAK_CLIENT_ID) in uc1._dag_holds + uc1.release_direct_access_grants() + assert admin.stored["directAccessGrantsEnabled"] is False + assert MARKER not in admin.stored["attributes"] + + def test_a_missing_login_client_skips_and_is_not_created() -> None: admin = _admin(None) with pytest.raises(pytest.skip.Exception, match=uc1.KEYCLOAK_CLIENT_ID): From 6a82248e3002073f2b9a2df0a8ae4c66fc01a87e Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Thu, 15 Oct 2026 18:25:50 +0300 Subject: [PATCH 10/15] fix(agent): retry an LLM rate limit (429) and request timeout (408) call_with_retry now retries HTTP 408 and 429 as transient, as it does a 5xx. When the retries are exhausted, raise_sanitized gives the access error (as for a 5xx), not UnparseableLLMResponseError. Before, one rate limit sent the onboarding to the DLQ. Also add LLMCredentialsError to the comments and docstrings that list the LLM errors (error_logging, the routes safety net, _digest_call). Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- src/aiac/agent/controller/routes.py | 5 ++-- src/aiac/agent/llm.py | 23 +++++++++++----- src/aiac/agent/policy_digester/digest.py | 4 +-- src/aiac/agent/shared/error_logging.py | 12 ++++---- test/unit/agent/test_llm.py | 35 ++++++++++++++++++++++++ 5 files changed, 63 insertions(+), 16 deletions(-) diff --git a/src/aiac/agent/controller/routes.py b/src/aiac/agent/controller/routes.py index cc74b338..cb6d4560 100644 --- a/src/aiac/agent/controller/routes.py +++ b/src/aiac/agent/controller/routes.py @@ -142,8 +142,9 @@ def _enforcement_precondition_error(_request: Request, exc: EnforcementPrecondit # sanitized the same way (never leaks, full detail logged). Registering the base-class net after # the specific handlers keeps the intent explicit; resolution itself is by specificity, not order — # Starlette walks the exception's MRO and picks the most-specific REGISTERED handler, so a mapped -# subclass (``LLMAccessError``, ``UnparseableLLMResponseError``, ``PolicyRulesBuilderError``, -# ``PolicyContradictionError``) always wins over this base handler and never falls through to 500. +# subclass (``LLMAccessError``, ``UnparseableLLMResponseError``, ``LLMCredentialsError``, +# ``PolicyRulesBuilderError``, ``PolicyContradictionError``) always wins over this base handler +# and never falls through to 500. @app.exception_handler(PolicyRulesBuilderBaseError) def _policy_builder_base_error(_request: Request, exc: PolicyRulesBuilderBaseError) -> JSONResponse: return _sanitized(exc, 500, "The policy build failed unexpectedly.") diff --git a/src/aiac/agent/llm.py b/src/aiac/agent/llm.py index ea686f33..1b25d760 100644 --- a/src/aiac/agent/llm.py +++ b/src/aiac/agent/llm.py @@ -60,6 +60,15 @@ class LLMCredentialsError(LLMError): # The HTTP statuses that mean the LLM endpoint refused the credentials, not the response. _CREDENTIALS_STATUSES = frozenset({401, 403}) +# The 4xx statuses that are transient for an LLM endpoint: 408 (request timeout) and 429 (rate limit). +# The next attempt can succeed, so they are retried like a 5xx. They are not unparseable responses. +_TRANSIENT_4XX_STATUSES = frozenset({408, 429}) + + +def _is_transient(err: BaseException) -> bool: + """``aiac.shared.upstream.is_transient``, plus HTTP 408 and 429 (``_TRANSIENT_4XX_STATUSES``).""" + return is_transient(err) or http_status(err) in _TRANSIENT_4XX_STATUSES + @dataclass(frozen=True) class LLMSettings: @@ -140,12 +149,12 @@ def call_with_retry(runnable: Any, messages: list[BaseMessage], *, settings: LLM what the runnable is (a structured-output binding, or a bare chat model) and what its result means. - Only transient failures (connection errors / timeouts / 5xx) are retried; a permanent failure + Only transient failures (connection errors / timeouts / 5xx / HTTP 408 and 429) are retried; a permanent failure (bad request, validation error) fails identically on every attempt, so it surfaces immediately. ``reraise=True`` hands back the ORIGINAL last exception (never a tenacity ``RetryError``) so the caller can classify it exactly as the retry loop did — see ``raise_sanitized``.""" retryer = Retrying( - retry=retry_if_exception(is_transient), + retry=retry_if_exception(_is_transient), stop=stop_after_attempt(settings.max_retries), wait=wait_exponential(multiplier=1, min=settings.backoff_min, max=settings.backoff_max), reraise=True, @@ -160,14 +169,14 @@ def raise_sanitized( unparseable_error: type[LLMError] = UnparseableLLMResponseError, credentials_error: type[LLMError] = LLMCredentialsError, ) -> NoReturn: - """Re-raise ``err`` as a typed, sanitized LLM error. A still-transient error means the retry - budget was exhausted against an unreachable LLM -> ``access_error``; an HTTP 401 / 403 means the - LLM refused the credentials -> ``credentials_error``; anything else means the LLM was reachable - but its response was unusable -> ``unparseable_error``. The messages are STATIC + """Re-raise ``err`` as a typed, sanitized LLM error. A still-transient error (also an HTTP 408 or + 429) means the retry budget was exhausted against an unreachable LLM -> ``access_error``; an + HTTP 401 / 403 means the LLM refused the credentials -> ``credentials_error``; anything else + means the LLM was reachable but its response was unusable -> ``unparseable_error``. The messages are STATIC (never ``str(err)``) so an endpoint / host / API key embedded in the raw error cannot leak; the raw error stays reachable via ``__cause__``. Callers pass their own subclasses to specialise the error vocabulary (the PRB folds these into ``PolicyRulesBuilderBaseError``).""" - if is_transient(err): + if _is_transient(err): raise access_error("LLM endpoint unreachable after exhausting transport retries") from err if http_status(err) in _CREDENTIALS_STATUSES: raise credentials_error("LLM endpoint rejected the credentials (HTTP 401/403)") from err diff --git a/src/aiac/agent/policy_digester/digest.py b/src/aiac/agent/policy_digester/digest.py index 39dfe794..b96289e2 100644 --- a/src/aiac/agent/policy_digester/digest.py +++ b/src/aiac/agent/policy_digester/digest.py @@ -41,8 +41,8 @@ def _digest_call(messages: list[BaseMessage]) -> str: """THE digester seam. Unit tests patch this. Delegates the client build + transport retry to the shared ``aiac.agent.llm`` seam (on the digester's own ``DIGEST_LLM_*`` settings profile) and returns the model's free-text content, folding any surviving failure into the shared sanitized - LLM errors (``LLMAccessError`` / ``UnparseableLLMResponseError``) so an endpoint / API key can - never leak.""" + LLM errors (``LLMAccessError`` / ``UnparseableLLMResponseError`` / ``LLMCredentialsError``) so + an endpoint / API key can never leak.""" settings = load_llm_settings(_DIGEST_NAMESPACE) try: result = call_with_retry(build_llm(settings), messages, settings=settings) diff --git a/src/aiac/agent/shared/error_logging.py b/src/aiac/agent/shared/error_logging.py index 1ebaee89..9c49bdbb 100644 --- a/src/aiac/agent/shared/error_logging.py +++ b/src/aiac/agent/shared/error_logging.py @@ -8,11 +8,13 @@ are no per-exception files: the container rootfs is read-only and ``/tmp`` is ephemeral, so persistence is the log pipeline's job, not this router's. -The four PRB errors share a base but are siblings (none inherits from another); the -report-carrying ``PolicyConflictError`` is a separate class. Both grant/deny findings -(``PolicyContradictionError`` intra-pass, ``PolicyConflictError`` cross-pass) aggregate on -the same ``aiac.onboarding.contradiction`` logger. The type→logger mapping is the authority -here — it does not recompute anything the callers do. +The five PRB errors (``PolicyRulesBuilderError``, ``LLMAccessError``, +``UnparseableLLMResponseError``, ``LLMCredentialsError``, ``PolicyContradictionError``) share a +base but are siblings (none inherits from another); the report-carrying ``PolicyConflictError`` +is a separate class. Both grant/deny findings (``PolicyContradictionError`` intra-pass, +``PolicyConflictError`` cross-pass) aggregate on the same ``aiac.onboarding.contradiction`` +logger. The type→logger mapping is the authority here — it does not recompute anything the +callers do. """ import logging diff --git a/test/unit/agent/test_llm.py b/test/unit/agent/test_llm.py index 42794901..88f34079 100644 --- a/test/unit/agent/test_llm.py +++ b/test/unit/agent/test_llm.py @@ -209,6 +209,41 @@ def test_raise_sanitized_rejected_credentials_give_credentials_error(cls, status assert ei.value.__cause__ is cause +def _status_error(status: int) -> openai.APIStatusError: + """The openai error for ``status``: ``RateLimitError`` for 429, the generic ``APIStatusError`` else.""" + return _openai_status_error(openai.RateLimitError if status == 429 else openai.APIStatusError, status) + + +@pytest.mark.parametrize("status", [408, 429], ids=["408", "429"]) +def test_call_with_retry_retries_request_timeout_and_rate_limit(monkeypatch: pytest.MonkeyPatch, status: int) -> None: + # A rate limit (429) or a request timeout (408) can clear on the next attempt: it is retried. + monkeypatch.setattr("time.sleep", lambda _s: None) # no real backoff in a unit test + runnable = MagicMock() + runnable.invoke.side_effect = [_status_error(status), "ok"] + assert call_with_retry(runnable, ["msg"], settings=_SETTINGS) == "ok" + assert runnable.invoke.call_count == 2 + + +@pytest.mark.parametrize("status", [408, 429], ids=["408", "429"]) +def test_exhausted_request_timeout_and_rate_limit_give_access_error( + monkeypatch: pytest.MonkeyPatch, status: int +) -> None: + # When the retries are exhausted, the error is an access error (as for a 5xx), not an unparseable one. + monkeypatch.setattr("time.sleep", lambda _s: None) + cause = _status_error(status) + runnable = MagicMock() + runnable.invoke.side_effect = cause + with pytest.raises(openai.APIStatusError) as raw: + call_with_retry(runnable, ["msg"], settings=replace(_SETTINGS, max_retries=3)) + assert raw.value is cause + assert runnable.invoke.call_count == 3 + with pytest.raises(LLMAccessError) as ei: + raise_sanitized(raw.value) + assert not isinstance(ei.value, UnparseableLLMResponseError) + assert _POISON not in str(ei.value) + assert ei.value.__cause__ is cause + + def test_raise_sanitized_other_4xx_stays_unparseable_error() -> None: # A 404 (e.g. a wrong base URL path) keeps its old classification. with pytest.raises(UnparseableLLMResponseError): From d6ca1cc9d71ca80c656110db57151359ada3366a Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Sun, 18 Oct 2026 20:38:14 +0300 Subject: [PATCH 11/15] fix(deploy): set the LLM request timeout to 60 s in the agent manifest The LLM endpoint can accept a request and then send no answer. A retry on a new connection answers in seconds. With a 120 s or 300 s timeout, one such call used all of the 300 s that the UC-1 system fixture waits for the onboarding, and the fixture failed with "agent did not converge". With 60 s, the agent drops the call and the retry still has time to finish. The code default in aiac.agent.llm stays at 120 s. It applies only when the key is not set. The specs and the deployment guide now give both values. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- docs/specs/components/aiac-agent.md | 2 +- docs/specs/components/aiac-agent/policy-rules-builder.md | 7 ++++--- k8s/agent-deployment.yaml | 4 +++- k8s/aiac-deployment-guide.md | 2 +- 4 files changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/specs/components/aiac-agent.md b/docs/specs/components/aiac-agent.md index 558c2566..f5fd6801 100644 --- a/docs/specs/components/aiac-agent.md +++ b/docs/specs/components/aiac-agent.md @@ -261,7 +261,7 @@ The `/apply/*` endpoints return bare HTTP status codes: `200 OK` on success (no | `LLM_MAX_RETRIES` | `3` | ConfigMap | | `LLM_RETRY_BACKOFF_MIN` | `1` | ConfigMap | | `LLM_RETRY_BACKOFF_MAX` | `30` | ConfigMap | -| `LLM_REQUEST_TIMEOUT` | `120` | ConfigMap | +| `LLM_REQUEST_TIMEOUT` | `60` | ConfigMap (the code default, if the key is not set, is `120`) | | `ONBOARD_LABEL_WAIT_ATTEMPTS` | `15` | env (optional; not set in `k8s/`; code default) | | `ONBOARD_LABEL_WAIT_BACKOFF` | `2.0` | env (optional; not set in `k8s/`; code default) | | `ONBOARD_CARD_WAIT_ATTEMPTS` | `15` | env (optional; not set in `k8s/`; code default) | diff --git a/docs/specs/components/aiac-agent/policy-rules-builder.md b/docs/specs/components/aiac-agent/policy-rules-builder.md index 9bb7d52d..a64dee9b 100644 --- a/docs/specs/components/aiac-agent/policy-rules-builder.md +++ b/docs/specs/components/aiac-agent/policy-rules-builder.md @@ -291,8 +291,9 @@ distinct: `aiac.agent.llm.call_with_retry`, which `_structured_call` calls (the Policy Digester uses the same seam), driven by **dedicated LLM knobs** (not the shared `UPSTREAM_MAX_RETRIES`): `LLM_MAX_RETRIES` (default `3`), `LLM_RETRY_BACKOFF_MIN` (`1`) and `LLM_RETRY_BACKOFF_MAX` (`30`). `is_transient` - classifies which failures retry (5xx / connect / timeout). `LLM_REQUEST_TIMEOUT` (default `120`) - bounds each request. The Phase-1 file read does **not** retry; it raises directly. + classifies which failures retry (5xx / connect / timeout). `LLM_REQUEST_TIMEOUT` (code default + `120`, `60` in the `aiac-agent-config` manifest) bounds each request. The Phase-1 file read does + **not** retry; it raises directly. `_structured_call` has three raise outcomes: @@ -616,7 +617,7 @@ and is **not yet enforced** here. |---|---|---| | `AIAC_POLICY_FILE` | Path to the whole-file access policy (default `/etc/aiac/policy.md`) | 1 | | `LLM_BASE_URL`, `LLM_MODEL`, `LLM_API_KEY` | LLM calls | 1 | -| `LLM_REQUEST_TIMEOUT` | Per-request LLM timeout, in seconds (default `120`) | 1 | +| `LLM_REQUEST_TIMEOUT` | Per-request LLM timeout, in seconds (code default `120`, `60` in the manifest) | 1 | | `LLM_MAX_RETRIES` | Transient-failure retry budget for the LLM seam (tenacity `stop_after_attempt`, default `3`) | 1 | | `LLM_RETRY_BACKOFF_MIN` | Minimum exponential backoff for the LLM seam, in seconds (default `1`) | 1 | | `LLM_RETRY_BACKOFF_MAX` | Maximum exponential backoff for the LLM seam, in seconds (default `30`) | 1 | diff --git a/k8s/agent-deployment.yaml b/k8s/agent-deployment.yaml index 3dc61cc4..3d623680 100644 --- a/k8s/agent-deployment.yaml +++ b/k8s/agent-deployment.yaml @@ -36,7 +36,9 @@ data: # of wedging /apply forever. Timeouts are transient, so _structured_call retries them # up to LLM_MAX_RETRIES (default 3, below) with exponential backoff before surfacing. # (The LLM seam uses LLM_MAX_RETRIES, not UPSTREAM_MAX_RETRIES.) - LLM_REQUEST_TIMEOUT: "120" + # 60, not the code default 120: the endpoint can hold a request open with no answer, and a + # retry on a new connection answers at once. A short timeout leaves time for that retry. + LLM_REQUEST_TIMEOUT: "60" UPSTREAM_MAX_RETRIES: "3" # Dedicated LLM retry cadence for the Policy Rules Builder seam # (graph._llm_retry_config). DELIBERATELY SEPARATE from UPSTREAM_MAX_RETRIES diff --git a/k8s/aiac-deployment-guide.md b/k8s/aiac-deployment-guide.md index 610bd774..acb3158e 100644 --- a/k8s/aiac-deployment-guide.md +++ b/k8s/aiac-deployment-guide.md @@ -432,7 +432,7 @@ kubectl rollout status deployment/aiac-agent -n aiac-system ``` - The apply writes the manifest value of each key, for example `AIAC_ENFORCEMENT_SIDE` - (`target-side`), `LOG_LEVEL` (`INFO`) and `LLM_REQUEST_TIMEOUT` (`120`). The block reads all + (`target-side`), `LOG_LEVEL` (`INFO`) and `LLM_REQUEST_TIMEOUT` (`60`). The block reads all live keys before the apply and patches them back before the restart. Thus an `agent-side` deployment stays `agent-side`, and a tuned key (for example `LLM_MAX_RETRIES` or `LLM_RETRY_BACKOFF_MAX`) keeps its live value. Only a key that is new in the manifest gets its From 97dbbf8da4b7fbf5ccb9c15a9274d6631a3269d2 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Mon, 19 Oct 2026 00:26:27 +0300 Subject: [PATCH 12/15] refactor(eval): move the agent-delegation scenario from test/system to eval/scenarios Only the eval lane uses scenario_eval_agent_delegation.py and its policy file. No system test reads them. The old "one exception" location made the batch digest converter skip the policy file. - Move both files to eval/scenarios/ with the other seven scenarios. - Import the scenario from eval.scenarios in test_policy_pipeline_eval.py. - Remove the "exception" text from docstrings, the PRD, and the eval docs. Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Oleg Blinder --- .../analysis/user-subject-across-token-exchange.md | 2 +- docs/evaluation/policy-eval-correctness-prb.md | 3 +-- .../policy-eval-robustness-consistency.md | 6 +++--- docs/evaluation/policy-eval-scenarios.md | 14 ++++---------- docs/specs/PRD.md | 2 +- .../scenarios}/policy.eval_agent_delegation.md | 0 .../scenarios}/scenario_eval_agent_delegation.py | 6 +----- eval/scenarios/scenario_eval_baseline.py | 2 +- .../scenario_eval_agent_delegation_perturbed.py | 5 ----- ...io_eval_agent_delegation_sensitive_perturbed.py | 3 --- eval/test_policy_pipeline_eval.py | 7 ++----- 11 files changed, 14 insertions(+), 36 deletions(-) rename {test/system => eval/scenarios}/policy.eval_agent_delegation.md (100%) rename {test/system => eval/scenarios}/scenario_eval_agent_delegation.py (94%) diff --git a/docs/analysis/user-subject-across-token-exchange.md b/docs/analysis/user-subject-across-token-exchange.md index 7b64b08d..624b3b48 100644 --- a/docs/analysis/user-subject-across-token-exchange.md +++ b/docs/analysis/user-subject-across-token-exchange.md @@ -190,7 +190,7 @@ So: - **Agent side:** one hop works, which is why the runbook's agent-side example (B.5) shows `dev-user`. It fails at the second agent of a chain, in both that agent's inbound and its outbound. AIAC models such a chain: the delegation scenario `dispatch-agent` → `customs-agent` - (`test/system/scenario_eval_agent_delegation.py`). + (`eval/scenarios/scenario_eval_agent_delegation.py`). The agent-side rows come from the code paths. They are not run live (§9). diff --git a/docs/evaluation/policy-eval-correctness-prb.md b/docs/evaluation/policy-eval-correctness-prb.md index 3f7301c4..f9c681c3 100644 --- a/docs/evaluation/policy-eval-correctness-prb.md +++ b/docs/evaluation/policy-eval-correctness-prb.md @@ -191,8 +191,7 @@ covered exactly as `policy-eval-scenarios.md` already documents them ## Policy-text rewrite -As part of this ticket, all 8 scenarios' policy `.md` files (`eval/scenarios/policy.eval_*.md`, -`test/system/policy.eval_agent_delegation.md`) and their 8 `eval/scenarios_perturbed/` +As part of this ticket, all 8 scenarios' policy `.md` files (`eval/scenarios/policy.eval_*.md`) and their 8 `eval/scenarios_perturbed/` semantic siblings were rewritten to read like real human-authored access-control policy rather than the pipeline's own internal vocabulary reflected back into the document a policy author would supposedly have written. The previous text opened with the literal internal-modeling phrase diff --git a/docs/evaluation/policy-eval-robustness-consistency.md b/docs/evaluation/policy-eval-robustness-consistency.md index 3be8ec51..a2f5c358 100644 --- a/docs/evaluation/policy-eval-robustness-consistency.md +++ b/docs/evaluation/policy-eval-robustness-consistency.md @@ -46,8 +46,8 @@ scenarios, and reuse that family's scenario corpus rather than defining their ow six siblings, plus their `policy.eval_*.md` files) were moved from `eval/` directly into a new `eval/scenarios/` subpackage, so the growing `eval/` directory doesn't flatten test modules, scenario-data modules, and (now) two more scenario-data variants (`scenarios_perturbed/`) into one -namespace. `scenario_eval_agent_delegation.py`/`policy.eval_agent_delegation.md` are unaffected — -they already lived at `test/system/` top level and stay there. `test_policy_pipeline_eval.py`'s +namespace. `scenario_eval_agent_delegation.py`/`policy.eval_agent_delegation.md` were first left +at `test/system/` top level; they later moved to `eval/scenarios/` too. `test_policy_pipeline_eval.py`'s imports were updated accordingly; no test logic changed. ## Description @@ -426,7 +426,7 @@ chart over time, not just show up as a one-off failed run with no lasting record robustness tests *and* the consistency suite identically for this scenario, both invariance (unedited) and sensitivity (edited) — proving it was a corpus gap, not an edit-mechanism bug. Fixed in two places: the committed digested `.md` (both original and perturbed sibling) now - states the read grant explicitly, and `agent-role-dispatcher`'s own description (`test/system/ + states the read grant explicitly, and `agent-role-dispatcher`'s own description (`eval/scenarios/ scenario_eval_agent_delegation.py` and its perturbed counterpart) now says "reading, creating, and updating" instead of just "creating and updating" — the same cascade-needs-its-own-textual- grounding principle this file already documents for `unreachable_resources`/`empty_descriptions` diff --git a/docs/evaluation/policy-eval-scenarios.md b/docs/evaluation/policy-eval-scenarios.md index 9f59dd0d..e542853c 100644 --- a/docs/evaluation/policy-eval-scenarios.md +++ b/docs/evaluation/policy-eval-scenarios.md @@ -15,21 +15,16 @@ Two independent groups of files, split by cost tier: -**Heavy scenarios (1, 3, 4, 6-10) — full pipeline, `eval` marker — under `eval/`, -except `agent_delegation`:** +**Heavy scenarios (1, 3, 4, 6-10) — full pipeline, `eval` marker — under `eval/`:** - `eval/test_policy_pipeline_eval.py` — the test module, `@pytest.mark.eval`. - `eval/scenarios/scenario_eval_baseline.py` (Scenario 1), + `scenario_eval_agent_delegation.py` (Scenario 3), `scenario_eval_unreachable_resources.py` (Scenario 4), `scenario_eval_ambiguous_clause.py` (Scenario 6), `scenario_eval_wildcard_grant.py` (Scenario 7), `scenario_eval_misleading_descriptions.py` (Scenario 8), `scenario_eval_confusable_agents.py` (Scenario 9), `scenario_eval_empty_descriptions.py` (Scenario 10) — pure-data scenario modules (mirroring `scenario.py`'s shape, generalized to lists/dicts of many entities), each isolating exactly one aspect at the minimal entity count that aspect needs. -- **`test/system/scenario_eval_agent_delegation.py`** (Scenario 3) — the one exception: - lives at the **top level** of `test/system/` (sibling of `launcher.py`/`scenario_uc1.py`), - not under `eval/` like the other seven. It isolates the agent-to-agent delegation - (`delegation_scopes`) mechanism, which is conceptually closer to the top-level fixed-scenario family than to - the `eval/` catalog's silent-gap/ambiguity/adversarial-authoring aspects. - A matching `policy.eval_.md` next to each scenario module above — the scenario's *source* policy text. Not read directly at runtime: the harness's `pipeline` fixture points `AIAC_POLICY_FILE` at that scenario's **committed digested** counterpart in @@ -324,8 +319,7 @@ See each scenario module's own module docstring (`scenario_eval_baseline.py`, `scenario_eval_empty_descriptions.py`) for the full entity list and role→access facts — these are the single source of truth (`INBOUND_PAIRS`/`OUTBOUND_SUBJECT_PAIRS`/`OUTBOUND_PAIRS`, plus `EXPECT_NO_REGO`/`IDENTITY_CONFUSION_PROBES` where applicable), not a second hand-maintained copy in -this document. Note that `scenario_eval_agent_delegation.py` lives at the top level of -`test/system/`, not under `eval/` like the other seven (see [Location](#location)). Scenarios +this document. Scenarios 2 and 5's cast is defined inline in their test modules' `_POLICY`/`_USER_ROLES`/`_ROLES` constants — see [Location](#location) for why the standalone `.md` mirrors are not what the tests actually read. @@ -340,7 +334,7 @@ scenarios (`KEYCLOAK_URL`, `KEYCLOAK_ADMIN_USERNAME`/`PASSWORD`, `AIAC_PDP_CONFI | Variable | Difference from `policy-pipeline.md` | |----------|----------------------------------------| | `KEYCLOAK_REALM` | Set per scenario module (`scenario.REALM_DEFAULT`), not a single fixed realm — eight distinct realms across the session. | -| `AIAC_POLICY_FILE` | Set per scenario to its **digested** policy in `eval/scenarios_digested/` (heavy scenarios only) — `digested_policy_path(scenario)`, same filename as `scenario.POLICY_FILE` regardless of which directory that scenario's source lives in, so `scenario_eval_agent_delegation.py` living one level up from the rest (see [Location](#location)) needs no special-casing here. | +| `AIAC_POLICY_FILE` | Set per scenario to its **digested** policy in `eval/scenarios_digested/` (heavy scenarios only) — `digested_policy_path(scenario)`, same filename as `scenario.POLICY_FILE` regardless of which directory that scenario's source lives in. | The light scenarios (2, 5) need only `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY` — no Keycloak, store, or OPA URLs, no `opa` binary. diff --git a/docs/specs/PRD.md b/docs/specs/PRD.md index eba7b399..5d927dbd 100644 --- a/docs/specs/PRD.md +++ b/docs/specs/PRD.md @@ -896,7 +896,7 @@ Beyond the marker-gated pytest tests above, individual tests are specified **one | `policy-pipeline` — `test_policy_pipeline.py` | End-to-end system test of the full identity→policy→enforcement pipeline — onboards `github-agent` + `github-tool` through the in-cluster UC-1 Controller (upserting the `AuthorizationPolicy` CRs) and asserts the deployed OPA plugin's allow/deny on real requests through AuthBridge. No `.rego` dump. `@pytest.mark.system`. | [testing/policy-pipeline.md](../testing/policy-pipeline.md) | | `uc1-onboarding-pipeline` — a **ladder** of UC-1 onboarding tests | Discovery-driven sibling of `policy-pipeline` validating the **phase-1** deliverable against **one** in-cluster AIAC stack (CR-backed writer, single abstract `policy.md`): `github-agent` + a simplified `github-tool` are deployed one at a time (each deploy registers a Keycloak client and fires the trigger); three gradual rungs drive **real event-driven UC-1 onboarding** (deploy → Keycloak SPI → NATS) — agent-only, agent→tool, tool→agent — and assert the deployed OPA plugin's allow/deny on real requests through AuthBridge (verdicts from `scenario_uc1.py`). Rungs 2/3 assert onboarding-**order-independence**. A fifth rung covers a failed onboarding (rollback + quarantine). A fourth two-policy rung is **deferred** (two-stack topology discarded). Same scenario facts/tables as `policy-pipeline`; Rego semantically similar (not byte-identical). `@pytest.mark.system`. | [testing/uc1-onboarding-pipeline.md](../testing/uc1-onboarding-pipeline.md) | | `shared-roles-integration` — `test_shared_roles.py` | The first test of the offline `integration` lane (D32): shared roles and shared scopes from the onboarding to the rendered CR, across namespaces and in one namespace, and the current role holders (agents and users) after a role-members event or a resync. The real resolver, Service Policy Builder, PRB graphs, PCE and writer app run between fakes for Keycloak, the Policy Model Store, the Kubernetes API and the LLM. The same harness runs the other files of the lane: `test_quarantine_lift.py`, `test_decommission_shared.py` and `test_snapshot_repair.py`. `@pytest.mark.integration`. | [testing/shared-roles-integration.md](../testing/shared-roles-integration.md) | -| `policy-eval-scenarios` — `test_policy_pipeline_eval.py` + guardrail tests | Generalized evaluation suite extending `policy-pipeline`'s single-agent/single-tool proof to ten scenarios: baseline-scale (many entities, names decoupled from roles, one agent→agent delegation grant), missing-details (emergent unreachability/zero-access under deny-by-default, a broad-sounding clause narrowed by an explicit qualifier, wildcard-grant expansion), adversarial-authoring (misleading names/descriptions, an identity/boundary-confusion probe, empty descriptions), and ambiguous-and-contradictory / adversarial-injection-and-edge-cases (whole-document `xfail` checks against the PRB directly, no Keycloak or `opa`). The eight heavy scenarios (scenario modules under `eval/scenarios/` except `agent_delegation`) assert full per-cell `opa eval` truth tables; the two light scenarios assert PRB-level rejection. The eight heavy scenarios carry `@pytest.mark.eval` (the five former `eval_*` markers collapsed into one flat `eval`); the two light guardrail tests live under `test/unit/agent/policy_rules_builder/` and carry `@pytest.mark.llm`. | [evaluation/policy-eval-scenarios.md](../evaluation/policy-eval-scenarios.md) | +| `policy-eval-scenarios` — `test_policy_pipeline_eval.py` + guardrail tests | Generalized evaluation suite extending `policy-pipeline`'s single-agent/single-tool proof to ten scenarios: baseline-scale (many entities, names decoupled from roles, one agent→agent delegation grant), missing-details (emergent unreachability/zero-access under deny-by-default, a broad-sounding clause narrowed by an explicit qualifier, wildcard-grant expansion), adversarial-authoring (misleading names/descriptions, an identity/boundary-confusion probe, empty descriptions), and ambiguous-and-contradictory / adversarial-injection-and-edge-cases (whole-document `xfail` checks against the PRB directly, no Keycloak or `opa`). The eight heavy scenarios (scenario modules under `eval/scenarios/`) assert full per-cell `opa eval` truth tables; the two light scenarios assert PRB-level rejection. The eight heavy scenarios carry `@pytest.mark.eval` (the five former `eval_*` markers collapsed into one flat `eval`); the two light guardrail tests live under `test/unit/agent/policy_rules_builder/` and carry `@pytest.mark.llm`. | [evaluation/policy-eval-scenarios.md](../evaluation/policy-eval-scenarios.md) | | `policy-eval-robustness-consistency` — `test_policy_pipeline_consistency.py` + `test_policy_pipeline_robustness.py` | Companion to `policy-eval-scenarios`, reusing its 8-scenario corpus to check the PRB's raw grant decisions (no OPA/PCE/k8s) for **consistency** (`@pytest.mark.eval`: N repeated runs on the same input, exact grant-set equality) and **robustness** (`@pytest.mark.eval`: two never-blended families, each its own metric — **invariance** under mechanical text/order perturbation and a hand-reworded semantic-sibling corpus under `eval/scenarios_perturbed/`, and **sensitivity** under a deterministic, meaning-changing mechanical edit per scenario (`SENSITIVITY_EDITS`) — all checked against the truth-table oracle). Mechanical-tier invariance/sensitivity feed the committed trend log; semantic-tier sensitivity is future work (#2467). No Keycloak/`opa` needed — only `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`. | [evaluation/policy-eval-robustness-consistency.md](../evaluation/policy-eval-robustness-consistency.md) | | `policy-eval-correctness-prb` — `test_policy_pipeline_correctness_prb.py` | Companion to `policy-eval-scenarios`/`policy-eval-robustness-consistency`, reusing the same 8-scenario corpus to score the PRB's raw grant/deny output (no OPA/PCE/k8s) against each scenario's truth table via a reusable, effect-aware scorer (`eval/correctness_scorer.py`): precision and recall tracked separately per gate and aggregated, plus a non-gating denial-precision figure for explicit `Deny` rules. `@pytest.mark.eval`, zero-tolerance over-grant gate; under-grants/incorrect denials reported only. No Keycloak/`opa` needed — only `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`. | [evaluation/policy-eval-correctness-prb.md](../evaluation/policy-eval-correctness-prb.md) | | `policy-eval-correctness-e2e` — `test_policy_pipeline_correctness_e2e.py` | Companion to `policy-eval-correctness-prb`, scoring the same 8-scenario corpus and the same reusable scorer one layer further downstream: real Keycloak provisioning → real Policy Rules Builder → real Policy Computation Engine → real `opa eval` against the rendered Rego, sourced from the rendered data maps (`subject_role_allow/deny_scopes`, `agent_role_scopes`) rather than per-pair decision probing. `@pytest.mark.eval`, same zero-tolerance over-grant gate. The shared `pipeline` fixture (`eval/test_policy_pipeline_eval.py`, also used by the other eval suites) now provisions all 8 scenarios concurrently via `ProcessPoolExecutor`. Needs `KEYCLOAK_URL` + admin creds + `LLM_BASE_URL`/`LLM_MODEL`/`LLM_API_KEY`, plus `opa` on `PATH`. | [evaluation/policy-eval-correctness-e2e.md](../evaluation/policy-eval-correctness-e2e.md) | diff --git a/test/system/policy.eval_agent_delegation.md b/eval/scenarios/policy.eval_agent_delegation.md similarity index 100% rename from test/system/policy.eval_agent_delegation.md rename to eval/scenarios/policy.eval_agent_delegation.md diff --git a/test/system/scenario_eval_agent_delegation.py b/eval/scenarios/scenario_eval_agent_delegation.py similarity index 94% rename from test/system/scenario_eval_agent_delegation.py rename to eval/scenarios/scenario_eval_agent_delegation.py index 4b902e5d..222dfcbd 100644 --- a/test/system/scenario_eval_agent_delegation.py +++ b/eval/scenarios/scenario_eval_agent_delegation.py @@ -2,11 +2,7 @@ Companion to ``scenario_eval_baseline.py`` (``eval/``) for ``test_policy_pipeline_eval.py`` (spec: ``docs/evaluation/policy-eval-scenarios.md``). -Unlike every other scenario in the family, this module lives directly under ``test/system/`` -(a sibling of ``scenario.py``/``scenario_uc1.py``), not under ``eval/`` — the one -deliberate exception in this suite's file layout, called out in the spec's *Location* section. It -is still imported and driven by the shared ``eval/test_policy_pipeline_eval.py`` harness; only its -file location differs, not its test wiring. +It is imported and driven by the shared ``eval/test_policy_pipeline_eval.py`` harness. This scenario isolates the suite's **one** aspect: the agent-to-agent ``target_scopes`` delegation mechanism (``src/aiac/policy/computation/engine.py``) — an Agent-typed service standing in as an diff --git a/eval/scenarios/scenario_eval_baseline.py b/eval/scenarios/scenario_eval_baseline.py index bc760b60..09104ad8 100644 --- a/eval/scenarios/scenario_eval_baseline.py +++ b/eval/scenarios/scenario_eval_baseline.py @@ -14,7 +14,7 @@ Unlike ``scenario_eval_baseline.py``'s previous revision, this scenario carries **no** agent-to-agent delegation grant — that mechanism now has its own dedicated scenario, -``scenario_eval_agent_delegation.py`` (``test/system/``). +``scenario_eval_agent_delegation.py``. Pure data: no imports beyond ``__future__``, mirroring ``scenario.py``. """ diff --git a/eval/scenarios_perturbed/scenario_eval_agent_delegation_perturbed.py b/eval/scenarios_perturbed/scenario_eval_agent_delegation_perturbed.py index 6d0a9b93..200a9300 100644 --- a/eval/scenarios_perturbed/scenario_eval_agent_delegation_perturbed.py +++ b/eval/scenarios_perturbed/scenario_eval_agent_delegation_perturbed.py @@ -1,11 +1,6 @@ """Semantic-perturbation sibling of ``scenario_eval_agent_delegation.py`` (spec: ``docs/evaluation/policy-eval-robustness-consistency.md``). -Note the asymmetry: the original lives at ``test/system/`` top level (a deliberate exception -in the base suite's file layout), but this perturbed sibling lives here in -``eval/scenarios_perturbed/`` alongside every other scenario's perturbed sibling — the robustness -suite treats all 8 scenarios uniformly regardless of where their originals happen to live. - Same structure (names, ``USERS``, all pair-lists are byte-identical to the original), reworded prose only. See ``scenario_eval_baseline_perturbed.py`` for the general rationale. diff --git a/eval/scenarios_perturbed/scenario_eval_agent_delegation_sensitive_perturbed.py b/eval/scenarios_perturbed/scenario_eval_agent_delegation_sensitive_perturbed.py index 041bc362..3b11efe6 100644 --- a/eval/scenarios_perturbed/scenario_eval_agent_delegation_sensitive_perturbed.py +++ b/eval/scenarios_perturbed/scenario_eval_agent_delegation_sensitive_perturbed.py @@ -9,9 +9,6 @@ *user* role gets the capability, not what the agent role itself covers). The truth delta is reused directly from ``SENSITIVITY_EDITS["agent_delegation"].removed``/``.added``. -Like its `_perturbed` sibling, lives under ``eval/scenarios_perturbed/`` even though the original -lives at ``test/system/`` top level, for the same uniformity reason. - Requires human sign-off before entering the corpus (spec §4) — see ``eval/scenarios_perturbed/ SIGNOFF.md``. diff --git a/eval/test_policy_pipeline_eval.py b/eval/test_policy_pipeline_eval.py index e8011497..78376046 100644 --- a/eval/test_policy_pipeline_eval.py +++ b/eval/test_policy_pipeline_eval.py @@ -35,10 +35,7 @@ generalized ``probe_eval.rego`` (parameterized by ``input.agent_id``) rather than the fixed ``probe.rego``. ``launcher.py`` is reused unmodified. -``scenario_eval_agent_delegation``'s data file is the one exception to the "everything lives in -``eval/``" rule — it sits at ``test/system/scenario_eval_agent_delegation.py`` (sibling of -``launcher.py``/``scenario_uc1.py``). Regardless of where a scenario's own source ``POLICY_FILE`` -lives, every scenario's ``AIAC_POLICY_FILE`` is pointed at its **committed digested** counterpart +Every scenario's ``AIAC_POLICY_FILE`` is pointed at its **committed digested** counterpart in ``eval/scenarios_digested/`` (``digested_policy_path``, same filename) — production feeds the PRB only digested policy (see ``docs/specs/digested-policy.md``), so this suite runs against the same input shape. @@ -125,6 +122,7 @@ sys.path.insert(0, str(SRC)) # so ``import aiac.*`` resolves from eval.scenarios import scenario_eval_ambiguous_clause as scn_ambiguous_clause # noqa: E402 +from eval.scenarios import scenario_eval_agent_delegation as scn_agent_delegation # noqa: E402 from eval.scenarios import scenario_eval_baseline as scn_baseline # noqa: E402 from eval.scenarios import scenario_eval_confusable_agents as scn_confusable_agents # noqa: E402 from eval.scenarios import scenario_eval_empty_descriptions as scn_empty_descriptions # noqa: E402 @@ -135,7 +133,6 @@ scenario_eval_unreachable_resources as scn_unreachable_resources, ) from eval.scenarios import scenario_eval_wildcard_grant as scn_wildcard_grant # noqa: E402 -from test.system import scenario_eval_agent_delegation as scn_agent_delegation # noqa: E402 from test.system.launcher import ( # noqa: E402 Service, require_env_or_skip, From 8a0071a76682d92125f2ac6d613da8c2068ca6a9 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Mon, 19 Oct 2026 01:43:52 +0300 Subject: [PATCH 13/15] style(eval): sort the scenario imports in the policy pipeline eval Commit 97dbbf8 added the agent-delegation import after the ambiguous-clause one. Ruff I001 (the CI lint job) needs agent before ambiguous. Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Oleg Blinder --- eval/test_policy_pipeline_eval.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/eval/test_policy_pipeline_eval.py b/eval/test_policy_pipeline_eval.py index 78376046..aa2f90ac 100644 --- a/eval/test_policy_pipeline_eval.py +++ b/eval/test_policy_pipeline_eval.py @@ -121,8 +121,8 @@ sys.path.insert(0, str(REPO_ROOT)) # so ``import test.system.*``/``eval.*`` resolves sys.path.insert(0, str(SRC)) # so ``import aiac.*`` resolves -from eval.scenarios import scenario_eval_ambiguous_clause as scn_ambiguous_clause # noqa: E402 from eval.scenarios import scenario_eval_agent_delegation as scn_agent_delegation # noqa: E402 +from eval.scenarios import scenario_eval_ambiguous_clause as scn_ambiguous_clause # noqa: E402 from eval.scenarios import scenario_eval_baseline as scn_baseline # noqa: E402 from eval.scenarios import scenario_eval_confusable_agents as scn_confusable_agents # noqa: E402 from eval.scenarios import scenario_eval_empty_descriptions as scn_empty_descriptions # noqa: E402 From 5f4ce92b70c420aabe0ae7ced3331c18b4409cc6 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Mon, 19 Oct 2026 02:34:36 +0300 Subject: [PATCH 14/15] fix(agent): retry 408/429 in the shared upstream rule, with Retry-After Move the transient 408/429 rule from the LLM seam into the shared aiac.shared.upstream.is_transient, so run_upstream also retries a rate limit of the Kubernetes API (API Priority and Fairness), the IdP and MCP. A 429 that names an exhausted quota (OpenAI insufficient_quota and similar billing codes) stays permanent, as before the 429 rule. The shared retry loop (upstream_retrying, used by run_upstream and by call_with_retry) waits the Retry-After of the error (seconds or an HTTP date) in place of the exponential backoff, capped at the backoff maximum. The LLM-only _is_transient wrapper is removed. raise_sanitized adds the HTTP status number to the access and unparseable messages ("(HTTP 404)"), so AIAC's own ERROR line names the cause; the endpoint and the key still cannot leak. Update the stale comments: builder.py lists LLMCredentialsError, and the ACK_WAIT sizing in stream.py uses LLM_MAX_RETRIES, the 60 s manifest timeout and the capped backoff / Retry-After wait (600 s is still above the 240 s worst case of one call). Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- docs/specs/components/aiac-agent.md | 2 +- .../aiac-agent/policy-rules-builder.md | 11 +- docs/specs/components/library-idp.md | 3 +- src/aiac/agent/eventbus/stream.py | 15 +- src/aiac/agent/llm.py | 53 +++---- src/aiac/agent/policy_rules_builder/graph.py | 7 +- .../uc/onboarding/policy_builder/builder.py | 10 +- src/aiac/idp/configuration/api.py | 26 +-- src/aiac/shared/upstream.py | 114 +++++++++++-- .../agent/policy_rules_builder/test_graph.py | 18 +-- test/unit/agent/test_llm.py | 65 ++++++++ test/unit/shared/test_upstream.py | 150 +++++++++++++++++- 12 files changed, 389 insertions(+), 85 deletions(-) diff --git a/docs/specs/components/aiac-agent.md b/docs/specs/components/aiac-agent.md index f5fd6801..c6931b5d 100644 --- a/docs/specs/components/aiac-agent.md +++ b/docs/specs/components/aiac-agent.md @@ -287,7 +287,7 @@ ChromaDB collections: `aiac-policies` and `aiac-domain-knowledge`. **Status: not The Agent keeps two retry layers distinct. -**Transport retries.** The Agent makes each upstream transport call up to `UPSTREAM_MAX_RETRIES` attempts in total (default `3`: the first attempt and two retries), with exponential backoff (`tenacity`; 1 s, 2 s, …, at most 30 s), before the error propagates. It retries only a transient failure: a connection error, a timeout or a `5xx`. A `4xx` is raised at once (see [`library-idp.md` → Transport retries](library-idp.md)). The retry primitive is the project-level shared `run_upstream(fn)` helper (`aiac/shared/upstream.py`). It is transport-agnostic: it re-raises the original exception after the final attempt. The Agent applies retry at the **transport boundary**, not at the agent call sites — inside the idp-library `Configuration` (its `_request` helper), inside the provision MCP helper (`_mcp_tools_list`), and inside the provision Kubernetes seam (`uc/onboarding/provision/kube.py`). Each caller then maps the re-raised failure to the upstream status below. +**Transport retries.** The Agent makes each upstream transport call up to `UPSTREAM_MAX_RETRIES` attempts in total (default `3`: the first attempt and two retries), with exponential backoff (`tenacity`; 1 s, 2 s, …, at most 30 s; a `Retry-After` header on the error replaces the backoff, up to the same 30 s), before the error propagates. It retries only a transient failure: a connection error, a timeout, a `5xx`, a `408` or a `429` (also the Kubernetes API Priority and Fairness `429`). A `429` that names an exhausted quota (for example `insufficient_quota`) and every other `4xx` are raised at once (see [`library-idp.md` → Transport retries](library-idp.md)). The retry primitive is the project-level shared `run_upstream(fn)` helper (`aiac/shared/upstream.py`). It is transport-agnostic: it re-raises the original exception after the final attempt. The Agent applies retry at the **transport boundary**, not at the agent call sites — inside the idp-library `Configuration` (its `_request` helper), inside the provision MCP helper (`_mcp_tools_list`), and inside the provision Kubernetes seam (`uc/onboarding/provision/kube.py`). Each caller then maps the re-raised failure to the upstream status below. **LLM-seam retries.** The Policy Rules Builder (PRB) retries its own LLM seam with dedicated knobs — `LLM_MAX_RETRIES`, `LLM_RETRY_BACKOFF_MIN`, and `LLM_RETRY_BACKOFF_MAX` (specified in [`aiac-agent/policy-rules-builder.md`](aiac-agent/policy-rules-builder.md)). `UPSTREAM_MAX_RETRIES` does **not** govern LLM calls. It stays for the IdP, MCP, and Kubernetes transport seams only. diff --git a/docs/specs/components/aiac-agent/policy-rules-builder.md b/docs/specs/components/aiac-agent/policy-rules-builder.md index a64dee9b..5df87b93 100644 --- a/docs/specs/components/aiac-agent/policy-rules-builder.md +++ b/docs/specs/components/aiac-agent/policy-rules-builder.md @@ -69,14 +69,16 @@ without touching the rest of the graph. ## Exceptions The PRB raises a small, closed hierarchy. Every message is **sanitized**: it never contains the -LLM endpoint, host, or API key. The HTTP column is the status a **single-entity caller** returns. +LLM endpoint, host, or API key. The `LLMAccessError` and `UnparseableLLMResponseError` messages end +with the HTTP status of the raw error when it has one (for example `(HTTP 404)`), so the ERROR log +line names the cause. The HTTP column is the status a **single-entity caller** returns. The async **retry class** is a separate axis — see the decoupling note below the table. | Exception | Where raised | HTTP (single-entity) | Async class | Notes | |---|---|---|---|---| | `PolicyRulesBuilderBaseError` | Base of the hierarchy — **never raised directly** | 500 | — | A safety net at the HTTP boundary | | `PolicyRulesBuilderError` | `_audit`, after `MAX_AUDIT_RETRIES` (3) ordinary rejections | 422 | permanent | The audit budget is exhausted | -| `LLMAccessError` | `_structured_call`, when **transient** failures (5xx / connect / timeout) are **exhausted** | 502 | **retryable** | Wraps the original error; message sanitized | +| `LLMAccessError` | `_structured_call`, when **transient** failures (5xx / connect / timeout / HTTP 408 / a rate-limit 429) are **exhausted** | 502 | **retryable** | Wraps the original error; message sanitized | | `UnparseableLLMResponseError` | `_structured_call`, when the LLM is **reachable but the response is unparseable / schema-invalid** | 502 | permanent | A non-transient parse or validation error; message sanitized | | `LLMCredentialsError` | `_structured_call`, when the LLM endpoint **refuses the credentials** (HTTP 401 / 403, for example a revoked API key) | 502 | permanent | Not retried; the key must be fixed first. Message sanitized | | `PolicyContradictionError` | `_audit`, on a **genuine** intra-focal grant/deny contradiction | 422 | permanent | Carries the focal entity + all contradictions; short-circuits past retry. UC1 aggregates it — see **Contradiction contract** | @@ -291,7 +293,10 @@ distinct: `aiac.agent.llm.call_with_retry`, which `_structured_call` calls (the Policy Digester uses the same seam), driven by **dedicated LLM knobs** (not the shared `UPSTREAM_MAX_RETRIES`): `LLM_MAX_RETRIES` (default `3`), `LLM_RETRY_BACKOFF_MIN` (`1`) and `LLM_RETRY_BACKOFF_MAX` (`30`). `is_transient` - classifies which failures retry (5xx / connect / timeout). `LLM_REQUEST_TIMEOUT` (code default + classifies which failures retry (5xx / connect / timeout / HTTP 408 / 429). A 429 that names an + exhausted quota (for example OpenAI `insufficient_quota`) is permanent and is not retried. A + `Retry-After` header on the error replaces the backoff, up to `LLM_RETRY_BACKOFF_MAX`. The loop is + the shared `aiac.shared.upstream.upstream_retrying`. `LLM_REQUEST_TIMEOUT` (code default `120`, `60` in the `aiac-agent-config` manifest) bounds each request. The Phase-1 file read does **not** retry; it raises directly. diff --git a/docs/specs/components/library-idp.md b/docs/specs/components/library-idp.md index efc52f82..1cde64af 100644 --- a/docs/specs/components/library-idp.md +++ b/docs/specs/components/library-idp.md @@ -181,7 +181,8 @@ status, message, response and notes. A transport failure (connection error, time |---|---| | Connection error, timeout | Yes | | `IdPHTTPError` with a `5xx` status (for example `502`, `503`) | Yes | -| `IdPHTTPError` with a `4xx` status (for example `400`, `404`, `409`) | No: raised at the first attempt | +| `IdPHTTPError` with a `408` or `429` status | Yes (a `Retry-After` header replaces the backoff, up to 30 s) | +| `IdPHTTPError` with another `4xx` status (for example `400`, `404`, `409`) | No: raised at the first attempt | The status is the status of the IdP Configuration Service, not of Keycloak. The service keeps a Keycloak `404` only on the reads that name one entity, and a Keycloak `409` only on some writes. It diff --git a/src/aiac/agent/eventbus/stream.py b/src/aiac/agent/eventbus/stream.py index a2cade60..d92c9b11 100644 --- a/src/aiac/agent/eventbus/stream.py +++ b/src/aiac/agent/eventbus/stream.py @@ -36,10 +36,17 @@ DLQ_SUBJECT = "aiac.apply.dlq" MAX_DELIVER = 5 -# Onboarding's LLM calls run synchronously inside the dispatch callback, each bounded by -# LLM_REQUEST_TIMEOUT (default 120s) and retried up to UPSTREAM_MAX_RETRIES times (default 3), -# across multiple structured calls (propose/audit). JetStream's own 30s default ack_wait would -# redeliver a message that's still being processed, so this is sized well above the worst case. +# Onboarding's LLM calls run synchronously inside the dispatch callback, across multiple structured +# calls (propose/audit). Each call is bounded by LLM_REQUEST_TIMEOUT (60 s in the aiac-agent-config +# manifest, 120 s code default) and makes up to LLM_MAX_RETRIES attempts (default 3; the LLM seam does +# not use UPSTREAM_MAX_RETRIES). Between attempts it waits the exponential backoff or the error's +# Retry-After, each wait at most LLM_RETRY_BACKOFF_MAX (default 30 s). Recheck with these numbers: the +# worst case of ONE call is 3 x 60 + 2 x 30 = 240 s with the manifest (3 x 120 + 2 x 30 = 420 s at the +# code default), so 600 s is still above it, and a Retry-After cannot make it longer (it is capped at +# the same 30 s). A normal onboarding, where each call answers in seconds, stays far below 600 s. It is +# not sized for an onboarding where several calls each use their full retry budget: NATS can then +# redeliver while the first delivery still runs. JetStream's own 30 s default ack_wait would redeliver a +# message that's still being processed even in the normal case. ACK_WAIT_SECONDS = 600.0 # JetStream's "stream name already in use with a different configuration" error code. diff --git a/src/aiac/agent/llm.py b/src/aiac/agent/llm.py index 1b25d760..5f87b210 100644 --- a/src/aiac/agent/llm.py +++ b/src/aiac/agent/llm.py @@ -23,9 +23,8 @@ from langchain_core.messages import BaseMessage from langchain_openai import ChatOpenAI from pydantic import SecretStr -from tenacity import Retrying, retry_if_exception, stop_after_attempt, wait_exponential -from aiac.shared.upstream import http_status, is_transient +from aiac.shared.upstream import http_status, is_transient, upstream_retrying _DEFAULT_LLM_REQUEST_TIMEOUT = 120.0 _DEFAULT_LLM_MAX_RETRIES = 3 @@ -60,15 +59,6 @@ class LLMCredentialsError(LLMError): # The HTTP statuses that mean the LLM endpoint refused the credentials, not the response. _CREDENTIALS_STATUSES = frozenset({401, 403}) -# The 4xx statuses that are transient for an LLM endpoint: 408 (request timeout) and 429 (rate limit). -# The next attempt can succeed, so they are retried like a 5xx. They are not unparseable responses. -_TRANSIENT_4XX_STATUSES = frozenset({408, 429}) - - -def _is_transient(err: BaseException) -> bool: - """``aiac.shared.upstream.is_transient``, plus HTTP 408 and 429 (``_TRANSIENT_4XX_STATUSES``).""" - return is_transient(err) or http_status(err) in _TRANSIENT_4XX_STATUSES - @dataclass(frozen=True) class LLMSettings: @@ -149,15 +139,15 @@ def call_with_retry(runnable: Any, messages: list[BaseMessage], *, settings: LLM what the runnable is (a structured-output binding, or a bare chat model) and what its result means. - Only transient failures (connection errors / timeouts / 5xx / HTTP 408 and 429) are retried; a permanent failure - (bad request, validation error) fails identically on every attempt, so it surfaces immediately. + The retry loop is the shared ``aiac.shared.upstream.upstream_retrying`` on the LLM knobs. Only + transient failures (``is_transient``: connection errors / timeouts / 5xx / HTTP 408 and 429) are + retried; a 429 for an exhausted quota (``insufficient_quota``) and other permanent failures (bad + request, validation error) fail identically on every attempt, so they surface immediately. A + ``Retry-After`` on the error replaces the exponential backoff, up to ``settings.backoff_max``. ``reraise=True`` hands back the ORIGINAL last exception (never a tenacity ``RetryError``) so the caller can classify it exactly as the retry loop did — see ``raise_sanitized``.""" - retryer = Retrying( - retry=retry_if_exception(_is_transient), - stop=stop_after_attempt(settings.max_retries), - wait=wait_exponential(multiplier=1, min=settings.backoff_min, max=settings.backoff_max), - reraise=True, + retryer = upstream_retrying( + attempts=settings.max_retries, backoff_min=settings.backoff_min, backoff_max=settings.backoff_max ) return retryer(runnable.invoke, messages) @@ -169,15 +159,20 @@ def raise_sanitized( unparseable_error: type[LLMError] = UnparseableLLMResponseError, credentials_error: type[LLMError] = LLMCredentialsError, ) -> NoReturn: - """Re-raise ``err`` as a typed, sanitized LLM error. A still-transient error (also an HTTP 408 or - 429) means the retry budget was exhausted against an unreachable LLM -> ``access_error``; an - HTTP 401 / 403 means the LLM refused the credentials -> ``credentials_error``; anything else - means the LLM was reachable but its response was unusable -> ``unparseable_error``. The messages are STATIC - (never ``str(err)``) so an endpoint / host / API key embedded in the raw error cannot leak; the - raw error stays reachable via ``__cause__``. Callers pass their own subclasses to specialise the - error vocabulary (the PRB folds these into ``PolicyRulesBuilderBaseError``).""" - if _is_transient(err): - raise access_error("LLM endpoint unreachable after exhausting transport retries") from err - if http_status(err) in _CREDENTIALS_STATUSES: + """Re-raise ``err`` as a typed, sanitized LLM error. A still-transient error (``is_transient``, + also an HTTP 408 or a rate-limit 429) means the retry budget was exhausted against an unreachable + LLM -> ``access_error``; an HTTP 401 / 403 means the LLM refused the credentials -> + ``credentials_error``; anything else (also a 429 for an exhausted quota, which is permanent) means + the LLM was reachable but its response was unusable -> ``unparseable_error``. The messages are + STATIC (never ``str(err)``) so an endpoint / host / API key embedded in the raw error cannot leak; + the raw error stays reachable via ``__cause__``. The access and unparseable messages end with the + HTTP status of the error (``" (HTTP 404)"``) when it has one: a number only, so the ERROR log line + (``log_by_type``) names the cause and still leaks nothing. Callers pass their own subclasses to + specialise the error vocabulary (the PRB folds these into ``PolicyRulesBuilderBaseError``).""" + status = http_status(err) + suffix = f" (HTTP {status})" if status is not None else "" + if is_transient(err): + raise access_error(f"LLM endpoint unreachable after exhausting transport retries{suffix}") from err + if status in _CREDENTIALS_STATUSES: raise credentials_error("LLM endpoint rejected the credentials (HTTP 401/403)") from err - raise unparseable_error("LLM returned an unparseable or schema-invalid response") from err + raise unparseable_error(f"LLM returned an unparseable or schema-invalid response{suffix}") from err diff --git a/src/aiac/agent/policy_rules_builder/graph.py b/src/aiac/agent/policy_rules_builder/graph.py index d2ff59c8..8558d5af 100644 --- a/src/aiac/agent/policy_rules_builder/graph.py +++ b/src/aiac/agent/policy_rules_builder/graph.py @@ -147,9 +147,10 @@ def _structured_call(schema: type[T], messages: list[BaseMessage]) -> T: surviving failure into the PRB's own typed, sanitized error subclasses (so consumers that catch ``PolicyRulesBuilderBaseError`` are unaffected). - Only transient failures (connection errors / timeouts / 5xx) are retried — a permanent - failure (e.g. a bad request or a validation error) fails identically on every attempt, so - it is surfaced immediately (consistent with ``aiac.shared.upstream``).""" + Only transient failures (``aiac.shared.upstream.is_transient``: connection errors / timeouts / + 5xx / HTTP 408 and 429) are retried — a permanent failure (e.g. a bad request, a validation + error, or a 429 for an exhausted quota) fails identically on every attempt, so it is surfaced + immediately.""" settings = load_llm_settings() # PRB profile: the bare LLM_* vars runnable = _build_llm().with_structured_output(schema) try: diff --git a/src/aiac/agent/uc/onboarding/policy_builder/builder.py b/src/aiac/agent/uc/onboarding/policy_builder/builder.py index 8d94a872..6e70574d 100644 --- a/src/aiac/agent/uc/onboarding/policy_builder/builder.py +++ b/src/aiac/agent/uc/onboarding/policy_builder/builder.py @@ -80,11 +80,11 @@ def build(service_id: str, service_type: ServiceType) -> list[PolicyRule]: # rule set is withheld) is accumulated as its ``(focal, contradictions)`` and the fan-out # CONTINUES, so every focal's contradiction is collected in one build rather than the report # stopping at whichever pair the loop happened to reach first. A HARD failure - # (PolicyRulesBuilderError / LLMAccessError / UnparseableLLMResponseError -- sibling - # PolicyRulesBuilderBaseError subclasses that are NOT PolicyContradictionError) is deliberately - # NOT caught here: it aborts the build immediately and propagates so the Orchestrator rolls - # back. We catch the narrow PolicyContradictionError ONLY -- never the base -- so a hard fault - # can never be silently folded into the report. + # (PolicyRulesBuilderError / LLMAccessError / UnparseableLLMResponseError / + # LLMCredentialsError -- sibling PolicyRulesBuilderBaseError subclasses that are NOT + # PolicyContradictionError) is deliberately NOT caught here: it aborts the build immediately + # and propagates so the Orchestrator rolls back. We catch the narrow PolicyContradictionError + # ONLY -- never the base -- so a hard fault can never be silently folded into the report. contradictions: list[PolicyContradictionError] = [] def _guarded(fn, *args) -> None: diff --git a/src/aiac/idp/configuration/api.py b/src/aiac/idp/configuration/api.py index b5f248ef..17df3f6f 100644 --- a/src/aiac/idp/configuration/api.py +++ b/src/aiac/idp/configuration/api.py @@ -16,11 +16,12 @@ class IdPHTTPError(RuntimeError): """A non-OK response from the IdP Configuration Service. It keeps the HTTP ``status`` and the - ``response``, so ``aiac.shared.upstream.is_transient`` retries a ``5xx`` and not a ``4xx``. It is - a ``RuntimeError`` subclass, so the callers that catch ``RuntimeError`` do not change, and the - message stays ``"HTTP {status}: {text}"``. A caller that must tell one status from another (for - example a ``404`` for a client that Keycloak has not committed yet) reads ``.status``, not the - message. A pickled or copied error keeps its status, text, response and notes.""" + ``response``, so ``aiac.shared.upstream.is_transient`` retries a ``5xx`` (and a ``408`` / ``429``) + and not another ``4xx``. It is a ``RuntimeError`` subclass, so the callers that catch + ``RuntimeError`` do not change, and the message stays ``"HTTP {status}: {text}"``. A caller that + must tell one status from another (for example a ``404`` for a client that Keycloak has not + committed yet) reads ``.status``, not the message. A pickled or copied error keeps its status, + text, response and notes.""" def __init__(self, status: int, text: str, response=None) -> None: super().__init__(f"HTTP {status}: {text}") @@ -153,13 +154,14 @@ def _request(self, method: str, path: str, **kwargs): so callers and tests keep a stable, mockable surface. Retries transient failures via ``run_upstream``: a connection error, a timeout or a ``5xx`` is tried again up to ``UPSTREAM_MAX_RETRIES`` attempts (default 3) with exponential backoff (1 s, 2 s, …); a - ``4xx`` is raised at once. A non-OK response raises :class:`IdPHTTPError` (``_check``), so - callers just consume ``resp.json()``. Retrying at this leaf boundary means composite methods - (``create_service_role`` / ``create_service_scope``) retry each sub-request without - compounding. Writes are retried too: a repeated write makes no duplicate, but a repeated - ``create_role`` / ``create_scope`` whose first attempt was committed answers ``409`` (see - ``library-idp.md``); ``create_service_role`` / ``create_service_scope`` take that ``409`` as - reuse (``_get_or_create``). + ``408`` / ``429`` is tried again too; another ``4xx`` is raised at once. A non-OK response + raises :class:`IdPHTTPError` (``_check``), so callers just consume ``resp.json()``. Retrying + at this leaf boundary means composite methods (``create_service_role`` / + ``create_service_scope``) retry each sub-request without compounding. Writes are retried + too: a repeated write makes no duplicate, but a repeated ``create_role`` / ``create_scope`` + whose first attempt was committed answers ``409`` (see ``library-idp.md``); + ``create_service_role`` / ``create_service_scope`` take that ``409`` as reuse + (``_get_or_create``). """ caller = getattr(requests, method.lower()) diff --git a/src/aiac/shared/upstream.py b/src/aiac/shared/upstream.py index a5f96799..ce3870ad 100644 --- a/src/aiac/shared/upstream.py +++ b/src/aiac/shared/upstream.py @@ -1,8 +1,9 @@ """Shared upstream-call helper (project-level). `run_upstream` wraps a zero-arg callable in a bounded retry loop for transient upstream -failures (IdP Configuration Service, Kubernetes API, MCP endpoints). It is deliberately -transport-agnostic: it retries then **re-raises the original exception**, leaving each +failures (IdP Configuration Service, Kubernetes API, MCP endpoints). ``upstream_retrying`` builds +the same loop with another budget and backoff (the LLM seam, ``aiac.agent.llm``, uses it). It is +deliberately transport-agnostic: it retries then **re-raises the original exception**, leaving each caller to map the failure to the right HTTP status at the service boundary (e.g. ``HTTPException(502)`` for IdP/Kubernetes — see the Error Handling table in ``aiac-agent.md``). The retry budget is read from ``UPSTREAM_MAX_RETRIES`` (default 3) at @@ -16,13 +17,28 @@ """ import os +from datetime import datetime, timezone +from email.utils import parsedate_to_datetime from typing import Callable, TypeVar -from tenacity import Retrying, retry_if_exception, stop_after_attempt, wait_exponential +from tenacity import RetryCallState, Retrying, retry_if_exception, stop_after_attempt, wait_exponential T = TypeVar("T") _DEFAULT_MAX_RETRIES = 3 +_DEFAULT_BACKOFF_MIN = 1.0 +_DEFAULT_BACKOFF_MAX = 30.0 + +# The 4xx statuses that can clear on the next attempt: 408 (request timeout) and 429 (rate limit, or +# Kubernetes API Priority and Fairness). They are retried like a 5xx. +_TRANSIENT_4XX_STATUSES = frozenset({408, 429}) + +# The error codes of a 429 that cannot clear by a wait: the account has no quota or no billing left +# (OpenAI ``insufficient_quota``, and the same idea on other OpenAI-compatible endpoints). Such a 429 +# is permanent, so it is not retried. Compared in lower case. +_QUOTA_ERROR_CODES = frozenset( + {"insufficient_quota", "quota_exceeded", "billing_hard_limit_reached", "billing_not_active"} +) def max_retries() -> int: @@ -52,11 +68,59 @@ def http_status(exc: BaseException) -> int | None: return None +def _error_codes(exc: BaseException) -> set[str]: + """The error codes that ``exc`` carries, in lower case: the ``code`` and ``type`` attributes (the + openai errors set them from the response body) and the ``code`` / ``type`` of a dict ``body``, also + one level down in ``body["error"]``. Only strings count.""" + found = [getattr(exc, "code", None), getattr(exc, "type", None)] + body = getattr(exc, "body", None) + for part in (body, body.get("error") if isinstance(body, dict) else None): + if isinstance(part, dict): + found += [part.get("code"), part.get("type")] + return {value.lower() for value in found if isinstance(value, str)} + + +def _quota_exhausted(exc: BaseException) -> bool: + """True when ``exc`` names a quota or billing limit that a wait cannot clear (``_QUOTA_ERROR_CODES``).""" + return not _error_codes(exc).isdisjoint(_QUOTA_ERROR_CODES) + + +def retry_after(exc: BaseException | None) -> float | None: + """The wait in seconds that the ``Retry-After`` header of the error's response asks for, or + ``None``. The header is read from ``.response.headers`` (``requests``, ``httpx`` / openai) or from + ``.headers`` (a Kubernetes ``ApiException``). Its value is seconds or an HTTP date; a date in the + past gives ``0``. Any other value (also a stand-in object in a unit test) gives ``None``.""" + if exc is None: + return None + for headers in (getattr(getattr(exc, "response", None), "headers", None), getattr(exc, "headers", None)): + try: + value = headers.get("Retry-After") or headers.get("retry-after") + except Exception: # noqa: BLE001 — no headers, or headers of an unknown shape + continue + if not isinstance(value, str) or not value.strip(): + continue + value = value.strip() + try: + return max(float(value), 0.0) + except ValueError: + pass + try: + when = parsedate_to_datetime(value) + except (TypeError, ValueError, IndexError): + continue + if when.tzinfo is None: + when = when.replace(tzinfo=timezone.utc) + return max((when - datetime.now(timezone.utc)).total_seconds(), 0.0) + return None + + def is_transient(exc: BaseException) -> bool: - """Only transient upstream failures are worth retrying: connection resets, timeouts, or a - 5xx server response. Permanent failures (4xx, value/validation errors) fail identically on - every attempt, so they are surfaced immediately rather than retried. Duck-typed so it - covers ``requests`` and Kubernetes client errors without importing either here.""" + """Only transient upstream failures are worth retrying: connection resets, timeouts, a 5xx + server response, or HTTP 408 / 429 (``_TRANSIENT_4XX_STATUSES``). A 429 that names an exhausted + quota or billing limit (``_QUOTA_ERROR_CODES``, for example OpenAI ``insufficient_quota``) cannot + clear, so it is permanent. Other permanent failures (other 4xx, value/validation errors) fail + identically on every attempt, so they are surfaced immediately rather than retried. Duck-typed so + it covers ``requests``, openai and Kubernetes client errors without importing any of them here.""" if isinstance(exc, (ConnectionError, TimeoutError)): # builtins (also OSError subclasses) return True # requests.exceptions.ConnectionError / Timeout are NOT subclasses of the builtins above; @@ -80,18 +144,38 @@ def is_transient(exc: BaseException) -> bool: if any(base.__name__ in transient_names for base in type(exc).__mro__): return True status = http_status(exc) + if status in _TRANSIENT_4XX_STATUSES: + return not (status == 429 and _quota_exhausted(exc)) return status is not None and status >= 500 -def run_upstream(fn: Callable[[], T]) -> T: - """Run ``fn`` with bounded retries (``UPSTREAM_MAX_RETRIES``, default 3) and exponential - backoff (1 s, 2 s, 4 s, … at most 30 s between attempts), retrying **only transient failures** - (``is_transient``) and reraising the last error so the caller can convert it to a 502. Returns - whatever ``fn`` returns (the return type is preserved for callers).""" - retryer = Retrying( +def upstream_retrying(*, attempts: int, backoff_min: float, backoff_max: float) -> Retrying: + """A tenacity ``Retrying`` that makes at most ``attempts`` attempts and retries **only transient + failures** (``is_transient``). Between attempts it waits the error's ``Retry-After`` + (``retry_after``) when there is one, else the exponential backoff from ``backoff_min``; each wait + is at most ``backoff_max``. ``reraise=True``: after the last attempt the ORIGINAL error is raised + (never a tenacity ``RetryError``).""" + backoff = wait_exponential(multiplier=1, min=backoff_min, max=backoff_max) + + def _wait(state: RetryCallState) -> float: + asked = retry_after(state.outcome.exception() if state.outcome else None) + return backoff(state) if asked is None else min(asked, backoff_max) + + return Retrying( retry=retry_if_exception(is_transient), - stop=stop_after_attempt(max_retries()), - wait=wait_exponential(multiplier=1, min=1, max=30), + stop=stop_after_attempt(attempts), + wait=_wait, reraise=True, ) + + +def run_upstream(fn: Callable[[], T]) -> T: + """Run ``fn`` with bounded retries (``UPSTREAM_MAX_RETRIES``, default 3) and exponential + backoff (1 s, 2 s, 4 s, … at most 30 s between attempts; a ``Retry-After`` replaces the backoff, + up to the same 30 s), retrying **only transient failures** (``is_transient``) and reraising the + last error so the caller can convert it to a 502. Returns whatever ``fn`` returns (the return + type is preserved for callers).""" + retryer = upstream_retrying( + attempts=max_retries(), backoff_min=_DEFAULT_BACKOFF_MIN, backoff_max=_DEFAULT_BACKOFF_MAX + ) return retryer(fn) diff --git a/test/unit/agent/policy_rules_builder/test_graph.py b/test/unit/agent/policy_rules_builder/test_graph.py index b0f86c3e..b162cc68 100644 --- a/test/unit/agent/policy_rules_builder/test_graph.py +++ b/test/unit/agent/policy_rules_builder/test_graph.py @@ -900,25 +900,23 @@ def test_structured_call_honors_llm_max_retries(monkeypatch): def test_structured_call_backoff_bounds_from_env(monkeypatch): - # The exponential-backoff bounds flow from LLM_RETRY_BACKOFF_MIN / _MAX into the retry wait. - # One attempt (no sleep) is enough: we assert the wait_exponential CONSTRUCTION bounds. - monkeypatch.setenv("LLM_MAX_RETRIES", "1") - monkeypatch.setenv("LLM_RETRY_BACKOFF_MIN", "7") - monkeypatch.setenv("LLM_RETRY_BACKOFF_MAX", "9") + # The exponential-backoff bounds flow from LLM_RETRY_BACKOFF_MIN / _MAX into the retry wait. The + # waits are recorded (no real sleep): 1, 2, 4, 8 s clamped to [3, 5]. + monkeypatch.setenv("LLM_MAX_RETRIES", "5") + monkeypatch.setenv("LLM_RETRY_BACKOFF_MIN", "3") + monkeypatch.setenv("LLM_RETRY_BACKOFF_MAX", "5") llm, _ = _failing_llm(ConnectionError("down")) + waits: list[float] = [] with ExitStack() as stack: stack.enter_context(patch("aiac.agent.policy_rules_builder.graph.get_policy_source", return_value=_Source())) stack.enter_context(patch("aiac.agent.policy_rules_builder.graph._build_llm", return_value=llm)) - stack.enter_context(patch("time.sleep")) - # The retry loop (and its wait_exponential) now lives in the shared aiac.agent.llm seam. - we = stack.enter_context(patch("aiac.agent.llm.wait_exponential")) + stack.enter_context(patch("time.sleep", side_effect=waits.append)) with pytest.raises(LLMAccessError): build_role_rules(_role(), [_scope("s-write", "write")]) - assert we.call_args.kwargs["min"] == 7 - assert we.call_args.kwargs["max"] == 9 + assert waits == [3, 3, 4, 5] def test_structured_call_decoupled_from_upstream_max_retries(monkeypatch): diff --git a/test/unit/agent/test_llm.py b/test/unit/agent/test_llm.py index 88f34079..1bb97f22 100644 --- a/test/unit/agent/test_llm.py +++ b/test/unit/agent/test_llm.py @@ -255,3 +255,68 @@ class MyCredentials(LLMCredentialsError): ... with pytest.raises(MyCredentials): raise_sanitized(_openai_status_error(openai.AuthenticationError, 401), credentials_error=MyCredentials) + + +def _rate_limit(*, retry_after: str | None = None, code: str | None = None) -> openai.RateLimitError: + """The ``openai.RateLimitError`` that the client raises for a 429: the response (with an optional + ``Retry-After``) and the body ``error`` object (openai passes ``body["error"]``).""" + headers = {"Retry-After": retry_after} if retry_after is not None else {} + response = httpx.Response( + 429, headers=headers, request=httpx.Request("POST", "http://llm.example/v1/chat/completions") + ) + body = {"message": _POISON, "type": code, "code": code} if code else None + return openai.RateLimitError(_POISON, response=response, body=body) + + +def test_call_with_retry_waits_the_retry_after_of_a_rate_limit(monkeypatch: pytest.MonkeyPatch) -> None: + waits: list[float] = [] + monkeypatch.setattr("time.sleep", waits.append) + runnable = MagicMock() + runnable.invoke.side_effect = [_rate_limit(retry_after="12"), _rate_limit(retry_after="600"), "ok"] + assert call_with_retry(runnable, ["msg"], settings=_SETTINGS) == "ok" + assert waits == [12.0, 30.0] # the Retry-After, capped at settings.backoff_max + + +def test_call_with_retry_uses_the_backoff_without_a_retry_after(monkeypatch: pytest.MonkeyPatch) -> None: + waits: list[float] = [] + monkeypatch.setattr("time.sleep", waits.append) + runnable = MagicMock() + runnable.invoke.side_effect = [_rate_limit(), _rate_limit(), "ok"] + assert call_with_retry(runnable, ["msg"], settings=replace(_SETTINGS, backoff_min=2.0)) == "ok" + assert waits == [2.0, 2.0] # wait_exponential from backoff_min + + +def test_a_rate_limit_for_an_exhausted_quota_is_not_retried_and_is_permanent(monkeypatch: pytest.MonkeyPatch) -> None: + # OpenAI answers 429 ``insufficient_quota`` when the account has no quota: a wait cannot clear it, + # so it is permanent (not the retryable access error), as before the 429 rule. + monkeypatch.setattr("time.sleep", lambda _s: pytest.fail("a quota 429 must not be retried")) + cause = _rate_limit(retry_after="1", code="insufficient_quota") + runnable = MagicMock() + runnable.invoke.side_effect = cause + with pytest.raises(openai.RateLimitError): + call_with_retry(runnable, ["msg"], settings=_SETTINGS) + runnable.invoke.assert_called_once() + with pytest.raises(UnparseableLLMResponseError) as ei: + raise_sanitized(cause) + assert not isinstance(ei.value, LLMAccessError) + assert _POISON not in str(ei.value) + assert ei.value.__cause__ is cause + + +@pytest.mark.parametrize( + ("cause", "error", "message"), + [ + (_openai_status_error(openai.NotFoundError, 404), UnparseableLLMResponseError, "response (HTTP 404)"), + (_openai_status_error(openai.InternalServerError, 503), LLMAccessError, "retries (HTTP 503)"), + (ValueError(_POISON), UnparseableLLMResponseError, "schema-invalid response"), + (TimeoutError(_POISON), LLMAccessError, "transport retries"), + ], + ids=["404", "503", "no-status", "timeout"], +) +def test_the_sanitized_message_names_the_status_and_nothing_else(cause, error, message) -> None: + # The status number is the one part of the raw error in the message: the ERROR log line names the + # cause (the rung-5 system test reads it), and the endpoint / key still cannot leak. + with pytest.raises(error) as ei: + raise_sanitized(cause) + assert str(ei.value).endswith(message) + assert _POISON not in str(ei.value) diff --git a/test/unit/shared/test_upstream.py b/test/unit/shared/test_upstream.py index 2895c86e..e04b306a 100644 --- a/test/unit/shared/test_upstream.py +++ b/test/unit/shared/test_upstream.py @@ -1,5 +1,8 @@ -"""Unit tests for aiac.shared.upstream — the status that ``is_transient`` reads from an error.""" +"""Unit tests for aiac.shared.upstream — the status that ``is_transient`` reads from an error, the +transient 408 / 429 rule, and the ``Retry-After`` wait of the shared retry loop.""" +from datetime import datetime, timedelta, timezone +from email.utils import format_datetime from unittest.mock import MagicMock import pytest @@ -7,7 +10,7 @@ from kubernetes.client.exceptions import ApiException from aiac.idp.configuration.api import IdPHTTPError -from aiac.shared.upstream import is_transient +from aiac.shared.upstream import is_transient, retry_after, run_upstream, upstream_retrying def _response(status: int) -> requests.Response: @@ -78,3 +81,146 @@ def test_a_bool_or_non_numeric_status_is_not_a_status(): assert is_transient(_StatusError(True)) is False assert is_transient(_StatusError("unavailable")) is False assert is_transient(_StatusError("503")) is True # a numeric string still counts, as before + + +# --------------------------------------------------------------------------- +# HTTP 408 / 429 are transient for every upstream; a 429 for an exhausted quota is not +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("status", [408, 429]) +def test_request_timeout_and_rate_limit_are_transient_for_every_seam(status): + # The Kubernetes API (API Priority and Fairness answers 429), the IdP library and MCP discovery. + assert is_transient(ApiException(status=status)) is True + assert is_transient(IdPHTTPError(status, "text", _response(status))) is True + assert is_transient(requests.HTTPError(response=_response(status))) is True + + +class _QuotaError(Exception): + """An error shaped like ``openai.RateLimitError``: ``status``, ``code`` and a dict ``body``.""" + + def __init__(self, status=429, *, code=None, body=None) -> None: + super().__init__(f"HTTP {status}") + self.status = status + self.code = code + self.body = body + + +@pytest.mark.parametrize( + "error", + [ + _QuotaError(code="insufficient_quota"), + _QuotaError(body={"code": "insufficient_quota"}), + _QuotaError(body={"error": {"type": "insufficient_quota", "code": None}}), + _QuotaError(code="QUOTA_EXCEEDED"), + _QuotaError(body={"code": "billing_hard_limit_reached"}), + ], + ids=["code", "body-code", "nested-type", "upper-case", "billing"], +) +def test_a_rate_limit_for_an_exhausted_quota_is_permanent(error): + # A wait cannot clear a quota or billing limit: it is not retried (as before the 429 rule). + assert is_transient(error) is False + + +def test_a_plain_rate_limit_is_transient(): + assert is_transient(_QuotaError(code="rate_limit_exceeded", body={"code": "rate_limit_exceeded"})) is True + assert is_transient(_QuotaError(body="not json")) is True + + +def test_a_quota_code_on_another_status_does_not_change_it(): + assert is_transient(_QuotaError(503, code="insufficient_quota")) is True # a 5xx stays transient + assert is_transient(_QuotaError(400, code="insufficient_quota")) is False + + +# --------------------------------------------------------------------------- +# retry_after — the Retry-After header of the error, in seconds +# --------------------------------------------------------------------------- + + +def _with_retry_after(value: str, status: int = 429) -> requests.HTTPError: + resp = _response(status) + resp.headers["Retry-After"] = value + return requests.HTTPError(response=resp) + + +def test_retry_after_in_seconds(): + assert retry_after(_with_retry_after("7")) == 7.0 + assert retry_after(_with_retry_after(" 1.5 ")) == 1.5 + assert retry_after(_with_retry_after("-3")) == 0.0 + + +def test_retry_after_as_an_http_date(): + when = datetime.now(timezone.utc) + timedelta(seconds=20) + wait = retry_after(_with_retry_after(format_datetime(when, usegmt=True))) + assert wait is not None and 15 <= wait <= 20 + assert retry_after(_with_retry_after("Wed, 21 Oct 2015 07:28:00 GMT")) == 0.0 # in the past + + +def test_retry_after_of_a_kubernetes_api_exception(): + exc = ApiException(status=429) + exc.headers = {"Retry-After": "2"} + assert retry_after(exc) == 2.0 + + +@pytest.mark.parametrize( + "exc", + [ + _with_retry_after("soon"), + requests.HTTPError(response=_response(429)), + requests.HTTPError(), + IdPHTTPError(429, "text", MagicMock()), # a stand-in response: its headers are not a header map + ValueError("no response"), + None, + ], + ids=["not-a-number-or-date", "no-header", "no-response", "mock-response", "plain-error", "none"], +) +def test_no_usable_retry_after_is_none(exc): + assert retry_after(exc) is None + + +# --------------------------------------------------------------------------- +# upstream_retrying / run_upstream — Retry-After replaces the backoff, with a cap +# --------------------------------------------------------------------------- + + +@pytest.fixture +def waits(monkeypatch: pytest.MonkeyPatch) -> list[float]: + recorded: list[float] = [] + monkeypatch.setattr("time.sleep", recorded.append) # tenacity sleeps via time.sleep at run time + monkeypatch.setenv("UPSTREAM_MAX_RETRIES", "3") + return recorded + + +def test_run_upstream_waits_the_retry_after_of_a_rate_limit(waits): + fn = MagicMock(side_effect=[_with_retry_after("5"), _with_retry_after("0"), "ok"]) + assert run_upstream(fn) == "ok" + assert waits == [5.0, 0.0] + + +def test_run_upstream_caps_the_retry_after(waits): + fn = MagicMock(side_effect=[_with_retry_after("3600"), "ok"]) + assert run_upstream(fn) == "ok" + assert waits == [30.0] # the backoff maximum of run_upstream + + +def test_run_upstream_uses_the_backoff_without_a_retry_after(waits): + fn = MagicMock(side_effect=ApiException(status=429)) + with pytest.raises(ApiException): + run_upstream(fn) + assert fn.call_count == 3 + assert waits == [1, 2] + + +def test_run_upstream_does_not_retry_a_quota_rate_limit(waits): + fn = MagicMock(side_effect=_QuotaError(code="insufficient_quota")) + with pytest.raises(_QuotaError): + run_upstream(fn) + fn.assert_called_once() + assert waits == [] + + +def test_upstream_retrying_uses_its_own_budget_and_cap(waits): + fn = MagicMock(side_effect=[_with_retry_after("60"), TimeoutError(), "ok"]) + retryer = upstream_retrying(attempts=5, backoff_min=2.0, backoff_max=10.0) + assert retryer(fn) == "ok" + assert waits == [10.0, 2.0] # Retry-After capped at backoff_max, then the backoff from backoff_min From 0a9e881ae55ef6e04f0f96abd5a7f60b2251ecc1 Mon Sep 17 00:00:00 2001 From: Oleg Blinder Date: Mon, 19 Oct 2026 02:34:36 +0300 Subject: [PATCH 15/15] fix(test): record each grant holder with an expiry, and release it for any import name The Direct Access Grants marker on the login client is now a JSON map of holder (one per pytest session: the xdist run id and the worker id) to expiry. A release removes its own holder and turns the grant off only when no other live holder remains, so two runs at the same time do not turn the grant off under each other. An expired entry, an empty map or the legacy "true" value is stale and is taken over as before. The session release in test/system/conftest.py finds the harness by its file, whatever name it was imported under, and logs a warning when a release cannot run. The cross-service test holds the grant before its first mutation, as _clean_slate does, so the skip cause is the same and a skip leaves no new realm. The rung-5 rollback test asserts on AIAC's own ERROR line with the injected status ("UnparseableLLMResponseError: ... (HTTP 404)"), not on the httpx INFO request line. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Oleg Blinder --- docs/specs/PRD.md | 2 +- docs/testing/uc1-onboarding-pipeline.md | 16 +- k8s/opa-kind-runbook.md | 9 +- test/system/conftest.py | 58 +++++- ...test_uc1_onboard_cross_service_conflict.py | 7 +- .../test_uc1_onboard_failure_rollback.py | 28 +-- test/system/uc1_onboard.py | 163 ++++++++++++--- .../uc/onboarding/test_uc1_login_client.py | 188 ++++++++++++++++-- 8 files changed, 392 insertions(+), 79 deletions(-) diff --git a/docs/specs/PRD.md b/docs/specs/PRD.md index 5d927dbd..e0a4af44 100644 --- a/docs/specs/PRD.md +++ b/docs/specs/PRD.md @@ -819,7 +819,7 @@ The Controller reads `AIAC_ENFORCEMENT_SIDE` at start, and an unknown value stop - **The namespace pipelines.** The `authbridge-runtime-config` ConfigMap of each service namespace must have `opa` in the inbound pipeline, and also `mcp-parser` for a tool. Under agent side, the agents' namespaces also need `mcp-parser` and `opa` in the outbound pipeline. Onboarding check #2 finds a missing plugin. - **Pod restarts.** The webhook injects the sidecar only on pod CREATE. After a change to the pipeline or to `injectTools`, restart the agent pods and the tool pods (`rossoctl.io/type=agent` and `rossoctl.io/type=tool`). - **The username precondition (D31).** The CRs key users by username, and `sub` is the username. So usernames must be unique, must not change and must never be used again. Keep `editUsernameAllowed: false` and `registrationAllowed: false` in the realm (the operator's realm template sets `registrationAllowed: false`; it does not set `editUsernameAllowed`, so that value is `false` only by the Keycloak default). An admin must never create a deleted username again: the new user would get the grants of the old user until AIAC writes the CRs again. -- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`hold_direct_access_grants`, once per session, before its first mutation) and turns it off again at session end if it turned it on; a client attribute records that, so the next run undoes a killed run. A manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). +- **The login clients (D31).** Each login client that calls AIAC agents directly must be in `PLATFORM_SOURCE_CLIENTS` (the `aiac-pdp-config` ConfigMap, default `rossoctl`). The IdP Configuration Service then links `aiac-username-sub` to it, so that the login token has `sub` = the username. The same setting also makes the client a platform client of the source gate (PDP Policy Writer). There is no manual `sub` mapper step: do not add a `sub` client mapper to a login client. The password grant of the test harness and of manual user-token probes needs Direct Access Grants on `rossoctl`. The system-test fixture enables it (`hold_direct_access_grants`, once per session, before its first mutation) and turns it off again at session end if it turned it on and no other live run holds it; a client attribute records the holder runs with an expiry, so concurrent runs do not turn it off under each other and a later run undoes a killed run. A manual probe needs it set by hand ([`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md#direct-access-grants-on-the-login-client)). - **The legacy token exchange (optional hardening).** AIAC does not need the Keycloak feature `token-exchange` (V1) to be off: the AuthBridge requests go to the standard exchange (V2), and a request that V1 handles still gets `sub` = the username for an onboarded target (§6). Keycloak recommends to turn V1 off, because then a request that V2 declines fails with `400` ("Standard token exchange is not enabled for the requested client") and does not silently go to a different engine. `KC_FEATURES` is in the Keycloak deployment of the rossoctl platform, not in this repo. If V1 stays on, AIAC needs no change. Detail: [`k8s/opa-kind-runbook.md`](../../k8s/opa-kind-runbook.md). diff --git a/docs/testing/uc1-onboarding-pipeline.md b/docs/testing/uc1-onboarding-pipeline.md index 8845ecb4..4bccf98d 100644 --- a/docs/testing/uc1-onboarding-pipeline.md +++ b/docs/testing/uc1-onboarding-pipeline.md @@ -226,8 +226,10 @@ cleanly** when any of it is absent (they never stand it up, and never false-pass `verify_login_subject` mints a login token through the login client `rossoctl` and checks that its `sub` is the username. Before the pre-run slate, the fixture enables Direct Access Grants on `rossoctl` (`hold_direct_access_grants`, once per session; the session end turns it off again if the - harness turned it on, and the client attribute `aiac.test.direct-access-grants-by-harness` records - that, so the next run undoes a killed run) and skips with the cause if the realm or `rossoctl` is + harness turned it on and no other live run holds it. The client attribute + `aiac.test.direct-access-grants-by-harness` records the holder runs, each with an expiry + (`AIAC_DAG_HOLD_TTL_SECONDS`, default 6 h), so two runs at the same time do not turn the grant off + under each other, and a later run undoes a killed run) and skips with the cause if the realm or `rossoctl` is absent or the admin cannot manage clients. It skips when it still cannot mint the token, and fails when `sub` is not the username. The `sub` comes from the client scope `aiac-username-sub`, the one source of the rule (D31). The IdP Configuration Service links it to @@ -485,10 +487,12 @@ changed combiner denies a pod that has no client CR, so a deleted CR means deny. scope of the disabled agent client (`test_rollback_keeps_the_subject_scope`). Provision keeps the link out of the created-manifest, so the rollback does not delete it (D31); - the agent has **no SPM** in the policy store; - - the Controller log shows the PRB's chat-completions request to the injected URL and its `404`, the - injected error (`UnparseableLLMResponseError`) and the move of the event to the dead-letter subject - `aiac.apply.dlq`. The `404` line proves that the injection caused the failure: other LLM faults - (for example a schema-invalid response) raise the same error type. + - the Controller log shows AIAC's own ERROR line of the injected error with its status + (`UnparseableLLMResponseError: ... (HTTP 404)`; the sanitized message ends with the HTTP status) and + the move of the event to the dead-letter subject `aiac.apply.dlq`. The `(HTTP 404)` proves that the + injection caused the failure: other LLM faults (for example a schema-invalid response) raise the + same error type with no status. The test does not read the httpx INFO request line (it depends on + `LOG_LEVEL` and on the httpx log format). - **Failed tool:** - the tool's CR, which its phase-3 onboarding wrote, is **deleted** (phase 3 records the CR as present, so "no CR" is a real transition), and the tool has **no SPM**; diff --git a/k8s/opa-kind-runbook.md b/k8s/opa-kind-runbook.md index 478c144b..25a06da9 100644 --- a/k8s/opa-kind-runbook.md +++ b/k8s/opa-kind-runbook.md @@ -228,9 +228,12 @@ login client `rossoctl` needs Direct Access Grants on that client. In a fresh realm it is off, and the mint fails with `unauthorized_client`. The system tests do not need this step: their fixture enables the grant itself, once per session (`hold_direct_access_grants` in `test/system/uc1_onboard.py`), and -turns it off again at session end if it turned it on. The client attribute -`aiac.test.direct-access-grants-by-harness` records that the harness turned it -on, so the next run undoes a run that was killed. +turns it off again at session end if it turned it on and no other run still +holds it. The client attribute `aiac.test.direct-access-grants-by-harness` +records which harness runs hold the grant, each with an expiry +(`AIAC_DAG_HOLD_TTL_SECONDS`, default 6 h). So two runs at the same time do not +turn the grant off under each other, and a later run undoes a run that was +killed. To enable it by hand, source the repo-root `.env` first. The snippet uses the same variables as the harness. It is a partial update: only the grant changes, diff --git a/test/system/conftest.py b/test/system/conftest.py index 15e8c412..8ebafa08 100644 --- a/test/system/conftest.py +++ b/test/system/conftest.py @@ -2,17 +2,63 @@ from __future__ import annotations +import logging import sys +from pathlib import Path import pytest +log = logging.getLogger(__name__) + +# The harness that holds Direct Access Grants on the login client (``hold_direct_access_grants``). +HARNESS_FILE = Path(__file__).resolve().with_name("uc1_onboard.py") + + +def _harness_modules() -> list: + """Each loaded copy of the harness, whatever name it was imported under (``test.system.uc1_onboard``, + ``uc1_onboard`` from another ``sys.path`` entry, or a module of ``--import-mode=importlib``). A copy + is a module whose file is ``HARNESS_FILE``. Each copy has its own holds, so each one is released.""" + found = [] + for module in list(sys.modules.values()): + path = getattr(module, "__file__", None) + try: + same = bool(path) and Path(path).resolve() == HARNESS_FILE + except (OSError, TypeError, ValueError): + continue + if same: + found.append(module) + return found + + +def release_harness_grants() -> None: + """Run ``release_direct_access_grants`` of each loaded copy of the harness (``_harness_modules``). + No copy loaded = no stack ran = nothing to release. A copy with no release function, or a release + that raises, is logged as a warning (the grant can then stay on until a later run takes it over); + it never fails the session teardown.""" + for module in _harness_modules(): + release = getattr(module, "release_direct_access_grants", None) + if not callable(release): + log.warning( + "the harness module %r has no release_direct_access_grants: Direct Access Grants on the " + "login client is not released by this session", + module.__name__, + ) + continue + try: + release() + except Exception: # noqa: BLE001 — a teardown hook: log, never fail the session end + log.warning( + "the release of Direct Access Grants by the harness module %r failed: the grant can stay on " + "until its holder entry expires and a later run takes it over", + module.__name__, + exc_info=True, + ) + @pytest.fixture(scope="session", autouse=True) def _release_direct_access_grants(): - """At the end of the session, turn Direct Access Grants off again on each login client that the - harness turned it on for (``uc1_onboard.hold_direct_access_grants``). Session scope: the grant is - turned on once, by the first stack of the session, and off once. A no-op when no stack ran.""" + """At the end of the session, release Direct Access Grants on each login client that the harness + holds (``uc1_onboard.hold_direct_access_grants``; ``release_harness_grants``). Session scope: the + grant is held once, by the first stack of the session, and released once. A no-op when no stack ran.""" yield - uc1 = sys.modules.get("test.system.uc1_onboard") # not imported = no stack ran = nothing to release - if uc1 is not None: - uc1.release_direct_access_grants() + release_harness_grants() diff --git a/test/system/disabled/test_uc1_onboard_cross_service_conflict.py b/test/system/disabled/test_uc1_onboard_cross_service_conflict.py index b08c4c62..ca049591 100644 --- a/test/system/disabled/test_uc1_onboard_cross_service_conflict.py +++ b/test/system/disabled/test_uc1_onboard_cross_service_conflict.py @@ -164,6 +164,10 @@ def test_cross_service_conflict_is_surfaced_as_422_conflict_report() -> None: keycloak_url = creds["KEYCLOAK_URL"] admin = uc1.connect_admin() + # Direct Access Grants on the login client, before the first mutation, as in ``uc1._clean_slate``: + # the same skip cause on the same cluster (a missing realm or login client), and a skip here leaves + # the cluster as it was (no new realm). Released at session end (``test/system/conftest.py``). + uc1.hold_direct_access_grants(admin, TEST_REALM) # Open the cleanup guard BEFORE the first shared-state mutation: the clean-slate steps below # (Agent CR delete, Keycloak cleanup, Policy Store clear, provisioning) already mutate shared # cluster state, so a failure mid-setup must still hit the ``finally`` teardown. @@ -172,9 +176,6 @@ def test_cross_service_conflict_is_surfaced_as_422_conflict_report() -> None: uc1.cleanup_provisioned(admin, TEST_REALM) # clean slate (Keycloak) uc1.clear_policy_store() # clean slate ONCE — NOT between phases (the agent must see the tool's rule) uc1.provision_realm_and_users(admin, TEST_REALM) # PRB reads the role universe - # After the provisioning: it creates the realm, so a missing login client is the true skip cause. - # Turned off again at session end if the harness turned it on. - uc1.hold_direct_access_grants(admin, TEST_REALM) uc1.verify_login_subject( keycloak_url=keycloak_url, realm=TEST_REALM, user="test-user", password=scn.USER_PASSWORD ) diff --git a/test/system/test_uc1_onboard_failure_rollback.py b/test/system/test_uc1_onboard_failure_rollback.py index 183d7506..f6d13b21 100644 --- a/test/system/test_uc1_onboard_failure_rollback.py +++ b/test/system/test_uc1_onboard_failure_rollback.py @@ -56,8 +56,10 @@ path is the same: both errors are in ``_ROLLBACK_ERRORS``, so the rollback and quarantine are identical. The error type alone does not prove that the injection caused the failure: every LLM fault that is not transient and not a refused credential (a schema-invalid response, another 4xx) also raises -``UnparseableLLMResponseError``. So the log check also asks for the PRB's chat-completions request to -``UNUSABLE_LLM_BASE_URL`` and its 404 (``_injected_request_line``). +``UnparseableLLMResponseError``. So the log check asks for AIAC's own ERROR line of the error with the +injected status (``_injected_error_line``: ``log_by_type`` logs the sanitized message, which ends with the +HTTP status, ``aiac.agent.llm.raise_sanitized``). It does not read the httpx INFO request line: that one +depends on ``LOG_LEVEL`` and on the httpx log format. **Re-onboarding trigger — the NATS subject.** ``publish_service_event`` publishes ``aiac.apply.service.`` (the subject + payload the ``aiac-event-listener`` SPI emits on @@ -120,14 +122,16 @@ # What the injection makes the consumer log (``log_by_type`` + the dead-letter move, both at ERROR). INJECTED_ERROR = "UnparseableLLMResponseError" +INJECTED_STATUS = 404 # the Controller's own app answers the injected path with 404 DLQ_SUBJECT = "aiac.apply.dlq" -def _injected_request_line() -> str: - """The Controller's httpx log line for the PRB call that the injection answers: the chat-completions - request to ``uc1.UNUSABLE_LLM_BASE_URL``, answered 404. Up to the status code, so the reason phrase - does not matter.""" - return f'HTTP Request: POST {uc1.UNUSABLE_LLM_BASE_URL.rstrip("/")}/chat/completions "HTTP/1.1 404' +def _injected_error_line() -> str: + """AIAC's own ERROR line for the build failure that the injection causes (``log_by_type``: + ``": "``). The sanitized message ends with the HTTP status of the raw error + (``aiac.agent.llm.raise_sanitized``), so the line proves the failure came from the injected 404, not + from another fault of the same type (a schema-invalid response has no status).""" + return f"{INJECTED_ERROR}: LLM returned an unparseable or schema-invalid response (HTTP {INJECTED_STATUS})" def _dlq_line(uuid: str) -> str: @@ -423,14 +427,14 @@ def test_failed_service_has_no_spm(run: dict, key: str) -> None: def test_failure_logged_and_dead_lettered(run: dict, key: str) -> None: """The Controller logs the build failure (its type) and moves the event to the DLQ as a **permanent** failure on the first delivery — the injection's error, never a redelivery loop. The - PRB's request to the injected URL and its 404 prove that the injection caused the failure, not - another LLM fault of the same type (module docstring).""" + error line with the injected 404 (``_injected_error_line``, AIAC's own ERROR line) proves that the + injection caused the failure, not another LLM fault of the same type (module docstring).""" obs = _failed(run, key) - assert _injected_request_line() in obs["logs"], ( - f"{key}: no {_injected_request_line()!r} in the Controller log — the build failed, but not on the " + assert INJECTED_ERROR in obs["logs"], f"{key}: no {INJECTED_ERROR!r} in the Controller log" + assert _injected_error_line() in obs["logs"], ( + f"{key}: no {_injected_error_line()!r} in the Controller log — the build failed, but not on the " "injected 404. Did the injection reach the Controller?" ) - assert INJECTED_ERROR in obs["logs"], f"{key}: no {INJECTED_ERROR!r} in the Controller log" assert _dlq_line(obs["uuid"]) in obs["logs"], f"{key}: no dead-letter line {_dlq_line(obs['uuid'])!r}" diff --git a/test/system/uc1_onboard.py b/test/system/uc1_onboard.py index 61374966..8bb42cff 100644 --- a/test/system/uc1_onboard.py +++ b/test/system/uc1_onboard.py @@ -103,6 +103,8 @@ import re import subprocess import sys +import time +import uuid from contextlib import contextmanager from dataclasses import dataclass from datetime import datetime, timedelta, timezone @@ -1054,17 +1056,88 @@ def _update_client(admin, realm: str, client_uuid: str, fields: dict) -> None: # Direct Access Grants on the login client. The harness mints user tokens with the password grant, so the # grant must be on. When the harness turns it on, it writes this client attribute in the same update. The -# marker is the record that the harness owns the grant, and it is in Keycloak, not in process memory: a -# run that is killed before its session teardown leaves the marker, and the next run takes the grant over -# and turns it off at its end. A grant that is on with no marker belongs to the realm (or to an operator: -# the runbook snippet removes the marker), and the harness never changes it. +# marker is the record of the harness sessions that hold the grant, and it is in Keycloak, not in process +# memory. Its value is a JSON object: holder id (``_dag_holder``) -> expiry (epoch seconds, the hold time + +# ``_dag_hold_ttl``). A release removes its own holder and turns the grant off only when no other live +# (not expired) holder remains, so two runs at the same time do not turn the grant off under each other. +# A run that is killed before its release leaves its holder; the entry expires, and the next run takes the +# grant over and turns it off at its end. A marker with no live holder, or a value that is not such an +# object (the legacy ``"true"`` of an earlier harness), is stale: it is taken over the same way. A grant +# that is on with no marker belongs to the realm (or to an operator: the runbook snippet removes the +# marker), and the harness never changes it. Keycloak has no compare-and-set: two holds at the same +# instant can lose one entry, and then the first release can turn the grant off early (a skip or a +# failed mint in the other run, never a false pass). DAG_MARKER_ATTRIBUTE = "aiac.test.direct-access-grants-by-harness" +# How long a holder entry stays live: longer than a whole system run. Override with this env var. +DAG_HOLD_TTL_ENV = "AIAC_DAG_HOLD_TTL_SECONDS" +DAG_HOLD_TTL_DEFAULT = 6 * 3600.0 + +# The run id of this process when pytest-xdist does not give one (a run with no ``-n``). +_PROCESS_RUN_ID = uuid.uuid4().hex + # The login clients whose grant the harness holds in this session: ``(realm, clientId)`` -> ``(admin, # client UUID)``. ``release_direct_access_grants`` empties it at session end (``test/system/conftest.py``). _dag_holds: dict[tuple[str, str], tuple[object, str]] = {} +def _dag_holder() -> str: + """The holder id of this pytest session: ``:``. The run id is the same in all xdist + workers of one run (``PYTEST_XDIST_TESTRUNUID``; else one id for this process). Each xdist worker is + its own pytest session with its own session-end release, so the worker id (``PYTEST_XDIST_WORKER``) + is part of the holder: the first worker to finish must not turn the grant off under its siblings.""" + run = os.environ.get("PYTEST_XDIST_TESTRUNUID") or _PROCESS_RUN_ID + return f"{run}:{os.environ.get('PYTEST_XDIST_WORKER', 'main')}" + + +def _dag_hold_ttl() -> float: + """The holder TTL in seconds (``DAG_HOLD_TTL_ENV``); a missing, non-numeric or non-positive value + gives the default.""" + try: + ttl = float(os.environ.get(DAG_HOLD_TTL_ENV, "")) + except ValueError: + return DAG_HOLD_TTL_DEFAULT + return ttl if ttl > 0 else DAG_HOLD_TTL_DEFAULT + + +def _now() -> float: + """The wall-clock time in epoch seconds (a seam for the unit tests).""" + return time.time() + + +def _dag_holders(client: dict) -> dict[str, float] | None: + """The holders that the marker of ``client`` records (holder id -> expiry), or ``None`` when the + client has no marker. A value that is not a JSON object of holder -> number (the legacy ``"true"``) + gives ``{}``: a stale marker with no holder.""" + raw = (client.get("attributes") or {}).get(DAG_MARKER_ATTRIBUTE) + if isinstance(raw, list): # an attribute read as a list of values + raw = raw[0] if raw else None + if not raw: + return None + try: + doc = json.loads(raw) + except (TypeError, ValueError): + return {} + if not isinstance(doc, dict): + return {} + return { + str(holder): float(expiry) + for holder, expiry in doc.items() + if isinstance(expiry, (int, float)) and not isinstance(expiry, bool) + } + + +def _live_others(holders: dict[str, float], me: str) -> dict[str, float]: + """The holders other than ``me`` whose entry has not expired.""" + now = _now() + return {holder: expiry for holder, expiry in holders.items() if holder != me and expiry > now} + + +def _marker_value(holders: dict[str, float]) -> str: + """The marker attribute value for ``holders`` (sorted, so the same holders give the same value).""" + return json.dumps({holder: round(expiry) for holder, expiry in sorted(holders.items())}) + + def _login_client_read_skip(realm: str, client_id: str, exc: Exception) -> str: """The skip message for a read of the login client that failed with ``exc``.""" if getattr(exc, "response_code", None) == 404: @@ -1085,11 +1158,13 @@ def hold_direct_access_grants(admin, realm: str, client_id: str = KEYCLOAK_CLIEN * The first call of the session reads the client. Later calls for the same client do nothing. * A grant that is on with no marker (``DAG_MARKER_ATTRIBUTE``) is the realm's own setting: it is not written, and it is not turned off later. - * A grant that is off is turned on, with the marker, in one partial update (``_update_client``). - * A marker from an earlier run that did not finish: this session takes the grant over. If the grant - is off with the marker on, it is turned on again (the marker stays). + * Else this session adds itself as a holder (``_dag_holder``, with an expiry) and turns the grant on, + in one partial update (``_update_client``). The update keeps the other live holders and drops the + expired ones. A marker with no live holder (a run that did not finish, or the legacy ``"true"``) is + taken over. A grant that is off with the marker on is turned on again. - In the last two cases ``release_direct_access_grants`` turns the grant off at session end. + ``release_direct_access_grants`` removes this holder at session end, and turns the grant off when no + other live holder remains. ``pytest.skip`` (with the cause) when the realm or the login client does not exist (nothing is created) or when the read or the update fails, for any reason (refused by the admin API, a transport @@ -1110,28 +1185,38 @@ def hold_direct_access_grants(admin, realm: str, client_id: str = KEYCLOAK_CLIEN f"the login client {client_id!r} does not exist in realm {realm!r}, so the harness cannot mint a " "user token through it (the realm owns its login client; the harness does not create it)" ) - marked = (client.get("attributes") or {}).get(DAG_MARKER_ATTRIBUTE) == "true" - if client.get("directAccessGrantsEnabled") and not marked: + holders = _dag_holders(client) + if client.get("directAccessGrantsEnabled") and holders is None: return - # Record the hold before the write: a write that raises after the server applied it is still undone. - _dag_holds[key] = (admin, client["id"]) - if marked: + me = _dag_holder() + others = _live_others(holders or {}, me) + if holders is not None and not others: log.warning( "an earlier run turned on Direct Access Grants on the login client %r in realm %r and did not " - "turn it off: this session takes it over and turns it off at its end", + "turn it off (no live holder in %r): this session takes it over and turns it off at its end", client_id, realm, + DAG_MARKER_ATTRIBUTE, ) - if client.get("directAccessGrantsEnabled"): - return - # The marker stays but the grant is off (an operator turned it off, a realm re-import, or a - # release that wrote only part of its update): turn the grant on again below. + elif others: + log.info( + "Direct Access Grants on the login client %r in realm %r is held by other live runs %s: this " + "session joins them", + client_id, + realm, + sorted(others), + ) + # Record the hold before the write: a write that raises after the server applied it is still undone. + _dag_holds[key] = (admin, client["id"]) try: _update_client( admin, realm, client["id"], - {"directAccessGrantsEnabled": True, "attributes": {DAG_MARKER_ATTRIBUTE: "true"}}, + { + "directAccessGrantsEnabled": True, + "attributes": {DAG_MARKER_ATTRIBUTE: _marker_value({**others, me: _now() + _dag_hold_ttl()})}, + }, ) except Exception as exc: # noqa: BLE001 — a refused write or a lost response: undo if applied, then skip _release_one(key) @@ -1142,48 +1227,66 @@ def hold_direct_access_grants(admin, realm: str, client_id: str = KEYCLOAK_CLIEN "(k8s/opa-kind-runbook.md, 'Direct Access Grants on the login client')." ) log.warning( - "enabled Direct Access Grants on the login client %r in realm %r (turned off again at session end)", + "holding Direct Access Grants on the login client %r in realm %r as %r (turned off again at session " + "end when no other run holds it)", client_id, realm, + me, ) def _release_one(key: tuple[str, str]) -> None: - """Turn the grant of one held login client off again, and remove the marker, if the marker is still - on the client. The client is read again first: when the marker is gone, someone else took the grant - over during the run (the runbook snippet removes the marker), and the grant is not changed. + """Remove this session's holder from one held login client, and turn its grant off (and remove the + marker) when no other live holder remains. The client is read again first: when the marker is gone, + someone else took the grant over during the run (the runbook snippet removes the marker), and the + grant is not changed. When another live holder remains, only this holder is removed (the expired + ones are dropped too) and the grant stays on. Best-effort: any error is logged, never raised, so it does not mask a failure that is propagating. - The marker stays when the write fails, so the next run turns the grant off.""" + The marker stays when the write fails, so the entry expires and a later run turns the grant off.""" admin, client_uuid = _dag_holds.pop(key) realm, client_id = key + me = _dag_holder() try: admin.change_current_realm(realm) client = admin.get_client(client_uuid) - if (client.get("attributes") or {}).get(DAG_MARKER_ATTRIBUTE) != "true": + holders = _dag_holders(client) + if holders is None: log.info( "Direct Access Grants on the login client %r in realm %r has no harness marker now: not changed", client_id, realm, ) return + others = _live_others(holders, me) + if others: + if me in holders: + _update_client(admin, realm, client_uuid, {"attributes": {DAG_MARKER_ATTRIBUTE: _marker_value(others)}}) + log.info( + "Direct Access Grants on the login client %r in realm %r stays on: other live runs %s hold it", + client_id, + realm, + sorted(others), + ) + return _update_client( admin, realm, client_uuid, {"directAccessGrantsEnabled": False, "attributes": {DAG_MARKER_ATTRIBUTE: ""}} ) except Exception as exc: # noqa: BLE001 — best-effort teardown: log, never mask the real failure log.error( - "cannot turn Direct Access Grants off again on the login client %r in realm %r: %s. The marker " - "%r stays, so the next run turns it off.", + "cannot release Direct Access Grants on the login client %r in realm %r: %s. The marker %r " + "stays, so its holder %r expires and a later run turns the grant off.", client_id, realm, exc, DAG_MARKER_ATTRIBUTE, + me, ) def release_direct_access_grants() -> None: - """At the end of the pytest session (``test/system/conftest.py``): turn Direct Access Grants off again - on each login client that ``hold_direct_access_grants`` holds (``_release_one``). A realm that had the - grant on is not changed. A no-op when no stack ran.""" + """At the end of the pytest session (``test/system/conftest.py``): release each login client that + ``hold_direct_access_grants`` holds (``_release_one``). A realm that had the grant on is not changed. + A no-op when no stack ran.""" for key in list(_dag_holds): _release_one(key) diff --git a/test/unit/agent/uc/onboarding/test_uc1_login_client.py b/test/unit/agent/uc/onboarding/test_uc1_login_client.py index aaeabebb..4c06f93c 100644 --- a/test/unit/agent/uc/onboarding/test_uc1_login_client.py +++ b/test/unit/agent/uc/onboarding/test_uc1_login_client.py @@ -1,12 +1,14 @@ """``uc1_onboard.hold_direct_access_grants`` turns on the password grant of the login client for the -session, and ``release_direct_access_grants`` turns it off again at session end if the harness turned it -on — pure unit tests. +session, and ``release_direct_access_grants`` turns it off again at session end when no other live harness +session holds it — pure unit tests. The harness mints user tokens through the login client ``rossoctl`` with the password grant, so the client needs Direct Access Grants on. ``_clean_slate`` (shared by ``onboarded_stack`` and ``pristine_stack``) holds the grant before its first mutation, and ``test/system/conftest.py`` releases -it at session end. The harness writes a marker attribute with the grant, so a run that is killed before -its release is undone by the next run, and a grant that someone else turned on is never turned off. A +it at session end. The harness writes a marker attribute with the grant: a JSON map of holder (one per +pytest session) -> expiry. A release turns the grant off only when no other live holder remains, a run +that is killed before its release expires and is undone by a later run, and a grant that someone else +turned on is never turned off. A grant that cannot be read or written is a skip with the cause. These tests use a stateful fake admin (no Keycloak) that applies the partial-update rules of Keycloak 26. They import the system harness like ``test_uc1_subject_scope.py`` does. @@ -15,6 +17,7 @@ from __future__ import annotations import sys +import types from pathlib import Path from unittest.mock import MagicMock @@ -29,16 +32,31 @@ LOGIN_UUID = "uuid-1" MARKER = uc1.DAG_MARKER_ATTRIBUTE +NOW = 1_000_000.0 # the fixed clock of the tests (``uc1._now``) +TTL = 3600.0 +ME = "run-a:main" # this session's holder id +OTHER = "run-b:main" # another live run @pytest.fixture(autouse=True) def _no_holds(monkeypatch: pytest.MonkeyPatch) -> None: - """Each test starts a fresh session: no login client is held.""" + """Each test starts a fresh session: no login client is held, a fixed clock and holder id.""" monkeypatch.setattr(uc1, "_dag_holds", {}) + monkeypatch.setattr(uc1, "_now", lambda: NOW) + monkeypatch.setenv(uc1.DAG_HOLD_TTL_ENV, str(TTL)) + monkeypatch.setattr(uc1, "_dag_holder", lambda: ME) -def _login(enabled: bool, *, marked: bool = False, client_id: str = uc1.KEYCLOAK_CLIENT_ID) -> dict: - attributes = {"keep.me": "yes", **({MARKER: "true"} if marked else {})} +def _mine() -> str: + return uc1._marker_value({ME: NOW + TTL}) + + +def _login( + enabled: bool, *, marked: bool = False, marker: str | None = None, client_id: str = uc1.KEYCLOAK_CLIENT_ID +) -> dict: + """The login client. ``marked`` puts the legacy ``"true"`` marker; ``marker`` puts the given value.""" + value = marker if marker is not None else ("true" if marked else None) + attributes = {"keep.me": "yes", **({MARKER: value} if value is not None else {})} return {"clientId": client_id, "id": LOGIN_UUID, "directAccessGrantsEnabled": enabled, "attributes": attributes} @@ -102,7 +120,7 @@ def test_the_grant_is_enabled_with_the_marker_in_one_partial_update() -> None: uc1.hold_direct_access_grants(admin, "rossoctl") admin.change_current_realm.assert_called_with("rossoctl") admin.update_client.assert_called_once_with( - LOGIN_UUID, {"directAccessGrantsEnabled": True, "attributes": {MARKER: "true"}} + LOGIN_UUID, {"directAccessGrantsEnabled": True, "attributes": {MARKER: _mine()}} ) assert admin.stored["attributes"]["keep.me"] == "yes" # other attributes are not written @@ -123,12 +141,18 @@ def test_the_grant_is_read_and_written_once_per_session() -> None: assert _grant_writes(admin) == [True, False] -def test_a_marker_left_by_a_killed_run_is_taken_over_and_released() -> None: - """The case of the review: an earlier run turned the grant on and was killed before its release. The - marker is still on the client, so this session turns the grant off at its end.""" - admin = _admin(_login(True, marked=True)) +@pytest.mark.parametrize( + "marker", + ["true", uc1._marker_value({"run-dead:main": NOW - 1}), "{}", "not json"], + ids=["legacy", "expired-holder", "no-holder", "garbage"], +) +def test_a_stale_marker_left_by_a_killed_run_is_taken_over_and_released(marker: str) -> None: + """The case of the review: an earlier run turned the grant on and was killed before its release. Its + marker has no live holder (the legacy ``"true"``, an expired entry, or an unreadable value), so this + session takes the grant over (it writes itself as the one holder) and turns it off at its end.""" + admin = _admin(_login(True, marker=marker)) uc1.hold_direct_access_grants(admin, "rossoctl") - admin.update_client.assert_not_called() # already on: nothing to write at entry + assert admin.stored["attributes"][MARKER] == _mine() # the expired holder is dropped uc1.release_direct_access_grants() assert admin.stored["directAccessGrantsEnabled"] is False assert MARKER not in admin.stored["attributes"] @@ -136,21 +160,95 @@ def test_a_marker_left_by_a_killed_run_is_taken_over_and_released() -> None: def test_a_marker_with_the_grant_off_turns_the_grant_on_and_is_released() -> None: """The case of the review: the marker is on the client but the grant is off (an operator turned it - off, a realm re-import, or a partial release). The session turns the grant on again, keeps the marker, - and turns the grant off at its end.""" + off, a realm re-import, or a partial release). The session turns the grant on again, writes itself as + the holder, and turns the grant off at its end.""" admin = _admin(_login(False, marked=True)) uc1.hold_direct_access_grants(admin, "rossoctl") admin.update_client.assert_called_once_with( - LOGIN_UUID, {"directAccessGrantsEnabled": True, "attributes": {MARKER: "true"}} + LOGIN_UUID, {"directAccessGrantsEnabled": True, "attributes": {MARKER: _mine()}} ) assert admin.stored["directAccessGrantsEnabled"] is True - assert admin.stored["attributes"][MARKER] == "true" + assert admin.stored["attributes"][MARKER] == _mine() assert ("rossoctl", uc1.KEYCLOAK_CLIENT_ID) in uc1._dag_holds uc1.release_direct_access_grants() assert admin.stored["directAccessGrantsEnabled"] is False assert MARKER not in admin.stored["attributes"] +# ====================================================================================== +# Two runs at the same time — each run is a holder; the grant goes off with the last one +# ====================================================================================== + + +def test_a_second_run_joins_the_live_holder_and_keeps_its_entry() -> None: + admin = _admin(_login(True, marker=uc1._marker_value({OTHER: NOW + 60}))) + uc1.hold_direct_access_grants(admin, "rossoctl") + assert uc1._dag_holders(admin.stored) == {OTHER: NOW + 60, ME: NOW + TTL} + + +def test_the_first_run_to_finish_does_not_turn_the_grant_off_under_another_live_run() -> None: + """The case of the review: run A and run B hold the grant. A finishes first: it removes only its own + entry, and the grant stays on for B. B then finishes and turns the grant off.""" + admin = _admin(_login(True, marker=uc1._marker_value({OTHER: NOW + 60}))) + uc1.hold_direct_access_grants(admin, "rossoctl") # this session (ME) joins OTHER + uc1.release_direct_access_grants() # ME finishes first + assert admin.stored["directAccessGrantsEnabled"] is True + assert uc1._dag_holders(admin.stored) == {OTHER: NOW + 60} + assert admin.update_client.call_args.args[1] == {"attributes": {MARKER: uc1._marker_value({OTHER: NOW + 60})}} + + +def test_the_last_live_run_to_finish_turns_the_grant_off(monkeypatch: pytest.MonkeyPatch) -> None: + admin = _admin(_login(False)) + uc1.hold_direct_access_grants(admin, "rossoctl") + # Another run joins during this session (as the test above, from its side). + admin.stored["attributes"][MARKER] = uc1._marker_value({ME: NOW + TTL, OTHER: NOW + 60}) + monkeypatch.setattr(uc1, "_now", lambda: NOW + 61) # the other run was killed: its entry expires + uc1.release_direct_access_grants() + assert admin.stored["directAccessGrantsEnabled"] is False + assert MARKER not in admin.stored["attributes"] + + +def test_an_expired_holder_does_not_keep_the_grant_on() -> None: + admin = _admin(_login(True, marker=uc1._marker_value({OTHER: NOW - 1, ME: NOW + TTL}))) + uc1._dag_holds[("rossoctl", uc1.KEYCLOAK_CLIENT_ID)] = (admin, LOGIN_UUID) + uc1.release_direct_access_grants() + assert admin.stored["directAccessGrantsEnabled"] is False + + +def test_the_holder_id_is_shared_by_the_run_and_names_the_worker(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.undo() # the real _dag_holder + monkeypatch.setenv("PYTEST_XDIST_TESTRUNUID", "run-123") + monkeypatch.setenv("PYTEST_XDIST_WORKER", "gw1") + assert uc1._dag_holder() == "run-123:gw1" + monkeypatch.delenv("PYTEST_XDIST_TESTRUNUID") + monkeypatch.delenv("PYTEST_XDIST_WORKER") + assert uc1._dag_holder() == f"{uc1._PROCESS_RUN_ID}:main" + + +@pytest.mark.parametrize( + ("raw", "ttl"), [("120", 120.0), ("0", uc1.DAG_HOLD_TTL_DEFAULT), ("x", uc1.DAG_HOLD_TTL_DEFAULT)] +) +def test_the_hold_ttl_comes_from_the_env(monkeypatch: pytest.MonkeyPatch, raw: str, ttl: float) -> None: + monkeypatch.setenv(uc1.DAG_HOLD_TTL_ENV, raw) + assert uc1._dag_hold_ttl() == ttl + + +@pytest.mark.parametrize( + ("value", "holders"), + [ + (None, None), + ("", None), + ("true", {}), + ('["a"]', {}), + ('{"a": 5, "b": "x", "c": true}', {"a": 5.0}), + (['{"a": 5}'], {"a": 5.0}), # an attribute read as a list of values + ], +) +def test_the_marker_value_is_read_as_holders(value, holders) -> None: + client = {"attributes": {} if value is None else {MARKER: value}} + assert uc1._dag_holders(client) == holders + + def test_a_missing_login_client_skips_and_is_not_created() -> None: admin = _admin(None) with pytest.raises(pytest.skip.Exception, match=uc1.KEYCLOAK_CLIENT_ID): @@ -206,6 +304,15 @@ def _applied_then_reset(uuid: str, fields: dict) -> None: assert uc1._dag_holds == {} +def test_a_failed_hold_that_joined_another_run_leaves_the_other_holder(monkeypatch: pytest.MonkeyPatch) -> None: + admin = _admin(_login(True, marker=uc1._marker_value({OTHER: NOW + 60}))) + admin.update_client.side_effect = KeycloakError(error_message="forbidden", response_code=403) + with pytest.raises(pytest.skip.Exception): + uc1.hold_direct_access_grants(admin, "rossoctl") + assert admin.update_client.call_count == 1 # ME is not in the marker, OTHER is live: nothing to undo + assert admin.stored["directAccessGrantsEnabled"] is True + + def test_another_login_client_can_be_held() -> None: admin = _admin(_login(False, client_id="other-login")) uc1.hold_direct_access_grants(admin, "rossoctl", client_id="other-login") @@ -258,7 +365,7 @@ def test_a_failed_release_is_logged_not_raised(caplog: pytest.LogCaptureFixture, uc1.hold_direct_access_grants(admin, "rossoctl") admin.update_client.side_effect = error uc1.release_direct_access_grants() # never masks a teardown failure - assert "cannot turn Direct Access Grants off again" in caplog.text + assert "cannot release Direct Access Grants" in caplog.text assert uc1._dag_holds == {} @@ -324,3 +431,48 @@ def test_the_body_failing_still_runs_the_teardown(steps: MagicMock, error: BaseE assert steps.mock_calls[0].args == ("teardown",) # the CR capture comes first assert [c.args[0] for c in steps.undeploy_workload.call_args_list] == ["github-tool", "github-agent"] steps._scrub_to_pristine.assert_called_once() + + +# ====================================================================================== +# test/system/conftest.py — the session release finds the harness under any import name +# ====================================================================================== + + +@pytest.fixture +def system_conftest(monkeypatch: pytest.MonkeyPatch): + """``test/system/conftest.py`` with only fake harness copies in ``sys.modules``.""" + from test.system import conftest + + monkeypatch.setattr(sys, "modules", {"test.system.conftest": conftest}) + return conftest + + +def _fake_harness(name: str, *, release=None) -> types.ModuleType: + module = types.ModuleType(name) + module.__file__ = str(Path(uc1.__file__)) + if release is not None: + module.release_direct_access_grants = release + return module + + +def test_the_release_finds_the_harness_under_another_import_name(system_conftest) -> None: + release = MagicMock() + sys.modules["uc1_onboard"] = _fake_harness("uc1_onboard", release=release) # another sys.path entry + sys.modules["other"] = types.ModuleType("other") # not the harness: no __file__ + system_conftest.release_harness_grants() + release.assert_called_once_with() + + +def test_the_release_does_nothing_when_no_harness_is_loaded(system_conftest, caplog) -> None: + system_conftest.release_harness_grants() + assert caplog.records == [] + + +def test_a_release_that_cannot_run_is_a_warning(system_conftest, caplog: pytest.LogCaptureFixture) -> None: + sys.modules["a"] = _fake_harness("a") # no release function + sys.modules["b"] = _fake_harness("b", release=MagicMock(side_effect=RuntimeError("boom"))) + system_conftest.release_harness_grants() # never raises + warnings = [r for r in caplog.records if r.levelname == "WARNING"] + assert len(warnings) == 2 + assert "has no release_direct_access_grants" in warnings[0].getMessage() + assert "release of Direct Access Grants by the harness module 'b' failed" in warnings[1].getMessage()