feat: generic HTTP external-metadata fetching evaluator (metadata/http) #184

Summary

Add a builtin that lets a policy issue a sub-request to an external HTTP service during request processing and use the response in the authorization decision — AuthPolicy's "generic HTTP" external metadata, adapted to PPE's embedded, capability-gated plugin model.

Concretely: a new plugin kind: metadata/http, invoked from authorization.pre_invocation as a run(name) step, that calls a user-defined endpoint (with request attributes mixed into URL / query / headers / body), parses the response, and publishes it into the attribute bag under a new metadata.<name>.* namespace so downstream predicates, PDPs (cel/opa/cedar), and assertions: can read it — typically combined with identity already resolved (subject.*, claim.*).

Motivation — parity gap

This closes the External metadata gap of the AuthPolicy/PPE feature comparison.

It also exercises three cross-cutting items that are currently weaker in PPE than in the AuthPolicy/Authorino and that matter specifically for a network callout:

• Rule/evaluator-level caching with user-defined dynamic cache keys
• Rule/evaluator-level metrics (+ user-defined CEL labels)
• Tracing (OpenTelemetry), data-plane traces

How it differs from what already exists

PPE already does authenticated/cached outbound HTTP in several places; this plugin should reuse that machinery, not reinvent it, and is distinct from each:

• identity/jwt JWKS fetching (crates/builtins/src/plugins/identity_jwt/config.rs:248) is the closest technical analog for HTTP-fetch-with-refresh-and-conditional-revalidation, but it is unauthenticated and fetches keys, not per-request metadata.
• delegator/oauth (crates/builtins/src/plugins/delegator_oauth/) already implements the OAuth2 client-credentials grant and a result cache — the two pieces this plugin needs for authenticated callouts.

Where it fits in the pipeline

• Hook / phase. Register an Unphased hook (fires once per request, like identity.resolve and token.delegate — see crates/ppe-core/src/hooks/metadata.rs:106), invoked from an authorization.pre_invocation step with run(name). This places the callout after identity resolution and before the authorization decision, matching AuthPolicy's metadata phase ordering.
• Output namespace. Introduce a new Metadata extension flattened into the bag as metadata.<instance-name>.* (dotted JSON), gated by a new read_metadata read capability (consistent with the capability catalog in docs/content/extensions.md). Rationale for a dedicated namespace rather than reusing claim.*/custom.*: the source is neither a verified claim nor host-defined custom state; keeping it separate preserves least-privilege gating and keeps audit/provenance honest.
• Merge-back mechanism. Same pattern as the JWT resolver: mutate a cloned payload and return PluginResult::modify_payload(...) (crates/builtins/src/plugins/identity_jwt/resolver.rs:842,927); add a typed slot + a CMF extractor in crates/ppe-apl-cmf/src/ so the response surfaces in the flat AttributeBag.

Why a new namespace rather than the existing custom.*

PPE already ingests host/filter-chain metadata through the custom.* namespace: the host populates Extensions.custom and extract_custom (crates/ppe-apl-cmf/src/custom.rs:19) bridges it into the bag. That slot is deliberately Unrestricted + Mutable, read_cap: None (crates/ppe-core/src/extensions/filter.rs:124) — ungated precisely because the host authored it from inside the trust boundary. External-HTTP metadata is a different category and should not reuse custom.*:

• Provenance. custom.* is host-vouched; this data came off the network (SSRF-guarded, authenticated, cacheable, can fail/stale). Routing it through the trusted host channel erases where the attribute came from — bad for audit and for trust reasoning.
• Least privilege. Every other resolved input (subject, claims, client, workload) is its own capability-gated slot, not dumped into custom.*. Network-fetched enrichment can carry sensitive remote data and should be gated the same way; custom.* is ungated by construction.
• Integrity. custom is Mutable; a fetched answer should be Immutable once resolved so a later plugin can't rewrite another service's response before the decision.
• Namespacing. Multiple fetchers need isolated metadata.<name>.* sub-namespaces; the flat host-owned custom map invites key collisions.
• No plugin-write path exists into custom today (write_cap: None; passed through on clone), so "merge" is not even the cheaper option.

Hence a dedicated Metadata extension → metadata.<name>.*, gated by read_metadata (open sub-question: gated vs ungated — recommended gated, because it crosses a network boundary; PDPs that read it then declare the cap).

Example wiring:

plugins:
  - name: geo
    kind: metadata/http
    hooks: [metadata.fetch]
    capabilities: [perform_http, read_subject, read_claims, read_headers, read_request]
    config: { ... }   # see "The fetching contract" below

routes:
  - tool: get_quote
    authentication: [corp-jwt]
    authorization:
      pre_invocation:
        - when: "authenticated && exists(subject.id)"   # conditional triggering
          do:   "run(geo)"
        - "metadata.geo.country in allowed_regions: allow"

The fetching contract

1. HTTP protocol

Everything maps onto the existing transport request type HttpRequest (crates/ppe-core/src/http.rs:83), so most of this is config → builder:

Attribute interpolation must be constrained to bag keys the plugin's declared read capabilities unlock (so a template cannot exfiltrate claim.* the instance was never granted). Credential-bearing namespaces (raw_credentials.*, http.request_headers.authorization) are the sharp case, and they span two distinct uses — both of which should be the operator's call, not PPE's:

• Placing the inbound token in the request body for the external service to verify — a delegated-credential check. The sub-request still authenticates as PPE itself; the caller's credential is data to be validated, not reused to authenticate. This is exactly what Kubernetes TokenReview does (see "Designed-for reuse").
• Reusing the inbound token as the sub-request's own outbound credential (e.g. copying it into the Authorization header) — token forwarding / impersonation.

Rather than a hard ban on either, interpolating a credential namespace is off by default and opt-in: it requires both the matching capability (read_inbound_credentials for raw_credentials.*, read_headers for the authorization header) and an explicit config acknowledgement, and the use is audited. This keeps the safe default without PPE deciding these patterns are off-limits — that call is the operator's.

The call itself goes through the host-services seam exactly as every other plugin does — svc.http_request(req, retry_policy) (crates/ppe-core/src/host.rs:178); PPE performs no HTTP of its own. The plugin must declare perform_http or the engine refuses to start and names the missing capability (crates/ppe-core/src/host.rs:344, docs/content/extensions.md "Gating an action rather than a slot").

Retry safety. Use RetryPolicy::idempotent() for GET and undelivered_only() for POST, keyed off HttpTransportError::may_have_reached_peer() (http.rs:440) — same discipline as the Vault provider (crates/builtins/src/secrets/vault/provider.rs:262).

2. TLS configurability

TLS trust is owned by the host-installed transport, not the plugin — this is a deliberate, documented property of the seam (crates/ppe-core/src/http.rs:471): one process, one connection pool, one trust store, one egress policy.

• Plaintext opt-in: a per-plugin insecure_http: bool (default false, rejects http://), consistent with identity/jwt, delegator/oauth, and vault (crates/builtins/src/plugins/delegator_oauth/config.rs:62).
• CA bundle / client-cert (mTLS) / skip-verify: the bundled HyperTransport pins webpki-roots with .with_no_client_auth() (crates/ppe/src/http_hyper.rs:254) and exposes none of these as YAML. A deployment that needs a private CA or mTLS to the metadata source does so by injecting its own HttpTransport via PolicyEngine::set_http_transport. A per-plugin TLS config struct (CA path, client cert/key, SNI) is a cross-cutting transport change rather than plugin-local — but it is in scope here, not a follow-up, because the Kubernetes TokenReview reuse (see "Designed-for reuse") cannot connect to the API server without a custom CA bundle and, for the mTLS variant, a client cert.

SSRF. The bundled transport refuses loopback / RFC1918 / link-local (incl. 169.254.169.254) / CGNAT 100.64.0.0/10 both pre-connect on IP literals and at DNS resolution time (crates/ppe-core/src/http_addr.rs:44, crates/ppe/src/http_hyper.rs:350), surfacing as HttpTransportError::Rejected. A metadata source on a private/in-cluster address therefore requires the host to build the transport with with_allow_private_destinations (http_hyper.rs:216) — the same constraint in-cluster Vault already carries. Document it; do not add a YAML hatch.

3. Authentication to the metadata source

A first-class auth: block (tagged enum, #[serde(tag = "kind")], matching the ClientSecretSource / DecodingKeySource convention). There is currently no generic outbound-credential primitive in the codebase — each plugin rolls its own — so this block is net-new and should become the reusable shape.

(a) Shared secret — a static credential attached to the callout, with the secret resolved through the secrets: block by value name, never a raw ref (docs/content/configuration.md, "Secrets and key material"):

auth:
  kind: shared_secret
  scheme: bearer            # bearer | header | basic
  header: Authorization     # for scheme: header (custom header name)
  secret: upstream_api_key  # a secrets.values entry
secrets:
  providers: { local: { kind: file, base_dir: /etc/ppe } }
  values:    { upstream_api_key: { provider: local, ref: metadata.key } }

Hold resolved secret material in Zeroizing<String> as delegator/oauth does (crates/builtins/src/plugins/delegator_oauth/delegator.rs:48). For backward compatibility also accept the per-plugin client_secret_source enum (env_var/file/literal).

(b) OAuth2 client-credentials — obtain an access token (RFC 6749 §4.4) and attach it to the callout, reusing delegator/oauth's machinery (GRANT_TYPE_CLIENT_CREDENTIALS, crates/builtins/src/plugins/delegator_oauth/delegator.rs:74; form_urlencode for the body; HTTP Basic client auth):

auth:
  kind: oauth2_client_credentials
  token_endpoint: "https://idp.example.com/realms/x/protocol/openid-connect/token"
  client_id: ppe-metadata
  client_secret: metadata_client_secret   # secrets.values entry
  scopes: [metadata.read]
  token_cache: { ttl_seconds: 3600 }       # cache the token, not just the response

The fetched token is cached independently of the response cache (§4), with TTL derived from the token response expires_in capped by ttl_seconds.

4. Supported response types

• json (default) — parse body, publish the object as metadata.<name>.* (dotted flatten; scalars/arrays map to AttributeValue variants in crates/ppe-apl-core/src/attributes.rs:32). Non-2xx or unparseable body → evaluator error (see fail-closed, below).
• text — publish the raw string as metadata.<name> (single scalar).
• Honor Content-Type for parse selection; reject a body exceeding max_response_bytes before parsing.
• Surface metadata.<name>.$status and metadata.<name>.$headers.* (operator opt-in) so a policy can branch on the callout's HTTP status without the plugin having to deny on non-2xx itself.

Cross-cutting capabilities

Caching of metadata responses (user-defined, dynamic keys)

This is the most important capability and the one with the sharpest open design question, because PPE deliberately forbids dynamic cache keys today.

What exists. The only result cache is delegator/oauth's DelegatedTokenCache (crates/builtins/src/plugins/delegator_oauth/cache/). Its key derivation (cache/key.rs) is HMAC-SHA256 under a per-process secret and deliberately excludes derived identity (security.subject.id) and refuses to cache when a request-data-bearing value (a rendered template) is in the key (NotCacheable::ResourceTemplate, cache/key.rs:19-45,346). DispatchCache (crates/ppe-apl-runtime/src/dispatch_plan.rs:741) caches compiled route plans, not results, and the valkey session store (crates/ppe-apl-runtime/src/session_store.rs:68) is a monotonic label union, not a general KV cache.

What this needs. Authorino caches metadata keyed on operator-chosen dynamic values (e.g. auth.identity.username). That is exactly what key.rs refuses to do — but the refusal is specific to a credential cache, where serving a cached token to the wrong principal is privilege escalation. A metadata response cache is different in kind: its payload is read-only enrichment, not a credential. Proposed resolution:

cache:
  enabled: true
  key: [subject.id, claim.tenant]   # user-defined lookup key; dynamic bag refs
  ttl_seconds: 300
  respect_cache_control: true       # honor upstream Cache-Control / ETag / 304
  max_entries: 10000                # bounded in-process LRU
  on_stale_refresh_error: serve_last_good | fail   # explicit staleness knob

Design points to settle (see Open questions):

• A cache keyed on subject.id is sound for enrichment iff (i) the key captures every attribute that varies the response, and (ii) the cached value is only ever used as enrichment feeding predicates, never re-emitted as a credential. This differs intentionally from the credential cache; cite cache/key.rs:19-45 for the contrast and document why metadata relaxes it.
• Reuse the HMAC-keyed CacheKey construction so cache keys are not guessable/forgeable, but allow derived-identity and request-data inputs that the token cache forbids.
• Conditional revalidation (ETag + If-None-Match, 304 as hit) already has a working implementation to copy from JWKS fetch (crates/builtins/src/plugins/identity_jwt/config.rs:442; HttpResponse::etag() / cache_max_age() / is_not_modified(), crates/ppe-core/src/http.rs:331-363).
• Backing store. In-process bounded LRU first (like DelegatedTokenCache and the api-key lookup cache). A distributed/shared cache (valkey) is a follow-up and would need a new store interface — the current SessionStore cannot hold arbitrary values.

Conditional triggering / skipping the callout

No per-plugin "skip unless condition" exists; gating is done by the surrounding APL rule predicate over the attribute bag — which is sufficient and idiomatic:

authorization:
  pre_invocation:
    - when: "authenticated && !exists(claim.department)"   # only call out if we lack it
      do:   "run(geo)"

Predicate leaves available: Comparison, IsTrue/IsFalse, exists(key), x in set (crates/ppe-apl-core/src/rules.rs:92). This covers AuthPolicy's "conditions" on a metadata source (skip when an attribute is already present, restrict to certain roles/paths, etc.). The plugin itself should additionally short-circuit to a cache hit before issuing the callout, so "skip the sub-request when we already have the answer" holds even inside a single request.

Observability — metrics and auditing

Current state (important). There are no metrics and no OpenTelemetry in the tree — the only observability dependency is tracing = "0.1" (Cargo.toml:102); no prometheus / metrics / opentelemetry crates, and no #[instrument]. Plugins emit tracing:: logs only. So this issue cannot "just add a Prometheus counter"; it should:

1. Now: emit structured tracing events for every callout — outcome (hit/miss/fetched/error), HTTP status, latency_ms, cache (hit/miss/revalidated), and the instance name — so behavior is observable via logs immediately. Emit an audit event per callout (the audit-logger reference plugin, reference/plugins/audit-logger/, is the sink pattern).
2. Define the target metric set this plugin will register once a metrics seam exists (counter by outcome, latency histogram, cache hit-ratio), with user-defined CEL labels — matching AuthPolicy's evaluator metrics + custom labels. Flag that the metrics seam itself is a broader cross-cutting prerequisite (the "Rule/evaluator-level metrics = No" row) and should be its own issue this one depends on, not scope-crept into here.

Trace-context propagation into the sub-request

Authorino propagates the trace into metadata callouts; PPE cannot today. The request.* extension carries trace and span ids (docs/content/extensions.md, Request extension → request.trace_id / request.span_id), but nothing injects them: HttpRequest has no trace fields and there is no traceparent handling anywhere. The host-services doc even notes the carrier should attach the request span (crates/ppe-core/src/host.rs:19) but no code does.

Proposal: add W3C traceparent (+ tracestate) injection at the transport seam so every plugin benefits, derived from request.trace_id/request.span_id, creating a child span for the callout. Because this touches HttpRequest / HostServices, land it as a small dedicated change this plugin consumes, rather than a metadata/http-only hack. Until it lands, the plugin should at minimum emit the ids on its tracing span so logs correlate.

Designed-for reuse: Kubernetes TokenReview authentication

The AuthPolicy comparison also lists Kubernetes TokenReview authentication as a PPE gap. A TokenReview is just "POST a JSON body to one HTTP endpoint and act on the JSON response", so the same HTTP-callout implementation should back it — this issue should factor the engine with that reuse in mind, not build a metadata/http-only silo.

How Authorino does it (and why it looks like it needs no config)

Authorino's evaluator (pkg/evaluators/identity/kubernetes_auth.go) is thin: it takes the inbound bearer token off the request, builds authv1.TokenReview{ Spec: { Token: reqToken, Audiences: audiencesWithDefault(host) } }, calls k8sClient.Create(ctx, tr) (a POST /apis/authentication.k8s.io/v1/tokenreviews), and reads Status.Authenticated + Status.User. It looks configuration-free because everything hard is hidden in the injected, shared k8sClient, which the manager built once from client-go's rest.InClusterConfig(). That gives Authorino, for free, because it is a Go controller running in the cluster:

• API server address from KUBERNETES_SERVICE_HOST / KUBERNETES_SERVICE_PORT (https://kubernetes.default.svc).
• TLS trust of the API server via the cluster CA at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.
• Client authentication to the API server via its own ServiceAccount bearer token at /var/run/secrets/kubernetes.io/serviceaccount/token (not mTLS, in the common in-cluster path), with client-go transparently reloading the bound (~hourly) token.

Generalizing to PPE

PPE may run outside Kubernetes, so none of the above can be assumed — which means the generic fetcher's job is exactly to make that hidden connection + auth explicit configuration. The mapping is almost one-to-one onto the contract above:

Two things make the TokenReview use not a plain metadata/http instance, and they define the seam to design for:

1. It is authentication, not enrichment. Its output binds to the subject identity (subject.id from status.user.username, subject.roles/teams from status.user.groups, gated authenticated), and it runs at identity.resolve (Unphased, pre-authz), not pre_invocation. So the clean factoring is a shared HTTP-callout core (request build + TLS + auth + cache + retry + SSRF + observability + trace) consumed by two thin output adapters: metadata/http (→ metadata.<name>.*) and an identity/kubernetes-tokenreview kind (→ subject identity). Same code path, different output binding and hook.
2. It must place the inbound credential in the request body. This is the headline case of the opt-in credential interpolation from §1: the TokenReview body legitimately needs the inbound token. It is gated by read_inbound_credentials and expressed as an explicit, named body field rather than free-form interpolation of raw_credentials.*, so the forwarding is deliberate, auditable, and scoped to the configured endpoint.

Consequence for this issue

Build the fetcher as a reusable HTTP-callout core, and promote the two TLS items (custom CA bundle and client cert/mTLS) from "candidate follow-up" (§2, Open question 5) to in-scope, because the TokenReview reuse cannot work without them. The TokenReview adapter itself (identity binding + inbound-token body field + the identity/kubernetes-tokenreview kind) can be a fast-follow issue that depends on this one — but the core must be shaped so it drops in without a rewrite. The same core also lines up the other AuthPolicy "call-an-endpoint" features (OAuth2 token introspection, UMA) as later adapters.

Fail-closed semantics

• perform_http is mandatory. Withholding it stops the call rather than degrading it — a metadata evaluator that silently skipped its callout would decide without the answer it needed (docs/content/extensions.md, "Gating an action rather than a slot").
• Default on_error: fail → a failed/timed-out/non-2xx callout denies (the plugin runs in the sequential band so it can block). Operators opt into fail-open with on_error: ignore/continue, a documented knob, exactly as elsewhere.
• Classify errors like JWKS does — Fatal (missing transport/capability, 4xx auth failure) vs Recoverable (timeout, 5xx) — so retry/soft-fail is principled (crates/builtins/src/plugins/identity_jwt/config.rs:740).
• Cache poisoning / staleness: a cached value must only ever feed predicates, never be re-emitted as a credential; on_stale_refresh_error makes the stale-serve decision explicit and alarmable.

Proposed config shape (full)

plugins:
  - name: geo
    kind: metadata/http
    hooks: [metadata.fetch]
    on_error: fail
    capabilities: [perform_http, read_subject, read_claims, read_headers, read_request]
    config:
      method: GET
      url: "https://geo.example.com/v1/users/{subject.id}/location"
      query:
        - { name: ip, from: http.request_headers.x-forwarded-for }
      headers:
        - { name: x-tenant, from: claim.tenant }
      timeout_seconds: 5
      connect_timeout_seconds: 2
      max_response_bytes: 1048576
      insecure_http: false
      auth:
        kind: oauth2_client_credentials
        token_endpoint: "https://idp.example.com/realms/x/protocol/openid-connect/token"
        client_id: ppe-metadata
        client_secret: metadata_client_secret
        scopes: [metadata.read]
        token_cache: { ttl_seconds: 3600 }
      response:
        type: json
      cache:
        enabled: true
        key: [subject.id, claim.tenant]
        ttl_seconds: 300
        respect_cache_control: true
        max_entries: 10000
        on_stale_refresh_error: serve_last_good

secrets:
  providers: { local: { kind: file, base_dir: /etc/ppe } }
  values:    { metadata_client_secret: { provider: local, ref: metadata.client-secret } }

routes:
  - tool: get_quote
    authentication: [corp-jwt]
    authorization:
      pre_invocation:
        - when: "authenticated && exists(subject.id)"
          do:   "run(geo)"
        - "metadata.geo.country in allowed_regions: allow"

Authoring notes (for the implementer)

• Template: identity/jwt. KIND const + PluginFactory (crates/builtins/src/plugins/identity_jwt/factory.rs:64), serde config with #[serde(deny_unknown_fields)], Plugin/HookHandler (crates/ppe-core/src/plugin.rs:74, crates/ppe-core/src/hooks/trait_def.rs:135).
• New Cargo feature (e.g. metadata-http), added to the builtins umbrella, and one line in register_builtins! (crates/ppe/src/lib.rs:180-229).
• New Metadata extension + read_metadata capability + CMF extractor (crates/ppe-apl-cmf/src/), publishing metadata.<name>.* into the bag.
• Unknown kind:, a declared-but-unreached plugin, and an unknown config key all fail the load by name — keep that contract.

Acceptance criteria

• kind: metadata/http registers behind its feature and via install_builtins.
• GET and POST callouts with dynamic URL / query / headers / body from the bag.
• Configurable timeout_seconds, connect_timeout_seconds, max_response_bytes.
• insecure_http knob; private-destination/SSRF behavior documented.
• TLS config: custom CA bundle and client cert/key (mTLS) — required for the Kubernetes TokenReview reuse.
• The callout logic is factored as a reusable core (request/TLS/auth/cache/retry/observability) that an identity/kubernetes-tokenreview adapter can consume without a rewrite; inbound-token-in-body is gated by read_inbound_credentials.
• auth: shared_secret (bearer / custom header / basic) via secrets: values.
• auth: oauth2_client_credentials with token caching.
• JSON and text response handling; result published under metadata.<name>.* and readable by cel/opa/cedar and assertions:.
• Response cache with user-defined (incl. dynamic) keys, TTL, bounded size, and ETag/Cache-Control revalidation.
• Conditional triggering via APL when: + in-request cache short-circuit.
• on_error: fail denies by default; ignore/continue documented fail-open.
• Structured tracing events + audit event per callout; target metric set and trace-propagation design captured (even if gated on the cross-cutting work).
• Docs: a docs/content/ page + a worked example, held by a loaded fixture.

Open questions

1. Dynamic cache keys vs cache/key.rs policy. Confirm that a metadata (non-credential) cache may key on derived identity / request data that the credential cache forbids, and nail the invariant that cached metadata is never re-emitted as a credential. This is the central security review item.
2. Hook model. A dedicated Unphased metadata.fetch hook invoked from pre_invocation, or model the whole thing as a pre_invocation step? Former matches identity.resolve/token.delegate; confirm preference.
3. Metrics seam dependency. Should this block on a general evaluator-metrics seam, or ship with tracing+audit now and add metrics when the seam lands?
4. Trace propagation scope. Land W3C traceparent injection at the transport seam (benefits all plugins) as a prerequisite, vs plugin-local for now?
5. Per-plugin TLS (CA / mTLS). Confirmed in scope (the TokenReview reuse needs it) — the open part is shape: a per-plugin TLS config struct, vs extending the host transport, vs both. Where does CA-bundle/client-cert config live so one process still shares one trust store where it wants to?
6. Distributed cache. In-process LRU only for v1, or is a shared (valkey) metadata cache in scope — noting the current SessionStore can't hold it?
7. TokenReview split. Land the reusable HTTP-callout core here and the identity/kubernetes-tokenreview adapter as a dependent fast-follow, or build both together? Either way, is identity/kubernetes-tokenreview the right kind, and does the inbound-token-in-body exception (gated by read_inbound_credentials) belong in the core or only in that adapter?