Skip to content

Commit eaca0fb

Browse files
authored
Compare scoped model reachability before config changes
1 parent ef181d5 commit eaca0fb

6 files changed

Lines changed: 439 additions & 0 deletions

File tree

‎docs/GATEWAY-POLICY-CHANGE.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Gateway permission-change preflight
2+
3+
## The concrete maintainer task
4+
5+
Before merging a gateway scope-to-model mapping or required-scope change, a maintainer wants to know: "Which of our supplied agent requests starts reaching a model, stops reaching it, or changes its refusal?" A generic Actas policy pack cannot answer that because gateway routing and scope mapping are separate constraints.
6+
7+
This command boots the actual gateway configuration, issues throwaway local tokens with its embedded Actas runtime, and sends actual in-process HTTP requests through signature verification, required-scope checks, known-route checks, scope-to-model mapping and MockProvider completion. It compares baseline and candidate behavior on the same explicit cases. It does not reimplement those decisions in a second policy engine.
8+
9+
## Run
10+
11+
```sh
12+
/path/to/gateway-dev-venv/bin/python scripts/gateway_policy_change.py \
13+
--gateway-source /path/to/clean/llmgate \
14+
--baseline baseline.json --candidate candidate.json --cases cases.json \
15+
> permission-change.json
16+
```
17+
18+
Exit 0 means no supplied-case HTTP status changed and candidate expectations match. Exit 1 means a change or mismatch requires review, even if the changed status was expected. Exit 2 is an input/source/runtime error, not acceptance.
19+
20+
Case schema, bounded to 32 cases:
21+
22+
```json
23+
[{"name":"cheap-worker-expensive-model","scope":["llm:complete"],"model":"expensive","expect_status":403}]
24+
```
25+
26+
The included fixture begins with `llm:complete` mapping only to `cheap`, then adds `expensive` to that same scope. The actual response changes from 403 with no provider execution to 200 with provider execution. The candidate's expected status is also changed to 200. The command still exits 1 and reports one newly reachable alias. That is the central workflow, not a cosmetic wrapper of the earlier runner.
27+
28+
## Executed checks
29+
30+
On clean gateway `ef181d541e1e958ead0f1ad2ea0ef7f89593e914`:
31+
32+
- Unchanged configuration: expected cheap request 200, out-of-scope model 403, missing required scope 403, unknown alias 404.
33+
- Mapping expansion: one supplied request newly reachable, HTTP 403 to 200, candidate expectations all match, exit 1.
34+
- Required-scope change: one newly reachable and one newly refused request, observed on native HTTP responses.
35+
- Removed configured alias: 403 becomes native 404, still flagged for review.
36+
- Real or mixed provider configuration: rejected before any native application construction. It is never silently stripped to a mock-only projection.
37+
- Inherited fake provider/Redis credentials: ignored. The request still completes locally.
38+
- Duplicate JSON members, invalid scope shapes, ambiguous enable flags and nondeterministic mock settings: rejected.
39+
40+
The original eight native-workflow regressions pass. Two additional focused regressions check missing dependencies and ordinary native runtime failures as sanitized error JSON/exit 2, not policy-change exit 1. The CLI fixture separately executes actual signed-token HTTP calls and records native module origins/hashes and the clean Git head. Reports include request case names and should remain private unless their inputs are deliberately shareable.
41+
42+
## Limits and next check
43+
44+
Only all-mock configurations with zero mock latency/failure rate are supported. The real production `deploy.config.json` contains real routes and therefore is explicitly unsupported. A maintainer can use a deliberately authored all-mock release fixture, but the report does not prove anything about declared or activated production real routes.
45+
46+
Tokens are newly issued per baseline/candidate run. This compares configurations for equivalent freshly issued local credentials, not migration compatibility of existing tokens or sessions. No production key is collected. The environment is cleared for native execution, socket connections are blocked, no configuration is installed, and no deployment or real-provider call is performed. MockProvider execution is explicitly part of the observed proof. These environment/socket guards are process-global and intended for trusted, single-process local use, not a concurrently hosted service. This is not a sandbox for malicious Python dependencies. Only trusted local code should run.
47+
48+
This does not assess request budgets, streaming/failover, cross-service issuer trust, PoP, distributed replay/revocation, all possible cases, live browser behavior or production safety. Mandatory post-production end-to-end checks remain separate. It is a concrete internal maintainer use case, not established external demand or a production authorization product.
49+
50+
This command complements the local native-contract preflight in `docs/LOCAL-PREFLIGHT.md`. Its supplied-case change report requires maintainer review; it never automatically approves or installs policy. Bounded gateway/Actas owner reviews checked the native boot, source boundary and claim meaning before integration. External user demand remains unvalidated.
Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"providers": [
3+
{
4+
"name": "local",
5+
"type": "mock",
6+
"models": [
7+
"cheap",
8+
"expensive"
9+
],
10+
"latency": 0
11+
}
12+
],
13+
"token_auth": {
14+
"enabled": true,
15+
"issuer": "local-issuer",
16+
"audience": "local-gateway",
17+
"scope_models": {
18+
"llm:complete": [
19+
"cheap"
20+
],
21+
"llm:premium": [
22+
"expensive"
23+
]
24+
}
25+
}
26+
}
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"providers": [
3+
{
4+
"name": "local",
5+
"type": "mock",
6+
"models": [
7+
"cheap",
8+
"expensive"
9+
],
10+
"latency": 0
11+
}
12+
],
13+
"token_auth": {
14+
"enabled": true,
15+
"issuer": "local-issuer",
16+
"audience": "local-gateway",
17+
"scope_models": {
18+
"llm:complete": [
19+
"cheap",
20+
"expensive"
21+
],
22+
"llm:premium": [
23+
"expensive"
24+
]
25+
}
26+
}
27+
}

‎examples/policy-change/cases.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
[
2+
{
3+
"name": "cheap",
4+
"scope": [
5+
"llm:complete"
6+
],
7+
"model": "cheap",
8+
"expect_status": 200
9+
},
10+
{
11+
"name": "expensive",
12+
"scope": [
13+
"llm:complete"
14+
],
15+
"model": "expensive",
16+
"expect_status": 200
17+
},
18+
{
19+
"name": "missing-required",
20+
"scope": [
21+
"llm:premium"
22+
],
23+
"model": "expensive",
24+
"expect_status": 403
25+
},
26+
{
27+
"name": "unknown-model",
28+
"scope": [
29+
"llm:complete"
30+
],
31+
"model": "absent",
32+
"expect_status": 404
33+
}
34+
]

‎scripts/gateway_policy_change.py‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
#!/usr/bin/env python3
2+
"""Compare actual local gateway responses for supplied scoped-token request cases."""
3+
import argparse
4+
import asyncio
5+
import hashlib
6+
import json
7+
import os
8+
from pathlib import Path
9+
import re
10+
import sys
11+
import socket
12+
import subprocess
13+
import inspect
14+
import tempfile
15+
from unittest.mock import patch
16+
17+
BOUNDARY = ('Supplied-case comparison on native mock-provider gateway requests with fresh local bearer tokens. '
18+
'Not a proof of production access, all possible requests, upstream health, distributed revocation, '
19+
'proof of possession or deployment safety. No configuration is installed.')
20+
IDENT = re.compile(r'[A-Za-z0-9][A-Za-z0-9._~:/@+-]{0,127}\Z')
21+
22+
23+
def read_json(path):
24+
raw = Path(path).read_bytes()
25+
if len(raw) > 65536:
26+
raise ValueError('input exceeds 64 KiB')
27+
def unique(pairs):
28+
d = {}
29+
for k, v in pairs:
30+
if k in d:
31+
raise ValueError('duplicate JSON field')
32+
d[k] = v
33+
return d
34+
return json.loads(raw, object_pairs_hook=unique), hashlib.sha256(raw).hexdigest()
35+
36+
37+
def text(x):
38+
return isinstance(x, str) and bool(IDENT.fullmatch(x))
39+
40+
41+
def validate_config(cfg):
42+
if not isinstance(cfg, dict) or not isinstance(cfg.get('providers'), list) or not 1 <= len(cfg['providers']) <= 8:
43+
raise ValueError('expected 1-8 mock providers')
44+
# Never instantiate an HTTPProvider or silently remove a declared real route.
45+
for p in cfg['providers']:
46+
if not isinstance(p, dict) or p.get('type') != 'mock':
47+
raise ValueError('only all-mock configurations are supported; real/mixed routes are not projected or activated')
48+
if not text(p.get('name')) or not isinstance(p.get('models'), list) or not 1 <= len(p['models']) <= 32 or not all(text(m) for m in p['models']):
49+
raise ValueError('invalid mock provider identifiers/models')
50+
if p.get('latency', .02) != 0 or p.get('fail_rate', 0) != 0:
51+
raise ValueError('set mock latency and fail_rate to zero for deterministic permission checks')
52+
ta = cfg.get('token_auth')
53+
if not isinstance(ta, dict) or ta.get('enabled') is not True or not text(ta.get('issuer')) or not text(ta.get('audience')):
54+
raise ValueError('enabled native token_auth with bounded issuer/audience required')
55+
if type(ta.get('demo', {}).get('enabled', False)) is not bool or ta.get('demo', {}).get('enabled', False):
56+
raise ValueError('delegation demo must be disabled for this local preflight')
57+
if not isinstance(ta.get('scope_models'), dict) or len(ta['scope_models']) > 32:
58+
raise ValueError('invalid scope mapping')
59+
for scope, models in ta['scope_models'].items():
60+
if not text(scope) or not isinstance(models, list) or len(models) > 32 or not all(text(m) for m in models):
61+
raise ValueError('invalid scope mapping values')
62+
if not text(ta.get('required_scope', 'llm:complete')):
63+
raise ValueError('invalid required scope')
64+
65+
66+
def validate_cases(cases):
67+
if not isinstance(cases, list) or not 1 <= len(cases) <= 32:
68+
raise ValueError('expected 1-32 request cases')
69+
names = set()
70+
for c in cases:
71+
if not isinstance(c, dict) or set(c) != {'name', 'scope', 'model', 'expect_status'}:
72+
raise ValueError('each case needs exactly name, scope, model and expect_status')
73+
if not text(c['name']) or c['name'] in names or not text(c['model']):
74+
raise ValueError('invalid or duplicate case identity')
75+
names.add(c['name'])
76+
if not isinstance(c['scope'], list) or len(c['scope']) > 32 or not all(text(s) for s in c['scope']) or len(set(c['scope'])) != len(c['scope']):
77+
raise ValueError('invalid case scopes')
78+
if type(c['expect_status']) is not int or c['expect_status'] not in (200, 401, 403, 404):
79+
raise ValueError('unsupported expected status')
80+
81+
82+
async def one_request(cfg, case):
83+
import httpx
84+
import llmgate.config as native_config
85+
from actas.tokens import issue_token
86+
original_create = native_config.create_app
87+
captured = {}
88+
def create(*args, **kw):
89+
captured['token_auth'] = kw['token_auth']
90+
return original_create(*args, **kw)
91+
with tempfile.TemporaryDirectory(prefix='gateway-policy-') as tmp:
92+
path = Path(tmp, 'config.json')
93+
path.write_text(json.dumps(cfg))
94+
with patch.dict(os.environ, {cfg.get('keys_env', 'LLMGATE_KEYS'): '{}'}, clear=False), patch.object(native_config, 'create_app', create):
95+
app = native_config.build_app(str(path))
96+
ta = captured['token_auth']
97+
token, _ = issue_token(ta.verifier.keystore, issuer=cfg['token_auth']['issuer'],
98+
subject='local-preflight-worker', scope=' '.join(case['scope']),
99+
audience=[cfg['token_auth']['audience']], expires_in=60)
100+
# Native app lifecycle, auth verification, required-scope check, allowlist,
101+
# known-route check and actual MockProvider completion all execute.
102+
async with app.router.lifespan_context(app):
103+
async with httpx.AsyncClient(transport=httpx.ASGITransport(app=app), base_url='http://local-preflight') as client:
104+
response = await client.post('/v1/chat/completions', headers={'Authorization': 'Bearer ' + token},
105+
json={'model': case['model'], 'messages': [{'role': 'user', 'content': 'local permission check'}], 'max_tokens': 1})
106+
calls = sum(route.provider.calls for route in app.state.router.routes)
107+
if response.status_code not in (200, 401, 403, 404):
108+
raise ValueError('native request produced an unsupported operational status')
109+
if (response.status_code == 200) != (calls > 0):
110+
raise ValueError('native response/provider execution evidence disagrees')
111+
return {'status': response.status_code, 'provider_executed': calls > 0}
112+
113+
114+
async def compare(baseline, candidate, cases):
115+
validate_config(baseline); validate_config(candidate); validate_cases(cases)
116+
changes = []
117+
# No caller credentials/store/env-based provider activation inherited.
118+
def no_network(*args, **kwargs):
119+
raise ValueError('network access is disabled during this local comparison')
120+
with patch.dict(os.environ, {}, clear=True), patch.object(socket.socket, 'connect', no_network), patch.object(socket.socket, 'connect_ex', no_network), patch.object(socket, 'create_connection', no_network):
121+
for case in cases:
122+
before = await one_request(baseline, case)
123+
after = await one_request(candidate, case)
124+
transition = 'newly_reachable' if before['status'] != 200 and after['status'] == 200 else 'newly_refused' if before['status'] == 200 and after['status'] != 200 else 'unchanged' if before['status'] == after['status'] else 'refusal_changed'
125+
changes.append({'name': case['name'], 'before': before, 'after': after, 'transition': transition,
126+
'candidate_expectation_matched': after['status'] == case['expect_status']})
127+
changed = any(x['before']['status'] != x['after']['status'] for x in changes)
128+
matched = all(x['candidate_expectation_matched'] for x in changes)
129+
return {'schema_version': 1, 'state': 'review_required' if changed or not matched else 'pass',
130+
'total': len(changes), 'newly_reachable': sum(x['transition'] == 'newly_reachable' for x in changes),
131+
'newly_refused': sum(x['transition'] == 'newly_refused' for x in changes),
132+
'refusal_changed': sum(x['transition'] == 'refusal_changed' for x in changes),
133+
'candidate_matched': sum(x['candidate_expectation_matched'] for x in changes),
134+
'cases': changes, 'boundary': BOUNDARY}
135+
136+
137+
def source_identity():
138+
import llmgate.config
139+
import llmgate.tokenauth
140+
import actas.tokens
141+
root = Path(inspect.getfile(llmgate.config)).resolve().parents[1]
142+
git = subprocess.run(['git', '-C', str(root), 'rev-parse', 'HEAD'], capture_output=True, text=True, timeout=10)
143+
status = subprocess.run(['git', '-C', str(root), 'status', '--porcelain'], capture_output=True, text=True, timeout=10)
144+
if git.returncode or status.returncode or status.stdout:
145+
raise ValueError('clean gateway Git source required')
146+
modules = {}
147+
for module in (llmgate.config, llmgate.tokenauth, actas.tokens):
148+
path = Path(inspect.getfile(module)).resolve()
149+
modules[module.__name__] = {'relative_path': str(path.relative_to(root)), 'sha256': hashlib.sha256(path.read_bytes()).hexdigest()}
150+
return {'gateway_git_head': git.stdout.strip(), 'native_imports': modules}
151+
152+
153+
def main():
154+
p = argparse.ArgumentParser(description=__doc__)
155+
for arg in ('baseline', 'candidate', 'cases'):
156+
p.add_argument('--' + arg, required=True)
157+
p.add_argument('--gateway-source', required=True, help='Clean gateway checkout containing its embedded Actas runtime')
158+
args = p.parse_args()
159+
try:
160+
expected_source = Path(args.gateway_source).resolve(strict=True)
161+
sys.path.insert(0, str(expected_source))
162+
baseline, bh = read_json(args.baseline); candidate, ch = read_json(args.candidate); cases, qh = read_json(args.cases)
163+
identity = source_identity()
164+
import llmgate.config
165+
if Path(inspect.getfile(llmgate.config)).resolve().parents[1] != expected_source:
166+
raise ValueError('native gateway import mismatch')
167+
report = asyncio.run(compare(baseline, candidate, cases))
168+
if source_identity() != identity:
169+
raise ValueError('native source changed during comparison')
170+
report['source'] = identity
171+
report['input_sha256'] = {'baseline': bh, 'candidate': ch, 'cases': qh}
172+
except Exception as exc:
173+
# Ordinary dependency/native execution failures are not policy regressions.
174+
# BaseException (KeyboardInterrupt/SystemExit) is deliberately not caught.
175+
report = {'schema_version': 1, 'state': 'error', 'error_type': type(exc).__name__, 'boundary': BOUNDARY}
176+
print(json.dumps(report, indent=2))
177+
return {'pass': 0, 'review_required': 1, 'error': 2}[report['state']]
178+
179+
180+
if __name__ == '__main__':
181+
sys.exit(main())

0 commit comments

Comments
 (0)