test(quota): qualify three-application distributed quotas by nerdalert · Pull Request #127 · praxis-proxy/grid

nerdalert · 2026-09-07T05:04:27Z

Summary

Extends the distributed token-quota topology and first-class qualification to prove three independent application quotas through one shared consumer endpoint.

application-a, application-b, and application-c authenticate with Basic Auth on the same listener. Praxis publishes the verified subject privately; Praxis AI uses that subject as the quota key; Grid routes admitted requests across the shared west, central, and east provider overlay.

This keeps three responsibilities separate:

Praxis establishes the authenticated subject.
Praxis AI admits or denies against that subject's distributed token budget.
Grid selects an eligible provider only after admission.

What the qualification proves

Three credentials use one endpoint and one quota rule configuration.
Each authenticated subject receives an independent quota.
The same subject shares one Valkey-backed budget across both consumer gateways, so scaling replicas does not multiply capacity.
Exhausting one subject denies it on both gateways while the other subjects remain admitted.
Missing, invalid, and cross-subject credentials are rejected before provider contact.
Denied quota requests do not reach a provider.
Admitted traffic covers all three attributable Grid provider sites.
Concurrent reservations do not exceed configured reservation capacity.
Natural window expiry restores admission.
Quota state survives consumer restart.
Valkey outage fails closed and recovery restores service.
NetworkPolicy positive and negative controls hold.
Accepted and serving overlay revisions converge before traffic.
Run-owned clusters, networks, and generated Forge state are cleaned automatically.

Topology changes

Keeps two consumer gateways with identical quota configuration and one shared listener.
Removes the temporary public identity gateway, projected identity headers, trusted-group filter, and private identity hop.
Uses key: authenticated_subject with one common rule and Valkey namespace; the verified subject partitions the ledger securely.
Retains the three provider sites and Grid round-robin overlay.
Adds run-scoped Forge state and generated-config ignore rules.

Quota contract

Admission is reservation-based because actual response usage is unknown before routing. The enforced invariant is:

active reserved tokens <= capacity

Settlement records actual usage after the response. Actual settled usage may exceed the reservation estimate without indicating over-admission. Provider distribution is sampled independently from tight quota boundaries.

Cross-repository dependencies

This PR is the Grid qualification and reference topology for a contract implemented across repositories:

feat(basic-auth): publish authenticated identity praxis#1108 publishes a successfully verified Basic Auth username as private request-local AuthenticatedIdentity.
feat(token-rate-limit): key quotas by authenticated subject ai#980 adds key: authenticated_subject, consumes that trusted identity, and hashes it into an opaque quota key.
This PR proves three independent subject quotas across gateway replicas and Grid providers.
The companion Demos PR packages the same topology for users.

The identity type is authentication-method-neutral. Basic Auth enables this demo now, while JWT/OIDC/OAuth-backed authentication can publish the same identity for reuse by the AI quota filter.

Related to #101 and praxis-proxy/ai#121.
Depends on praxis-proxy/praxis#1108 and praxis-proxy/ai#980.

Validation

Two fresh Kind qualifications passed all 9 scenarios with automatic teardown. Static validation included xtask tests, workspace Clippy, formatting, make test, make doc, make lint, Forge validation, and git diff --check.

Evidence and generated .forge.resolved.*.yaml files are excluded from the commit.

Landing order

Keep this PR open until Praxis #1108 is released and AI #980 consumes that release. Then rebuild the feature-enabled AI image from committed dependencies and rerun the qualification before merge.

Companion demo: praxis-proxy/demos#20.

Exercise three Basic Auth subjects through one shared endpoint and verify independent subject-keyed quotas across two consumer gateways. Preserve the existing routing, concurrency, expiry, restart, outage, NetworkPolicy, evidence, and cleanup coverage while removing the temporary identity-gateway design. Signed-off-by: Brent Salisbury <bsalisbu@redhat.com>

nerdalert · 2026-09-17T18:08:17Z

This feature needs to go in as an experimental flag.

hexfusion

the implementation is solid, three apps on one listener and one token_rate_limit rule, each keeping its own per-subject budget, is really neat, and testing concurrency plus a gateway restart against the shared counter is nice coverage.

what i'd want to settle before is what the per-user budget is meant to be. the value here from use-case feels like the openrouter shape. one endpoint fronting every site, all usage metered per user through it. that makes the budget global, not per-site. so the shared counter's behavior under partition and upgrade is really the core of the distributed system problem.

is this a hard gate or a meter. the readme calls it hard enforcement, but settled usage can exceed the reservation and isn't treated as a second cap, so today it's a hard cap on concurrent reservations and a soft meter on tokens. worth being explicit, because hard enforcement is a much higher bar for the store.
reservation_timeout is 30s, under both the 60s window and a plausible long generation. a request over 30s frees its own hold mid-flight and over-admits on one node with no partition, and a late settlement writes against an expired hold. seems like it should be at least the max request time.
is the failover path exercised? one non-ha valkey means only total-outage fail-closed is tested. an async replica promoted under partition can lose an acked reservation and fail open, spending a budget twice. fail-closed-on-outage is a narrower guarantee than failover safety.
a couple i couldn't confirm from the diff: is settle idempotent under retry, and is reserve a single atomic op rather than get-then-set that could race even on a healthy valkey.

mostly it'd help to hear the intended use case and rollout. how many gateway replicas, how valkey (could we use postgres?) is deployed and failed over, max_tokens in real traffic, and the upgrade strategy. that pins down which of these actually bite.

hexfusion · 2026-09-17T23:44:39Z

+                gateway,
+                app.credential,
+            )?;
+            denied_on_both = denied_on_both && record(results, result) == Outcome::QuotaDenied;


nit: && short-circuits, so record() gets skipped once this is false and those probes miss the log. same at 1701.

hexfusion · 2026-09-17T23:44:39Z

+        .lines()
+        .map(|line| {
+            let lower = line.to_ascii_lowercase();
+            if ["authorization:", "password:", "token:", "secret:", "privatekey:"]


nit: the redaction misses the json form, inline secrets, and the redis://:pass@host url. can we add a test covering those cases?

hexfusion · 2026-09-17T23:44:39Z

+        hash ^= u32::from(byte);
+        hash = hash.wrapping_mul(0x0100_0193);
+    }
+    let third_octet = 20 + (hash % 200);


question: does this only give 200 subnets? two runs could hash to the same one and collide.

hexfusion · 2026-09-17T23:44:39Z

 /// Scenario: valid Alice auth is admitted on both gateways with attribution.
-fn scenario_valid_auth(results: &mut Vec<HttpResult>) -> Result<ScenarioResult, Box<dyn std::error::Error>> {
+/// Return whether attribution covers every and only configured provider site.
+fn covers_all_provider_sites(distribution: &BTreeMap<String, u32>) -> bool {


nit: this might be flaky. only 6 probes cover it, and a transport retry can double-advance the round robin and drop a provider site.

hexfusion · 2026-09-17T23:44:39Z

+            return Ok(result);
+        }
+        results.push(result);
+        std::thread::park_timeout(RESTART_PROBE_INTERVAL);


park_timeout can wake early on a spurious wakeup, and with no deadline recheck it can burn the restart retries before a slow gateway is ready. use sleep?

Suggested change

std::thread::park_timeout(RESTART_PROBE_INTERVAL);

std::thread::sleep(RESTART_PROBE_INTERVAL);

hexfusion · 2026-09-17T23:44:39Z

+    // into the topology (one application-scoped listener per app). They are used
+    // as-is; nothing is regenerated from a fixture, so deployed configuration is
+    // exactly what is reviewed in the repository.
+    let _ = state_dir;


question: is this used? state_dir is referenced two lines down.

hexfusion · 2026-09-17T23:44:40Z

-/// Alice principal password used only by this qualification topology.
-const ALICE_PASS: &str = "alice-secret";
+/// Application-A principal used by the legacy quota scenarios.
+const ALICE_USER: &str = "application-a";


nit: the const is named ALICE_USER but the value is application-a, not a huge deal but could we make this a little more clear?

hexfusion · 2026-09-17T23:44:40Z

-      - main
-
+- name: proxy
+  address: 0.0.0.0:8443


minor: plaintext on 8443 is deceiving, since the same port runs mtls on the provider side later. use a plaintext-looking port or add a note.

hexfusion · 2026-09-17T23:44:40Z


 /// Validate an IPv4 CIDR without accepting host-only or IPv6 forms.
-fn validate_ipv4_cidr(cidr: &str) -> Result<(), ForgeError> {
+pub(crate) fn validate_ipv4_cidr(cidr: &str) -> Result<(), ForgeError> {


minor: this only rejects a prefix over 32, so host bits (10.240.5.7/24) and /31 or /32 slip through. could tighten it for user-supplied subnets.

nerdalert · 2026-09-19T05:20:11Z

Holding on converting to experimental.

nerdalert force-pushed the fix/three-app-distributed-quota branch from 620ea90 to a185c14 Compare September 7, 2026 19:55

nerdalert changed the title ~~test(quota): add three-application distributed qualification~~ test(quota): qualify three-application distributed quotas Sep 7, 2026

shaneutt added this to the v0.3.0 milestone Sep 17, 2026

shaneutt added this to AI Grid Network - Inference Fundamentals Sep 17, 2026

github-project-automation Bot moved this to Next in AI Grid Network - Inference Fundamentals Sep 17, 2026

shaneutt requested review from hexfusion and jordigilh September 17, 2026 21:56

shaneutt assigned hexfusion Sep 17, 2026

shaneutt moved this from Next to Review in AI Grid Network - Inference Fundamentals Sep 17, 2026

shaneutt added this to Advanced Token-Based Rate-Limiting Sep 17, 2026

github-project-automation Bot moved this to Todo in Advanced Token-Based Rate-Limiting Sep 17, 2026

shaneutt removed this from AI Grid Network - Inference Fundamentals Sep 17, 2026

hexfusion reviewed Sep 18, 2026

View reviewed changes

nerdalert added the holding-pattern Waiting for discussions or contributor updates in order to proceed label Sep 19, 2026

Navigation Menu

test(quota): qualify three-application distributed quotas - #127

test(quota): qualify three-application distributed quotas#127
nerdalert wants to merge 1 commit into
praxis-proxy:mainfrom
nerdalert:fix/three-app-distributed-quota

nerdalert commented Sep 7, 2026 •

edited

Loading

Uh oh!

nerdalert commented Sep 17, 2026

Uh oh!

hexfusion left a comment

Uh oh!

hexfusion Sep 17, 2026

Uh oh!

hexfusion Sep 17, 2026

Uh oh!

hexfusion Sep 17, 2026

Uh oh!

hexfusion Sep 17, 2026

Uh oh!

hexfusion Sep 17, 2026 •

edited

Loading

Uh oh!

hexfusion Sep 17, 2026

Uh oh!

hexfusion Sep 17, 2026 •

edited

Loading

Uh oh!

hexfusion Sep 17, 2026

Uh oh!

hexfusion Sep 17, 2026

Uh oh!

nerdalert commented Sep 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	std::thread::park_timeout(RESTART_PROBE_INTERVAL);
	std::thread::sleep(RESTART_PROBE_INTERVAL);

Conversation

nerdalert commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What the qualification proves

Topology changes

Quota contract

Cross-repository dependencies

Validation

Landing order

Uh oh!

nerdalert commented Sep 17, 2026

Uh oh!

hexfusion left a comment

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

hexfusion Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

nerdalert commented Sep 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nerdalert commented Sep 7, 2026 •

edited

Loading

hexfusion Sep 17, 2026 •

edited

Loading

hexfusion Sep 17, 2026 •

edited

Loading