app-policy/checker: differential tests for Evaluate against the policyscale oracle - #13896
Draft
dimitri-nicolo wants to merge 2 commits into
Draft
app-policy/checker: differential tests for Evaluate against the policyscale oracle#13896dimitri-nicolo wants to merge 2 commits into
dimitri-nicolo wants to merge 2 commits into
Conversation
…benchmarks Lift the two synthetic policy sets out of the checker benchmarks into a package that renders the same set three ways: a policystore.PolicyStore and endpoint for engine and collector benchmarks, the ToDataplane updates that load it through ProcessUpdate, and Calico resources (Tier, GlobalNetworkPolicy, GlobalNetworkSet) for applying it to a cluster. A number measured at the engine, in the collector and on a node then describes the same policy set. The fixture also carries an oracle: Fixture.Expect computes the trace the engine must report for a flow from the generator's own model of each rule, independently of the matching code, and MatchingFlow / DeniedFlow / Sampler produce flows aimed at chosen depths of the walk or following a flow model (miss fraction, repeat fraction). Later changes to the engine (compiled policies, a verdict cache, evaluation off the collector's main loop) are gated on differential tests built on these. Composite applies both measured shapes to one endpoint: 19,992 ingress rules, 18,662 egress rules, 7,571 IP sets. BenchmarkEvaluateComposite measures it in the three cases that matter for the collector; the two existing benchmarks keep their cases and their numbers. hack/cmd/policyscale renders a preset as YAML, or prints sampled flows with their expected verdicts, for node-level runs. Tracking: CORE-13316.
…yscale oracle Run Evaluate over a corpus drawn from each policyscale preset (the denied flow, flows aimed at every 257th rule, sampled flows following a flow model, and replays as UDP, with an invalid protocol and with nil addresses) and require the trace to equal the one the fixture's oracle computes from its own model of the rules, in both policy scopes. The harness takes two evaluators, so a second implementation of the walk (compiled policies, a verdict cache, evaluation on another goroutine) is checked against Evaluate the same way before it is switched on. The named cases cover what a generated corpus cannot reach: nil addresses against positive and negated references, named and negated named ports, IP+port sets, negated and missing sets, protocols out of range, by name, by number and negated, staged policies in both scopes, HTTP criteria against an L4 flow, Log rules, Pass into the next tier and into profiles, and a policy missing from the store. runNamedCases takes an evaluator for the same reason. Tracking: CORE-13316.
dimitri-nicolo
added
docs-not-required
Contributor
CI triage — CalicoRecommendation: Dig deeper before merging Likely caused by this PR
workflow_id: 047eec87-b64f-465e-ac07-25acdb148ddb |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Second of the stack for CORE-13316, stacked on #13895 (the policyscale generator); review the last commit only.
Adds the differential harness that gates every later change to the evaluation path:
TestEvaluateAgreesWithOraclerunsEvaluateover a corpus drawn from each policyscale preset (baseline, baseline with missing sets, the egress allow-list, and the composite) — the denied flow, flows aimed at every 257th rule, 250 sampled flows per direction following a flow model, and replays as UDP, with an invalid protocol and with nil addresses — and requires the trace to equal the one the fixture's oracle computes from its own model of the rules, in bothPolicyScopes.assertEquivalenttakes twoevaluators, so a second implementation of the walk (the compile-once path in Compile dikastes/collector policies once instead of interpreting per flow #13267, a verdict cache, evaluation on another goroutine) is checked againstEvaluatethe same way before it is switched on.TestEvaluateNamedCasesis the hand-built matrix for what a generated corpus cannot reach: nil addresses against positive and negated references, named and negated named ports, IP+port sets, negated and missing sets, protocols out of range / by name / by number / negated, staged policies in both scopes (including a tier that is all staged contributing nothing to the enforced verdict), HTTP criteria against an L4 flow, Log rules, Pass into the next tier and into profiles, and a policy missing from the store.runNamedCasestakes an evaluator for the same reason.No production code changes. The corpus test runs in about 5 s.
Testing
Release note:
AI assistance: Written with Claude Code (Claude Fable 5.1); the author reviewed the diff and ran the tests above.
By opening this PR you take responsibility for every line in it, and you agree to explain the change yourself during review rather than routing review comments back through an agent. See AI_POLICY.md.
🤖 Generated with Claude Code