ci: policy evaluation benchmark block publishing to the Lens trend store - #13901
Draft
dimitri-nicolo wants to merge 6 commits into
Draft
ci: policy evaluation benchmark block publishing to the Lens trend store#13901dimitri-nicolo wants to merge 6 commits into
dimitri-nicolo wants to merge 6 commits into
Conversation
…benchmarks Lift the two synthetic policy sets out of the checker benchmarks into a package that renders the same set three ways: a policystore.PolicyStore and endpoint for engine and collector benchmarks, the ToDataplane updates that load it through ProcessUpdate, and Calico resources (Tier, GlobalNetworkPolicy, GlobalNetworkSet) for applying it to a cluster. A number measured at the engine, in the collector and on a node then describes the same policy set. The fixture also carries an oracle: Fixture.Expect computes the trace the engine must report for a flow from the generator's own model of each rule, independently of the matching code, and MatchingFlow / DeniedFlow / Sampler produce flows aimed at chosen depths of the walk or following a flow model (miss fraction, repeat fraction). Later changes to the engine (compiled policies, a verdict cache, evaluation off the collector's main loop) are gated on differential tests built on these. Composite applies both measured shapes to one endpoint: 19,992 ingress rules, 18,662 egress rules, 7,571 IP sets. BenchmarkEvaluateComposite measures it in the three cases that matter for the collector; the two existing benchmarks keep their cases and their numbers. hack/cmd/policyscale renders a preset as YAML, or prints sampled flows with their expected verdicts, for node-level runs. Tracking: CORE-13316.
…yscale oracle Run Evaluate over a corpus drawn from each policyscale preset (the denied flow, flows aimed at every 257th rule, sampled flows following a flow model, and replays as UDP, with an invalid protocol and with nil addresses) and require the trace to equal the one the fixture's oracle computes from its own model of the rules, in both policy scopes. The harness takes two evaluators, so a second implementation of the walk (compiled policies, a verdict cache, evaluation on another goroutine) is checked against Evaluate the same way before it is switched on. The named cases cover what a generated corpus cannot reach: nil addresses against positive and negated references, named and negated named ports, IP+port sets, negated and missing sets, protocols out of range, by name, by number and negated, staged policies in both scopes, HTTP criteria against an L4 flow, Log rules, Pass into the next tier and into profiles, and a policy missing from the store. runNamedCases takes an evaluator for the same reason. Tracking: CORE-13316.
…neration
A verdict is a function of the endpoint's applicable rules, the store's
IP sets and the flow. For a flow with no L7 attributes (every flow the
collector evaluates) the criteria in match() reduce to protocol, source
and destination address, destination port and, only where a rule looks
at it, source port; identity and HTTP criteria match such a flow
whatever the rule says. So a store can remember the verdict for that
key and answer a flow that repeats an earlier flow's endpoints on a new
source port without walking the policy set, which on a large policy
set is the difference between a millisecond and a map lookup.
PolicyStore gains a Generation that ProcessUpdate moves on every update
but InSync, and an optional VerdictCache bound to it: entries are valid
for one generation and the cache starts over when the store moves on,
so a cached verdict never outlives the policies, IP sets or endpoints it
was computed from. Whether the source port is part of the key is decided
once per endpoint, scope, direction and generation by walking the
endpoint's applicable rules. The cache is bounded and starts over when
full. Evaluate consults it when the store has one; failed evaluations
are not cached; flows carrying identity or HTTP attributes (Dikastes
requests) and flows with a nil address bypass it.
The collector gets the cache through a PolicyStoreManager option so that
every store the manager creates, including on reconnect, carries one,
and reports it as felix_collector_policy_eval_cache_{hits,misses,resets,
evictions}_total. FlowLogsPolicyEvaluationCacheSize sizes it (default
65536; 0 disables). It is a config-file / environment-variable parameter
with no FelixConfiguration field yet, so that it picks to release
branches without an API change; the field can follow on master.
On the composite reference set (18,662 egress rules) with the sampler's
flow model, per evaluation: 1.00 ms uncached; 0.72 ms cached with no
deliberate repeats (11% hit ratio from coincidental ones), 0.31 ms at a
50% repeat rate (62% hits), 0.08 ms at 90% (92% hits).
Tests: the differential corpus and the named cases through a cached
store, twice, against an uncached one; invalidation on IP set delta,
policy update and policy removal through ProcessUpdate; source-port
keying; L7 and nil-address bypass; generation, capacity and option
behaviour in policystore; collector wiring.
Tracking: CORE-13316.
…uation
BenchmarkCollectorPolicyEval drives a real collector, wired for
evaluation only (a lookups cache that knows one local workload, a policy
store loaded with the policyscale composite set and that workload's
endpoint, no readers, no reporters, main loop not running), with
conntrack updates for flows drawn from the sampler's model, and
measures the two paths on the collector's main goroutine: handleCtInfo
for a new tuple, which creates the Data and evaluates it inline, and one
full re-evaluation sweep over a fixed live population. Each reports
ns/flow, the flow rate one goroutine could sustain at that cost, and the
verdict cache's hit ratio when caching.
This is the K2 level of the benchmark definition in
felix/design/flow-logs-policy-evaluation.md: the number that says
whether the collector, not just the engine, meets the per-node target.
On one laptop core, 200 new flows per case:
NewFlow/Egress/Uncached 1.24 ms per flow ~800 flows/s
NewFlow/Ingress/Uncached 0.82 ms per flow ~1,200 flows/s
NewFlow/Egress/Cached 0.38 ms per flow ~2,600 flows/s (59% hits)
NewFlow/Ingress/Cached 0.09 ms per flow ~10,700 flows/s (93% hits)
Sweep/Uncached 1.04 ms per live flow
Sweep/Cached 0.48 us per live flow (a sweep over an
unchanged store is all hits)
Tracking: CORE-13316.
The design doc for the user-mode policy evaluation the flow-log collector runs for pending (staged) policy verdicts: the two evaluation paths and the invariants they keep, the performance target and the three-level benchmark (engine, collector, node) that defines it, the correctness gates every new evaluator runs through, the verdict cache, the planned off-main-loop evaluation and sweep invalidation, the backport-first constraints, and the failure modes a reviewer should look for. Indexed from felix/DESIGN.md; the collector as a whole stays with the flow-logs-collector sub-design. Tracking: CORE-13316.
A Semaphore block that runs the engine (K1) and collector (K2) benchmarks of the pending-policy evaluation on the policyscale reference set whenever app-policy, felix/collector or hack/perf change, and in the scheduled builds, and publishes one hack/perf document per case to the Lens trend store through send-perf-results, as the nftables dataplane benchmark does. Trend tracking, not a per-PR gate. hack/perf/perfdoc is the producer side of the hack/perf contract for Go benchmarks: a Recorder measures the timed loop (wall ns, bytes and allocs per op) and writes <dir>/<family>/<name>.json when the artifacts directory is set. Both benchmarks use it under POLICY_EVAL_PERF_ARTIFACTS_DIR; make bench-policy-eval in app-policy and felix runs them in go-build with the variable set. The benchmark_data_policy_eval index template pins the field types. Tracking: CORE-13316.
dimitri-nicolo
added
docs-not-required
Contributor
CI triage — CalicoRecommendation: Dig deeper before merging Likely caused by this PR
workflow_id: 12e3e554-ef1f-41de-bd67-bbb098be617d |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Sixth of the stack for CORE-13316, stacked on #13900; review the last commit only.
Adds a Semaphore block, Policy evaluation benchmark, that runs the engine (K1,
BenchmarkEvaluateCompositeandBenchmarkEvaluateVerdictCacheinapp-policy/checker) and collector (K2,BenchmarkCollectorPolicyEvalinfelix/collector) benchmarks of the pending-policy evaluation on the policyscale reference set wheneverapp-policy,felix/collectororhack/perfchange, and in the scheduled builds, and publishes one hack/perf document per case to the Lens trend store throughsend-perf-results. Same shape as the nftables dataplane benchmark block (#45): trend tracking so that a regression in the evaluation path shows up as a step on a graph, not a per-PR gate.hack/perf/perfdocis the producer side of thehack/perfcontract for Go benchmarks:Start/Finishmeasure the timed loop (wall ns, bytes, allocs per op) and write<dir>/<family>/<name>.jsonwhen the artifacts directory is set; nothing happens when it is not, which is the default outside CI.POLICY_EVAL_PERF_ARTIFACTS_DIRwith the scenario scalars (case, direction, cached, scale, ns/flow, flows/s, hit ratio).make bench-policy-evalinapp-policyandfelixruns them in go-build with the variable set toartifacts/perf.hack/perf/index-templates/benchmark_data_policy_eval.jsonpins the field types for the new family..semaphore/semaphore.ymland.semaphore/semaphore-scheduled-builds.ymlare regenerated from the block.Testing
The block itself has not run in CI yet (it will on this PR, since it changes
app-policyandhack/perf). Lens credentials come from the samebanzai-secretssecret the nftables block uses.Release note:
AI assistance: Written with Claude Code (Claude Fable 5.1); the author reviewed the diff and ran the commands above.
By opening this PR you take responsibility for every line in it, and you agree to explain the change yourself during review rather than routing review comments back through an agent. See AI_POLICY.md.
🤖 Generated with Claude Code