felix/collector: collector-level benchmark of the pending-policy evaluation - #13899
Draft
dimitri-nicolo wants to merge 4 commits into
Draft
felix/collector: collector-level benchmark of the pending-policy evaluation#13899dimitri-nicolo wants to merge 4 commits into
dimitri-nicolo wants to merge 4 commits into
Conversation
…benchmarks Lift the two synthetic policy sets out of the checker benchmarks into a package that renders the same set three ways: a policystore.PolicyStore and endpoint for engine and collector benchmarks, the ToDataplane updates that load it through ProcessUpdate, and Calico resources (Tier, GlobalNetworkPolicy, GlobalNetworkSet) for applying it to a cluster. A number measured at the engine, in the collector and on a node then describes the same policy set. The fixture also carries an oracle: Fixture.Expect computes the trace the engine must report for a flow from the generator's own model of each rule, independently of the matching code, and MatchingFlow / DeniedFlow / Sampler produce flows aimed at chosen depths of the walk or following a flow model (miss fraction, repeat fraction). Later changes to the engine (compiled policies, a verdict cache, evaluation off the collector's main loop) are gated on differential tests built on these. Composite applies both measured shapes to one endpoint: 19,992 ingress rules, 18,662 egress rules, 7,571 IP sets. BenchmarkEvaluateComposite measures it in the three cases that matter for the collector; the two existing benchmarks keep their cases and their numbers. hack/cmd/policyscale renders a preset as YAML, or prints sampled flows with their expected verdicts, for node-level runs. Tracking: CORE-13316.
…yscale oracle Run Evaluate over a corpus drawn from each policyscale preset (the denied flow, flows aimed at every 257th rule, sampled flows following a flow model, and replays as UDP, with an invalid protocol and with nil addresses) and require the trace to equal the one the fixture's oracle computes from its own model of the rules, in both policy scopes. The harness takes two evaluators, so a second implementation of the walk (compiled policies, a verdict cache, evaluation on another goroutine) is checked against Evaluate the same way before it is switched on. The named cases cover what a generated corpus cannot reach: nil addresses against positive and negated references, named and negated named ports, IP+port sets, negated and missing sets, protocols out of range, by name, by number and negated, staged policies in both scopes, HTTP criteria against an L4 flow, Log rules, Pass into the next tier and into profiles, and a policy missing from the store. runNamedCases takes an evaluator for the same reason. Tracking: CORE-13316.
…neration
A verdict is a function of the endpoint's applicable rules, the store's
IP sets and the flow. For a flow with no L7 attributes (every flow the
collector evaluates) the criteria in match() reduce to protocol, source
and destination address, destination port and, only where a rule looks
at it, source port; identity and HTTP criteria match such a flow
whatever the rule says. So a store can remember the verdict for that
key and answer a flow that repeats an earlier flow's endpoints on a new
source port without walking the policy set, which on a large policy
set is the difference between a millisecond and a map lookup.
PolicyStore gains a Generation that ProcessUpdate moves on every update
but InSync, and an optional VerdictCache bound to it: entries are valid
for one generation and the cache starts over when the store moves on,
so a cached verdict never outlives the policies, IP sets or endpoints it
was computed from. Whether the source port is part of the key is decided
once per endpoint, scope, direction and generation by walking the
endpoint's applicable rules. The cache is bounded and starts over when
full. Evaluate consults it when the store has one; failed evaluations
are not cached; flows carrying identity or HTTP attributes (Dikastes
requests) and flows with a nil address bypass it.
The collector gets the cache through a PolicyStoreManager option so that
every store the manager creates, including on reconnect, carries one,
and reports it as felix_collector_policy_eval_cache_{hits,misses,resets,
evictions}_total. FlowLogsPolicyEvaluationCacheSize sizes it (default
65536; 0 disables). It is a config-file / environment-variable parameter
with no FelixConfiguration field yet, so that it picks to release
branches without an API change; the field can follow on master.
On the composite reference set (18,662 egress rules) with the sampler's
flow model, per evaluation: 1.00 ms uncached; 0.72 ms cached with no
deliberate repeats (11% hit ratio from coincidental ones), 0.31 ms at a
50% repeat rate (62% hits), 0.08 ms at 90% (92% hits).
Tests: the differential corpus and the named cases through a cached
store, twice, against an uncached one; invalidation on IP set delta,
policy update and policy removal through ProcessUpdate; source-port
keying; L7 and nil-address bypass; generation, capacity and option
behaviour in policystore; collector wiring.
Tracking: CORE-13316.
…uation
BenchmarkCollectorPolicyEval drives a real collector, wired for
evaluation only (a lookups cache that knows one local workload, a policy
store loaded with the policyscale composite set and that workload's
endpoint, no readers, no reporters, main loop not running), with
conntrack updates for flows drawn from the sampler's model, and
measures the two paths on the collector's main goroutine: handleCtInfo
for a new tuple, which creates the Data and evaluates it inline, and one
full re-evaluation sweep over a fixed live population. Each reports
ns/flow, the flow rate one goroutine could sustain at that cost, and the
verdict cache's hit ratio when caching.
This is the K2 level of the benchmark definition in
felix/design/flow-logs-policy-evaluation.md: the number that says
whether the collector, not just the engine, meets the per-node target.
On one laptop core, 200 new flows per case:
NewFlow/Egress/Uncached 1.24 ms per flow ~800 flows/s
NewFlow/Ingress/Uncached 0.82 ms per flow ~1,200 flows/s
NewFlow/Egress/Cached 0.38 ms per flow ~2,600 flows/s (59% hits)
NewFlow/Ingress/Cached 0.09 ms per flow ~10,700 flows/s (93% hits)
Sweep/Uncached 1.04 ms per live flow
Sweep/Cached 0.48 us per live flow (a sweep over an
unchanged store is all hits)
Tracking: CORE-13316.
dimitri-nicolo
added
docs-not-required
Contributor
CI triage — CalicoRecommendation: Dig deeper before merging Likely caused by this PR
workflow_id: 19805093-7628-4726-be0e-8ebb03e3713f |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fourth of the stack for CORE-13316, stacked on #13897; review the last commit only.
BenchmarkCollectorPolicyEvalis the collector-level (K2) benchmark from the benchmark definition: it drives a realcollector— wired for evaluation only, with a lookups cache that knows one local workload, a policy store loaded with the policyscale composite set and that workload's endpoint, no readers and no reporters, main loop not running — with conntrack updates for flows drawn from the sampler's model (a tenth miss every rule, the rest aimed uniformly at the walk, half repeating an earlier flow's endpoints on a new source port), and measures the two paths on the collector's main goroutine:handleCtInfofor a tuple the collector has not seen, which creates theDataand evaluates it inline.ns/flowis the main-loop cost of a new flow;flows/sthe rate one goroutine could sustain if it did nothing else.ns/flowis the sweep's cost per live flow.Both with and without the verdict cache from #13897. Tuples are made unique so every update is a new flow; the local workload's address is outside every generated CIDR and IP set so the walk depends only on the remote address and ports.
Numbers (one laptop core,
-benchtime 200xfor NewFlow,3xfor Sweep):Three things the numbers say. The uncached new-flow cost on this set is about a millisecond, i.e. the few-hundred-flows-per-second ceiling reported from the field, reproduced on a laptop with no cluster. The cache buys what the traffic's repeat rate allows and no more, which is why the engine work in #13898 and the off-main-loop design still matter for the miss path. And a sweep over a store that has not changed is almost free with the cache, because every flow is a hit; on a busy cluster the store changes every few seconds and empties it, which is exactly the case the planned sweep invalidation addresses.
Testing
felix/deps.txtis regenerated because the collector's tests now importapp-policy/policyscale.Release note:
AI assistance: Written with Claude Code (Claude Fable 5.1); the author reviewed the diff and ran the benchmarks above.
By opening this PR you take responsibility for every line in it, and you agree to explain the change yourself during review rather than routing review comments back through an agent. See AI_POLICY.md.
🤖 Generated with Claude Code