Problem
nft set membership is deliberately never persisted. internal/network/policy/render.go:19-23
says so explicitly — the rendered artifact carries set schemas only, and the daemon's statusz
poll loop owns the contents. So after a reboot the boot oneshot replays
/etc/solo-provisioner/network-weaver-workload-policy.nft with every membership set declared
empty, and they stay empty until the daemon completes its first successful poll.
The daemon already does what it can to shrink that window. traffic_shaper_monitor.go:330-345
reconciles once on entry rather than waiting for the first tick, with back-off retry, precisely
so a reboot converges quickly:
// Reconcile once on entry so the daemon converges immediately on startup
// (including after a host reboot) without waiting a full interval for the
// first tick. The nft set elements are not boot-persistent; this entry probe
// is what rehydrates them as soon as the daemon starts and the BN statusz
// endpoint is reachable.
But that only bounds the window by block-node startup time, which on a cold boot is however
long the BN pod takes to become ready and serve statusz — minutes. The nft loader has replayed
the table from local disk long before that. And on the pod-discovery path (no base_url
configured) the entry reconcile is a silent no-op, so convergence waits for pod discovery plus
up to one full poll interval on top; the comment at traffic_shaper_monitor.go:338-342 calls
this out.
What an empty set means differs sharply by set, and this is what makes it worth fixing:
| Set |
Behaviour while empty |
Impact |
bn-restricted |
no drop rule matches |
fail-open — peers the BN has declared restricted are not blocked |
bn-publisher, bn-partner-out, bn-backfill |
stamp rules match nothing |
traffic lands in the HTB default class — unclassified, not dropped |
The classification half is a bounded performance degradation. The bn-restricted half is a
security gap: for the first several minutes of every boot, the quarantine list is empty.
This also blocks #981. A loop that re-asserts the weaver tables after a third party removes them
will recreate the sets empty and have nothing to rebuild membership from, reproducing this exact
window on every re-assert rather than only at boot.
Proposed fix
Persist the last successfully-applied membership to a daemon-owned state file and replay it
at daemon startup, before the first poll.
Sketch, not a design:
- Write on each successful
Reconciler.Apply (internal/blocknode/shaper/reconciler.go:161),
after ApplySets reports applied. The digest the reconciler already computes is a natural
staleness marker to store alongside it.
- Somewhere under
/var/lib/solo-provisioner/, not /etc/solo-provisioner/. It is derived
runtime state, not operator configuration.
- Not rendered into the
.nft artifact. Doing so would make the daemon rewrite the
operator's artifact every tick, and in the firewall package's case would collide with
parse.go recovering operator state by reading elements back out of the file.
- Replayed by the daemon, not by the boot oneshot, so the "operator owns the artifact,
daemon owns membership" split stays intact.
Open questions
Should classification sets be replayed at all, or only the deny sets? Staleness cuts opposite
ways. A stale restricted list over-blocks, which is the safe direction for a quarantine. A stale
partner list mis-stamps traffic for peers that are no longer partners — arguably worse than being
briefly unclassified, which is what happens today. A defensible first cut is deny sets only.
How stale is too stale? A node that has been off for a week would replay a week-old list. Some
bound (refuse to replay beyond N hours, log and start empty) may be wanted, at which point the
fail-open window returns for that case by choice rather than by omission.
Interaction with #978. If the statusz-derived restricted CIDRs also land in a host-firewall
set, that set needs the same treatment — and it is on the input hook, where the table is
enforcing, so the fail-open window there is more consequential than on forward.
Acceptance
Problem
nft set membership is deliberately never persisted.
internal/network/policy/render.go:19-23says so explicitly — the rendered artifact carries set schemas only, and the daemon's statusz
poll loop owns the contents. So after a reboot the boot oneshot replays
/etc/solo-provisioner/network-weaver-workload-policy.nftwith every membership set declaredempty, and they stay empty until the daemon completes its first successful poll.
The daemon already does what it can to shrink that window.
traffic_shaper_monitor.go:330-345reconciles once on entry rather than waiting for the first tick, with back-off retry, precisely
so a reboot converges quickly:
But that only bounds the window by block-node startup time, which on a cold boot is however
long the BN pod takes to become ready and serve statusz — minutes. The nft loader has replayed
the table from local disk long before that. And on the pod-discovery path (no
base_urlconfigured) the entry reconcile is a silent no-op, so convergence waits for pod discovery plus
up to one full poll interval on top; the comment at
traffic_shaper_monitor.go:338-342callsthis out.
What an empty set means differs sharply by set, and this is what makes it worth fixing:
bn-restrictedbn-publisher,bn-partner-out,bn-backfillThe classification half is a bounded performance degradation. The
bn-restrictedhalf is asecurity gap: for the first several minutes of every boot, the quarantine list is empty.
This also blocks #981. A loop that re-asserts the weaver tables after a third party removes them
will recreate the sets empty and have nothing to rebuild membership from, reproducing this exact
window on every re-assert rather than only at boot.
Proposed fix
Persist the last successfully-applied membership to a daemon-owned state file and replay it
at daemon startup, before the first poll.
Sketch, not a design:
Reconciler.Apply(internal/blocknode/shaper/reconciler.go:161),after
ApplySetsreports applied. The digest the reconciler already computes is a naturalstaleness marker to store alongside it.
/var/lib/solo-provisioner/, not/etc/solo-provisioner/. It is derivedruntime state, not operator configuration.
.nftartifact. Doing so would make the daemon rewrite theoperator's artifact every tick, and in the firewall package's case would collide with
parse.gorecovering operator state by reading elements back out of the file.daemon owns membership" split stays intact.
Open questions
Should classification sets be replayed at all, or only the deny sets? Staleness cuts opposite
ways. A stale restricted list over-blocks, which is the safe direction for a quarantine. A stale
partner list mis-stamps traffic for peers that are no longer partners — arguably worse than being
briefly unclassified, which is what happens today. A defensible first cut is deny sets only.
How stale is too stale? A node that has been off for a week would replay a week-old list. Some
bound (refuse to replay beyond N hours, log and start empty) may be wanted, at which point the
fail-open window returns for that case by choice rather than by omission.
Interaction with #978. If the statusz-derived restricted CIDRs also land in a host-firewall
set, that set needs the same treatment — and it is on the
inputhook, where the table isenforcing, so the fail-open window there is more consequential than on
forward.Acceptance
.nftartifactbn-restrictedis populated from persisted state within seconds of boot, rather thanwaiting for the block node to come up and serve statusz
.nftartifacts are unchanged — nothing in this path rewrites theminternal/network/policy/render.go's "membership is never persisted" contract is updatedto say where it now lives