Skip to content

feat(daemon): persist statusz-derived nft set membership so it survives a reboot #990

Description

@brunodam

Problem

nft set membership is deliberately never persisted. internal/network/policy/render.go:19-23
says so explicitly — the rendered artifact carries set schemas only, and the daemon's statusz
poll loop owns the contents. So after a reboot the boot oneshot replays
/etc/solo-provisioner/network-weaver-workload-policy.nft with every membership set declared
empty
, and they stay empty until the daemon completes its first successful poll.

The daemon already does what it can to shrink that window. traffic_shaper_monitor.go:330-345
reconciles once on entry rather than waiting for the first tick, with back-off retry, precisely
so a reboot converges quickly:

// Reconcile once on entry so the daemon converges immediately on startup
// (including after a host reboot) without waiting a full interval for the
// first tick. The nft set elements are not boot-persistent; this entry probe
// is what rehydrates them as soon as the daemon starts and the BN statusz
// endpoint is reachable.

But that only bounds the window by block-node startup time, which on a cold boot is however
long the BN pod takes to become ready and serve statusz — minutes. The nft loader has replayed
the table from local disk long before that. And on the pod-discovery path (no base_url
configured) the entry reconcile is a silent no-op, so convergence waits for pod discovery plus
up to one full poll interval on top; the comment at traffic_shaper_monitor.go:338-342 calls
this out.

What an empty set means differs sharply by set, and this is what makes it worth fixing:

Set Behaviour while empty Impact
bn-restricted no drop rule matches fail-open — peers the BN has declared restricted are not blocked
bn-publisher, bn-partner-out, bn-backfill stamp rules match nothing traffic lands in the HTB default class — unclassified, not dropped

The classification half is a bounded performance degradation. The bn-restricted half is a
security gap: for the first several minutes of every boot, the quarantine list is empty.

This also blocks #981. A loop that re-asserts the weaver tables after a third party removes them
will recreate the sets empty and have nothing to rebuild membership from, reproducing this exact
window on every re-assert rather than only at boot.

Proposed fix

Persist the last successfully-applied membership to a daemon-owned state file and replay it
at daemon startup, before the first poll.

Sketch, not a design:

  • Write on each successful Reconciler.Apply (internal/blocknode/shaper/reconciler.go:161),
    after ApplySets reports applied. The digest the reconciler already computes is a natural
    staleness marker to store alongside it.
  • Somewhere under /var/lib/solo-provisioner/, not /etc/solo-provisioner/. It is derived
    runtime state, not operator configuration.
  • Not rendered into the .nft artifact. Doing so would make the daemon rewrite the
    operator's artifact every tick, and in the firewall package's case would collide with
    parse.go recovering operator state by reading elements back out of the file.
  • Replayed by the daemon, not by the boot oneshot, so the "operator owns the artifact,
    daemon owns membership" split stays intact.

Open questions

Should classification sets be replayed at all, or only the deny sets? Staleness cuts opposite
ways. A stale restricted list over-blocks, which is the safe direction for a quarantine. A stale
partner list mis-stamps traffic for peers that are no longer partners — arguably worse than being
briefly unclassified, which is what happens today. A defensible first cut is deny sets only.

How stale is too stale? A node that has been off for a week would replay a week-old list. Some
bound (refuse to replay beyond N hours, log and start empty) may be wanted, at which point the
fail-open window returns for that case by choice rather than by omission.

Interaction with #978. If the statusz-derived restricted CIDRs also land in a host-firewall
set, that set needs the same treatment — and it is on the input hook, where the table is
enforcing, so the fail-open window there is more consequential than on forward.

Acceptance

  • Successfully-applied statusz-derived membership is persisted outside the .nft artifact
  • The daemon replays it at startup, before the first statusz poll completes
  • bn-restricted is populated from persisted state within seconds of boot, rather than
    waiting for the block node to come up and serve statusz
  • The persisted state is superseded by the first successful poll, and never wins over it
  • Which sets are replayed (deny only vs. all) is an explicit, documented decision
  • The operator-owned .nft artifacts are unchanged — nothing in this path rewrites them
  • internal/network/policy/render.go's "membership is never persisted" contract is updated
    to say where it now lives

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions