Problem
There is no policy governing when a solo-provisioner CLI upgrade should upgrade or
restart solo-provisioner-daemon. The current behaviour is not a decision — it is what
two unrelated early-returns happen to produce:
- A CLI upgrade never restarts the daemon.
ensureBlockNodeDaemon
(cmd/cli/commands/block/node/daemon_offer.go) returns early when the service is
running, so NewDaemonServiceInstallWorkflow — which ends in an unconditional
RestartService (internal/workflows/steps/step_daemon.go) — never runs on an
already-provisioned host.
- The daemon can never be upgraded through the catalog path.
InstallDaemonBinaryStep
skips the download when installer.IsInstalled() reports true, and that method returns
softwareState.Installed, a plain bool (pkg/software/base_installer.go). It is
version-agnostic: once a daemon has ever been installed, no newer one is ever fetched.
So the effective policy is "install once, never upgrade, never restart". That is a
defensible default, but nothing states it, nothing enforces it, and nothing reports when
the daemon on disk stops matching the daemon in memory.
The gap becomes visible now that CLI self-install places the co-built daemon binary on the
host (#973). The rename cannot disturb a running daemon — it keeps its own inode — but the
on-disk and running versions can now diverge with no surface reporting it. The failure mode
is not restart churn; it is an unrelated reboot silently adopting a daemon version nobody
decided to roll out.
Why version equality is the wrong trigger
The obvious fix — "CLI version changed, so replace the daemon" — treats CLI version
changed as a proxy for the daemon must change. Those are different questions, and the
proxy over-triggers: it forces a daemon replacement (and eventually a restart) on every
release, including the majority where no daemon code changed at all.
Content comparison cannot rescue it either. The version string is stamped into the daemon
binary via -ldflags (LDFLAGS in Taskfile.yaml), so every release produces a
byte-different daemon binary even when zero daemon source changed. Checksums cannot
discriminate "the daemon actually changed" from "the version stamp moved".
Proposed fix
Declare a compatibility contract rather than comparing versions for equality.
1. The CLI declares the daemon range it supports. A minDaemonVersion (and optionally
a max) as a field on the solo-provisioner-daemon catalog entry in
pkg/software/infrastructure-catalog.yaml, or an embedded constant. Bumping the minimum
becomes a deliberate release-time act — "this CLI needs the new daemon RPC / the new nft
set layout" — instead of an accident of the version stamp.
2. Decide against the running daemon, not the installed CLI. The daemon's --version
already emits JSON and the parser exists (daemonVersionOutput in step_daemon.go).
| Running daemon |
Action |
| Satisfies the declared range |
Nothing. No copy, no restart. |
| Below the minimum |
Required upgrade: replace the binary and restart, logging the reason explicitly (e.g. daemon 1.2.0 < minimum 1.3.0 required by CLI 1.5.0) |
| Not installed |
Install, as today |
pkg/semver.CheckVersionRequirements(progVersion, minimum, maximum) already exists and has
no production caller — it is exactly this primitive.
3. Never restart implicitly for an in-range daemon. Provide an explicit escape hatch
for operators who want the newest build regardless — solo-provisioner daemon service upgrade, or a --force-daemon-upgrade flag.
4. Make skew visible. daemon service check should report the running version, the
on-disk version, the CLI's required range, and whether a restart is pending. Whatever the
policy, silent divergence between disk and memory is the part that bites.
5. Make IsInstalled() version-aware, or stop using it as the upgrade gate. As long as
it is a bool, the catalog path can never deliver a newer daemon even when one is required.
Restart cost, for calibration
Worth recording, because it sets how conservative the policy needs to be: a daemon restart
is a brief reconciliation gap, not a traffic interruption.
- Nothing tears down enforcement on shutdown. The nft sets and tc qdiscs live in the
kernel; the monitors simply stop. Existing classification keeps being enforced while the
daemon is down.
- The pod-lifecycle watcher performs a full
List on startup and dispatches an upsert for
every existing block-node pod (internal/daemon/blocknode/pod_watcher.go), so veth HTB
state is re-asserted rather than lost.
- The shaper
Reconciler holds no authoritative in-memory state
(internal/blocknode/shaper/reconciler.go): it re-derives desired membership from
statusz, reads live nft, and writes only the diff.
The residual exposure is that membership changes published during the downtime are not
applied until the daemon returns. With the statusz poll interval at 5 min, a restart window
is small relative to the polling cadence itself.
This lowers the stakes of getting the policy wrong, but it does not make version-coupled
restarts desirable — it argues for a moderate policy rather than a paranoid one.
Acceptance
Related Issues
Problem
There is no policy governing when a
solo-provisionerCLI upgrade should upgrade orrestart
solo-provisioner-daemon. The current behaviour is not a decision — it is whattwo unrelated early-returns happen to produce:
ensureBlockNodeDaemon(
cmd/cli/commands/block/node/daemon_offer.go) returns early when the service isrunning, so
NewDaemonServiceInstallWorkflow— which ends in an unconditionalRestartService(internal/workflows/steps/step_daemon.go) — never runs on analready-provisioned host.
InstallDaemonBinaryStepskips the download when
installer.IsInstalled()reports true, and that method returnssoftwareState.Installed, a plain bool (pkg/software/base_installer.go). It isversion-agnostic: once a daemon has ever been installed, no newer one is ever fetched.
So the effective policy is "install once, never upgrade, never restart". That is a
defensible default, but nothing states it, nothing enforces it, and nothing reports when
the daemon on disk stops matching the daemon in memory.
The gap becomes visible now that CLI self-install places the co-built daemon binary on the
host (#973). The rename cannot disturb a running daemon — it keeps its own inode — but the
on-disk and running versions can now diverge with no surface reporting it. The failure mode
is not restart churn; it is an unrelated reboot silently adopting a daemon version nobody
decided to roll out.
Why version equality is the wrong trigger
The obvious fix — "CLI version changed, so replace the daemon" — treats CLI version
changed as a proxy for the daemon must change. Those are different questions, and the
proxy over-triggers: it forces a daemon replacement (and eventually a restart) on every
release, including the majority where no daemon code changed at all.
Content comparison cannot rescue it either. The version string is stamped into the daemon
binary via
-ldflags(LDFLAGSinTaskfile.yaml), so every release produces abyte-different daemon binary even when zero daemon source changed. Checksums cannot
discriminate "the daemon actually changed" from "the version stamp moved".
Proposed fix
Declare a compatibility contract rather than comparing versions for equality.
1. The CLI declares the daemon range it supports. A
minDaemonVersion(and optionallya max) as a field on the
solo-provisioner-daemoncatalog entry inpkg/software/infrastructure-catalog.yaml, or an embedded constant. Bumping the minimumbecomes a deliberate release-time act — "this CLI needs the new daemon RPC / the new nft
set layout" — instead of an accident of the version stamp.
2. Decide against the running daemon, not the installed CLI. The daemon's
--versionalready emits JSON and the parser exists (
daemonVersionOutputinstep_daemon.go).daemon 1.2.0 < minimum 1.3.0 required by CLI 1.5.0)pkg/semver.CheckVersionRequirements(progVersion, minimum, maximum)already exists and hasno production caller — it is exactly this primitive.
3. Never restart implicitly for an in-range daemon. Provide an explicit escape hatch
for operators who want the newest build regardless —
solo-provisioner daemon service upgrade, or a--force-daemon-upgradeflag.4. Make skew visible.
daemon service checkshould report the running version, theon-disk version, the CLI's required range, and whether a restart is pending. Whatever the
policy, silent divergence between disk and memory is the part that bites.
5. Make
IsInstalled()version-aware, or stop using it as the upgrade gate. As long asit is a bool, the catalog path can never deliver a newer daemon even when one is required.
Restart cost, for calibration
Worth recording, because it sets how conservative the policy needs to be: a daemon restart
is a brief reconciliation gap, not a traffic interruption.
kernel; the monitors simply stop. Existing classification keeps being enforced while the
daemon is down.
Liston startup and dispatches an upsert forevery existing block-node pod (
internal/daemon/blocknode/pod_watcher.go), so veth HTBstate is re-asserted rather than lost.
Reconcilerholds no authoritative in-memory state(
internal/blocknode/shaper/reconciler.go): it re-derives desired membership fromstatusz, reads live nft, and writes only the diff.
The residual exposure is that membership changes published during the downtime are not
applied until the daemon returns. With the statusz poll interval at 5 min, a restart window
is small relative to the polling cadence itself.
This lowers the stakes of getting the policy wrong, but it does not make version-coupled
restarts desirable — it argues for a moderate policy rather than a paranoid one.
Acceptance
binary replacement and no restart.
restarts, logging which bound was violated.
daemon service checkreports running version, on-disk version, required range, andpending-restart state.
(i.e. the install gate is no longer a version-agnostic bool).
Related Issues
self-install and surfaced the absence of an upgrade policy.