recovery drill

.context/TASKS.md

@@ -36,7 +36,7 @@ the name-based policy work.
 
 ### Phase 1: Correctness & Broken Things `#priority:high`
 - [x] Fix broken CI integration test #source:jira.xml #added:2026-07-14 #done:2026-07-16 (stale: CI has been green for several weeks; closed on user confirmation)
-- [ ] Fix broken recovery/restore flow #source:jira.xml #added:2026-07-14 (code review 2026-07-16: no live breakage found; awaiting the scripted drill in Phase 3 to confirm and close)
+- [x] Fix broken recovery/restore flow #source:jira.xml #added:2026-07-14 #done:2026-07-16 (the drill exposed a real boot-order deadlock: an un-initialized Nexus never started its listener, so the emergency restore route was unreachable when every Keeper died; fixed by recovering from Keepers in the background, and the recovery loop now stands down once an operator restore supplies the root key. Verified live by make drill-recovery)
 - [x] Fix `spike cipher` stream mode (broken; owner: Murat); JSON mode fix unblocks encryption-as-a-service demo/docs #source:jira.xml #added:2026-07-14 #done:2026-07-15 (stale: both cipher streaming and file modes verified passing via the make start checks on 2026-07-15)
 - [ ] Retry sqlite operations with exponential backoff on transient locks (all of `app/nexus/internal/state/persist`) → ideas/research-db-resilience.md #source:jira.xml #added:2026-07-14
 - [ ] Bound the Bootstrap keeper-wait loop with a configurable timeout/max-attempts instead of looping forever → ideas/research-db-resilience.md #source:jira.xml #added:2026-07-14
@@ -49,8 +49,8 @@ the name-based policy work.
 - [ ] Make `make test` concurrent again (currently serialized by env setup) → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14
 - [ ] Add integration tests: root key cached/recovered/not-re-initialized; secret & policy CRUD; Pilot denies when Nexus uninitialized / warns when unreachable → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14
 - [ ] Raise CLI command coverage to 60%+ via unit + HTTP-mock tests; fix `t.Skip()`ed tests; DI-refactor `sendShardsToKeepers` → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14
-- [ ] `start.sh` should exercise recovery/restore and encryption/decryption #source:jira.xml #added:2026-07-14
-- [ ] Scripted live recovery/restore drill: once `make start` completes cleanly, run `spike operator recover`, kill Nexus and the Keepers, restart Nexus alone, feed the shards back via `spike operator restore` (scriptable via stdin since fix/operator-restore), and verify a pre-crash secret reads back. Rationale: the 2026-07-16 code review found no live breakage (shard-index fidelity intact end to end; guards use exact SPIFFE role matching, unaffected by the policy-name migration), so only a drill can prove the Phase 1 "recovery/restore is broken" claim stale and close both tasks. Needs the recover/restore role entries (spire-server-entry-recover-register.sh / -restore-register.sh), which make start does not register by default. #added:2026-07-16
+- [x] `start.sh` should exercise recovery/restore and encryption/decryption #source:jira.xml #added:2026-07-14 #done:2026-07-16 (encryption/decryption checks live in start.sh since the policy-validation rework; recovery/restore is exercised by make drill-recovery, kept as a separate second-terminal script deliberately so the crash simulation never runs inside the normal startup path)
+- [x] Scripted live recovery/restore drill: once `make start` completes cleanly, run `spike operator recover`, kill Nexus and the Keepers, restart Nexus alone, feed the shards back via `spike operator restore` (scriptable via stdin since fix/operator-restore), and verify a pre-crash secret reads back. Rationale: the 2026-07-16 code review found no live breakage (shard-index fidelity intact end to end; guards use exact SPIFFE role matching, unaffected by the policy-name migration), so only a drill can prove the Phase 1 "recovery/restore is broken" claim stale and close both tasks. Needs the recover/restore role entries (spire-server-entry-recover-register.sh / -restore-register.sh), which make start does not register by default. #added:2026-07-16 #done:2026-07-16 (implemented as hack/bare-metal/drill/recovery-drill.sh behind make drill-recovery; the drill first exposed the Nexus boot-order deadlock, then passed end to end once it was fixed)
 
 ### Phase 4: Policy & Secrets `#priority:medium`
 - [ ] Add a `list` permission type; scope `spike secret list` to the caller's allowed path patterns → ideas/research-list-permission.md #source:jira.xml #added:2026-07-14

app/nexus/internal/initialization/initialization.go

@@ -17,8 +17,10 @@ import (
 // backend store type. The function handles three initialization modes:
 //
 // 1. SPIKE-Keeper-based initialization (SQLite and Lite backends):
-//   - Initializes the backing store from SPIKE Keeper instances
+//   - Initializes the backing store from SPIKE Keeper instances in the
+//     background, so the caller can start serving immediately
 //   - Starts a background goroutine for periodic shard synchronization
+//     once the backing store is initialized
 //
 // 2. In-memory initialization (Memory backend):
 //   - Initializes an empty in-memory backing store without the root key
@@ -46,16 +48,21 @@ func Initialize(source *workloadapi.X509Source) {
 
 	if requireBackingStoreToBootstrap := env.BackendStoreTypeVal() == env.Sqlite ||
 		env.BackendStoreTypeVal() == env.Lite; requireBackingStoreToBootstrap {
-		// Initialize the backing store from SPIKE Keeper instances.
-		// This is only required when the SPIKE Nexus needs bootstrapping.
-		// For modes where bootstrapping is not required (such as in-memory mode),
-		// SPIKE Nexus should be initialized internally.
-		recovery.InitializeBackingStoreFromKeepers(source)
+		// Initialize the backing store from SPIKE Keeper instances in the
+		// background so the caller can start the mTLS listener right away.
+		// While the root key is missing, the router only exposes the
+		// operator recover/restore emergency routes (see route/base), so
+		// serving early is safe. It is also required: a break-the-glass
+		// restore feeds shards through this very listener when every
+		// SPIKE Keeper has lost its shard. Blocking here would deadlock
+		// that flow, because SPIKE Nexus would never start listening.
+		go func() {
+			recovery.InitializeBackingStoreFromKeepers(source)
 
-		// Lazy evaluation in a loop:
-		// If bootstrapping is successful, start a background process to
-		// periodically sync shards.
-		go recovery.SendShardsPeriodically(source)
+			// If bootstrapping is successful, start a background process
+			// to periodically sync shards.
+			go recovery.SendShardsPeriodically(source)
+		}()
 
 		return
 	}

app/nexus/internal/initialization/recovery/recovery.go

@@ -76,6 +76,16 @@ func InitializeBackingStoreFromKeepers(source *workloadapi.X509Source) {
 	_, err := retry.Forever(ctx, func() (bool, *sdkErrors.SDKError) {
 		log.Debug(fName, "message", "retry attempt", "time", time.Now().String())
 
+		// The operator may have restored the root key through the
+		// emergency restore route while this loop was retrying. In that
+		// case initialization already happened; stop polling the SPIKE
+		// Keepers and let the periodic shard sync take over.
+		if !state.RootKeyZero() {
+			log.Info(fName,
+				"message", "root key already present; skipping recovery")
+			return true, nil
+		}
+
 		// Early check: avoid unnecessary function call if the source is nil
 		if source == nil {
 			warnErr := *sdkErrors.ErrSPIFFENilX509Source.Clone()

app/nexus/internal/initialization/recovery/standdown_test.go

@@ -0,0 +1,54 @@
+//    \\ SPIKE: Secure your secrets with SPIFFE. — https://spike.ist/
+//  \\\\\ Copyright 2024-present SPIKE contributors.
+// \\\\\\\ SPDX-License-Identifier: Apache-2.0
+
+package recovery
+
+import (
+	"testing"
+	"time"
+
+	"github.com/spiffe/spike-sdk-go/crypto"
+	"github.com/spiffe/spike-sdk-go/security/mem"
+
+	state "github.com/spiffe/spike/app/nexus/internal/state/base"
+)
+
+// TestInitializeBackingStoreFromKeepers_RootKeyAlreadyPresent verifies
+// that the Keeper recovery loop stands down promptly when the root key
+// is already present, which is what happens when an operator restores
+// the system through the emergency restore route while the loop is
+// still retrying. Before the standdown check, this call would retry
+// forever (the nil source alone never stops it), so the test guards
+// the boot path that keeps the restore flow deadlock-free.
+func TestInitializeBackingStoreFromKeepers_RootKeyAlreadyPresent(
+	t *testing.T,
+) {
+	rk := &[crypto.AES256KeySize]byte{}
+	for i := range rk {
+		rk[i] = byte(i + 1)
+	}
+	state.SetRootKey(rk)
+
+	// SetRootKey rejects zero keys by design, so zero the key directly
+	// under the lock, the same way the state package's own tests reset it.
+	t.Cleanup(func() {
+		state.LockRootKey()
+		defer state.UnlockRootKey()
+		mem.ClearRawBytes(state.RootKeyNoLock())
+	})
+
+	done := make(chan struct{})
+	go func() {
+		InitializeBackingStoreFromKeepers(nil)
+		close(done)
+	}()
+
+	select {
+	case <-done:
+		// The loop noticed the root key and returned.
+	case <-time.After(5 * time.Second):
+		t.Fatal("InitializeBackingStoreFromKeepers did not stand down" +
+			" although the root key is present")
+	}
+}

hack/bare-metal/drill/recovery-drill.sh

@@ -0,0 +1,247 @@
+#!/usr/bin/env bash
+
+#    \\ SPIKE: Secure your secrets with SPIFFE. — https://spike.ist/
+#  \\\\\ Copyright 2024-present SPIKE contributors.
+# \\\\\\\ SPDX-License-Identifier: Apache-2.0
+
+# Live recovery/restore drill for the bare-metal dev environment.
+#
+# Run this from the repository root, in a second terminal, after
+# `make start` has completed cleanly (SPIRE server and agent, three
+# Keepers, and Nexus with a persistent backend are all running). Run it
+# from the same shell environment as `make start` so a restarted Nexus
+# inherits the same configuration.
+#
+# The drill proves the break-the-glass runbook end to end
+# (see https://spike.ist/operations/recovery/):
+#
+#   1. Writes a marker secret while the system is healthy.
+#   2. Exports recovery shards via `spike operator recover`.
+#   3. Simulates a crash: kills Nexus and all Keepers (the Keepers lose
+#      their in-memory shards, so auto-recovery is impossible).
+#   4. Restarts Nexus alone and feeds the shards back one by one via
+#      `spike operator restore`, using its non-interactive stdin mode.
+#   5. Verifies the marker secret reads back after the restore.
+#
+# The Pilot's SPIRE entry is rotated through the recover and restore
+# roles along the way; a trap reverts it to the superuser role on every
+# exit path. The Keepers stay down when the drill ends: run `make kill`
+# and `make start` afterward to return to a pristine environment.
+
+set -u
+
+DRILL_MARKER_PATH="drill/recovery-marker"
+DRILL_MARKER_VALUE="drill-$(date +%s)-$$"
+RECOVERY_DIR="${SPIKE_PILOT_RECOVERY_DIR:-$HOME/.spike/recover}"
+NEXUS_LOG="$(mktemp -t spike-drill-nexus.XXXXXX.log)"
+
+say() {
+  echo ""
+  echo "drill: $*"
+}
+
+fail() {
+  echo "" >&2
+  echo "drill: FAIL: $*" >&2
+  exit 1
+}
+
+# Returns the Pilot's current role by asking the SPIRE server which of
+# the known role SPIFFE IDs has a registration entry.
+current_pilot_role() {
+  for role in superuser recover restore; do
+    if spire-server entry show \
+      -spiffeID "spiffe://spike.ist/spike/pilot/role/$role" 2>/dev/null |
+      grep -q "Entry ID"; then
+      echo "$role"
+      return 0
+    fi
+  done
+  echo "unknown"
+}
+
+# Reverts the Pilot's entry to the superuser role no matter which role
+# the drill died in. Runs on every exit path via the trap below.
+ensure_superuser_role() {
+  case "$(current_pilot_role)" in
+  recover)
+    say "reverting the Pilot entry from recover to superuser..."
+    ./hack/bare-metal/entry/spire-server-entry-recover-revert.sh
+    ;;
+  restore)
+    say "reverting the Pilot entry from restore to superuser..."
+    ./hack/bare-metal/entry/spire-server-entry-restore-revert.sh
+    ;;
+  superuser)
+    :
+    ;;
+  *)
+    echo "drill: WARNING: could not determine the Pilot role;" >&2
+    echo "drill: restore it manually with" >&2
+    echo "drill:   ./hack/bare-metal/entry/spire-server-entry-su-register.sh" >&2
+    ;;
+  esac
+}
+
+trap ensure_superuser_role EXIT
+
+# Retries a command until it succeeds or the attempts run out.
+retry() {
+  local attempts="$1"
+  local delay="$2"
+  shift 2
+
+  local i
+  for ((i = 1; i <= attempts; i++)); do
+    if "$@"; then
+      return 0
+    fi
+    sleep "$delay"
+  done
+  return 1
+}
+
+# --- Preflight ------------------------------------------------------------
+
+[ -x ./bin/spike ] || fail "run this from the repository root" \
+  "(./bin/spike not found; did make start build the binaries?)"
+
+for b in spike spire-server; do
+  command -v "$b" >/dev/null 2>&1 || fail "'$b' is not on PATH"
+done
+
+command -v spike | xargs test "$(pwd)/bin/spike" -ef ||
+  fail "PATH resolves 'spike' outside $(pwd)/bin"
+
+pgrep -x nexus >/dev/null || fail "Nexus is not running (run make start)"
+pgrep -x keeper >/dev/null || fail "no Keepers running (run make start)"
+pgrep -x spire-server >/dev/null || fail "SPIRE server is not running"
+
+if [ "${SPIKE_NEXUS_BACKEND_STORE:-}" = "memory" ]; then
+  fail "the drill needs a persistent backend;" \
+    "unset SPIKE_NEXUS_BACKEND_STORE"
+fi
+
+if [ "$(current_pilot_role)" != "superuser" ]; then
+  fail "the Pilot entry is not in the superuser role;" \
+    "revert it before running the drill"
+fi
+
+# --- 1. Write a marker secret while healthy -------------------------------
+
+say "writing the marker secret ($DRILL_MARKER_PATH)..."
+spike secret put "$DRILL_MARKER_PATH" value="$DRILL_MARKER_VALUE" ||
+  fail "could not write the marker secret"
+
+# Note: the Pilot prints its output on stderr (cobra's Print* default),
+# so merge the streams before grepping, like the other harness scripts.
+spike secret get "$DRILL_MARKER_PATH" 2>&1 |
+  grep -q "$DRILL_MARKER_VALUE" ||
+  fail "could not read the marker secret back while healthy"
+
+# --- 2. Export recovery shards ---------------------------------------------
+
+say "rotating the Pilot entry to the recover role..."
+./hack/bare-metal/entry/spire-server-entry-recover-register.sh ||
+  fail "could not register the recover role"
+
+recover_once() {
+  spike operator recover 2>&1 | grep -q "recovery directory"
+}
+
+say "exporting recovery shards (retrying while the SVID rotates)..."
+retry 5 3 recover_once || fail "spike operator recover did not succeed"
+
+shard_count=$(ls "$RECOVERY_DIR"/spike.recovery.*.txt 2>/dev/null | wc -l)
+[ "$shard_count" -ge 2 ] ||
+  fail "expected at least 2 shard files in $RECOVERY_DIR," \
+    "found $shard_count"
+say "exported $shard_count shards to $RECOVERY_DIR"
+
+say "rotating the Pilot entry back to superuser..."
+./hack/bare-metal/entry/spire-server-entry-recover-revert.sh ||
+  fail "could not revert the recover role"
+
+# --- 3. Simulate the crash --------------------------------------------------
+
+say "simulating a crash: killing Nexus and all Keepers..."
+pkill -x nexus
+pkill -x keeper
+
+crash_complete() {
+  ! pgrep -x nexus >/dev/null && ! pgrep -x keeper >/dev/null
+}
+retry 10 1 crash_complete || fail "Nexus/Keeper processes did not exit"
+say "Nexus and Keepers are down; Keeper shards are lost"
+
+# --- 4. Restart Nexus alone and restore ------------------------------------
+
+say "restarting Nexus alone (log: $NEXUS_LOG)..."
+nohup ./hack/bare-metal/startup/start-nexus.sh \
+  >"$NEXUS_LOG" 2>&1 &
+
+say "rotating the Pilot entry to the restore role..."
+./hack/bare-metal/entry/spire-server-entry-restore-register.sh ||
+  fail "could not register the restore role"
+
+restored=0
+for shard_file in "$RECOVERY_DIR"/spike.recovery.*.txt; do
+  say "feeding $(basename "$shard_file")..."
+
+  feed_shard() {
+    output=$(spike operator restore <"$shard_file" 2>&1)
+    # Two conditions are transient and worth retrying: Nexus is still
+    # starting (comms errors), and the SPIRE agent has not yet rotated
+    # the Pilot SVID to the restore role (access_unauthorized).
+    if echo "$output" | grep -q "Failed to communicate"; then
+      return 1
+    fi
+    if echo "$output" | grep -q "access_unauthorized"; then
+      return 1
+    fi
+    return 0
+  }
+
+  retry 20 2 feed_shard ||
+    fail "could not feed $(basename "$shard_file"):" \
+      "Nexus unreachable or the restore SVID never arrived"
+
+  echo "$output"
+  if echo "$output" | grep -q "restored and ready"; then
+    restored=1
+    break
+  fi
+done
+
+[ "$restored" -eq 1 ] ||
+  fail "fed all shards but Nexus did not report itself restored"
+
+say "rotating the Pilot entry back to superuser..."
+./hack/bare-metal/entry/spire-server-entry-restore-revert.sh ||
+  fail "could not revert the restore role"
+
+# --- 5. Verify the pre-crash secret ----------------------------------------
+
+verify_marker() {
+  spike secret get "$DRILL_MARKER_PATH" 2>&1 |
+    grep -q "$DRILL_MARKER_VALUE"
+}
+
+say "verifying the pre-crash marker secret..."
+retry 10 2 verify_marker ||
+  fail "the marker secret did not read back after the restore"
+
+# --- Cleanup ----------------------------------------------------------------
+
+say "cleaning up: deleting the marker secret and the shard files..."
+spike secret delete "$DRILL_MARKER_PATH" >/dev/null 2>&1
+rm -f "$RECOVERY_DIR"/spike.recovery.*.txt
+
+say "PASS: recovery/restore drill completed successfully."
+echo ""
+echo "  A secret written before the crash survived the loss of Nexus"
+echo "  and every Keeper, and was readable after a shard-based restore."
+echo ""
+echo "  Note: the Keepers are still down. Run 'make kill' and then"
+echo "  'make start' to return to a pristine environment."
+echo ""

makefiles/BareMetal.mk

@@ -38,6 +38,16 @@ build:
 build-spire:
 	./hack/bare-metal/build/build-spire.sh
 
+.PHONY: drill-recovery
+# Live recovery/restore drill: proves the break-the-glass runbook end
+# to end. Run it in a second terminal after `make start` has completed
+# cleanly. The drill kills Nexus and the Keepers, restores Nexus from
+# operator recovery shards, and verifies that a pre-crash secret reads
+# back. The Keepers stay down afterward; run `make kill` and then
+# `make start` to reset the environment.
+drill-recovery:
+	./hack/bare-metal/drill/recovery-drill.sh
+
 # Registry an entry to the SPIRE server for the demo app.
 demo-register-entry:
 	./examples/consume-secrets/demo-register-entry.sh