Skip to content

fix(remote): stop --exit-on-error reporting unattempted hosts as failures - #186

Open
tas50 wants to merge 1 commit into
mainfrom
fix/exit-on-error-failure-count
Open

fix(remote): stop --exit-on-error reporting unattempted hosts as failures#186
tas50 wants to merge 1 commit into
mainfrom
fix/exit-on-error-failure-count

Conversation

@tas50

@tas50 tas50 commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

The defect

RunMany cancelled a shared context to stop the queue after the first failure. Every host still queued then took the cancelled-context branch and came back with ExitCode: 255, which countRemoteFailures duly counted:

first results: [{"host":"h00","stderr":"boom","exit_code":1},
                {"host":"h01","exit_code":255,"error":"context canceled"},
                {"host":"h02","exit_code":255,"error":"context canceled"}, ...]

hosts actually attempted = 1, hosts reported as failed = 20 (of 20)

So one failing command across twenty hosts told the operator node ssh failed on 20 host(s), when nineteen were never contacted. That is the opposite of useful during an incident: it turns "one box is unhappy" into "the fleet is down".

Cancelling also contradicted the flag's own help text:

--exit-on-error: stop launching new SSH sessions after the first failure

It promises to stop launching, but a cancelled context also aborts sessions already in flight, and kills hosts mid-dial with a confusing dial ssh: context canceled.

The fix

  • Stop launching via a plain atomic.Bool instead of cancelling the context, so commands already running are left to finish. That is what the flag says it does.
  • Hosts that never ran come back with a new Skipped field rather than a fabricated failure exit code. Nothing ran on them, so nothing is known about them.
  • countRemoteFailures ignores skipped hosts, and the error message distinguishes the two:
node ssh failed on 1 host(s); 19 not attempted after --exit-on-error

The caller's context is still honoured, so Ctrl-C still stops the run; those hosts are reported as skipped too, which is accurate.

Tests

  • TestRunManyExitOnErrorMarksRemainingSkipped — 20 hosts, one failure; asserts exactly one failure, the rest skipped with no bogus exit code, and that the runner was invoked once.
  • TestRunManyExitOnErrorLetsInFlightWorkFinish — uses a barrier runner that blocks until all four commands are genuinely in flight, then releases them, proving none is aborted. (My first attempt at this test was flaky: the stub returned instantly, so the failure was observed before the other jobs were dispatched. The barrier makes it deterministic.)
  • TestRunManyWithoutExitOnErrorRunsEveryHost — the flag is off by default and must change nothing.

Verified with -race -count=10. go test ./... -race, go vet ./..., and gofmt -l . are clean.

node ssh already has acceptance coverage (TestNodeSSHSkipSearchAgainstSSHServer) and this adds no command, so the coverage manifest is unchanged.

@tas50
tas50 force-pushed the fix/exit-on-error-failure-count branch 3 times, most recently from 52f7320 to 20e79be Compare September 8, 2026 17:38
…ures

RunMany cancelled a shared context to stop the queue after the first
failure. Every host still queued then took the cancelled-context branch
and came back with ExitCode 255, so countRemoteFailures counted it. One
failing command across twenty hosts reported:

  node ssh failed on 20 host(s)

when nineteen of them were never contacted.

Cancelling also contradicted the flag's own help text. "--exit-on-error:
stop launching new SSH sessions after the first failure" promises to stop
launching, but a cancelled context also aborts sessions already running.

Use a plain atomic flag to stop launching, leaving in-flight commands to
finish, and mark hosts that never ran as Skipped rather than failed. The
error now distinguishes the two:

  node ssh failed on 1 host(s); 19 not attempted after --exit-on-error

Signed-off-by: Tim Smith <tim@mondoo.com>
@tas50
tas50 force-pushed the fix/exit-on-error-failure-count branch from 20e79be to 74b1578 Compare September 8, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant