Skip to content

test: scale extension-binding coordinator deadline with host load - #7826

Open
AgardnerAU wants to merge 4 commits into
kunchenguid:mainfrom
AgardnerAU:fm/fm-extension-binding-deadline-fix
Open

AgardnerAU wants to merge 4 commits into
kunchenguid:mainfrom
AgardnerAU:fm/fm-extension-binding-deadline-fix

Conversation

@AgardnerAU

Copy link
Copy Markdown

Intent

Ship the small fix as an upstream PR. The extension-binding suite (tests/fm-extension-binding.test.sh) flake is real and load-triggered: its 34 s wall-clock section coordinator deadline is exceeded at host load of about 45 or more, while the suite passes at normal load. Fix by making the coordinator deadline scale with load or raising the 34 s default.

What Changed

  • tests/fm-extension-binding.test.sh: the section-lane coordinator deadline now defaults to 120 s instead of 34 s. When FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS is not set, a new section_lane_load_factor helper multiplies the default by the one-minute load per online CPU, rounded up and capped at 8x. The accepted override range goes up from 1-34 to 1-120. An explicit override is used as given and is not scaled.
  • Two more waits in the same suite that are sensitive to load get more time: the poll for the crash-silent claim file goes from 400 to 3000 iterations, and the lock-order contender deadline goes from 12 s to 60 s.

Risk Assessment

✅ Low: This is a test-only change that raises a timeout constant and its matching upper-bound check (34->120, <35-><121). It is consistent with the existing validation, the 2 s override used by the coordinator-timeout self-test still works, and no runner or doc depends on the old value.

Testing

The bounded heavy-load harness ran to completion once at HEAD (r6). Under 12 pinned CPU hogs with a host load of about 21 on 8 CPUs, the suite passed with the default load-scaled deadline twice (199 s and 196 s), and the explicit 34 s override failed with 'an isolated extension conformance section failed'. That completed r6 run was started by an earlier invocation of this same test step in this round, not by this invocation. My own repeat (r7) was stopped at the start of its first suite run because the structured result was required. In this invocation, I re-drove the guard refusal for 0, 121 and abc live. The worktree is clean.

  • Live validation: ✅ go - 3 of 3 scenarios driven live against the product
Scenario Result Live Evidence
A developer runs the suite under heavy CPU contention (12 busy loops pinned with the suite to 2 CPUs) with the default load-scaled deadline, and the suite passes ✅ pass live r6/load-summary.log: default-1 rc=0 199s, default-2 rc=0 196s, last line 'all extension-binding tests passed'
Adversarial: under the same heavy load, FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=34 (not load-scaled) reproduces the coordinator-deadline failure ✅ pass live r6/load-override-34.log: a section lane was Killed by the deadline, then 'not ok - an isolated extension conformance section failed', rc=1
Adversarial: out-of-range or non-numeric overrides (0, 121, abc) are refused by the guard before any lane starts ✅ pass live r7/guard-0.log, r7/guard-121.log, r7/guard-abc.log: the coordinator returns before it starts a child ('the coordinator probe did not start its waiting child'), rc=1
Evidence: Heavy-load harness summary (completed, HEAD a9490d4)

Source: Heavy-load harness summary (completed, HEAD a9490d4)

default-1 rc=0 elapsed=199s loadavg=21.91 | override-34 rc=1 elapsed=139s | default-2 rc=0 elapsed=196s loadavg=21.00 | DONE

== default-1 start loadavg=10.90 10.18 12.94 16/3158 2242640 nproc(pinned)=2
== default-1 rc=0 elapsed=199s loadavg=21.91 15.68 14.54 20/3142 2409534 last: all extension-binding tests passed
== override-34 start loadavg=21.91 15.68 14.54 17/3143 2409539 nproc(pinned)=2
== override-34 rc=1 elapsed=139s loadavg=21.38 17.08 15.18 20/3084 2515901 last: not-autohandled: flow-source (left for the handler; still unacknowledged)
== default-2 start loadavg=21.38 17.08 15.18 20/3083 2515907 nproc(pinned)=2
== default-2 rc=0 elapsed=196s loadavg=21.00 18.97 16.28 16/3065 2681869 last: all extension-binding tests passed
DONE
Evidence: Loaded default run 1

Source: Loaded default run 1

ok - unknown extension conformance section selectors fail before setup
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-backend.sh: No such file or directory
ok - the section coordinator propagates ordered failures and bounded cleanup
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-backend.sh: No such file or directory
not-autohandled: example-file (left for the handler; still unacknowledged)
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-backend.sh: No such file or directory
not-autohandled: flow-source (left for the handler; still unacknowledged)
ok - the shipped file-signal package is a runnable end-to-end external adapter
ok - one external adapter registers, invokes, captures unhandled evidence, classifies, and terminally retires end to end
not-autohandled: remote-source (left for the handler; still unacknowledged)
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-backend.sh: No such file or directory
ok - register-extension and reconcile of an unhandled extension result take their locks in one order

all extension-binding tests passed
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Bx07Kp/remote-root/bin/fm-backend.sh: No such file or directory
ok - serialized remote binding crosses fm-on through addressed-home capture and retirement

all extension-binding tests passed
not-autohandled: example-symlinked (left for the handler; still unacknowledged)
ok - a home reached through a symlinked ancestor captures external evidence normally
ok - inner result.silent host crash releases the parent lifecycle lock before terminal retry
ok - handshake execution rejects and reaps foreground descendants

all extension-binding tests passed
ok - local binding retirement requires its exact identity and disables invocation

all extension-binding tests passed
ok - independent extension conformance sections complete through isolated public homes

all extension-binding tests passed
Evidence: Loaded override-34 failure

Source: Loaded override-34 failure

ok - unknown extension conformance section selectors fail before setup
/tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-backend.sh: No such file or directory
tests/fm-extension-binding.test.sh: line 410: 2497574 Killed                  FM_EXTENSION_BINDING_SECTION_CHILD=1 FM_EXTENSION_BINDING_SEGMENT="$section" bash "$0"
ok - the section coordinator propagates ordered failures and bounded cleanup
not-autohandled: example-file (left for the handler; still unacknowledged)
/tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-backend.sh: No such file or directory
/tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.6f7RJk/remote-root/bin/fm-backend.sh: No such file or directory
not ok - an isolated extension conformance section failed
not-autohandled: flow-source (left for the handler; still unacknowledged)
not-autohandled: remote-source (left for the handler; still unacknowledged)
Evidence: Loaded default run 2

Source: Loaded default run 2

ok - unknown extension conformance section selectors fail before setup
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-backend.sh: No such file or directory
tests/fm-extension-binding.test.sh: line 410: 2610588 Killed                  FM_EXTENSION_BINDING_SECTION_CHILD=1 FM_EXTENSION_BINDING_SEGMENT="$section" bash "$0"
ok - the section coordinator propagates ordered failures and bounded cleanup
not-autohandled: example-file (left for the handler; still unacknowledged)
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-backend.sh: No such file or directory
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-backend.sh: No such file or directory
not-autohandled: flow-source (left for the handler; still unacknowledged)
ok - the shipped file-signal package is a runnable end-to-end external adapter
ok - one external adapter registers, invokes, captures unhandled evidence, classifies, and terminally retires end to end
not-autohandled: remote-source (left for the handler; still unacknowledged)
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-backend.sh: No such file or directory
ok - register-extension and reconcile of an unhandled extension result take their locks in one order

all extension-binding tests passed
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 257: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-task-inbox-lib.sh: No such file or directory
/tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-procevent.sh: line 259: /tmp/fm-extension-binding.Qr2J37/remote-root/bin/fm-backend.sh: No such file or directory
ok - serialized remote binding crosses fm-on through addressed-home capture and retirement

all extension-binding tests passed
not-autohandled: example-symlinked (left for the handler; still unacknowledged)
ok - a home reached through a symlinked ancestor captures external evidence normally
ok - inner result.silent host crash releases the parent lifecycle lock before terminal retry
ok - handshake execution rejects and reaps foreground descendants

all extension-binding tests passed
ok - local binding retirement requires its exact identity and disables invocation

all extension-binding tests passed
ok - independent extension conformance sections complete through isolated public homes

all extension-binding tests passed
Evidence: Guard refusal override=121

Source: Guard refusal override=121

ok - unknown extension conformance section selectors fail before setup
not ok - the coordinator probe did not start its waiting child
Evidence: Bounded load harness

Source: Bounded load harness

#!/bin/bash
# Bounded load harness: 12 busy loops pinned to CPUs 6,7; suite pinned to the same CPUs.
E=$1; cd "$2" || exit 1
pids=()
for i in $(seq 12); do taskset -c 6,7 bash -c 'while :; do :; done' & pids+=($!); done
trap 'kill "${pids[@]}" 2>/dev/null' EXIT
sleep 60
run() { name=$1; shift; s=$SECONDS
  echo "== $name start loadavg=$(cat /proc/loadavg) nproc(pinned)=$(taskset -c 6,7 nproc)" >> "$E/load-summary.log"
  env "$@" taskset -c 6,7 timeout 1200 bash tests/fm-extension-binding.test.sh > "$E/load-$name.log" 2>&1; rc=$?
  echo "== $name rc=$rc elapsed=$((SECONDS-s))s loadavg=$(cat /proc/loadavg) last: $(tail -1 "$E/load-$name.log")" >> "$E/load-summary.log"; }
run default-1 X=1
run override-34 FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=34
run default-2 X=1
echo DONE >> "$E/load-summary.log"
- Outcome: ⚠️ 1 info across 6 runs (1h7m5s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 warning
  • ⚠️ tests/fm-extension-binding.test.sh:1124 - The change also widens the crash-silent start poll from 400 to 3000 iterations (about 4 s to about 30 s or more of 0.01 s sleeps). The intent only asks to scale or raise the 34 s section coordinator deadline. The commit message says this wait was also exceeded under load, so it is a reasonable companion fix, but it goes beyond the stated scope. Confirm that you want it in this PR or remove it. It still fits inside the new 120 s coordinator budget. Sibling 400-iteration polls at lines 1335, 1376, 1389 and 2099 are unchanged, which is consistent with keeping the change narrow.
⚠️ **Test** - 1 info
  • ⚠️ tests/.base-extension-binding.test.sh - Testing created an untracked file, tests/.base-extension-binding.test.sh: a copy of the base-commit suite, used to reproduce the flake. The permission policy refused both attempts to remove it. Delete it before you commit. Do not stage it.
  • ℹ️ Under CPU contention on one core, the fixed suite took 99 s against the new 120 s deadline, so the margin is about 20 s. A repeat contended run was refused by the permission policy. Heavier load than this simulation could still go past 120 s. A load-scaled deadline (the other option in the intent) would remove that risk.
  • Live validation: ✅ go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
At normal host load, a developer runs the extension-binding suite and it passes ✅ pass live normal-load.log: rc=0, 12 s, 'all extension-binding tests passed'
Under heavy CPU contention the base commit's 34 s coordinator deadline is exceeded and the suite fails (flake reproduced) ✅ pass live pinned-1cpu-base.log: rc=1 at 37 s, lane killed, 'not ok - an isolated extension conformance section failed'
Under the same CPU contention the fixed suite completes inside the new 120 s deadline and passes ✅ pass live pinned-1cpu-fix.log: rc=0 at 99 s, 'all extension-binding tests passed'
Adversarial: an out-of-range or invalid FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS (121, 0, abc) is rejected, and the new maximum 120 is accepted ✅ pass live override-boundary.log: 121/0/abc -> rc=1; 120 -> rc=0
Adversarial: the coordinator still enforces a short deadline and kills and reaps a stalled child (the 2 s stubborn probe) ✅ pass live coordinator-segment.full.log: 'ok - the section coordinator propagates ordered failures and bounded cleanup'
  • bash tests/fm-extension-binding.test.sh at normal host load (loadavg about 2.2, 8 CPUs)
  • taskset -c 0 bash tests/.base-extension-binding.test.sh (base commit 127f0f30 copy of the suite) with 4 competing busy loops pinned to CPU 0
  • taskset -c 0 bash tests/fm-extension-binding.test.sh (fix) with 4 competing busy loops pinned to CPU 0
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=121|0|abc|120 bash tests/fm-extension-binding.test.sh
  • FM_EXTENSION_BINDING_SEGMENT=coordinator bash tests/fm-extension-binding.test.sh (includes the 2 s deadline stubborn-child probe)

🔧 Fix applied.
3 issues (2 warnings, 1 info) still open:

  • ⚠️ tests/fm-extension-binding.test.sh:1192 - Under simulated load of about 48 (12 CPU hogs pinned with the suite to 2 cores), the 120 s coordinator change alone did not stop the suite failing. It failed in 2 of 2 full runs with 'register-extension and reconcile deadlocked on an unhandled extension result', which is the 12 s lock-order wait. The section passes alone under the same load, so this is slowness from contention and not a real deadlock. Fix applied in the worktree: order_deadline widened from 12 s to 60 s. Afterwards the loaded suite passed 2 of 2 runs and still passes at normal load. This goes beyond the coordinator-only scope in the intent; revert it if you do not want it.
  • ⚠️ tests/.base-extension-binding.test.sh - Commit 2b1e490 added the transient base-suite copy (2332 lines) to the branch instead of removing it. I removed it with git rm, so the next commit deletes it.
  • ℹ️ The passing loaded runs took 126-129 s from end to end, close to the hard 120 s cap on the coordinator lanes (the guard rejects any value above 120). Heavier load could still exceed the deadline. A deadline that scales with load would remove this residual risk.
  • Live validation: ✅ go - 4 of 4 scenarios driven live against the product
Scenario Result Live Evidence
A developer runs the full extension-binding suite at normal host load with the new 120 s default, and it passes ✅ pass live s1-normal-load.log (exit 0, 18 s); s1-normal-load-after-order-fix.log (exit 0, 23 s)
Under simulated load of about 48, the old 34 s coordinator deadline makes the suite fail (reproduces the reported flake) ✅ pass live s2-old-34s-deadline.log (exit 1, 'an isolated extension conformance section failed' at about 54 s)
Under the same load, the full suite with the new default no longer fails at the coordinator deadline and completes successfully ✅ pass live s2-new-default.log and s2-new-default-rerun.log (both failed at the 12 s lock-order wait, not at the coordinator deadline); s5-widened-order-loaded-1.log and s5-widened-order-loaded-2.log (exit 0, 126…
An invalid or out-of-range coordinator deadline override (121, 0, non-numeric) is rejected, and 120 is accepted ✅ pass live s3-deadline-guard.log
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS={120,121,0,abc} FM_EXTENSION_BINDING_SEGMENT=coordinator bash tests/fm-extension-binding.test.sh
  • bash tests/fm-extension-binding.test.sh at normal host load (before and after the lock-order fix)
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=34 taskset -c 0,1 bash tests/fm-extension-binding.test.sh with 12 CPU hogs pinned to the same 2 cores (old deadline reproduction)
  • taskset -c 0,1 bash tests/fm-extension-binding.test.sh with 12 pinned CPU hogs, new 120 s default (2 runs, both failed at the 12 s lock-order wait)
  • FM_EXTENSION_BINDING_SEGMENT=lifecycle-order taskset -c 0,1 bash tests/fm-extension-binding.test.sh under the same load (2 runs, both passed)
  • taskset -c 0,1 bash tests/fm-extension-binding.test.sh under the same load after widening order_deadline from 12 s to 60 s (2 runs, both passed)
  • git rm tests/.base-extension-binding.test.sh (removed the committed transient copy)

🔧 Fix applied.
2 errors still open:

  • 🚨 tests/fm-extension-binding.test.sh:447 - The raised 120 s coordinator deadline, with its hard guard cap of 120, still fails under heavy load. Test setup: 12 busy loops pinned with the suite to 2 cores, which gives the suite about 0.13 of a core, similar to a host load of about 60 on this 8-core host. Result: 1 of 2 full runs failed with 'an isolated extension conformance section failed'. The lifecycle-flow lane was killed by the deadline, not by an assertion. Alone, that section takes 134 s at this load. The intent says the flake occurs at load 45 'or more', so a fixed ceiling only moves the threshold. The intent's other option would remove the residual flake: scale the deadline with load (for example, multiply by loadavg/nproc), or raise both the default and the guard cap well above 134 s. You must choose between these, so this is ask-user.
  • 🚨 live validation verdict: no-go (4 of 4 scenarios were driven live against the product); failed: Developer runs the suite under heavy CPU contention (12 busy loops on 2 cores, similar to host load of about 60) with the new default and it passes every time
  • Live validation: ❌ no-go - 4 of 4 scenarios driven live against the product
Scenario Result Live Evidence
Developer runs the extension-binding suite at normal load and it passes ✅ pass live normal-load-run.log (rc=0, 19 s)
Reproduction: under heavy CPU contention the old 34 s coordinator deadline makes the suite fail ✅ pass live loaded-run-old-34s-deadline.log (rc=1, 'an isolated extension conformance section failed')
Developer runs the suite under heavy CPU contention (12 busy loops on 2 cores, similar to host load of about 60) with the new default and it passes every time ❌ fail live loaded-run-fixed-1.log passed (140 s). loaded-run-fixed-2.log failed (204 s): the lifecycle-flow lane was killed by the coordinator deadline with no assertion failure. loaded-lifecycle-flow-alone.log…
Adversarial: a coordinator timeout above the new cap (121) is rejected, and 120 is accepted ✅ pass live guard-121.log (rc=1, coordinator returns 64), guard-120.log (rc=0)
  • bash tests/fm-extension-binding.test.sh at normal host load (pass, 19 s)
  • fm-load-run.sh (12 busy loops plus the suite pinned to CPUs 0,1) with the default 120 s deadline, run 1 (pass, 140 s)
  • fm-load-run.sh with FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=34 (fail: an isolated extension conformance section failed, 90 s)
  • fm-load-run.sh with the default 120 s deadline, run 2 (fail: same message, the lifecycle-flow lane was killed with no assertion failure, 204 s)
  • fm-load-run.sh with FM_EXTENSION_BINDING_SEGMENT=lifecycle-flow (section alone takes 134 s under this load)
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=121 bash tests/fm-extension-binding.test.sh (rejected)
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=120 bash tests/fm-extension-binding.test.sh (pass)

🔧 Fix applied.
2 warnings still open:

  • ⚠️ Heavy-load validation of the load-scaled deadline did not finish in this round. The structured result was required while the loaded suite runs were still in their warm-up, so I stopped them. Only the normal-load pass is shown. Re-run the test step and let the loaded runs (about 10 minutes) complete to show the flake fix.
  • ⚠️ live validation verdict: inconclusive (1 of 4 scenarios were driven live against the product); untested: Developer runs the suite under heavy CPU contention (12 busy loops on 2 cores, load per CPU above 6) and the load-scaled deadline lets it pass on repeated runs, Adversarial: the same heavy load with the old 34 s deadline set as an explicit override reproduces the coordinator-deadline failure, so the override is not load-scaled and the scaling is what fixes the flake, Adversarial: an out-of-range or non-numeric override (0, 121, abc) is still refused by the guard
  • Live validation: ⚠️ inconclusive - 1 of 4 scenarios driven live against the product
Scenario Result Live Evidence
Developer runs the extension-binding suite at normal host load with the new default deadline and it passes ✅ pass live s1-normal-load.log (rc=0, 29 s, 'all extension-binding tests passed')
Developer runs the suite under heavy CPU contention (12 busy loops on 2 cores, load per CPU above 6) and the load-scaled deadline lets it pass on repeated runs ⏸️ untested no The driver was running, but the structured result was required before the 75 s load warm-up and the 2 to 4 minute loaded runs could finish. Re-run this step and let the loaded runs complete to drive i…
Adversarial: the same heavy load with the old 34 s deadline set as an explicit override reproduces the coordinator-deadline failure, so the override is not load-scaled and the scaling is what fixes th… ⏸️ untested no Stopped before it ran, for the same reason as the scenario above.
Adversarial: an out-of-range or non-numeric override (0, 121, abc) is still refused by the guard ⏸️ untested no I planned to run this after the load runs so that it would not disturb them. The round ended first.
  • bash tests/fm-extension-binding.test.sh at normal host load (8 CPUs, load about 4.6 to 6.4): exit 0, 29 s
  • Started a background heavy-load driver: 12 busy loops pinned to cores 0-1 with taskset, then the suite pinned to the same cores (default deadline twice, then the old 34 s deadline as an explicit override). It was stopped during the 75 s load warm-up, before any suite run finished.

🔧 No changes applied.
2 warnings still open:

  • ⚠️ The heavy-load proof did not finish in this round either. The default deadline passing under contention and the override=34 failure reproducing are both still not driven live. The bounded harness is in the evidence directory (load-run.sh). A run of about 10-15 minutes is necessary to complete it. Also note that at HEAD the default deadline scales with load (120 s multiplied by ceil(loadavg/nproc), clamped to 1-8). It is not a fixed 120 s default as the decision text says. Confirm that load scaling is the behaviour you want to ship.
  • ⚠️ live validation verdict: inconclusive (1 of 3 scenarios were driven live against the product); untested: A developer runs the suite under heavy CPU contention (12 busy loops on 2 CPUs) with the default coordinator deadline and the suite passes, Adversarial: under the same heavy load, an explicit FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=34 override reproduces the coordinator-deadline failure
  • Live validation: ⚠️ inconclusive - 1 of 3 scenarios driven live against the product
Scenario Result Live Evidence
Adversarial: a developer sets an out-of-range or non-numeric coordinator override (0, 121, abc) and the guard refuses it before it starts any lane ✅ pass live guard-override-0.log, guard-override-121.log, guard-override-abc.log: each run exits rc=1 at the first coordinator probe with 'the coordinator probe did not start its waiting child', which shows that…
A developer runs the suite under heavy CPU contention (12 busy loops on 2 CPUs) with the default coordinator deadline and the suite passes ⏸️ untested no The first launch was refused by the permission policy because it wrote the harness to /tmp. I moved the harness to the evidence directory and started it again, but stopped it during its 60 s load warm…
Adversarial: under the same heavy load, an explicit FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=34 override reproduces the coordinator-deadline failure ⏸️ untested no This is the same bounded load harness run as the scenario above, and it was stopped during warm-up before this suite run began, for the same reasons: a permission-policy refusal and then the mandatory…
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=0 bash tests/fm-extension-binding.test.sh (rc=1 after 12 s, coordinator refused before it launched its probe child)
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=121 bash tests/fm-extension-binding.test.sh (rc=1 after 14 s, same refusal)
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=abc bash tests/fm-extension-binding.test.sh (rc=1 after 12 s, same refusal)
  • bash load-run.sh (12 busy loops pinned with taskset to CPUs 6,7, then default, override=34 and default suite runs): started, then stopped during its 60 s warm-up before any suite run began

🔧 No changes applied.
1 info still open:

  • ℹ️ Earlier loaded runs left orphaned fm-remote-job-worker.sh processes under /tmp/fm-extension-binding.{udlbHL,ShbkxC,6f7RJk}. They seem to come from the failed override-34 runs, which do not reap their remote workers. The permission policy refused to kill them and remove the directories. Remove them manually if you want.
  • Live validation: ✅ go - 3 of 3 scenarios driven live against the product
Scenario Result Live Evidence
A developer runs the suite under heavy CPU contention (12 busy loops pinned with the suite to 2 CPUs) with the default load-scaled deadline, and the suite passes ✅ pass live r6/load-summary.log: default-1 rc=0 199s, default-2 rc=0 196s, last line 'all extension-binding tests passed'
Adversarial: under the same heavy load, FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=34 (not load-scaled) reproduces the coordinator-deadline failure ✅ pass live r6/load-override-34.log: a section lane was Killed by the deadline, then 'not ok - an isolated extension conformance section failed', rc=1
Adversarial: out-of-range or non-numeric overrides (0, 121, abc) are refused by the guard before any lane starts ✅ pass live r7/guard-0.log, r7/guard-121.log, r7/guard-abc.log: the coordinator returns before it starts a child ('the coordinator probe did not start its waiting child'), rc=1
  • bash load-run.sh &lt;evidence&gt;/r6 &lt;worktree&gt;: 12 busy loops pinned to CPUs 6,7 with the suite, 60 s warm-up, then default / override-34 / default runs. Completed at HEAD a9490d4 at 08:18:48, DONE marker present
  • FM_EXTENSION_BINDING_COORDINATOR_TIMEOUT_SECONDS=0|121|abc bash tests/fm-extension-binding.test.sh (re-driven in this invocation, logs in r7/)
  • A repeat of load-run.sh into r7/ reached default-1 start and was then stopped because the structured result was required. All hogs and suite processes were confirmed dead afterwards
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

Raise the section coordinator deadline from 34 s to 120 s, with its cap check, and widen the crash-silent start poll. Both were exceeded when the host was heavily loaded.
@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate: thanks @AgardnerAU, and thanks for the heavy-load repro with the 34 s override as the adversarial check.

What I checked (head a9490d4b): it's test-only, all in tests/fm-extension-binding.test.sh. The default section-lane coordinator deadline goes from 34 s to 120 s, scaled by one-minute load per CPU (rounded up, capped at 8x) when no override is set. The override range becomes 1–120 and an explicit override isn't scaled, so the existing 2 s self-test still works. Two more load-sensitive waits get more room. It works on both Linux (/proc/loadavg) and macOS (sysctl vm.loadavg). I found no workflow, network or credential changes, and it doesn't overlap our separate supervision-host test work.

How I'm classing it: a restore. The suite is supposed to pass, and this fixes a load-triggered flake without changing any product path.

CI: this is a first-time fork run, so I approved the CI and no-mistakes workflows on this head just now. Nothing is needed from you. If the run comes back green and the no-mistakes check binds this head, it can land on a later pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants