Fix: don't cancel the sibling matrix legs when one OS fails (Part 2/2) - #652
Open
AmaadMartin wants to merge 1 commit into
Open
Conversation
The run-tests matrix runs three OSes with GitHub's default fail-fast, so the first leg to fail cancels the other two and the run yields no signal from the OSes that were fine. An intermittent per-OS timeout therefore costs the whole matrix result, not just its own leg. The trade is that a genuinely broken PR now burns all three runners instead of being cut short after the first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
No public issue. Stacked on Fix: stop the app_loader discovery test billing its fixture setup to the first it() (Part 1/2) #506 (Part 1/2), which fixes the
app_loaderintegration-test timeouts themselves; this part is the CI-side half and is split out so it can be judged on its own merits.Problem:
run-testsruns a three-OS matrix (ubuntu-latest,windows-latest,macos-latest) under GitHub's defaultfail-fast: true. The first leg to fail cancels the other two, so an intermittent per-OS failure costs the whole matrix result rather than its own leg: the run reports nothing about the OSes that were fine, and the usual response — re-run everything — pays for all three again.This is what made the
app_loaderdiscovery flake so expensive. It failed onmacos-latestandwindows-latestwhileubuntu-latestwas healthy, and each occurrence cancelled the healthy leg too, leaving an inconclusive run.Solution: set
fail-fast: falseon therun-testsstrategy. Each OS now reports its own result, so an intermittent failure on one runner is visible as an intermittent failure on one runner, and a re-run is only needed for the leg that failed.The tradeoff, stated plainly: a genuinely broken PR now burns all three runners to completion instead of being cut short after the first failure. That is the intended trade — one flake no longer destroys the signal from the other two OSes — but it is a repo-wide policy change with a real cost, which is exactly why it is its own PR instead of a hunk inside a test fix.
Nothing else in the workflow changes: the OS list, the
envblock and every step are untouched.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
There is no code here to unit-test — the change is three lines of GitHub Actions configuration (one key plus a two-line comment), and matrix cancellation semantics cannot be exercised from a test runner. It is verified structurally instead: the workflow parses, and the
run-testsstrategy resolves to exactlywith the OS list and the
envblock byte-identical tomain.npm run format:checkglobs**/*.tsonly, so this file is outside it; its existing quoting is left exactly as-is rather than reflowed into the diff.CI on this PR is absent, not green. The
validationworkflow ison: pull_request: branches: [main], and this PR targets the Part 1 branch, so no test job triggers for it — onlyauto-assign, which validates nothing. Validated locally on the pushed commit14ce6cc3instead:The behavioural proof is the CI run on the parent PR, whose tree differs from this one by exactly the three YAML lines above: Part 1's validation run is green on all three legs (
macos-latest6m42s,ubuntu-latest5m39s,windows-latest9m27s), so this change is stacked on a matrix that is currently healthy, and the next intermittent failure will show the difference — one red leg and two conclusive green ones.Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
Because this PR targets the Part 1 branch rather than
main, thevalidationworkflow (on: pull_request: branches: [main]) does not trigger for it — there are no CI checks to read on this PR, and that is expected rather than a failure. To observe the change directly once it is on amain-targeted branch: make one OS fail deliberately (e.g. aprocess.platform-guarded failing assertion) and confirm the other two legs still run to completion and report their own results, where previously they were cancelled.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.