ci: space the ci-build image pull retries across a registry outage - #1211
Open
zancas wants to merge 1 commit into
Open
ci: space the ci-build image pull retries across a registry outage#1211zancas wants to merge 1 commit into
zancas wants to merge 1 commit into
Conversation
On 2026-07-25 one integration bucket lost all egress to registry-1.docker.io for over four minutes (run 30166451562): every dial timed out after 60 seconds, podman's default three back-to-back retries burned out inside the outage, and the bucket's fail-all step cancelled the whole run. The two sibling buckets pulled the same tag successfully in the same window, so this was a single runner's network, not the registry or the tag. Pull with --retry 8 --retry-delay 45s so the attempts span roughly thirteen minutes instead of four. A healthy pull is unaffected, and a tag that was never published still fails on the first attempt because podman does not retry 4xx registry responses. The flags need podman 4.7; the ubuntu-24.04 runners ship 4.9. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens the integration buckets' ci-build image pull against transient registry outages. Based on
devso every PR benefits; the feat/nym stack picks it up on its next rebase.The failure this prevents
In run 30166451562 (PR #1208) the "accounts" bucket lost all egress to
registry-1.docker.iofor over four minutes: four consecutivedial tcp <ip>:443: i/o timeoutfailures against four different registry IPs. Podman's default three retries fire one second apart, so all of them burned inside the outage, the step failed with exit 125, and the bucket's fail-all step cancelled the entire run, killing the APK build mid-compile. The two sibling buckets pulled the identical tag successfully in the same window: one runner's network, not the registry and not the tag.The change
podman pull --retry 8 --retry-delay 45s. The attempts now span roughly thirteen minutes instead of four, long enough to ride out an egress blip while the bucket would otherwise be idling toward the ci-gate artifact hold anyway. A healthy pull is unchanged. A tag that was never published still fails on the first attempt, because podman retries only errors it classifies as transient (timeouts, 5xx, 429) and never 4xx responses such as manifest-unknown.The flags require podman 4.7; the ubuntu-24.04 runners ship 4.9.
🤖 Generated with Claude Code