ci: Pin openSUSE to second-to-last Tumbleweed snapshot; use metalink for the default mirror#4366
ci: Pin openSUSE to second-to-last Tumbleweed snapshot; use metalink for the default mirror#4366martinpitt wants to merge 3 commits into
Conversation
|
Wow, a green CI run! That's been a while 🎉 |
This comment was marked as resolved.
This comment was marked as resolved.
|
See #4365 (comment) -- sadly, only using the metalink is structurally broken. We need both, I'm afraid 😢 and dear openSUSE, pleeease fix your mirror situation. 🙏 |
martinpitt
force-pushed
the
opensuse-snapshot
branch
from
4031cba to
28a41d8
Compare
martinpitt
changed the title
|
The opensuse extension tests all failed, they don't use the snapshot of the main image build. Fixed. |
martinpitt
force-pushed
the
opensuse-snapshot
branch
from
28a41d8 to
47af17b
Compare
If tests build an image with a particular `Snapshot=` (as the next commit will do), we need extension builds to use the same snapshot, so that (1) they match the base image, and (2) avoid trying to download newer packages from mirrors which don't have these yet. Read the snapshot from the main mkosi.local.conf and pass it along in `ImageConfig`, for the `--directory ...` extension builds in tests/test_extension.py.
martinpitt
force-pushed
the
opensuse-snapshot
branch
from
47af17b to
fc09616
Compare
This comment was marked as resolved.
This comment was marked as resolved.
martinpitt
force-pushed
the
opensuse-snapshot
branch
from
fc09616 to
5139dff
Compare
|
two failures, both with the same cause:
On the good side, the rpm download failover and the consistent snapshot selection now work! Neither me nor Claude have any idea what else to try now. So I think that's about as good as it gets. At least slc-mirror is reachable again, so if @behrmann or @daandemeyer could retry that failed test, I'd appreciate -- it should hopefully go green now. Also, ready for review! @Firstyear @cyphar I think this does what we discussed in #4365 -- but second pair of eyes appreciated! |
behrmann
left a comment
behrmann
left a comment
There was a problem hiding this comment.
Thanks! I've not seen CI this green in a long time.
openSUSE Tumbleweed US mirrors frequently lag behind freshly published snapshots. On US-based GitHub action runners the geo-redirector routes to US mirrors that haven't rsync'd the latest snapshot yet, so zypper/dnf get 404s and timeouts on every mirror and abort. (This never reproduces in Germany, as the primary mirrors in Nürnberg sync very quickly.) Pin the second-to-last published snapshot for openSUSE image builds, which should have had time to propagate to all mirrors. Fixes half of systemd#4365.
Point the openSUSE repositories at download.opensuse.org's per-file metalink instead of a plain baseurl when using the default mirror. download.opensuse.org routes each client to a single sticky mirror, so when that mirror is slow, overloaded or has not yet synced a freshly published snapshot, downloads often fail. Note that `download.max_silent_tries` doesn't save us here, as it's just going to hit the same mirror again. The metalink is a ranked list of mirrors that libdnf/libzypp fall back across on download failure. Custom mirrors (`Mirror=`) don't serve a metalink, so they keep baseurl. This fixes the other half of systemd#4365
martinpitt
force-pushed
the
opensuse-snapshot
branch
from
5139dff to
2db2cde
Compare
|
So, LGTM in principle, but opensuse/arch and opensuse/opensuse are still running into timeouts. fedora/arch is running into a cooldown it can't handle, so that's different, but I'll leave looking at the suse failures for your, by now, trained eye. |
|
@behrmann that's the same issue as above: slc-mirror.opensuse.org doesn't respond:
sadly, zypper/opensuse repos seem to have no solution for that, I'm at my wit's end 😢 It seems to be working again now. fedora/arch cooldown: same, no idea -- we can't immediately retry a 429 error, that would only put even more load on the mirror. |
|
I sent #4368 with a version without the metalink change. Since both approaches are broken in different ways, I suppose when in doubt use the less intrusive one. |
|
I am shelving this. metalink would be better (that's what zypper is designed to handle well), but with that being so unreliable, there's no point in this. |
2 participants
See #4365 for details. It will fix this kind of failure and also this kind.