mCaptcha: Make setup less prone to crashing on startup - #1464
Conversation
|
CC @ethancedwards8, since you also looked into this for #1463. |
|
Nope, CI also runs into this… And it also runs into this issue on |
|
Yeah. It'll work on my machine but not in CI. Very frustrating |
|
Should we just disable these tests for now, and open an issue about re-enabling them in a less-flaky state later? We had to do that afew times for Heads in the past. (For debugging this, I'd start to patch |
|
I have an example for the cargo dependency patching, one sec. |
|
How about also waiting for port? It amounts to the same thing as we have roughly, but is more idiomatic. |
|
Got somewhere by doing this though: mcaptcha.overrideAttrs (oa: {
cargoDeps = oa.cargoDeps.overrideAttrs (oa2: {
buildCommand = oa2.buildCommand + ''
realdir="$(realpath $out/libmcaptcha-0.2.4)"
rm "$out/libmcaptcha-0.2.4"
cp -r --no-preserve=mode "$realdir" "$out/libmcaptcha-0.2.4"
pushd "$out/libmcaptcha-0.2.4"
# apply patches here
popd
'';
});
})This code here takes the very first module listing that mcaptcha gets from its request to redis. This then gets checked later to make sure it's the But the order in which redis replies its loaded modules doesn't seem consistent. On some runs, it's: Other times, it's: In the case of the latter, libmcaptcha errors out because |
|
I've added a commit that applies a patch to Edit: Submitted upstream as mCaptcha/libmcaptcha#15. I'm going to bed now. |
9661419 to
fed443c
Compare
|
Fails again... different reason this time, though: |
|
The actual error is likely because of: > my_own_services # [ 12.585359] systemd[1]: Starting Redis Server - redis-mcaptcha...[ 12.711141] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
> my_own_services #
> my_own_services # [ 12.717063] cfg80211: failed to load regulatory.dbSo we'll probably need to enable hardware.wirelessRegulatoryDatabase in the build machines. Edit: Nevermind, reran the build and now it passes, but it's still odd that this happens ... |
Same message appears on successful runs, so that doesn't seem to be a real issue.
I don't see the log of that anymore on the PR, and digging through buildbot evals to find it is annoying for me to do rn, but I think it was just taking too long to finish the postgresql initial setup which caused this to get tripped? We could prolly give it abit more time by waiting for postgresql startup first, and then also waiting for postgresql init finish. Or increasing the timeout on that |
- Wait for network-online.target To avoid/reduce likeliness of error about network being down - Wait for postgresql target instead of service Service is when postgresql is started, target when initial setup scripts have actually completed - Wait for redis when creating it locally Happened to work out fine so far, but no reason not to make sure they're launched in the correct order - Fix postgresql authentication setting in bring-your-own-services test Was erroring about IPv6 host not being allowed. Just allow all hosts for this test.
Sometimes, mcaptcha_cache is not the first entry in the module listing that libmcaptcha receives. In that case, it determines that the module is not loaded, and mcaptcha crashes. Check all returned modules, not just the first.
CI managed to time out while waiting for the target at least once. We can give it abit more leeway by waiting for the service first, then the target.
bf0307a to
8c70641
Compare
fricklerhandwerk
left a comment
fricklerhandwerk
left a comment
There was a problem hiding this comment.
Nice, thanks @OPNA2608
4 participants
network-online.targetTo avoid/reduce likeliness of error about network being down
Service is when
postgresqlis started, target when initial setup scripts have actually completedHappened to work out fine so far, but no reason not to make sure they're launched in the correct order
Was erroring about IPv6 host not being allowed. Just allow all hosts for this test.
I still have one flakiness issue with
projects.mCaptcha.nixos.tests.create-locallyon my machine, where mcaptcha thinks that redis is not launched with the mcaptcha module despite redis in the service log explicitly mentioning that it loaded it:And once it fails its initial setup at that point, things are in a broken state and it will seemingly not be able to recover from that:
I'm hoping that this flakiness is just limited to my machine and not CI, but it's not great either way…