test(cluster): raise scheduledbackups cleanup timeout to 10m#2
Closed
aqeelat wants to merge 44 commits into
Closed
test(cluster): raise scheduledbackups cleanup timeout to 10m#2aqeelat wants to merge 44 commits into
aqeelat wants to merge 44 commits into
Conversation
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
…g#776) Signed-off-by: Nolan Gilley <nkgilley@gmail.com> Signed-off-by: Jan Larwig <jan@larwig.com> Signed-off-by: Itay Grudev <itay+github.com@grudev.com> Co-authored-by: Nolan Gilley <nkgilley@gmail.com> Co-authored-by: Itay Grudev <itay+github.com@grudev.com>
…e-pg#927) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…#926) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
…uster Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
…only Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
…ions Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
The barman-plugin-migration chainsaw test fails consistently in CI. The migration itself works correctly (verified against operator source and the plugin migration docs); the failures are in test assertions of transient states, plus one secret-name typo. - Migration 'Upgrading cluster' phase (step 4): status.phase is set for only a fraction of a second (the operator models rollouts as PhaseUpgrade -> PhaseWaitingForInstancesToBeActive, see tests/e2e/probes_test.go), and helm --wait does not gate CNPG CRDs, so polling always missed it. Replaced with a polling assert on the persistent UpgradingInstance Event. - Backup 'running' phase (step 5): the backup completes in ~10s, faster than polling. Replaced with a polling assert on the persistent backup 'Starting' Event (backup_controller.go:337). - PITR secret name (step 8): referenced recovery-backup-pitr-superuser (release name) instead of recovery-backup-pitr-cluster-superuser (cluster name), causing CreateContainerConfigError. Fixed. Verified end-to-end in isolated fork CI (identical test( cluster ) workflow), including PITR recovery. Signed-off-by: Abdullah Alaqel <abdullah.t.aqeel@gmail.com> Signed-off-by: Abdullah Alaqeel <abdullah.t.aqeel@gmail.com>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
Signed-off-by: Itay Grudev <itay@verito.digital>
The barman-plugin-scheduledbackups test intermittently fails with 'context deadline exceeded' during cleanup. The test logic passes; the failure is the namespace teardown exceeding the 5m cleanup budget. Root cause: the CNPG instance pod has terminationGracePeriodSeconds=1800 and its graceful Postgres shutdown (bounded by smartShutdownTimeout ~180s) takes ~2-3m. On GitHub runners (kind-in-Docker on a constrained VM) that shutdown stretches past 5m under load, so the namespace finalization times out. No finalizer is stuck (Cluster and ObjectStore carry none), and the ObjectStore has no plugin finalizer. Verified by reproducing on an isolated, well-resourced cluster: full cleanup completes in ~3m (pod teardown ~2m + namespace GC ~40s). The 5m budget is simply too tight for the slow-runner case. Raise cleanup to 10m. The shutdown is bounded by smartShutdownTimeout, so this won't grow unbounded; 10m gives safe headroom on GitHub runners. Signed-off-by: Abdullah Alaqeel <abdullah.t.aqeel@gmail.com>
Owner
Author
|
Verification complete (run green). Closing this throwaway PR; the fix is upstreamed as cloudnative-pg#940. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verification-only throwaway PR. Bumps the barman-plugin-scheduledbackups cleanup timeout from 5m to 10m to fix intermittent 'context deadline exceeded' teardown failures on GitHub runners. Will be deleted after the run.