test(cluster): raise scheduledbackups cleanup timeout to 10m#940
Closed
aqeelat wants to merge 1 commit into
Closed
test(cluster): raise scheduledbackups cleanup timeout to 10m#940aqeelat wants to merge 1 commit into
aqeelat wants to merge 1 commit into
Conversation
The barman-plugin-scheduledbackups test intermittently fails with 'context deadline exceeded' during cleanup. The test logic passes; the failure is the namespace teardown exceeding the 5m cleanup budget. Root cause: the CNPG instance pod has terminationGracePeriodSeconds=1800 and its graceful Postgres shutdown (bounded by smartShutdownTimeout ~180s) takes ~2-3m. On GitHub runners (kind-in-Docker on a constrained VM) that shutdown stretches past 5m under load, so the namespace finalization times out. No finalizer is stuck (Cluster and ObjectStore carry none), and the ObjectStore has no plugin finalizer. Verified by reproducing on an isolated, well-resourced cluster: full cleanup completes in ~3m (pod teardown ~2m + namespace GC ~40s). The 5m budget is simply too tight for the slow-runner case. Raise cleanup to 10m. The shutdown is bounded by smartShutdownTimeout, so this won't grow unbounded; 10m gives safe headroom on GitHub runners. Signed-off-by: Abdullah Alaqeel <abdullah.t.aqeel@gmail.com>
dosubot
Bot
added
the
size:XS
Author
|
Closing in favor of a proper fix: set a low |
itay-grudev
pushed a commit
that referenced
this pull request
Jul 6, 2026
The barman-plugin-scheduledbackups test intermittently fails with 'context deadline exceeded' during cleanup. The chart forces immediate: true on scheduled backups (templates/scheduled-backups.yaml), so a backup is still running when cleanup starts; Barman blocks instance shutdown until it completes, pushing teardown past the 5m cleanup budget on GitHub runners. Expose cluster.stopDelay in the chart (values + schema + template), mapping to the CNPG Cluster spec.stopDelay, which also sets the pod's terminationGracePeriodSeconds. The scheduledbackups test sets stopDelay: 30 so the instance is force-stopped promptly at teardown, interrupting the still-running backup. That's acceptable because this test only verifies a backup is started from the ScheduledBackup; backup success is covered by barman-plugin-backup-restore. Result: passes reliably and runs in ~75s (down from ~6m, previously timing out). Verified green in fork CI. This is the 'help it clean up faster' approach rather than raising the cleanup timeout. Supersedes #940. Supplements #924. Signed-off-by: Abdullah Alaqeel <abdullah.t.aqeel@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
barman-plugin-scheduledbackupstest intermittently fails withcontext deadline exceededduring cleanup. The test logic passes; only namespace teardown exceeds the 5m budget.Root cause: the CNPG instance pod has
terminationGracePeriodSeconds=1800and graceful Postgres shutdown (bounded bysmartShutdownTimeout~180s) takes ~2–3m. On GitHub runners (kind-in-Docker on a constrained VM) that stretches past 5m under load. No stuck finalizer — Cluster and ObjectStore carry none (the plugin sets no ObjectStore finalizer).Verified by reproducing on an isolated, well-resourced cluster: full cleanup completes in ~3m (pod teardown ~2m + namespace GC ~40s). The 5m budget is simply too tight for the slow-runner case.
Fix: raise
cleanup: 5m → 10m. Shutdown is bounded bysmartShutdownTimeout, so this will not grow unbounded; 10m gives safe headroom. Confirmed green in fork CI.Supplements #924.