[test] Stabilize SinkSavepointITCase.testRecoverFromSavepoint - #8741
Merged
JingsongLi merged 1 commit intoJul 20, 2026
Merged
Conversation
testRecoverFromSavepoint intermittently hangs a whole CI job instead of failing within its own budget. For example the "UTCase and ITCase Flink 2.x on JDK 11" run https://github.com/apache/paimon/actions/runs/29686582857/job/88191764951 (surfaced while validating apache#8736) ran the full 2h workflow cap and was cancelled, with this test the only class that started and never reported completion. Two independent problems compound. First, the retry loop wraps jobClient.stopWithSavepoint(...).get() in a broad catch (Exception e). JUnit 5 enforces @timeout(180) in SAME_THREAD mode by delivering a single interrupt to the test thread; when that interrupt lands in Future.get() it surfaces as InterruptedException, which the broad catch swallowed. The interrupt flag is then cleared and no further interrupt is ever scheduled, so @timeout(180) was permanently defeated and the test looped until the 2h job-level cap rather than failing at 180s. The catch now rethrows when the exception carries an InterruptedException, so the deadline fires as intended with a diagnosable stack. Second, the outer loop restarted the job from a savepoint until it reached FINISHED, with no bound on the number of cycles. Each cycle pays a full Flink job-startup plus restore cost, so on a starved runner (the same run had LookupJoinITCase take 30 minutes) the number of cycles needed to drain the input can explode and legitimately exceed the timeout. The number of stop-with-savepoint / restore cycles is now capped; after that the job is allowed to run to completion uninterrupted, which keeps total runtime bounded while still exercising savepoint recovery. The final wait also fails fast if the job reaches a terminal state without finishing, so a dead job is reported clearly instead of as an opaque timeout. Test-only change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
SinkSavepointITCase.testRecoverFromSavepointintermittently hangs an entire CI job instead of failing within its own budget. For example the "UTCase and ITCase Flink 2.x on JDK 11" job at https://github.com/apache/paimon/actions/runs/29686582857/job/88191764951 (which surfaced while validating #8736) ran the full 2h workflow cap and was cancelled; this test was the only class that loggedRunning ...and never reported completion.Two independent problems compound.
First, the retry loop wraps
jobClient.stopWithSavepoint(...).get()in a broadcatch (Exception e). JUnit 5 enforces@Timeout(180)inSAME_THREADmode by delivering a single interrupt to the test thread; when that interrupt lands insideFuture.get()it surfaces asInterruptedException, which the broad catch swallowed. The interrupt flag is then cleared and no further interrupt is ever scheduled, so@Timeout(180)was permanently defeated and the test looped until the 2h job-level cap rather than failing at 180s. The catch now rethrows when the exception carries anInterruptedException, so the deadline fires as intended, with a stack that points at where the test is stuck.Second, the outer loop restarted the job from a savepoint until it reached
FINISHED, with no bound on the number of cycles. Each cycle pays a full Flink job-startup plus restore cost, so on a starved runner (the same run hadLookupJoinITCasetake about 30 minutes) the number of cycles needed to drain the input can explode and legitimately exceed the timeout. The number of stop-with-savepoint / restore cycles is now capped; after that the job is allowed to run to completion uninterrupted, which keeps total runtime bounded while still exercising savepoint recovery. The final wait also fails fast if the job reaches a terminal state without finishing, so a dead job is reported clearly instead of as an opaque timeout.This is a test-only change; no production code is touched.
Tests
SinkSavepointITCase.testRecoverFromSavepoint(the modified test itself).