Skip to content

[MINOR][SS][TEST] Use a managed temp dir for the checkpoint in KafkaRealTimeIntegrationSuite - #58212

Open
Kino1994 wants to merge 1 commit into
apache:masterfrom
Kino1994:fix/kafka-rtm-checkpoint-location
Open

[MINOR][SS][TEST] Use a managed temp dir for the checkpoint in KafkaRealTimeIntegrationSuite#58212
Kino1994 wants to merge 1 commit into
apache:masterfrom
Kino1994:fix/kafka-rtm-checkpoint-location

Conversation

@Kino1994

Copy link
Copy Markdown

What changes were proposed in this pull request?

Use a managed temp directory for the checkpointLocation of the e2e stateless test in KafkaRealTimeIntegrationSuite:

- .option("checkpointLocation", Files.createTempDirectory("some-prefix").toFile.getName)
+ .option("checkpointLocation", Utils.createTempDir().getCanonicalPath)

java.nio.file.Files was only used by that line, so its import is dropped.

Why are the changes needed?

Files.createTempDirectory(...) returns the full path of the directory it creates, but .toFile.getName keeps only the last segment of it:

full path   : /tmp/some-prefix7700179876677442263
getName()   : some-prefix7700179876677442263
isAbsolute  : false

So the option is a relative path, which Spark resolves against the working directory. Each run of the suite leaks two directories, neither of which is cleaned up:

  • the temp directory that was created and then never used, under java.io.tmpdir
  • the checkpoint itself, written into the module directory as connector/kafka-0-10-sql/some-prefix<N>

The second one is not covered by .gitignore, so it shows up as untracked in git status for anyone who runs the module's tests. After two full runs of sql-kafka-0-10 locally:

./connector/kafka-0-10-sql/some-prefix6311020086553351455
./connector/kafka-0-10-sql/some-prefix9434080927785378187
./target/tmp/some-prefix6311020086553351455
./target/tmp/some-prefix9434080927785378187

The suffixes pair up, one pair per run.

Utils.createTempDir() is what the rest of the module already uses for this: it is absolute, lives under java.io.tmpdir, and is deleted on JVM exit. The other five checkpointLocation usages in connector/kafka-0-10-sql all pass an absolute path from a managed temp directory; this line was the only one that did not.

Does this PR introduce any user-facing change?

No. Test-only.

How was this patch tested?

Ran KafkaRealTimeIntegrationSuite and confirmed the test still passes and that no some-prefix* directory is left behind afterwards, in the module directory or under target/tmp.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

…ealTimeIntegrationSuite

`Files.createTempDirectory("some-prefix").toFile.getName` keeps only the last
segment of the created path, so the checkpointLocation is a relative path that
Spark resolves against the working directory. Every run of the suite therefore
leaves two directories behind: the temp directory that was created and never
used, and the checkpoint itself, written into the module directory as an
untracked `connector/kafka-0-10-sql/some-prefix*` and never cleaned up.

Use `Utils.createTempDir().getCanonicalPath`, as the rest of the module's tests
do. It is absolute, lives under `java.io.tmpdir`, and is removed on JVM exit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@uros-b

uros-b commented Aug 22, 2026

Copy link
Copy Markdown
Member

LGTM, thank you @Kino1994!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants