HIVE-29483: Implement an integration testing mechanism with real Tez on Yarn including resource localization - #6640
Conversation
hemanthumashankar0511
changed the title
hemanthumashankar0511
changed the title
| import java.nio.file.Files; | ||
| import java.nio.file.Path; | ||
|
|
||
| /** |
There was a problem hiding this comment.
I think we don't need this comment entirely naming itself quite self explainatory
| * | ||
| * Run with: mvn -Pitests,tez-yarn -pl itests/tez-yarn-it -Dtest=TezYarnClusterContainerTest test | ||
| */ | ||
| public class TezYarnClusterContainerTest { |
There was a problem hiding this comment.
please use TestTezYarnClusterContainer
| import java.util.regex.Matcher; | ||
| import java.util.regex.Pattern; | ||
|
|
||
| /** |
There was a problem hiding this comment.
please get rid of these large chunk of explainatory comments
try to keep them coincise to convey the purpose wherever the methods and classes are complex enough and need some explicit explanation
please avoid these * Run with: kind of comments
| import java.util.Set; | ||
| import java.util.stream.Stream; | ||
|
|
||
| /** |
There was a problem hiding this comment.
same https://github.com/apache/hive/pull/6640/changes#r3681063334
and here comments are explaining the same redundant thing over and over so please remove those
There was a problem hiding this comment.
Hey @ramitg254 Thanks for pointing it out..I'll get rid of these unwanted comments
|
3 participants
What changes were proposed in this pull request?
Adds an opt-in itests/tez-yarn-it module with a Docker/Testcontainers-based HDFS+YARN cluster and an end-to-end test that verifies hive-exec.jar is localized when Tez runs on YARN.
Why are the changes needed?
MiniTezCluster / TestMiniTezCliDriver do not exercise real YARN resource localization, so regressions like skipping hive-exec.jar from commonLocalResources are not caught today.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Ran locally with mvn test -Pitests,tez-yarn -pl itests/tez-yarn-it -Dtest=TestTezYarnLocalization#testQuerySucceedsWithAppJar, and also validated on GitHub Actions (ubuntu-latest, native Linux amd64) using the same Maven command after mvn clean install -DskipTests -Pitests -pl itests/tez-yarn-it -am; the full end-to-end test is reliable on native Linux and macOS