Skip to content

[AURON #2392] Support Spark 4.2 #2404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
XorSum wants to merge 3 commits into
base: master
Choose a base branch
from
Open

[AURON #2392] Support Spark 4.2 #2404

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5a9c3d3
Support Spark 4.2.0
XorSum Jul 17, 2026
8d072a7
Update .github/workflows/build-amd64-releases.yml
XorSum Jul 20, 2026
fc6eb77
update CONTRIBUTING.md and pom.xml
XorSum Aug 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .github/workflows/build-amd64-releases.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ jobs:
runs-on: ${{ matrix.runner }}
strategy:
matrix:
sparkver: [spark-3.0, spark-3.1, spark-3.2, spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1]
sparkver: [spark-3.0, spark-3.1, spark-3.2, spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1, spark-4.2]
scalaver: [ 2.12, 2.13 ]
javaver: [ 8, 21 ]
auronver: [9.0.0-SNAPSHOT]
Expand Down Expand Up @@ -70,17 +70,24 @@ jobs:
scalaver: '2.12'
- sparkver: spark-4.1
scalaver: '2.12'
- sparkver: spark-4.2
scalaver: '2.12'
- sparkver: spark-4.0
javaver: '8'
- sparkver: spark-4.1
javaver: '8'
- sparkver: spark-4.2
javaver: '8'
# Spark 4.x JDK 21 builds are handled by explicit rockylinux8 include entries below.
- sparkver: spark-4.0
scalaver: '2.13'
javaver: '21'
- sparkver: spark-4.1
scalaver: '2.13'
javaver: '21'
- sparkver: spark-4.2
scalaver: '2.13'
javaver: '21'
include:
# Spark 4.x uses rockylinux8 image (with JDK 21)
- sparkver: spark-4.0
Expand All @@ -95,6 +102,12 @@ jobs:
auronver: 9.0.0-SNAPSHOT
runner: ubuntu-24.04
image: rockylinux8
- sparkver: spark-4.2
scalaver: '2.13'
javaver: '21'
auronver: 9.0.0-SNAPSHOT
runner: ubuntu-24.04
image: rockylinux8

steps:
- uses: actions/checkout@v7
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/build-arm-releases.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
runs-on: ${{ matrix.runner }}
strategy:
matrix:
sparkver: [spark-3.0, spark-3.1, spark-3.2, spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1]
sparkver: [spark-3.0, spark-3.1, spark-3.2, spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1, spark-4.2]
auronver: [9.0.0-SNAPSHOT]
scalaver: [2.12, 2.13]
javaver: [ 8, 21 ]
Expand Down Expand Up @@ -69,10 +69,14 @@ jobs:
scalaver: '2.12'
- sparkver: spark-4.1
scalaver: '2.12'
- sparkver: spark-4.2
scalaver: '2.12'
- sparkver: spark-4.0
javaver: '8'
- sparkver: spark-4.1
javaver: '8'
- sparkver: spark-4.2
javaver: '8'

steps:
- uses: actions/checkout@v7
Expand Down
6 changes: 5 additions & 1 deletion .github/workflows/build-macos-releases.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
runs-on: ${{ matrix.runner }}
strategy:
matrix:
sparkver: [spark-3.0, spark-3.1, spark-3.2, spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1]
sparkver: [spark-3.0, spark-3.1, spark-3.2, spark-3.3, spark-3.4, spark-3.5, spark-4.0, spark-4.1, spark-4.2]
auronver: [9.0.0-SNAPSHOT]
scalaver: [2.12, 2.13]
javaver: [8, 21]
Expand Down Expand Up @@ -69,10 +69,14 @@ jobs:
scalaver: '2.12'
- sparkver: spark-4.1
scalaver: '2.12'
- sparkver: spark-4.2
scalaver: '2.12'
- sparkver: spark-4.0
javaver: '8'
- sparkver: spark-4.1
javaver: '8'
- sparkver: spark-4.2
javaver: '8'

steps:
- uses: actions/checkout@v7
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tpcds-reusable.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ jobs:
if: steps.cache-spark-bin.outputs.cache-hit != 'true'
run: |
SPARK_PATH="spark/spark-${{ steps.get-dependency-version.outputs.sparkversion }}"
if [[ ${{ inputs.scalaver }} = "2.13" && "${{ inputs.sparkver }}" != "spark-4.0" && "${{ inputs.sparkver }}" != "spark-4.1" ]]; then
if [[ ${{ inputs.scalaver }} = "2.13" && "${{ inputs.sparkver }}" != "spark-4.0" && "${{ inputs.sparkver }}" != "spark-4.1" && "${{ inputs.sparkver }}" != "spark-4.2" ]]; then
SPARK_FILE="spark-${{ steps.get-dependency-version.outputs.sparkversion }}-bin-${{ inputs.hadoop-profile }}-scala${{ inputs.scalaver }}.tgz"
else
SPARK_FILE="spark-${{ steps.get-dependency-version.outputs.sparkversion }}-bin-${{ inputs.hadoop-profile }}.tgz"
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/tpcds.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,3 +112,13 @@ jobs:
scalaver: '2.13'
hadoop-profile: 'hadoop3'
sparktests: 'true'

test-spark-42-jdk21-scala-2-13:
name: Test spark-4.2 JDK21 Scala-2.13
uses: ./.github/workflows/tpcds-reusable.yml
with:
sparkver: spark-4.2
javaver: '21'
scalaver: '2.13'
hadoop-profile: 'hadoop3'
sparktests: 'true'
3 changes: 2 additions & 1 deletion CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ Auron provides a unified build script `auron-build.sh` that supports both local
Run `./auron-build.sh --help` to see all available options, including:

- `--pre` or `--release`: Build profile
- `--sparkver`: Spark version (3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1)
- `--sparkver`: Spark version (3.0, 3.1, 3.2, 3.3, 3.4, 3.5, 4.0, 4.1, 4.2)
- `--scalaver`: Scala version (2.12, 2.13)
- `--celeborn`, `--uniffle`, `--paimon`, `--iceberg`: Optional integrations
- `--skiptests`: Skip unit tests (default: true)
Expand All @@ -142,6 +142,7 @@ integration is not supported for that Spark version and must not be passed to
| 3.5 | 2.12 | 8+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | 1.10.1 | 0.15 |
| 4.0 | 2.13 | 17+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | 1.10.1 | — |
| 4.1 | 2.13 | 17+ | `rockylinux8` | 1.18 | 0.6 | 0.10 | 1.2 | — | — |
| 4.2 | 2.13 | 17+ | `rockylinux8` | - | - | - | - | — | — |

Spark 4.x requires Scala 2.13 and JDK 17 or later. In Docker mode,
`auron-build.sh` selects JDK 17 for Spark 4.x, and the `rockylinux8` image applies that
Expand Down
2 changes: 1 addition & 1 deletion auron-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
# Define constants for supported component versions
# -----------------------------------------------------------------------------
SUPPORTED_OS_IMAGES=("centos7" "ubuntu24" "rockylinux8" "debian11" "azurelinux3")
SUPPORTED_SPARK_VERSIONS=("3.0" "3.1" "3.2" "3.3" "3.4" "3.5" "4.0" "4.1")
SUPPORTED_SPARK_VERSIONS=("3.0" "3.1" "3.2" "3.3" "3.4" "3.5" "4.0" "4.1" "4.2")

@slfan1989 slfan1989 Aug 1, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding Spark 4.2 support! Could you also update CONTRIBUTING.md? Its supported-version list and compatibility matrix currently stop at Spark 4.1, and the document requires them to be updated when support changes. Please add Spark 4.2 with Scala 2.13/JDK 17+ and note that Iceberg and Hudi are unsupported.

@XorSum XorSum Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slfan1989 Thanks for the review! I've updated both files

SUPPORTED_SCALA_VERSIONS=("2.12" "2.13")
SUPPORTED_CELEBORN_VERSIONS=("0.5" "0.6")
# Currently only one supported version, but kept plural for consistency
Expand Down
4 changes: 2 additions & 2 deletions auron-spark-tests/common/src/test/scala/org/apache/spark/sql/SparkExpressionTestsBase.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,7 @@ trait SparkExpressionTestsBase
Column(expression)
}

@sparkver("4.0 / 4.1")
@sparkver("4.0 / 4.1 / 4.2")
private def columnFromExpression(expression: Expression): Column = {
new Column(org.apache.spark.sql.classic.ExpressionColumnNode(expression))
}
Expand All @@ -398,7 +398,7 @@ trait SparkExpressionTestsBase
_spark.internalCreateDataFrame(rows, schema)
}

@sparkver("4.0 / 4.1")
@sparkver("4.0 / 4.1 / 4.2")
private def internalCreateDataFrame(
rows: org.apache.spark.rdd.RDD[InternalRow],
schema: StructType): org.apache.spark.sql.classic.DataFrame = {
Expand Down
2 changes: 1 addition & 1 deletion auron-spark-ui/src/main/scala/org/apache/spark/sql/execution/ui/AuronAllExecutionsPage.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ private[ui] class AuronAllExecutionsPage(parent: AuronSQLTab) extends WebUIPage(
UIUtils.headerSparkPage(request, "Auron", buildInfoSummary(sqlStore.buildInfo()), parent)
}

@sparkver("4.0 / 4.1")
@sparkver("4.0 / 4.1 / 4.2")
override def render(request: jakarta.servlet.http.HttpServletRequest): Seq[Node] = {
UIUtils.headerSparkPage(request, "Auron", buildInfoSummary(sqlStore.buildInfo()), parent)
}
Expand Down
58 changes: 58 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1156,6 +1156,64 @@
</build>
</profile>

<profile>
<id>spark-4.2</id>
<properties>
<shimName>spark-4.2</shimName>
<scalaTestVersion>3.2.9</scalaTestVersion>
<sparkVersion>4.2.0</sparkVersion>
<parquetVersion>1.17.0</parquetVersion>
<shortSparkVersion>4.2</shortSparkVersion>
<nettyVersion>4.2.13.Final</nettyVersion>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>${maven-enforcer-plugin.version}</version>
<executions>
<execution>
<id>spark42-enforce-java-scala-version</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<!-- Spark 4.2 requires JDK 17+ and Scala 2.13.x -->
<requireJavaVersion>
<version>[17,)</version>
<message>Spark 4.2 requires JDK 17 or higher. Current: ${java.version}</message>
</requireJavaVersion>
<requireProperty>
<property>scalaLongVersion</property>
<regex>2\.13\.\d+</regex>
<regexMessage>Spark 4.2 requires Scala 2.13.x. Current: ${scalaLongVersion}</regexMessage>
</requireProperty>
</rules>
</configuration>
</execution>
<execution>
<id>spark42-disallow-iceberg</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireProperty>
<property>icebergEnabled</property>
<regex>false</regex>
<regexMessage>Spark 4.2 is not supported for Iceberg. Use Spark 3.4-4.0 or disable icebergEnabled.</regexMessage>
</requireProperty>
</rules>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>

<profile>
<id>jdk-8</id>
<activation>
Expand Down
4 changes: 2 additions & 2 deletions ...-shims-spark/src/main/scala/org/apache/spark/sql/auron/InterceptedValidateSparkPlan.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import org.apache.auron.sparkver

object InterceptedValidateSparkPlan extends Logging {

@sparkver("3.2 / 3.3 / 3.4 / 3.5 / 4.0 / 4.1")
@sparkver("3.2 / 3.3 / 3.4 / 3.5 / 4.0 / 4.1 / 4.2")
def validate(plan: SparkPlan): Unit = {
import org.apache.spark.sql.execution.adaptive.BroadcastQueryStageExec
import org.apache.spark.sql.execution.auron.plan.NativeRenameColumnsBase
Expand Down Expand Up @@ -79,7 +79,7 @@ object InterceptedValidateSparkPlan extends Logging {
throw new UnsupportedOperationException("validate is not supported in spark 3.0.3 or 3.1.3")
}

@sparkver("3.2 / 3.3 / 3.4 / 3.5 / 4.0 / 4.1")
@sparkver("3.2 / 3.3 / 3.4 / 3.5 / 4.0 / 4.1 / 4.2")
private def errorOnInvalidBroadcastQueryStage(plan: SparkPlan): Unit = {
import org.apache.spark.sql.execution.adaptive.InvalidAQEPlanException
throw InvalidAQEPlanException("Invalid broadcast query stage", plan)
Expand Down
Loading
Loading