[SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity - #58222
Closed
szehon-ho wants to merge 2 commits into
Closed
[SPARK-58940][SDP][DOC] Add migration notes for Declarative Pipelines case sensitivity#58222szehon-ho wants to merge 2 commits into
szehon-ho wants to merge 2 commits into
Conversation
… case sensitivity SPARK-58517 made Declarative Pipelines honor spark.sql.caseSensitive when inferring and evolving pipeline table schemas, but described the change as affecting only unreleased branches, so no migration note was added. The previous case-sensitive-only behavior did ship in 4.1.x and 4.2.x, and the change also introduces a new error condition when flows disagree on the effective value.
anew
approved these changes
Aug 22, 2026
anew
left a comment
anew
left a comment
Contributor
There was a problem hiding this comment.
This looks good to me. Thanks @szehon-ho !
uros-b
approved these changes
Aug 22, 2026
Member
|
+1, thank you @szehon-ho and @anew! |
AnishMahto
approved these changes
Aug 22, 2026
dongjoon-hyun
approved these changes
Aug 23, 2026
HyukjinKwon
approved these changes
Aug 24, 2026
…P migration notes Surviving column spelling depends on declared schema, incremental streaming tables, and materialized views, not only on persisted columns.
szehon-ho
added a commit
that referenced
this pull request
Aug 24, 2026
… case sensitivity ### What changes were proposed in this pull request? Adds two entries to the "Upgrading from Spark SQL 4.2 to 4.3" section of `docs/sql-migration-guide.md`, documenting the behavior change made by SPARK-58517 (#57722): - Declarative Pipelines now honors `spark.sql.caseSensitive` when inferring and evolving pipeline table schemas. Under case-insensitive resolution (the default), column names differing only in case identify the same column, so flows contribute a single column rather than one per spelling, and the note states which spelling survives. - All flows writing to one pipeline table must agree on the effective `spark.sql.caseSensitive`, or the update fails with the new `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` error condition. Documentation only; no code change. ### Why are the changes needed? SPARK-58517 first ships in 4.3.0 (merged to `master`, `branch-4.x`, and `branch-4.3`), and the behavior it changed has been in place since 4.1.0: `SchemaMergingUtils.mergeSchemas` merged positionally, leaving `StructType.merge`'s `caseSensitive = true` default; `DataflowGraph.inferredSchema` merged with `reduce(SchemaMergingUtils.mergeSchemas)`; and `SchemaInferenceUtils.diffSchemas` keyed columns on exact field names. Users upgrading from 4.1.x or 4.2.x will see two observable differences that a migration note should cover: - A target schema that previously evolved to carry both spellings, then failed later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`, now folds into a single column. - A pipeline whose flows disagree on `spark.sql.caseSensitive` now fails up front with `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`. ### Does this PR introduce _any_ user-facing change? No. This is a documentation-only change. It documents a user-facing change that was already made by SPARK-58517. ### How was this patch tested? No tests; documentation only. The error condition names in the notes were checked against the tree (`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` in `error-conditions.json`, `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA` in `GraphErrors.scala`), as was the "lowest flow identifier wins" rule (`SchemaInferenceUtils.inferSchemaFromFlows` sorts by `_.identifier.unquotedString`). The link form `declarative-pipelines-programming-guide.html` matches the existing link in `docs/index.md`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 (Cursor) Closes #58222 from szehon-ho/docs-sdp-case-sensitivity-migration-note. Authored-by: Szehon Ho <szehon.apache@gmail.com> Signed-off-by: Szehon Ho <szehon.apache@gmail.com> (cherry picked from commit 5a26f48) Signed-off-by: Szehon Ho <szehon.apache@gmail.com>
szehon-ho
added a commit
that referenced
this pull request
Aug 24, 2026
… case sensitivity ### What changes were proposed in this pull request? Adds two entries to the "Upgrading from Spark SQL 4.2 to 4.3" section of `docs/sql-migration-guide.md`, documenting the behavior change made by SPARK-58517 (#57722): - Declarative Pipelines now honors `spark.sql.caseSensitive` when inferring and evolving pipeline table schemas. Under case-insensitive resolution (the default), column names differing only in case identify the same column, so flows contribute a single column rather than one per spelling, and the note states which spelling survives. - All flows writing to one pipeline table must agree on the effective `spark.sql.caseSensitive`, or the update fails with the new `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` error condition. Documentation only; no code change. ### Why are the changes needed? SPARK-58517 first ships in 4.3.0 (merged to `master`, `branch-4.x`, and `branch-4.3`), and the behavior it changed has been in place since 4.1.0: `SchemaMergingUtils.mergeSchemas` merged positionally, leaving `StructType.merge`'s `caseSensitive = true` default; `DataflowGraph.inferredSchema` merged with `reduce(SchemaMergingUtils.mergeSchemas)`; and `SchemaInferenceUtils.diffSchemas` keyed columns on exact field names. Users upgrading from 4.1.x or 4.2.x will see two observable differences that a migration note should cover: - A target schema that previously evolved to carry both spellings, then failed later with errors such as `COLUMN_ALREADY_EXISTS` or `AMBIGUOUS_REFERENCE`, now folds into a single column. - A pipeline whose flows disagree on `spark.sql.caseSensitive` now fails up front with `CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY`. ### Does this PR introduce _any_ user-facing change? No. This is a documentation-only change. It documents a user-facing change that was already made by SPARK-58517. ### How was this patch tested? No tests; documentation only. The error condition names in the notes were checked against the tree (`CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY` in `error-conditions.json`, `UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMA` in `GraphErrors.scala`), as was the "lowest flow identifier wins" rule (`SchemaInferenceUtils.inferSchemaFromFlows` sorts by `_.identifier.unquotedString`). The link form `declarative-pipelines-programming-guide.html` matches the existing link in `docs/index.md`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 (Cursor) Closes #58222 from szehon-ho/docs-sdp-case-sensitivity-migration-note. Authored-by: Szehon Ho <szehon.apache@gmail.com> Signed-off-by: Szehon Ho <szehon.apache@gmail.com> (cherry picked from commit 5a26f48) Signed-off-by: Szehon Ho <szehon.apache@gmail.com>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds two entries to the "Upgrading from Spark SQL 4.2 to 4.3" section of
docs/sql-migration-guide.md, documenting the behavior change made bySPARK-58517 (#57722):
spark.sql.caseSensitivewhen inferring andevolving pipeline table schemas. Under case-insensitive resolution (the
default), column names differing only in case identify the same column, so
flows contribute a single column rather than one per spelling, and the note
states which spelling survives.
spark.sql.caseSensitive, or the update fails with the newCONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITYerror condition.Documentation only; no code change.
Why are the changes needed?
SPARK-58517 first ships in 4.3.0 (merged to
master,branch-4.x, andbranch-4.3), and the behavior it changed has been in place since 4.1.0:SchemaMergingUtils.mergeSchemasmerged positionally, leavingStructType.merge'scaseSensitive = truedefault;DataflowGraph.inferredSchemamerged with
reduce(SchemaMergingUtils.mergeSchemas); andSchemaInferenceUtils.diffSchemaskeyed columns on exact field names. Usersupgrading from 4.1.x or 4.2.x will see two observable differences that a
migration note should cover:
later with errors such as
COLUMN_ALREADY_EXISTSorAMBIGUOUS_REFERENCE,now folds into a single column.
spark.sql.caseSensitivenow fails upfront with
CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITY.Does this PR introduce any user-facing change?
No. This is a documentation-only change. It documents a user-facing change that
was already made by SPARK-58517.
How was this patch tested?
No tests; documentation only. The error condition names in the notes were
checked against the tree (
CONFLICTING_PIPELINE_FLOW_CASE_SENSITIVITYinerror-conditions.json,UNABLE_TO_INFER_PIPELINE_TABLE_SCHEMAinGraphErrors.scala), as was the "lowest flow identifier wins" rule(
SchemaInferenceUtils.inferSchemaFromFlowssorts by_.identifier.unquotedString). The link formdeclarative-pipelines-programming-guide.htmlmatches the existing link indocs/index.md.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5 (Cursor)