Spark: Make the view stored-schema coercion configurable - #17499
Open
bmorck wants to merge 1 commit into
Open
Conversation
ResolveViews rebuilds a view's output from the stored schema, by position, wrapping each column in an UpCast. UpCast only widens, so a view whose stored type is narrower than what its SQL produces cannot be read at all, and there is no way to relax it. Add spark.sql.iceberg.view.schema-binding-mode, taking its mode names and coercions from Spark's ViewSchemaMode: BINDING (UpCast, the default and current behaviour), COMPENSATION (an ANSI cast, allowing narrowing) and TYPE_EVOLUTION (no cast, so the view reports the types its SQL produces). All three keep the stored column name and metadata. When the conf is unset, Spark's spark.sql.legacy.viewSchemaBindingMode and viewSchemaCompensation are honored instead, reproducing how SessionCatalog.castColToType treats SchemaUnsupported.
bmorck
marked this pull request as ready for review
August 3, 2026 18:51
bmorck
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change adds a new
spark.sql.iceberg.view.schema-binding-modeproperty, used inResolveViewsin Spark 4.1. Prior to this property,ResolveViewsalways wraps each output column in an UpCast that only does widening type coercion. So if the stored schema of the view has a narrower type than what the SQL actually produces, we get an error like the following:The new
spark.sql.iceberg.view.schema-binding-modeintroduces 3 modes to relax this behavior, taking its mode names from Spark'sViewSchemaMode:BINDINGUpCast(col, storedType)— the current behaviour, and the defaultCOMPENSATIONCast(col, storedType, ansiEnabled = true)— narrowing allowedTYPE_EVOLUTIONSpark's has the
spark.sql.legacy.viewSchemaBindingModeandspark.sql.legacy.viewSchemaCompensationconfs to relax this behavior on the v1SessionCatalogview path, but this doesn't apply to v2 views.This change was adapted from: #17453