Skip to content

[SPARK-58936][SQL] Add partial view updates to ViewCatalog - #58208

Open
manuzhang wants to merge 1 commit into
apache:masterfrom
manuzhang:codex/add-view-changes
Open

[SPARK-58936][SQL] Add partial view updates to ViewCatalog#58208
manuzhang wants to merge 1 commit into
apache:masterfrom
manuzhang:codex/add-view-changes

Conversation

@manuzhang

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR adds a partial-update API for catalog views:

  • Add ViewChange with ordered SetProperty and RemoveProperty changes.
  • Add ViewCatalog.alterView(Identifier, ViewChange...).
  • Provide a backward-compatible default that loads the latest view, applies the changes while preserving all typed view metadata, and calls replaceView.
  • Route ALTER VIEW ... SET TBLPROPERTIES and ALTER VIEW ... UNSET TBLPROPERTIES through one alterView change batch.
  • Keep full metadata operations such as ALTER VIEW ... AS and schema-binding changes on replaceView.

Catalogs can override alterView to translate the full ordered batch into a native atomic update.

Why are the changes needed?

The current V2 property ALTER executors rebuild a complete View from metadata captured during analysis and call replaceView. That prevents connectors from applying native property validation and atomic partial updates, and a full replacement can overwrite unrelated metadata changed concurrently.

A ViewChange API, analogous to TableChange, gives connectors the requested operation instead of a reconstructed full view. The default implementation preserves compatibility for existing ViewCatalog implementations.

Does this PR introduce any user-facing change?

Yes. This adds an evolving connector API for partial view updates. Existing catalogs remain compatible through the default implementation, and SQL SET/UNSET behavior is unchanged. Catalogs that override alterView can provide native validation and atomic update semantics.

How was this patch tested?

  • build/sbt catalyst/Test/compile
  • build/sbt sql/Test/compile
  • build/sbt 'catalyst/testOnly org.apache.spark.sql.connector.catalog.ViewCatalogSuite'
  • SPARK_LOCAL_IP=127.0.0.1 SPARK_LOCAL_HOSTNAME=localhost build/sbt 'sql/testOnly org.apache.spark.sql.execution.command.v2.AlterViewSetTblPropertiesSuite org.apache.spark.sql.execution.command.v2.AlterViewUnsetTblPropertiesSuite'
  • build/sbt catalyst/checkstyle catalyst/scalastyle catalyst/Test/scalastyle sql/scalastyle sql/Test/scalastyle

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)

Co-authored-by: Codex <codex@openai.com>
@manuzhang
manuzhang force-pushed the codex/add-view-changes branch from 4add20e to 7eb5bc2 Compare August 21, 2026 16:05
@manuzhang manuzhang closed this Aug 24, 2026
@manuzhang manuzhang reopened this Aug 24, 2026
@manuzhang
manuzhang marked this pull request as ready for review August 24, 2026 13:57
@manuzhang

Copy link
Copy Markdown
Member Author

@szehon-ho @aokolnychyi @cloud-fan This is the feature gap we found when implementing Iceberg integration with Spark 4.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant