[spark] Support partition DDL for catalog-managed Format Table partitions#8751
Merged
JingsongLi merged 3 commits intoJul 21, 2026
Merged
Conversation
This was referenced Jul 20, 2026
sundapeng
changed the title
sundapeng
force-pushed
the
split-v5/format-table-partition-source-spark
branch
3 times, most recently
from
July 21, 2026 03:18
b20819b to
12470fe
Compare
3 tasks
A Format Table whose partitions come from the catalog can be given the partition DDL that implies: ADD and DROP PARTITION, and MSCK REPAIR TABLE to reconcile a table whose directories and catalog have drifted apart — after a backfill by an older writer, say. ADD forwards the batch and its IF NOT EXISTS flag to the catalog without looking anything up first, so rejecting a duplicate stays the catalog's decision and the batch stays atomic; it then creates the directory, so the new partition reads as empty rather than as a missing path. DROP unregisters before deleting, so a failed deletion leaves nothing readable behind, and it only touches partitions the catalog actually knows. A repair never deletes data at all, and refuses to act on a listing it could not complete. SHOW PARTITIONS needs no special support: it already asks the scan, which now asks the catalog. Tables whose partitions come from the filesystem keep their previous behaviour and say so when asked for this DDL.
sundapeng
force-pushed
the
split-v5/format-table-partition-source-spark
branch
from
July 21, 2026 04:15
12470fe to
ec556a2
Compare
JingsongLi
reviewed
Jul 21, 2026
| * the persisted value always wins, warning only when it actually disagrees, instead of failing | ||
| * every format table load in that session. | ||
| */ | ||
| private def normalizeCatalogManagedPartitionOptions( |
Contributor
There was a problem hiding this comment.
Do we need this? Just remove it?
JingsongLi
reviewed
Jul 21, 2026
| // with the table FileIO client-side. A directory-deletion failure stays unregistered so | ||
| // partially deleted data is not exposed again. | ||
| PaimonFormatTablePartitionDdlExec.refreshingCache(refreshCache) { | ||
| table.dropFormatTablePartitions( |
Contributor
There was a problem hiding this comment.
Why put these implementation into PaimonPartitionManagement? Just in this class?
JingsongLi
reviewed
Jul 21, 2026
| .map { | ||
| case _: SparkTable => true | ||
| case formatTable: PaimonFormatTable => | ||
| PaimonFormatTablePartitionDdlExec.usesCatalogManagedPartitions(formatTable) |
Contributor
There was a problem hiding this comment.
Why the different treatment? What’s the difference between the file system and the metadata?
- Treat metastore.partitioned-table as a strict persisted property for a Format Table: a conflicting dynamic Spark option now fails the table load (via FormatTable#copy) instead of being silently ignored with a warning. - Move the Format Table partition DDL (add/drop/registration lookup) off the shared PaimonPartitionManagement trait onto PaimonFormatTable, dropping the nullable partitionManager hook every BaseTable inherited and the reverse FormatTable casts; restore the trait's FileStoreTable gate on toPaimonPartitions. - Collapse the DROP PARTITION rewrite back to a single UnresolvedPaimonRelation rule, extended to recognize catalog-managed Format Tables, so the parser no longer duplicates the extractor or depends on the execution package.
Classify the DROP target through RewriteSparkDDLCommands' own injected catalog manager instead of the SparkSession.active singleton, so a catalog-managed Format Table resolves against the session the command actually runs in. The singleton object cached the first session's catalog manager for the whole JVM, so in a multi-session run a REST-catalog table could not be found and the rewrite silently did not fire. Delete the now-unused UnresolvedPaimonRelation. Also cover three gaps: ADD rejects a partition value that escapes the table location, ADD keeps the partition registered when directory creation fails, and refreshingCache surfaces a refresh failure when the operation itself succeeds.
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Adds partition DDL for a Format Table whose partitions come from the catalog —
metastore.partitioned-table = trueon an internal table in a REST catalog (#8750):SHOW PARTITIONSneeds no change — it goes through the scan.ADD forwards the batch and its
IF NOT EXISTSflag to the catalog without aclient-side existence check, so the batch stays atomic, and creates the
partition directory so the new partition reads as empty. DROP unregisters before
deleting and only touches registered partitions, so data awaiting registration
is never removed and a failed deletion leaves nothing readable. MSCK diffs the
directories against the registration using the raw directory values, never
touches data files, and fails closed on an incomplete listing.
A Format Table whose partitions come from the filesystem keeps its current
behaviour and reports an explicit error for this DDL.
Builds on #8750 (merged). Supersedes #8713, #8729, #8730.
Tests
FormatTablePartitionRepairTest,OptionUtilsTest(33) andFormatTablePartitionDdlPlanningTest,FormatTablePartitionManagementTest,CatalogManagedPartitionMsckRepairTest,CatalogManagedPartitionLoadTest(47),including
ALTER TABLE ... ADD PARTITIONend to end against a REST catalog andMSCK with fault injection.