[spark] Add managed Format Table partition administration by sundapeng · Pull Request #8730 · apache/paimon

sundapeng · 2026-07-19T07:45:49Z

Relationship

Third part of splitting #8713; stacked on the DDL part #8729 (the diff shown here includes the
two previous commits until they merge — please review only the [spark] Add managed Format Table partition administration commit). Marked draft until the DDL part lands.

Purpose

Administration and repair on top of the DDL part: two procedures, MSCK REPAIR TABLE
and SHOW PARTITIONS.

Main changes

sys.list_format_table_partitions(table, where?, limit?, page_token?): paged listing.
The continuation token is the catalog's own page token, passed through verbatim. With
a where filter, each call returns the matching rows of raw catalog pages, skipping
ahead past fully-filtered pages so an empty result always means the listing is
exhausted; filtered pages can carry fewer than limit rows.
sys.sync_format_table_metadata(table, mode=ADD|DROP|SYNC, dry_run):
filesystem-to-catalog reconciliation. dry_run defaults to true; discovery uses
raw directory values (round-trips with writer-registered values) and fails on a
mid-scan listing error instead of producing a truncated diff (a DROP diff from a
truncated listing would unregister live partitions). Applies diffs in bounded batches;
both directions converge on rerun. Spark cache is refreshed after any non-preview
attempt, including ambiguous failures.
MSCK REPAIR TABLE [ADD|DROP|SYNC] PARTITIONS reuses the sync engine (bare MSCK =
ADD, Hive semantics; DROP is metadata-only). Unmanaged Format Tables and native
Paimon tables keep Spark's rejection.
SHOW PARTITIONS backed by the catalog, capped with a pointer to the paged procedure.

Behavior notes

This first version keeps the procedure surface minimal on purpose: no scope
restriction and no target parameter on the sync procedure (it always reconciles all
partition metadata, exactly like MSCK), and filtered listing pages are allowed to be
sparse instead of maintaining a resumable dense-page cursor. Both can be added later
without breaking the contract.

Tests

Procedure suite (26), MSCK end-to-end against a mock REST server (10), planning suite
extension (23), SHOW partitions exec tests (6).

Route managed Format Table partition discovery through the catalog so scans read exactly the registered partitions and an empty registration reads as empty instead of falling back to directory listing. Validate partition locations against the table path before any read or delete. Register written partitions at commit time in bounded batches, keep IF NOT EXISTS registration idempotent for concurrent writers, and fall back to unmanaged directory discovery when no catalog manages the table.

Route ALTER TABLE ADD/DROP PARTITION for catalog-managed Format Tables through a Spark-side partition gateway. ADD forwards the whole batch with the IF NOT EXISTS flag so the catalog can apply it atomically and creates partition directories; strict batch atomicity stays with the catalog service. DROP unregisters partitions before deleting their directories, expands partial specs through bounded catalog pagination, and never touches data that is not registered. Session-level overrides of the managed flag are ignored when classifying tables, and unmanaged Format Tables fail with an explicit unsupported error.

Add list_format_table_partitions and sync_format_table_metadata procedures for catalog-managed Format Tables. Listing pages through the catalog and embeds a filter fingerprint in page tokens so stale cursors are rejected. The sync procedure diffs the filesystem against the catalog registration, defaults to dry-run, and applies repair diffs in bounded batches. MSCK REPAIR TABLE reuses the same sync engine through an interception ahead of Spark's v2 rejection, and SHOW PARTITIONS runs through a bounded, paginated executor.

JingsongLi · 2026-07-20T05:04:33Z

+`MSCK REPAIR TABLE` fail with an error.
+
+`SHOW PARTITIONS` throws an error when the number of partitions exceeds
+`spark.paimon.format-table.show-partitions.max-results` (default: 10000). For larger tables, use


Hive behavior?

JingsongLi · 2026-07-20T05:04:50Z

+remove the option, use `ALTER TABLE my_table UNSET TBLPROPERTIES ('metastore.partitioned-table')`.
+On a REST catalog, an existing Format Table whose partitions were never registered reads as empty
+until you register them with `MSCK REPAIR TABLE my_table` or
+`CALL sys.sync_format_table_metadata`.


Why not just use MSCK?

JingsongLi · 2026-07-20T05:06:20Z

+     * catalog, or {@code null} if they can.
+     */
+    @Nullable
+    private static String managedFormatTableLoadProblem(


Just throw exception, remove this.

JingsongLi · 2026-07-20T05:08:06Z

+    // Keyed by identifier full name: listings themselves live in per-instance caches, so sharing
+    // a generation counter across table incarnations (or across catalogs using the same name) can
+    // only cause an extra invalidation, never a stale read.
+    private static final Cache<String, AtomicLong> GENERATIONS =


Remove this and all cache.

JingsongLi · 2026-07-20T05:08:34Z

+    // after deserialization.
+    @Nullable private transient Catalog catalog;
+
+    public FormatTableCatalogProvider(Identifier identifier, CatalogLoader catalogLoader) {


Just use CatalogLoader, remove this FormatTableCatalogProvider.

JingsongLi · 2026-07-20T05:09:05Z

+                null);
+    }
+
+    public FormatTableCommit(


Remove old constructor.

JingsongLi · 2026-07-20T05:09:42Z

        }
    }

+    private RuntimeException partitionRegistrationFailure(RuntimeException cause) {


Why need a method?

sundapeng · 2026-07-20T17:10:48Z

Closing in favour of a re-split of this work.

The implementation changed enough since this split that updating the branch in
place would replace every line and leave the review comments pointing at code
that no longer exists:

the serializable provider with its 30s listing cache and JVM-wide generation
counter is gone, replaced by a partition manager that creates one catalog per
operation and closes it;
the managed scan subclass is gone, folded into FormatTableScan;
the two procedures are gone; MSCK REPAIR TABLE is the only entry point;
the feature no longer reuses metastore.partitioned-table. It has its own
option, format-table.partition-source = filesystem | rest, so a
catalog-level default set for Paimon tables cannot turn it on for a Format
Table by accident.

The review comments here are all addressed; the new PRs describe how.

Replaced by:

sundapeng mentioned this pull request Jul 19, 2026

[core][spark] Support managed Format Table partition operations #8713

Closed

3 tasks

sundapeng added 3 commits July 19, 2026 18:42

sundapeng force-pushed the split-v4/format-table-managed-spark-admin branch from c3ad3ad to 02eebd0 Compare July 19, 2026 11:43

sundapeng mentioned this pull request Jul 19, 2026

[core][spark] Support catalog-managed Format Table reads and writes #8728

Closed

JingsongLi reviewed Jul 20, 2026

View reviewed changes

This was referenced Jul 20, 2026

[core][rest] Support catalog-managed partitions for Format Table reads and writes #8750

Merged

[spark] Support partition DDL for catalog-managed Format Table partitions #8751

Merged

sundapeng closed this Jul 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[spark] Add managed Format Table partition administration#8730

[spark] Add managed Format Table partition administration#8730
sundapeng wants to merge 3 commits into
apache:masterfrom
sundapeng:split-v4/format-table-managed-spark-admin

sundapeng commented Jul 19, 2026 •

edited

Loading

Uh oh!

JingsongLi Jul 20, 2026

Uh oh!

JingsongLi Jul 20, 2026

Uh oh!

JingsongLi Jul 20, 2026

Uh oh!

JingsongLi Jul 20, 2026

Uh oh!

JingsongLi Jul 20, 2026

Uh oh!

JingsongLi Jul 20, 2026

Uh oh!

JingsongLi Jul 20, 2026

Uh oh!

sundapeng commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sundapeng commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relationship

Purpose

Main changes

Behavior notes

Tests

Uh oh!

JingsongLi Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

sundapeng commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sundapeng commented Jul 19, 2026 •

edited

Loading