[core][rest] Support catalog-managed partitions for Format Table reads and writes - #8750
Merged
JingsongLi merged 1 commit intoJul 21, 2026
Conversation
This was referenced Jul 20, 2026
sundapeng
changed the title
sundapeng
force-pushed
the
split-v5/format-table-partition-source-core
branch
from
July 20, 2026 17:27
98d10d7 to
6111ecd
Compare
JingsongLi
reviewed
Jul 21, 2026
| .withDescription("Format table uses paimon or engine."); | ||
|
|
||
| public static final ConfigOption<PartitionSource> FORMAT_TABLE_PARTITION_SOURCE = | ||
| key("format-table.partition-source") |
Contributor
There was a problem hiding this comment.
Why not reusing partition metastore option?
sundapeng
force-pushed
the
split-v5/format-table-partition-source-core
branch
from
July 21, 2026 02:38
6111ecd to
423b9e0
Compare
A partitioned Format Table finds its partitions by walking the table directory. That is how Hive works and it stays the default, but it means a directory appearing under the table is a partition, whether or not anyone meant it to be, and that listing a large table costs a full traversal of object storage. With metastore.partitioned-table=true on an internal Format Table in a REST catalog, the catalog becomes the source of truth instead: a scan reads the partitions it has registered, and a batch write registers the ones it wrote. A directory nobody registered is not part of the table, and an empty catalog means an empty table — reads never quietly fall back to the filesystem, since that would defeat the point. A registered partition whose directory is missing reads as empty, matching Hive, where ADD PARTITION before the first insert is ordinary. The option already gates the catalog partition API for Paimon tables in a REST catalog, so a Format Table using it for the same purpose is the same idea, not a second one. On a Format Table it only takes effect in a REST catalog and only for an internal table under the paimon implementation; asking for it elsewhere fails rather than handing back a table reading its partitions from somewhere it did not ask for. Whether a table's partitions come from the catalog is then answered in one place, by whether loading it produced a partition manager. The partition metadata itself is reached through a manager the table carries: paging, per-request batching and pattern encoding are its business, and it creates one catalog per operation and closes it, so the table stays serializable without owning a client. Engine support for the partition DDL this makes possible follows separately.
sundapeng
force-pushed
the
split-v5/format-table-partition-source-core
branch
from
July 21, 2026 03:18
423b9e0 to
bc3cae9
Compare
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
A partitioned Format Table discovers its partitions by listing the table
directory. This adds a second source for an internal Format Table in a REST
catalog, reusing the option that a REST catalog already uses to let it own a
Paimon table's partitions:
The catalog then owns the partitions: a scan reads the ones registered there and
a batch write registers the ones it wrote. An unregistered directory is not part
of the table, and reads do not fall back to the filesystem. A registered
partition with no directory reads as empty, as in Hive.
metastore.partitioned-tablekeeps its meaning on a Paimon table (mirrorpartitions into the metastore) and takes on this second one only for a Format
Table in a REST catalog, which is consistent — the REST server already gates the
partition API on it. The two never collide: a table is catalog-managed only when
loading it produced a partition manager, and that happens only for an internal
Format Table, in a REST catalog, under the paimon implementation. Everywhere else
the option keeps its existing meaning or is inert, so nothing a Paimon table or a
catalog default sets can change a Format Table by accident. Requesting it where it
cannot be served — a non-REST catalog, an external table,
format-table.implementation=engine— fails the load rather than readingpartitions from somewhere it did not ask for.
FormatTablecarries aFormatTablePartitionManagerholding aCatalogLoader;it creates one catalog per operation and closes it, so the table stays
serializable. Paging and per-request batching are internal to it.
No REST protocol change: the endpoints and DTOs from #8707 are untouched.
Spark partition DDL follows in #8751. Splits #8713; supersedes #8728, #8729,
#8730.
Tests
CatalogUtilsTest,CatalogFormatTablePartitionManagerTest,CatalogManagedPartitionScanTest,FormatTableCommitTest,FormatTableCompatibilityTest,FileSystemCatalogTest,MockRESTCatalogTest,PartitionPathUtilsTest— 272 tests.