[core][rest] Push Format Table partition filters to REST catalog - #8785
Merged
JingsongLi merged 4 commits intoJul 22, 2026
Merged
Conversation
Contributor
|
Why not using Paimon Predicate? cc @plusplusjiajia |
sundapeng
force-pushed
the
format-table-partition-filter-rest-api
branch
2 times, most recently
from
July 21, 2026 17:20
ccd3184 to
3feccd2
Compare
sundapeng
changed the title
sundapeng
force-pushed
the
format-table-partition-filter-rest-api
branch
2 times, most recently
from
July 21, 2026 18:04
ba1a312 to
059f464
Compare
sundapeng
changed the title
sundapeng
force-pushed
the
format-table-partition-filter-rest-api
branch
5 times, most recently
from
July 21, 2026 19:26
9999866 to
7974e93
Compare
sundapeng
changed the title
The request carries the partition predicate in its filter field as the JSON serialization of a Paimon Predicate tree. The server evaluates it in a best-effort way: a tree it cannot restore or re-anchor counts as always-true, so the response is a superset of the matching partitions and never misses one, and callers keep applying the predicate on the returned partitions. Pages may be sparse; only an empty nextPageToken ends the listing. The mock server records received filter requests so tests can assert the transport path.
Send the partition predicate of a managed Format Table scan to the partitions/list-by-filter endpoint together with the existing name prefix. The predicate travels as its own flat JSON serialization, so the server evaluates exactly the tree the scan evaluates locally and there is no translation step; the scan keeps no pushdown policy of its own — whatever partition predicate exists goes to the manager. Request sizing is left to the HTTP layer, and the server weakens what it cannot evaluate instead of rejecting it, returning a superset (today that includes temporal and decimal literals, whose JSON round trip stays unrestorable until the literal serde is hardened upstream). Falling back is stateless and stays inside the one catalog the operation loaded: a catalog without the endpoint (an UnsupportedOperationException, into which the REST catalog translates HTTP 501, or the endpoint 404) costs one rejected probe per filtered scan and the same call retries the plain prefix listing, so behavior is identical across copies, serialization and processes. The per-partition filter in the plan is unchanged, and a Spark suite pins the chain end to end: a Spark SQL partition predicate must reach the filtered listing endpoint and restore to the pushed tree.
sundapeng
force-pushed
the
format-table-partition-filter-rest-api
branch
from
July 21, 2026 19:37
7974e93 to
9b577f5
Compare
sundapeng
changed the title
sundapeng
changed the title
sundapeng
marked this pull request as ready for review
July 22, 2026 01:10
JingsongLi
reviewed
Jul 22, 2026
| pageToken, | ||
| pattern)); | ||
| } catch (Exception e) { | ||
| if (!indicatesNoFilterListing(e)) { |
Contributor
There was a problem hiding this comment.
Do not catch this, must support list by filter.
JingsongLi
requested changes
Jul 22, 2026
JingsongLi
left a comment
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
Please update rest-catalog-open-api.yaml
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Push partition predicates for catalog-managed Format Tables to a REST partition-listing endpoint. This lets range, non-leading equality,
IN, and other partition filters reduce catalog results whileFormatTableScankeeps exact local filtering.Follow-up to #8750.
Changes
POST .../partitions/list-by-filter, reusingListPartitionsResponse.PredicateJSON together with the existing leading partition-name prefix; no separate filter grammar or translation layer is introduced.nextPageTokenis empty, including sparse or empty pages, and fail fast on repeated tokens.Cataloginstance for the operation.Testing
RESTCatalogTest: predicate transport, prefix conjunction, pagination, and superset behavior.CatalogFormatTablePartitionManagerTest: filtered pagination, sparse pages, repeated-token detection, prefix enforcement, fallback, andCataloglifecycle.CatalogManagedPartitionScanTest: catalog-managed partition visibility and final local filtering.ManagedFormatTablePartitionPruningTest: Spark SQL predicates reach the REST endpoint end to end.Notes