[python][daft] Push filters into Paimon scan planning#8744
Merged
Conversation
Daft passes datasource filters through each Pushdowns request, but PaimonDataSource still relied on mutable state populated by a legacy push_filters callback. The current DataSource path never populated that state, so native Parquet scans skipped Paimon file pruning and scan explain used a separate path. Derive the Paimon predicate from the current Pushdowns for both reads and explain, remove serialized filter state, and add regression coverage that verifies actual task pruning matches explain output. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
PaimonDataSourcerelied on mutable filter state populated by the legacypush_filters()callback. Daft now passes filters through eachPushdownsrequest, so normal native Parquet scans skipped Paimon file pruning.Derive predicates directly from the current
Pushdownsfor reads and explain, remove the serialized filter state, and verify that actual tasks and explain output report consistent file pruning.Tests