[core] Support row-count based data file rolling (target-file-num-rows)#8739
Draft
XiaoHongbo-Hope wants to merge 1 commit into
Draft
[core] Support row-count based data file rolling (target-file-num-rows)#8739XiaoHongbo-Hope wants to merge 1 commit into
XiaoHongbo-Hope wants to merge 1 commit into
Conversation
XiaoHongbo-Hope
force-pushed
the
de_row_rolling
branch
3 times, most recently
from
July 22, 2026 10:35
af59abf to
db36b3a
Compare
Files roll by size only today. On data-evolution tables with very wide rows (e.g. vector/embedding columns), a size-bounded file still holds too many rows, so a later add-column / update-by-row-id OOMs materializing that row range. Lance handles this with max_rows_per_file; this adds the same to Paimon. - New option target-file-num-rows (default Long.MAX_VALUE, disabled). Rolling triggers on row count or target-file-size, whichever first; bundle-granular. - Wired through the append write path (incl. the dedicated BLOB/vector writer) and the PK data + changelog writers. Data-evolution base writes therefore roll into row-bounded files, so a later add-column materializes bounded ranges. - Append, bucketed and clustering compaction selection treat a file that reached either target (bytes or rows) as complete, so a row-capped byte-small file is not re-selected forever; files with a deletion vector are still compacted. - Fail fast on non-positive target-file-num-rows; regenerate docs. Data-evolution compaction still merges into a single file for now: making it emit multiple row-bounded files also needs the deletion-vector rewriter to support multi-output and the DE compaction planner to be row-aware, which is left as a follow-up. This PR bounds the data-evolution write path. Tests cover the write primitive, bundle granularity, dedicated writer, DE write rolling and a rolled base + rolled column stitched read, and no-progress avoidance in the append, bucketed (incl. full compaction) coordinators.
XiaoHongbo-Hope
force-pushed
the
de_row_rolling
branch
from
July 22, 2026 11:02
db36b3a to
475bb07
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Files roll by size only. On data-evolution tables with very wide rows (e.g.
vector/embedding columns), a size-bounded file still holds too many rows, so a later
add-column/update-by-row-idOOMs materializing that row range. Lance handles thiswith
max_rows_per_file; this adds the same to Paimon.target-file-num-rows(defaultLong.MAX_VALUE, disabled). Rolls on rowcount or
target-file-size, whichever first; bundle-granular.data + changelog writers, so data-evolution base writes roll into row-bounded files.
as complete, so a row-capped byte-small file isn't re-selected forever; files with a
deletion vector are still compacted.
Scope: this PR bounds the DE write path. DE compaction still merges into a
single file — emitting multiple row-bounded files also needs the deletion-vector rewriter
to support multi-output and a row-aware DE planner, left as a follow-up.
Tests
New test methods added in this PR:
RollingFileWriterTest#testRollingByRows,#testRollingByRowsWithBundleDedicatedFormatRollingFileWriterTest#testRollingByRowsDataEvolutionTableTest#testDataEvolutionWriteRollsByRows,#testDataEvolutionReadWithRolledColumnsAppendOnlySimpleTableTest#testCompactionRespectsRowLimitAppendCompactCoordinatorTest#testFilesAtRowCapAreNotRecompactedBucketedAppendCompactManagerTest#testRowCompleteFilesAreNotPickedFullCompactTaskTest#testRowCompleteFilesAreNotFullCompacted