Skip to content

[core] Support column placement policies for MAP shared-shredding#8743

Merged
JingsongLi merged 1 commit into
apache:masterfrom
lxy-9602:map-shredding-lru
Jul 20, 2026
Merged

[core] Support column placement policies for MAP shared-shredding#8743
JingsongLi merged 1 commit into
apache:masterfrom
lxy-9602:map-shredding-lru

Conversation

@lxy-9602

Copy link
Copy Markdown
Contributor

Purpose

This is a subtask of MAP shared-shredding support.

This PR adds configurable physical column placement policies for MAP shared-shredding.

It introduces the per-field option:

fields.<field-name>.map.shared-shredding.column-placement-policy

Supported policies are:

  • plain: place fields according to the MAP entry order.
  • sequential: place fields according to their dictionary IDs.
  • lru: preserve recently used field-to-column assignments and evict the least recently used columns. This is the default.

The shared allocator now owns common file-level metadata collection, while each policy provides its own row allocation strategy. The selected policy is propagated through MapSharedShreddingWritePlanFactory and applied by MapSharedShreddingRowConverter.

Schema validation now rejects unknown placement policies. Existing physical-layout tests explicitly use plain, while new tests cover all three policies.

Tests

  • Added unit tests for option parsing and schema validation.
  • Added allocator tests for plain, sequential, and lru.
  • Added write-plan and row-converter policy tests.
  • Added ORC/Parquet table-level E2E tests covering all three policies, footer metadata, overflow placement, and logical read-back.
  • Ran the related paimon-common and paimon-core test suites.
  • Spotless and git diff --check passed.

@lxy-9602 lxy-9602 changed the title [Core] Support column placement policies for MAP shared-shredding [core] Support column placement policies for MAP shared-shredding Jul 20, 2026
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 5a70451 into apache:master Jul 20, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants