fix(inkless:consolidation): don't fence a consolidating leader below the seal#677
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR prevents a switched diskless leader (classic-to-diskless seal committed) from being fenced offline when LEO < seal if the topic can rebuild its classic prefix from remote storage (consolidating diskless with remote enabled). This ensures the ConsolidationReconciler can arm consolidation and recover a wiped leader instead of leaving the partition offline and causing KAFKA_STORAGE_ERROR on reads.
Changes:
- Update
ReplicaManager.maybeReconcileSwitchedLeaderto avoid fencing leaders below the seal when remote recovery is possible. - Extend the reconciliation docstring to document the new recovery path and why fencing blocks it.
- Add a unit test covering the “consolidating leader below seal with remote enabled stays online” case and adjust the test harness to correctly enable
UnifiedLog.remoteLogEnabled().
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| core/src/main/scala/kafka/server/ReplicaManager.scala | Changes switched-leader reconciliation behavior for LEO < seal to preserve availability for remote rebuild. |
| core/src/test/scala/unit/kafka/server/ReplicaManagerInklessTest.scala | Adds coverage for the new recovery behavior and fixes test setup so remote log is actually enabled in UnifiedLog. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
viktorsomogyi
force-pushed
the
svv/ts-unification-leader-below-seal
branch
from
7faa8c0 to
53c392e
Compare
…the seal maybeReconcileSwitchedLeader fenced a switched leader offline whenever its LEO fell below the committed seal. It runs during applyLocalLeadersDelta, before the ConsolidationReconciler, so the wiped-leader recovery from #673 never ran. The reconciler only sees online partitions, and a wiped consolidating leader (LEO 0 < seal) was offline before it could rebuild. Every fetch and offset request then failed with KAFKA_STORAGE_ERROR. For a consolidating diskless topic with remote storage enabled, [0, seal) is in the remote tier and can be rebuilt, so leave the partition online and let the reconciler arm consolidation at the current LEO. Non-consolidating topics still fence, since their classic prefix has no remote copy and LEO < seal is unrecoverable corruption. Co-authored-by: Cursor <cursoragent@cursor.com>
viktorsomogyi
force-pushed
the
svv/ts-unification-leader-below-seal
branch
from
53c392e to
e526d64
Compare
jeqo
reviewed
Jul 1, 2026
jeqo
left a comment
jeqo
left a comment
Contributor
There was a problem hiding this comment.
LGMT, just a small logging improvement
| s"classic-to-diskless start offset $seal; cannot catch up from another replica. " + | ||
| s"Marking the partition offline as its local log is corrupt below the committed seal.") | ||
| markPartitionOffline(tp) | ||
| if (isConsolidatingPartition(partition) && log.remoteLogEnabled()) { |
Contributor
There was a problem hiding this comment.
personal note: this can be simplified when #678 lands -- not a blocker
jeqo
approved these changes
Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
maybeReconcileSwitchedLeader fenced a switched leader offline whenever its LEO fell below the committed seal. It runs during applyLocalLeadersDelta, before the ConsolidationReconciler, so the wiped-leader recovery from #673 never ran. The reconciler only sees online partitions, and a wiped consolidating leader (LEO 0 < seal) was offline before it could rebuild. Every fetch and offset request then failed with KAFKA_STORAGE_ERROR.
For a consolidating diskless topic with remote storage enabled, [0, seal) is in the remote tier and can be rebuilt, so leave the partition online and let the reconciler arm consolidation at the current LEO. Non-consolidating topics still fence, since their classic prefix has no remote copy and LEO < seal is unrecoverable corruption.
Co-authored-by: Cursor cursoragent@cursor.com