test(inkless:consolidation): add read-from-remote system tests, harness support by viktorsomogyi · Pull Request #654 · aiven/inkless

viktorsomogyi · 2026-06-16T17:59:53Z

System tests covering the read-from-remote-after-prune path for all three consolidation entry points:

born-consolidated,
classic-to-consolidated (switched), and
untiered-diskless -> consolidated (upgrade).

Includes the KafkaService harness support the tests rely on (atomic alter_topic_configs, per-service consolidation toggle).

Copilot

Pull request overview

Adds Inkless consolidation system tests that validate read-from-remote behavior after WAL pruning and complete local log loss across multiple consolidation entry paths, plus KafkaService harness support for atomic multi-config topic alters required by some transitions.

Changes:

Add born-consolidated durability system test verifying earliest/first-fetch-from-0 after wipe+restart.
Add switched classic→diskless consolidated durability system test covering classic-epoch boundary reads after prune + wipe.
Add upgrade test covering untiered-diskless → consolidation-enabled transition and end-to-end readability.
Extend KafkaService with alter_topic_configs() to apply multiple topic configs in one kafka-configs --add-config invocation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
tests/kafkatest/tests/inkless/consolidation_switched_read_from_remote_after_prune_test.py	New switched-topic durability test; includes offset/read helpers using console-consumer.
tests/kafkatest/tests/inkless/consolidation_read_from_remote_after_prune_test.py	New born-consolidated durability test verifying earliest and fetch-from-0 after wipe+restart.
tests/kafkatest/tests/inkless/consolidation_enable_on_untiered_diskless_test.py	New upgrade test for enabling consolidation on an existing untiered diskless topic.
tests/kafkatest/services/kafka/kafka.py	Adds atomic multi-config topic alter helper (`alter_topic_configs`).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

+        and locate the remote segments. ``allow_fail``: a data dir may not hold a
+        replica of this partition."""
+        for data_dir in (self.kafka.DATA_LOG_DIR_1, self.kafka.DATA_LOG_DIR_2):
+            node.account.ssh("rm -rf -- %s/%s-*" % (data_dir, topic), allow_fail=True)


+                if len(parts) == 3 and parts[0] == topic and parts[1] == str(partition):
+                    return int(parts[2])
+        except Exception as e:  # noqa: BLE001 - best effort; tool may not be ready
+            self.logger.warn("Failed to read offset (time=%s) for %s-%d: %s"


+                        first_offset = int(match.group(1))
+                    num_read += 1
+        except Exception as e:  # noqa: BLE001 - best effort; tool may time out
+            self.logger.warn("Failed to read from %s-%d at offset %d: %s"


+                if match:
+                    records.append((int(match.group(1)), int(match.group(2))))
+        except Exception as e:  # noqa: BLE001 - best effort; tool may time out
+            self.logger.warn("Failed to read records from %s-%d at offset %d: %s"


…ne system test Add ConsolidationPipelineTest for a born-consolidated topic, asserting the full WAL -> local log -> remote pipeline: consolidation JMX gauges are exported, the MinIO tiered-storage prefix grows, the Postgres control plane prunes the WAL, and every acked record is still consumable. Consolidation is enabled per-test via KafkaService(consolidation=True) (also flips the isolated controller), so it runs in the unified inkless ducktape suite without a run-wide --globals flag. JmxTool runs on brokers only since the controller never exposes the consolidation MBeans. Add ConsolidationVerifier helper (JMX gauges, MinIO via mc, control plane via psql) and install mc + postgresql-client in the ducker image.

…the seal maybeReconcileSwitchedLeader fenced a switched leader offline whenever its LEO fell below the committed seal. It runs during applyLocalLeadersDelta, before the ConsolidationReconciler, so the wiped-leader recovery from #673 never ran. The reconciler only sees online partitions, and a wiped consolidating leader (LEO 0 < seal) was offline before it could rebuild. Every fetch and offset request then failed with KAFKA_STORAGE_ERROR. For a consolidating diskless topic with remote storage enabled, [0, seal) is in the remote tier and can be rebuilt, so leave the partition online and let the reconciler arm consolidation at the current LEO. Non-consolidating topics still fence, since their classic prefix has no remote copy and LEO < seal is unrecoverable corruption. Co-authored-by: Cursor <cursoragent@cursor.com>

…ss support System tests covering the read-from-remote-after-prune path for all three consolidation entry points: - born-consolidated, - classic-to-consolidated (switched), and - untiered-diskless -> consolidated (upgrade). Includes the KafkaService harness support the tests rely on (atomic alter_topic_configs) and a shared ConsolidationVerifier helper for the JMX/object-store/control-plane assertions and the common topic, switch, offset and read helpers. Co-authored-by: Cursor <cursoragent@cursor.com>

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch 2 times, most recently from 7bd6055 to 055dd90 Compare June 16, 2026 19:37

viktorsomogyi force-pushed the svv/ts-unification-recover-leader branch from 3437d4c to 986aaa3 Compare June 23, 2026 08:44

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 055dd90 to 6451701 Compare June 23, 2026 08:45

viktorsomogyi force-pushed the svv/ts-unification-recover-leader branch 2 times, most recently from d29c23c to 17d63d7 Compare June 23, 2026 12:17

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 6451701 to 6e205fa Compare June 23, 2026 12:59

viktorsomogyi requested a review from Copilot June 23, 2026 13:00

Copilot started reviewing on behalf of viktorsomogyi June 23, 2026 13:00 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

viktorsomogyi changed the base branch from svv/ts-unification-recover-leader to svv/ts-unification-systest-test1 June 23, 2026 14:27

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 6e205fa to 253d3c7 Compare June 23, 2026 14:28

viktorsomogyi force-pushed the svv/ts-unification-systest-test1 branch from 1395419 to b31f319 Compare June 23, 2026 14:30

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch 3 times, most recently from d78b81f to 643ff3d Compare June 23, 2026 14:44

viktorsomogyi requested a review from Copilot June 23, 2026 14:44

Copilot started reviewing on behalf of viktorsomogyi June 23, 2026 14:45 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread tests/kafkatest/tests/inkless/consolidation_read_from_remote_after_prune_test.py Outdated

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 643ff3d to 24eea0d Compare June 23, 2026 14:51

viktorsomogyi requested a review from Copilot June 23, 2026 14:52

Copilot started reviewing on behalf of viktorsomogyi June 23, 2026 14:52 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread tests/kafkatest/services/kafka/kafka.py Outdated

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 24eea0d to 380bfa8 Compare June 23, 2026 15:01

viktorsomogyi requested a review from Copilot June 23, 2026 15:02

Copilot started reviewing on behalf of viktorsomogyi June 23, 2026 15:02 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread tests/kafkatest/tests/inkless/consolidation_enable_on_untiered_diskless_test.py Outdated

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 380bfa8 to 8a416c4 Compare June 23, 2026 15:27

viktorsomogyi requested a review from Copilot June 23, 2026 15:29

Copilot started reviewing on behalf of viktorsomogyi June 23, 2026 15:30 View session

Copilot AI reviewed Jun 23, 2026

View reviewed changes

Comment thread tests/kafkatest/tests/inkless/consolidation_enable_on_untiered_diskless_test.py Outdated

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch 2 times, most recently from 02de918 to e2756bf Compare June 24, 2026 11:38

viktorsomogyi force-pushed the svv/ts-unification-systest-test1 branch from b31f319 to 5f769d0 Compare June 26, 2026 14:56

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from e2756bf to d6aaa2d Compare June 26, 2026 14:58

viktorsomogyi marked this pull request as ready for review June 29, 2026 08:09

viktorsomogyi requested review from giuseppelillo and jeqo June 29, 2026 08:10

viktorsomogyi force-pushed the svv/ts-unification-systest-test1 branch from 5f769d0 to ec364e5 Compare June 29, 2026 08:45

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from d6aaa2d to 1353235 Compare June 29, 2026 12:01

Base automatically changed from svv/ts-unification-systest-test1 to svv/ts-unification-recover-leader June 29, 2026 12:46

viktorsomogyi force-pushed the svv/ts-unification-recover-leader branch 3 times, most recently from c551884 to 89f76e2 Compare June 29, 2026 15:53

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 1353235 to 7ef5195 Compare June 29, 2026 15:58

viktorsomogyi changed the base branch from svv/ts-unification-recover-leader to svv/ts-unification-systest-test1 June 29, 2026 16:00

viktorsomogyi marked this pull request as draft June 29, 2026 16:01

viktorsomogyi force-pushed the svv/ts-unification-systest-test1 branch from 21101ef to 162024b Compare June 30, 2026 09:44

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 7ef5195 to a5d0879 Compare June 30, 2026 09:54

viktorsomogyi requested a review from Copilot June 30, 2026 09:55

Copilot started reviewing on behalf of viktorsomogyi June 30, 2026 09:55 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

viktorsomogyi force-pushed the svv/ts-unification-systest-test1 branch from 162024b to 3b59a48 Compare June 30, 2026 11:38

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from a5d0879 to 9b7ae16 Compare June 30, 2026 11:43

viktorsomogyi and others added 2 commits June 30, 2026 16:58

viktorsomogyi force-pushed the svv/ts-unification-recovery-systest branch from 9b7ae16 to 0011cb9 Compare June 30, 2026 14:58

Base automatically changed from svv/ts-unification-systest-test1 to main June 30, 2026 14:59

viktorsomogyi changed the base branch from main to svv/ts-unification-leader-below-seal June 30, 2026 15:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(inkless:consolidation): add read-from-remote system tests, harness support#654

test(inkless:consolidation): add read-from-remote system tests, harness support#654
viktorsomogyi wants to merge 3 commits into
svv/ts-unification-leader-below-sealfrom
svv/ts-unification-recovery-systest

viktorsomogyi commented Jun 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

viktorsomogyi commented Jun 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants