[split 20/22] executor: offset re-insert fix, volatile pause barrier, per-thread cache invalidation generations, schema-cache TTL - #1375
Open
minguyen9988 wants to merge 1 commit into
Conversation
…che invalidation generations, schema-cache TTL Wires the new core classes into the batch path. The heart of the race-condition fixes. Merge after config, db-ops, core-classes and batch PRs. - Duplicate-insert fix: an already-flushed batch waiting on offset commit was re-INSERTED on every retry pass (one record observed 1,000 times on the combined PR). A flushed batch now only re-checks committability - never re-inserts. - ClickHouseBatchExecutor.isPaused becomes volatile: it is the DDL-vs-DML barrier, written by the Debezium thread and read in the batch threads' spin loop with no other happens-before edge. Without volatile the read can be hoisted and a batch thread can apply DML against a schema mid-DDL - silent corruption, not a stall. - Cache invalidation by generation: the previous remove-on-read signal let the FIRST worker thread consume the invalidation, leaving every other thread on a stale DbWriter (stale column list) forever. Per-table monotonic generation counters let every thread invalidate independently; plus a schema-cache TTL (default 1h) so a missed DDL self-heals. CacheInvalidationManagerTest / CacheInvalidationManagerTtlTest. - DebeziumOffsetManagement.acknowledgeRecord for the single-record path; beginFlush no longer collides with an in-flight flush (the original OffsetStorageWriter "already flushing" crash from the production log). - Integrity gate + freeze manager + error classifier wired into ClickHouseBatchRunnable / ClickHouseBatchWriter; DbWriter carries the cache generation. Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PR: based on the previous PR in the series (
omniwatcher/split-19-batch-statement); this PR's own diff is only its listed files. Merge the series in order; after the predecessor merges, retarget this PR to2.10.0.Wires the new core classes into the batch path. The heart of the race-condition fixes. Merge after config, db-ops, core-classes and batch PRs.
Part of the split of #1353 into independently mergeable sub-PRs (each <= 10 files), so the 2.10.0 branch can absorb the fixes incrementally.
Split out of #1353, which this series replaces. Each sub-PR is <= 10 files; the union of all 22 reproduces the #1353 tree exactly (verified by tree SHA).