Repository navigation
dbt Core 2.0 parity: Materialization incremental #704
Description
Sub-issue of #660 — feature-parity tracking for the ClickHouse adapter in dbt Core 2.0 / Fusion versus the Python dbt-clickhouse adapter.
Covers the incremental materialization. On main all five strategies (legacy, append/inserts_only, delete_insert, insert_overwrite, microbatch), Python-exact strategy resolution/validation and on_schema_change work. Distributed variants are tracked in #707.
Status legend: ✅ implemented — merged and working on dbt-core 2.0 origin/main · 🟡 partial — partly working on origin/main; Notes say what works vs fails · 🚧 in progress — implemented in an open PR (linked in Notes) · 🔴 not started — not working anywhere · Combined status "A-🚧->B": the feature is at A on origin/main today; an open PR is driving it to B (🟡 partially working / ✅ fully working)
| Feature | Subfeatures / details | Status | Notes (PR) |
|---|---|---|---|
Materialization: incremental |
(overall) | ✅ | Merged: all strategies, resolution/validation, on_schema_change (dbt-labs/dbt#16023, #16079, #16090) |
| Base incremental flow | Full v1 skeleton: create / full-refresh swap / insert-only path / strategy dispatch, backup + intermediate relations, cleanup tail | ✅ | Strategy block v1-identical since dbt-labs/dbt#16090 |
legacy strategy |
Default: new-data temp + anti-join + swap (EXCHANGE when the server supports it, rename fallback otherwise) | ✅ | dbt-labs/dbt#16090 (resolution); EXCHANGE swap with rename fallback in dbt-labs/dbt#16200 |
append strategy / inserts_only config |
Pure-append path via clickhouse__insert_into; also taken when unique_key is none |
✅ | inserts_only key in dbt-labs/dbt#16023 |
insert_overwrite strategy |
ALTER TABLE … REPLACE PARTITION per partition, with the 26.6+ allow_replace_partition_from_empty_source gate |
✅ | Gate in dbt-labs/dbt#16079 |
insert_only config |
Alias row: the real key is inserts_only |
✅ | dbt-labs/dbt#16023 |
unique_key handling |
string / list join / empty-string / empty-list normalization; compound keys | ✅ | Macro logic v1-identical. Seed non-Nullable typing (dbt-labs/dbt#15912) made the unique-key suite reliable |
incremental_predicates / predicates config |
Extra predicates on the delete_insert/microbatch DELETE | ✅ | Validation ("cannot apply predicates with X strategy") in dbt-labs/dbt#16090 |
Cloud (Shared engine) delete+insert consistency |
select_sequential_consistency = 1 on the DELETE's subquery |
✅ | dbt-labs/dbt#16079 (port of dbt-clickhouse#715) |
adapter.calculate_incremental_strategy / validate_incremental_strategy |
Python-exact resolution (None/default → delete_insert if use_lw_deletes else legacy; +→_) + cross-config validation with impl.py-identical messages |
✅ | dbt-labs/dbt#16090, with unit tests; parse mode resolves through the same function |
on_schema_change |
append_new_columns / sync_all_columns / fail; adapter.check_incremental_schema_changes (system.columns vs DESCRIBE diff, query_settings= kwarg, Python-identical "out of sync" error) + clickhouse__apply_column_changes, codec-aware ALTERs |
✅ | dbt-labs/dbt#16090 |
Column string_type |
Must render String, never FixedString(n) (breaks append_new_columns ALTERs) |
✅ | dbt-labs/dbt#16082 |
delete_insert strategy |
Lightweight DELETE + INSERT | ✅ | dbt-labs/dbt#16090 |
microbatch strategy |
delete_insert machinery + event-time predicates | ✅ | dbt-labs/dbt#16090. The suspected Jinja truthiness crash on __dbt_internal_microbatch_event_time_end is unreachable (v2 injects both batch bounds as datetimes) |
Activity