Repository navigation
dbt Core 2.0: dbt v2 specific functionality #736
Description
Sub-issue of #660. Functionality that only exists in dbt Core 2.0 / Fusion / dbt platform (no dbt Core 1.x counterpart in dbt-clickhouse), tracked here so the ClickHouse adapter can support it as well. Unlike the parity sub-issues, most of these items depend on engine or platform pieces owned by dbt Labs; the Notes column says which part is on our side and which is not. It also holds two engine behaviours that are not dbt Core 1.x parity work: usage tracking in the HTTP User-Agent and the relation-cache population timing.
Status legend:
- ✅ implemented — merged and working on dbt-core 2.0 origin/main
- 🟡 partial — partly working on origin/main; Notes say what works vs fails
- 🚧 in progress — implemented in an open PR (linked in Notes)
- 🔴 not started — not working anywhere
| Feature | What it means | Status | Notes |
|---|---|---|---|
| dbt v2 specific functionality | (overall) | 🔴 | Everything built on the engine's SQL understanding is off for ClickHouse because the SQL frontend has no ClickHouse dialect; only dbt platform column lineage works, and only with a project-level quoting workaround (fix requested from dbt Labs in dbt-labs/dbt#16646). Metadata-based source freshness is not implemented (dbt source freshness panics). Record/replay rebuilds relations without their ClickHouse catalog stamps. dbt/<version> is in the HTTP User-Agent but not always first (waits on adbc_clickhouse#85). The relation cache is filled lazily while nodes run (dbt-labs/dbt#16454) |
| Column-level lineage in dbt platform | dbt platform's column lineage view, built from the artifacts a dbt Core 2.0 run produces | 🟡 | Works when the root dbt_project.yml disables quoting (quoting: {database: false, schema: false, identifier: false}), so compiled SQL renders relations as default.my_model. Fails with the default ClickHouse quoting: the lineage inspection reports SQL parsing error — Invalid expression / Unexpected token at the first backtick-quoted relation in compiled_code (e.g. select * from \default`.`my_first_dbt_model`). Because the engine writes no column-lineage artifacts for ClickHouse (see the engine-side row), dbt platform falls back to parsing compiled_code` with a dialect that rejects backtick identifiers. Fix is on the dbt Labs side: a ClickHouse dialect mapping in the platform parser, requested as step 1 of dbt-labs/dbt#16646. The quoting workaround is diagnostic only: unquoted identifiers break relation names that need quoting in ClickHouse |
| Record/replay of adapter calls | dbt Core 2.0 can record every adapter call of a run and replay it later without a warehouse (--fs-record / --fs-replay); relations that cross that boundary are serialized and rebuilt by the adapter (time_machine/serializable_impls.rs) |
🟡 | Works: a relation's database, schema, identifier, type and quoting round-trip. Fails: the ClickHouse catalog stamps are not serialized, so a replayed relation is rebuilt with defaults instead of what the catalog said at record time — can_on_cluster (a relation recorded false for a Replicated database replays as true under a clustered profile and gains ON CLUSTER), can_exchange (a replay renders a rename where the recording ran EXCHANGE TABLES), mvs_pointing_to_it, is_refreshable, refreshable_append. Missing: serialize each stamp as a tri-state and restore it on replay, the way the serializer already handles Databricks' is_delta. Validation caveat: the open-source adapter factory ignores the replay mode, so --fs-record / --fs-replay runs execute live for every adapter; the serializer can only be exercised through unit tests until that engine piece (dbt Labs) is available |
Usage tracking in the HTTP User-Agent |
dbt Core 2.0 identifies itself first in system.query_log.http_user_agent, as dbt/<version> …, so ClickHouse telemetry can attribute queries to dbt |
🟡 | Works: dbt/<version> is in the User-Agent through the driver's clickhouse.client.product_info option (dbt-labs/dbt#16156); DDL/INSERT statements log dbt/2.0.0-rc.1 adbc_clickhouse/0.1.1 clickhouse-rs/0.15.2 (…). Not yet: always first — Arrow reads (SELECT/DESCRIBE) log clickhouse-ext-arrow/0.1.0 dbt/2.0.0-rc.1 … because clickhouse-ext-arrow tags the query last and clickhouse-rs prints the last-added product first (ClickHouse/clickhouse-rs#469, fixed by ClickHouse/clickhouse-rs#484 and released in clickhouse-ext-arrow 0.2.0; the driver still has to adopt that release together with ADBC 0.24 and Arrow 59 — ClickHouse/adbc_clickhouse#85 —, then publish, then dbt Core 2.0 bumps its driver pin; no dbt-side code change expected). Extra key/value pairs in the parenthesised comment are not possible either (clickhouse-rs hardcodes that section). Not dbt Core 1.x parity: that adapter sends dbt-adapters/x dbt-clickhouse/y clickhouse-connect/z (lv:py/…; os:…; os_user:…) |
| ClickHouse dialect in the SQL frontend | The engine's dialect-aware SQL layer (Dialect enum, dialect_of(adapter_type), per-dialect ANTLR lexer) that every static feature below is built on |
🔴 | Missing: (1) an ANTLR lexer crate dbt-lexer-clickhouse under crates/dbt-sql/ (exposing the SEMI_COLON/UNPAIRED_TOKEN tokens the statement splitter dispatches on, registered in dbt-sql-utils' SUPPORTED_DIALECTS and lexer dispatch); (2) a Dialect::ClickHouse variant plus the dialect_of(ClickHouse) arm (crates/dbt-common/src/adapter.rs, today None). The lexer is buildable from open inputs: the Rust-target ANTLR tool is released at sdf-labs/antlr4, the dbt-antlr4 runtime is on crates.io, and ClickHouse's own utils/antlr/ClickHouseLexer.g4 is the grammar (needs an UNPAIRED_TOKEN catch-all and a $tag$ heredoc rule added). The other lexers' grammars live in closed parser crates and the semantic-analysis frontend that consumes Dialect is closed source, so the variant and the grammar's placement need dbt Labs (step 2 of dbt-labs/dbt#16646). Unblocks: replacing the interim hand-written statement scanner (dbt-labs/dbt#16873: heredocs, nested comments, backslash escapes; its tests become the lexer's acceptance tests) with the lexer, and the dialect_of-gated engine paths that return early for ClickHouse today (state-aware defer / run-cache relation parsing, SQL source extraction) |
Static analysis (--static-analysis baseline / strict) |
Compile-time SQL validation, schema inference and type checking without hitting the warehouse; drives the metadata index (--write-metadata), unit-test and data-test static evaluation |
🔴 | Forced to off for every ClickHouse invocation: the adapter is not in STATIC_ANALYSIS_SUPPORTED_ADAPTERS (crates/dbt-adapter-core/src/lib.rs) and disable_static_analysis_if_not_supported overrides any --static-analysis flag or static_analysis model config. Visible in artifacts as static_analysis_off_reason: "configuredoff" on every node. Depends on the dialect row |
Column-level lineage (engine-side, dbt docs / parquet) |
--write-lineage column lineage parquet consumed by the local docs site (dbt docs generate) and by dbt platform when present |
🔴 | The column_lineage hook returns no edges when static analysis is off, and the run logs column-level lineage requires --static-analysis strict; no column lineage written (crates/dbt-main/src/metadata.rs). The generated docs site therefore has no column lineage for ClickHouse projects. Depends on the static analysis row |
| Source freshness from warehouse metadata | dbt source freshness on sources without loaded_at_field: dbt Core 2.0 asks the adapter for each source table's last modification time (freshness_inner → last_altered/is_view) instead of running a query |
🔴 | ClickHouseMetadataAdapter::freshness_inner is todo!(), so dbt source freshness panics for ClickHouse sources (dbt-labs/dbt#14653). Ours to implement: read metadata_modification_time from system.tables (or the max part modification_time from system.parts) for the requested relations. dbt Core 1.x never had a metadata path for ClickHouse (only the loaded_at_field query), so this is new functionality, not parity |
| Relation cache population timing | The relation cache is filled once per schema before the first node runs (dbt-core populate_adapter_cache), so every materialization sees the same existing_relation state (mvs_pointing_to_it, can_exchange, …) |
🔴 | Engine behaviour, not adapter code: dbt Core 2.0 lists a schema lazily when the first model asks for it (and concurrent misses list it again, the last one overwriting), so the state a model sees depends on thread timing; dbt run and dbt run --select <model> can disagree and TestMVWithExplicitTargetForcesOnSchemaChangeFailInTable is flaky. Filed as dbt-labs/dbt#16454; engine fix pending (the per-schema listing itself is in place since dbt-labs/dbt#16429). The Fusion distribution has a pre-run "downloading relations" warm-up that the open dbt-oss build lacks (only defer and dev-clone warm the cache there), so dbt-oss is the more exposed build |
Upstream: dbt-labs/dbt#14653 (source freshness), dbt-labs/dbt#16646 (column-level lineage: platform-side dialect mapping, then a ClickHouse dialect in the engine).
Reacted by Bentsi Leviav
Activity