Repository navigation
dbt Core 2.0 parity: Tests / catalog / docs #710
Description
Sub-issue of #660 — feature-parity tracking for the ClickHouse adapter in dbt Core 2.0 / Fusion versus the Python dbt-clickhouse adapter.
Covers tests, catalog, and docs. Data tests, the test-extra-config macros, catalog generation, persist_docs, the docs-generate suite (on 1.10-style yml) and unit tests (including the engine ordering that runs a unit test's given schema fetch after the seed backing that source()) are merged, including persist_docs descriptions containing ;, and the macro tree is byte-identical to v1. docs generate command parity and a generic replacement for the ClickHouse special cases in unit-test schema inference are ⚪️ future work.
Status legend: ✅ implemented — merged and working on dbt-core 2.0 origin/main · 🟡 partial — partly working on origin/main; Notes say what works vs fails · 🚧 in progress — implemented in an open PR (linked in Notes) · 🔴 not started — not working anywhere · ⚪️ future work — explicitly deferred, not being solved now (also listed in the future-work sub-issue) · Combined status "A-🚧->B": the feature is at A on origin/main today; an open PR is driving it to B (🟡 partially working / ✅ fully working)
| Feature | Subfeatures / details | Status | Notes (PR) |
|---|---|---|---|
| Tests / catalog / docs | (overall) | ✅ | Merged: data tests, extra configs, catalog macros, persist_docs (dbt-labs/dbt#16082; descriptions containing ; via dbt-labs/dbt#16873), docs-generate suite, unit tests (dbt-labs/dbt#16153), unit-test render ordered after the seed backing a given source() (merged, closes dbt-labs/dbt#16202); persist_docs.sql byte-identical to v1. Deferred: docs generate parity and generic unit-test schema inference (#731) |
| Data tests | Standard and generic tests | ✅ | TestGenericTests, TestSingularTests, TestSingularTestsEphemeral pass |
| Data tests extra configs | clickhouse__get_test_sql + clickhouse__place_limit (LIMIT injection needs the -- end_of_sql marker from get_model_settings) |
✅ | Macros byte-identical to v1; the -- end_of_sql marker comes from get_model_settings (dbt-labs/dbt#16079). Generic test configs (severity, warn_if/error_if, where, limit, store_failures) are engine-side. Test limit injection has no direct integration-suite coverage |
catalog.sql (get_catalog / get_catalog_relations) |
Catalog generation macros; dbt compile --write-catalog |
✅ | Merged (closes dbt-labs/dbt#15896): both macros synced from dbt-clickhouse#657 plus Rust record-batch parsers for the ADBC Arrow wire types, unit-tested. Exercised end-to-end by the docs-generate suite |
| Unit-test materialization | unit.sql; v2 executes unit tests through its own renderable rather than this macro |
✅ | Macro byte-identical to v1. Unit tests run since dbt-labs/dbt#16153 (the expected-schema probe no longer relies on a session-scoped TEMPORARY table, which ClickHouse never lists in system.columns) |
| Generic-test execution ordering | EscapeSingleQuotesBackslash: UNKNOWN_TABLE …test_escape_single_quotes |
✅ | Not an ordering problem: the model SQL contains \', which the statement splitter could not lex, so the CREATE was silently dropped. Fixed on main (the splitter no longer drops unlexable statements) and completed by dbt-labs/dbt#16153 (ClickHouse statements routed to a lexer that tokenizes backslash escapes, so multi-statement SQL splits correctly too) |
| Unit-test schema inference (v2 renderable) | ClickHouse forced onto the query-schema DESCRIBE path; always refetch given/expected fixture schemas (no-TTL schema cache goes stale across rebuilds); dbt-core fixture-error message parity | ✅ | dbt-labs/dbt#16153; the string_type normalization half in dbt-labs/dbt#16082. Both ClickHouse branches are adapter-type special cases inside the engine's unit-test renderable; replacing them with generic engine behaviour is ⚪️ future work (row below, #731) |
unit.sql double-execution fix |
Drop the run_query() wrap around get_create_table_as_sql (upstream dbt-clickhouse#690) |
✅ | dbt-labs/dbt#16153, byte-identical to v1. Parity-only: v2 never runs the adapter's unit macro (the engine builds and executes the unit-test SQL itself), so the double execution cannot occur in v2 |
Unit test given on a seed-backed source() read before the seed has run |
Ordering a unit test's render-time given schema fetch after the seed that creates that source() in the same dbt build (static analysis off) |
✅ | Merged on main (closes dbt-labs/dbt#16202): the task graph waits for the seed backing an overlapping source() before rendering the unit test, so a first dbt build on a fresh schema passes (dbt1308 … fetching schema for unit test upstream relation from given no longer occurs) |
QuotedColumn comment test |
TestPersistDocsCommentOnQuotedColumn |
✅ | Passes on main (the fixture's materialized key without + is handled on the harness side) |
dbt docs generate suite |
BaseDocsGenerate ×2, BaseDocsGenReferences |
✅ | Passes on main with 1.10-style yml (the upstream fixtures' pre-1.10 docs/meta/tags and properties-level seed quote_columns are rejected by v2 as dbt1060/dbt1013 — an engine yml policy, migrated on the harness side). --no-compile is the deferred row below |
persist_docs |
Comment DDL | ✅ | Relation and column comments persist on incremental, table and view (dbt-labs/dbt#16082; BaseComment ×2, PersistDocsColumnMissing, QuotedColumn pass), including descriptions containing ; (dbt-labs/dbt#16873; TestPersistDocs passes) |
persist_docs with ; in a description |
Splitting the ALTER … MODIFY COMMENT $dbt_comment_literal_block$…$ heredoc |
✅ | dbt-labs/dbt#16873: ClickHouse statements are split by a ClickHouse-aware scanner ($tag$ heredocs, nested /* */, #/// comment lines, backslash escapes), so the comment DDL reaches the server intact for tables and views. Interim until the ClickHouse dialect row in #736 lands |
persist_docs.sql byte-parity |
The validate_doc_columns warn call (v2 warns about descriptions for columns the relation lacks; v1 silently filtered them) |
✅ | Byte-identical: dbt-clickhouse adopted v2's validate_doc_columns call (warns about descriptions for columns the relation lacks; needs dbt-adapters ≥ 1.22.10), so the v2 macro is the reference |
dbt docs generate command parity |
docs generate --no-compile; docs-site asset copying (asset-paths → target/assets) |
⚪️ | Deferred (#731). v2's docs generate builds a static site from the parquet index and writes no catalog.json; the catalog comes from compile --write-catalog, which has no --no-compile and copies no assets |
| Unit-test schema inference without adapter special cases | Replace the two ClickHouse-only branches in the engine's unit-test renderable (query-schema inference, forced schema refetch) with generic engine behaviour | ⚪️ | Deferred (#731). Unit tests work through the special cases; the forced refetch disables the schema cache for ClickHouse and is the part to roll back |
Activity