Skip to content

fix(materialized_view): add ephemeral CTEs to each view of a multi-view model - #768

Open
DG47 wants to merge 4 commits into
ClickHouse:mainfrom
DG47:fix/multiple-mv-ephemeral-ctes
Open

DG47 wants to merge 4 commits into
ClickHouse:mainfrom
DG47:fix/multiple-mv-ephemeral-ctes

Conversation

@DG47

@DG47 DG47 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #521.

A materialized view model with several --<name>:begin / --<name>:end views fails when a view selects from an ephemeral model:

Code: 60. DB::Exception: Unknown table expression identifier '__dbt__cte__people_ephemeral' ... (UNKNOWN_TABLE)

dbt injects the CTEs of ephemeral models into the model SQL only once (inject_ctes_into_sql in dbt-core). Depending on how the first view starts, they end up either before the first marker (with __dbt__cte__x as (...) --mv1:begin select ...) or inside the with clause of the first view. clickhouse__extract_mv_views only keeps the text between the markers, so the views were created without the CTEs. A model without markers was not affected, because its single view is the whole SQL.

clickhouse__extract_mv_views now reads the injected CTEs from model.extra_ctes and, for each marked view that does not already contain them, adds them: merged into the view's own with clause if it has one (after leading comments, and after recursive), otherwise as a new with clause before the query. Views of models that use no ephemeral models, and models without markers, are unchanged. The target table is still created from the whole model SQL, which already had the CTEs.

Tests

  • Unit (tests/unit/test_extract_mv_views.py): CTEs injected before the first marker, CTEs injected into the first view's with clause, a later view with its own WITH clause after a comment, several ephemeral models, and the unchanged cases. The four new cases that expect added CTEs fail on main.
  • Integration (TestMultipleMVFromEphemeral in test_multiple_materialized_views.py): a two-view model selecting from an ephemeral model, with and without a with clause in the first view. It runs --full-refresh and then a normal run (view query update), and checks that new source rows reach the target table through both views. Both cases fail on main with UNKNOWN_TABLE and pass with the fix.

Commands, against a local clickhouse/clickhouse-server 26.9.17.8:

  • pytest tests/unit: 102 passed
  • DBT_CH_TEST_PORT=18123 pytest tests/integration/adapter/materialized_view tests/integration/adapter/basic: 80 passed, 1 skipped
  • make lint: passes

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A changelog entry was added following Updating the Changelog (user-facing, concise, PR link)

Note

Low Risk
Localized change to MV view splitting macros with broad unit/integration coverage; no auth or data-migration behavior.

Overview
Fixes multi-view materialized_view models that ref() an ephemeral model, which previously failed at run time with Unknown table expression identifier '__dbt__cte__...'.

dbt injects ephemeral CTEs into compiled SQL once (before the first --name:begin marker or inside the first view’s WITH). clickhouse__extract_mv_views only kept text between markers, so later views were deployed without those CTEs. The macro now reads model.extra_ctes, and for any marked view that does not already contain that SQL, clickhouse__add_ctes_to_view_sql prepends or merges the CTEs into the view’s WITH (including after leading comments). Single-view models and multi-view models without ephemerals are unchanged; the target table still uses the full model SQL.

Unit and integration tests cover CTE placement edge cases and end-to-end dbt run / view query updates.

Reviewed by Cursor Bugbot for commit a1b9862. Bugbot is set up for automated code reviews on this repo. Configure here.

DG47 added 2 commits October 24, 2026 06:25
…ew model

dbt injects the CTEs of ephemeral models into the model SQL only once:
before the first `--<name>:begin` marker, or into the `with` clause of
the first view. The marker extraction then gave views without the CTEs,
so a view selecting from an ephemeral model failed with
`Unknown table expression identifier '__dbt__cte__...'`.

Each extracted view that does not contain the injected CTEs now gets
them, merged into its own `with` clause when it has one.

Fixes ClickHouse#521

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit d498c50. Configure here.

@DG47

DG47 commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

A materialized view model with several --<name>:begin / --<name>:end views fails with Unknown table expression identifier '__dbt__cte__...' when a view selects from an ephemeral model, because dbt injects the CTEs once and clickhouse__extract_mv_views only keeps the text between the markers. This fixes #521 by reading the CTEs from model.extra_ctes and adding them to each marked view that lacks them, merged into an existing with clause or prepended as a new one. @canbekley @koletzilla, I'd appreciate a review; the unit and integration tests are described in the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Creating materialized views using TO table syntax does not work with ephemeral models

1 participant