[alepietrobon]

Summary

Fixes #784. Related: #662 (snapshot tracking issue), #291, #562.

After the first build, snapshots could silently skip keys that are new in the source, and hard_deletes='invalidate' never closed keys removed from it. Every run still reported OK.

Cause

Both staging macros (timestamp and check strategy) found these rows with the ANSI anti-join idiom:

from insertions_source_data as source_data
left outer join snapshotted_data on <key join>
where snapshotted_data.dbt_unique_key is null
   or (snapshotted_data.dbt_unique_key is not null and <row_changed>)

With ClickHouse's default join_use_nulls = 0, an unmatched LEFT JOIN row is filled with type defaults, not NULL. A non-nullable key becomes 0 / '', so is null is never true:

• New keys are inserted only if row_changed happens to be true for the default-filled snapshot row:
  • timestamp strategy: snapshotted_data.dbt_valid_from < source.updated_at. If updated_at is non-nullable, the default dbt_valid_from is 1970-01-01 and the comparison is true by accident. If updated_at is Nullable, the default is NULL and the new key is dropped.
  • check strategy: a new key is dropped when every check column equals its type default, for example all NULL for Nullable check columns.
• Hard deletes (left join deletes_source_data ... where source_data.dbt_unique_key is null) never match a non-nullable key, so removed keys are never closed.

This is why the existing tests passed: seeds load as non-nullable types (Int32, DateTime, ...), so BaseSnapshotTimestamp only hits the accidental 1970-01-01 < updated_at case. No test covered hard_deletes='invalidate'.

In a real project on 1.10.3 (single String key, Nullable(DateTime) updated_at), a grown source ended with 4,920 of 9,775 current keys in the snapshot, and 0 of 4,855 removed keys were closed. With this fix: 9,775 current, 4,855 closed, none closed wrongly.

Fix

Use LEFT ANTI JOIN, which does not depend on NULL-filling. It keeps using the existing unique_key_join_on helper, so single and list unique_keys (#751) both work:

• insertions: new keys come from left anti join snapshotted_data, and changed keys from an inner join on row_changed (the same join updates already uses). The two are combined with union all.
• deletes: from snapshotted_data left anti join deletes_source_data. The right-side source_data.* columns are type defaults. They only keep the column layout for the positional union all; the merge reads only dbt_change_type, dbt_scd_id and dbt_valid_to from a delete row, as before.

Alternative considered: append settings join_use_nulls = 1 to the staging query. I decided against it. It is session-dependent, it interacts with the SETTINGS that create_table_as emits, and it makes the right-hand columns Nullable, which changes the staging table's types. An anti join is correct under either setting.

hard_deletes='new_record' (#562) is out of scope.

Tests

New tests/integration/adapter/basic/test_snapshot_new_and_deleted_keys.py uses a seed with a non-nullable key and Nullable name / updated_at (column_types). After the first snapshot it adds two keys (one with a NULL check column) and removes one. It then asserts the new keys are current, the removed key is closed, and a further run changes nothing. It covers timestamp and check strategies, each with a single and a compound unique_key (4 classes).

• On main: all 4 fail. The removed key is never closed; the timestamp strategy drops both new keys; the check strategy drops the one whose check column is NULL.
• With the fix: all 4 pass.
• Regression: tests/integration/adapter/basic/test_snapshot_*.py, grants, dbt_clone: 21 passed, 1 skipped (test_distributed_grants, not on a cluster) on ClickHouse 26.10. test_snapshot_*.py also passes on 26.3.
• Not run: cluster mode (DBT_CH_TEST_CLUSTER).
• make lint passes.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches core snapshot SQL generation for inserts and hard-delete invalidation, so incorrect joins could misclassify rows; mitigated by focused anti-join logic and new integration coverage but still impacts snapshot data correctness.

Overview
ClickHouse snapshot staging now finds brand-new keys and hard deletes using LEFT ANTI JOIN plus a targeted inner join, instead of LEFT JOIN ... WHERE key IS NULL, which failed when join_use_nulls stays at the default (0) and defaulted keys aren’t actually NULL.

The insertions CTE now unions “purely new keys” with “existing keys whose monitored columns changed,” and deletes under hard_deletes='invalidate' now anti-joins snapshotted rows against the current source so removed keys reliably close even for non-nullable keys.

CHANGELOG calls out that this fixes silently dropped new snapshot rows (especially with nullable updated_at / check columns) and invalidate hard deletes never closing rows.

A new integration suite test_snapshot_new_and_deleted_keys.py seeds nullable fields, adds/removes rows, and covers timestamp vs check strategies with scalar and compound unique_keys to lock the regression in.

^{Reviewed by Cursor Bugbot for commit 5ed3f66. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

🟢 Approval recommended

No active review issues block approval of the snapshot key-handling changes.

No open issues 🎉

What changed in this PR

Fixes snapshot handling of new and removed keys under ClickHouse’s default join_use_nulls = 0, addressing #784 and the hard-delete invalidation issue tracked in #662.

Changes:

• Uses anti joins to detect new and deleted keys, while retaining inner joins for changed rows.
• Adds regression coverage for timestamp and check strategies with single and compound keys, nullable columns, and repeat-run stability.
• Documents the fix in the changelog; hard_deletes='new_record' remains out of scope.

File	Description
tests/​integration/​adapter/​basic/​test_snapshot_new_and_deleted_keys.py	Adds four regression cases for new-key insertion and hard-delete invalidation.
dbt/​include/​clickhouse/​macros/​materializations/​snapshot.sql	Replaces NULL-based unmatched-row detection with anti joins in both staging macros.
CHANGELOG.md	Records the user-visible snapshot fixes and links to the PR.

_{Review effort: Balanced}