fix(microbatch): filter Date event_time columns without a string cast by DG47 · Pull Request #759 · ClickHouse/dbt-clickhouse

DG47 · 2026-10-19T13:56:44Z

Summary

Fixes #527.

A microbatch model whose event_time is a Date (or Date32) column fails on its first batch:

Code: 53. DB::Exception: Cannot convert string '2020-01-01 00:00:00' to type Date: while executing function greaterOrEquals on arguments __table2.event_date Date ... (TYPE_MISMATCH)

The error comes from the filter dbt adds to upstream ref()s of a microbatch model, rendered by ClickHouseRelation._render_event_time_filtered. It compared the column with a 'YYYY-MM-DD hh:mm:ss' string literal. ClickHouse casts that literal to the column type, and a value with a time part cannot be cast to Date.

The filter now compares the column with toDateTime('YYYY-MM-DD hh:mm:ss'). That works against Date, Date32, DateTime and DateTime64 columns. It is also the form the microbatch strategy already uses for its delete predicates on the target table (incremental.sql), so both sides of a batch now use the same boundaries. For DateTime columns without an explicit time zone the result does not change. The same filter is used by --sample mode.

Tests:

Unit: tests/unit/test_relation.py::test_render_event_time_filtered covers the start+end, start-only, end-only and empty cases.
Integration: TestMicrobatchIncrementalDateEventTime runs the standard BaseMicrobatch scenario on a Date event_time column. It fails on main with the error above and passes with this change.

Checklist

Unit and integration tests covering the common scenarios were added
A changelog entry was added following Updating the Changelog (user-facing, concise, PR link)

Note

Medium Risk
Changes SQL predicates for microbatch incremental and event-time sampling; incorrect boundaries could skip or mis-delete rows, though behavior is narrowly scoped and heavily tested.

Overview
Fixes microbatch incremental runs (and upstream/--sample event-time filters) that failed when event_time is Date or Date32, because batch boundaries were compared using 'YYYY-MM-DD hh:mm:ss' string literals that ClickHouse cannot cast to Date.

ClickHouseRelation._render_event_time_filtered now builds predicates with a shared toDateTime64(..., 0) helper instead of quoted datetime strings. The microbatch delete/insert predicates in incremental.sql are aligned to the same toDateTime64 form so upstream filtering and target deletes share identical batch windows.

Using DateTime64 also avoids DateTime’s 1970–2106 clamping, so batches before 1970 or after 2106 work for Date32 / DateTime64 columns. Coverage adds unit tests for filter rendering and integration tests for Date, Date32, and pre-1970 DateTime64 microbatch scenarios.

^{Reviewed by Cursor Bugbot for commit f45a498. Bugbot is set up for automated code reviews on this repo. Configure here.}

The event_time filter that dbt applies to upstream refs of a microbatch model compared the column with a 'YYYY-MM-DD hh:mm:ss' string literal. ClickHouse casts the literal to the column type, which fails for Date and Date32 columns (TYPE_MISMATCH). Compare with toDateTime(...) instead, which works for Date, Date32, DateTime and DateTime64 and matches the predicates the microbatch strategy already uses on the target table. Fixes ClickHouse#527

DG47 · 2026-10-30T14:44:38Z

@koletzilla could you take a look at this when you get a chance? It fixes #527, where a microbatch model with a Date or Date32 event_time column failed on its first batch with a TYPE_MISMATCH error because the upstream ref() filter compared the column with a 'YYYY-MM-DD hh:mm:ss' string literal. The filter now uses toDateTime(...), matching the form the delete predicates in incremental.sql already use, and there are unit and integration tests covering it. Happy to rebase or adjust anything you'd like changed.

…-event-time

Copilot

🟡 Changes recommended

The new toDateTime boundaries exclude valid historical dates and can shift sampling windows when server and column timezones differ, requiring corrections and regression coverage.

2 open issues

What changed in this PR

This PR addresses #527 by changing upstream event-time filters to use typed datetime boundaries, allowing microbatch models with Date event-time columns to run without string-conversion errors. The same rendering change applies to --sample.

Changes:

Add a helper that renders filter boundaries with toDateTime().
Add unit coverage for boundary combinations and an integration scenario using a Date column.
Document the fix in the changelog.

File	Description
`tests/unit/test_relation.py`	Tests paired, start-only, end-only, and empty filter bounds.
`tests/integration/adapter/incremental/test_incremental_microbatch.py`	Adds a microbatch scenario with a `Date` event-time column.
`dbt/adapters/clickhouse/relation.py`	Replaces bare string boundaries with typed datetime expressions.
`CHANGELOG.md`	Records the date-column filtering fix.

_{Review effort: Balanced}

DG47 · 2026-11-03T16:48:22Z

+def _to_datetime_literal(value: datetime) -> str:
+    # A bare string literal is cast to the column type, and '2020-01-01 00:00:00' cannot be cast to Date
+    # or Date32. A DateTime value compares with Date, Date32, DateTime and DateTime64 columns alike.
+    return f"toDateTime('{value.strftime('%Y-%m-%d %H:%M:%S')}')"


The delete predicates in incremental.sql have rendered the boundaries as toDateTime('...') since the microbatch strategy was added, so they are in the session timezone, and the upstream filter now uses the same form. On 26.9 with session_timezone='Europe/Paris', toDateTime64('...', 0) resolves to the same instant as toDateTime('...') for DateTime, DateTime('UTC') and DateTime64(3, 'UTC') columns. Before this PR the filter on a DateTime('UTC') column used the column zone while the delete used the server zone, so on a non-UTC server a re-run of a batch did not delete the rows it had inserted in the last hour of the window; one form on both sides fixes that. Forcing 'UTC' would change the result for plain DateTime columns on non-UTC servers, which is a separate decision from #527, so the zone stays as it was.

DG47 · 2026-11-03T16:48:24Z

+    # or Date32. A DateTime value compares with Date, Date32, DateTime and DateTime64 columns alike.
+    return f"toDateTime('{value.strftime('%Y-%m-%d %H:%M:%S')}')"


Agreed: toDateTime('1960-06-15 00:00:00') clamps to 1970-01-01 00:00:00, so a Date32 or DateTime64 window before 1970 selected nothing. Both the filter (relation.py) and the delete predicates (incremental.sql) now render toDateTime64('...', 0), which compares correctly with Date, Date32, DateTime and DateTime64 columns on 26.9 and keeps the 1960 bounds. The unit test covers 1960 and 2200 boundaries, and two new integration scenarios run the microbatch flow with begin=1960-01-01 on Date32 and DateTime64 columns; both produced an empty model before the change.

toDateTime() clamps every value to 1970-2106, so a Date32 or DateTime64 event_time column with rows before 1970 got an empty batch window: the filter on upstream models selected nothing and the delete predicates on the model matched nothing. Render both as toDateTime64('...', 0), which compares with Date, Date32, DateTime and DateTime64 columns alike and covers 1900-2299. The delete predicates in incremental.sql are changed in the same way so the rows a batch deletes are the rows it selects. Adds the pre-1970 and post-2106 cases to the unit test and two microbatch integration scenarios with a 1960 begin on Date32 and DateTime64 columns.

DG47 · 2026-11-03T16:48:26Z

@BentsiLeviav since your review request I switched the boundary literal to toDateTime64('...', 0) in both the upstream filter and the microbatch delete predicates, so pre-1970 Date32 and DateTime64 windows are no longer clamped, and added unit and integration coverage for a 1960 begin. The SQL evidence is in the replies on the two Copilot threads.

BentsiLeviav requested a review from koletzilla October 19, 2026 15:10

DG47 added 2 commits October 20, 2026 12:02

Link changelog entry to the PR

a47e60a

DG47 force-pushed the fix/microbatch-date-event-time branch from 76fb61b to a47e60a Compare October 20, 2026 16:38

Merge remote-tracking branch 'upstream/main' into fix/microbatch-date…

f67bf34

…-event-time

BentsiLeviav requested a balanced review from Copilot November 3, 2026 15:10

Copilot started reviewing on behalf of BentsiLeviav November 3, 2026 15:10

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix(microbatch): filter Date event_time columns without a string cast - #759

fix(microbatch): filter Date event_time columns without a string cast#759
DG47 wants to merge 4 commits into
ClickHouse:mainfrom
DG47:fix/microbatch-date-event-time

DG47 commented Oct 19, 2026 •

edited by cursor Bot

Loading

Uh oh!

DG47 commented Oct 30, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

DG47 Nov 3, 2026

Uh oh!

DG47 Nov 3, 2026

Uh oh!

DG47 commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		# or Date32. A DateTime value compares with Date, Date32, DateTime and DateTime64 columns alike.
		return f"toDateTime('{value.strftime('%Y-%m-%d %H:%M:%S')}')"

Conversation

DG47 commented Oct 19, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

DG47 commented Oct 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

DG47 Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

DG47 Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

DG47 commented Nov 3, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DG47 commented Oct 19, 2026 •

edited by cursor Bot

Loading