Skip to content

Fix #918: support Time (TimeInterval) identifiers in DuckDB time operators - #958

Merged
albertohernandez1995 merged 2 commits into
1.9.Xfrom
cr-918
Aug 4, 2026
Merged

Fix #918: support Time (TimeInterval) identifiers in DuckDB time operators#958
albertohernandez1995 merged 2 commits into
1.9.Xfrom
cr-918

Conversation

@javihern98

@javihern98 javihern98 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

On the DuckDB backend, every time operator crashed with TypeError: cannot unpack non-iterable NoneType object when the Data Set's time identifier was declared "type": "Time" (VTL TimeInterval, e.g. 2001-01-01/2001-12-31). _resolve_time_identifier only matched Date and TimePeriod, so it fell off the end and returned None to its three unpacking call sites. The pandas engine handled these Data Sets fine, so the two backends silently diverged.

flow_to_stock, stock_to_flow, timeshift and fill_time_series now return on DuckDB exactly what the pandas engine returns. Building the cross-backend fixtures then surfaced seven more defects — two on the DuckDB Date path and five in the pandas engine — all fixed here.

1. TimeInterval support on the DuckDB backend

How the frequency is derived. A TimeInterval series is characterised by the duration of a single interval, which the pandas engine (Time._classify_interval_period) reports as Y/S/Q/M/W/D, or as a P<n>Y<n>M<n>D fallback for spans with no canonical period indicator. Both forms reduce to the same thing — a (months, days) offset — so the new macros carry the frequency as a STRUCT and materialise it as a native DuckDB INTERVAL for generate_series and for date arithmetic. That covers the non-canonical durations (P2Y, P7M, P1Y2M, P14D, …) with no special casing.

Note

The frequency is a STRUCT and not an INTERVAL because DuckDB normalises a month to 30 days when comparing intervals: INTERVAL 30 DAY = INTERVAL 1 MONTH evaluates true, which silently misclassifies 2020-01-01/2020-01-31 and 2020-01-01/2020-03-31.

  • sql/time_operators.sqlvtl_interval_start_date/_end_date/_start_ts/_end_ts, vtl_interval_months/_days, vtl_interval_is_canonical, vtl_interval_nonzero, vtl_interval_freq, vtl_interval_freq_to_step, vtl_interval_step, vtl_interval_build, vtl_interval_shift.
  • Transpiler/__init__.pyTimeInterval added to _resolve_time_identifier; new TimeInterval branch in visit_BinOp_timeshift; new _fill_time_series_interval, which steps the two endpoints separately and pairs the k-th of each, mirroring Fill_time_series.fill_time_intervals.
  • io/_execution.py — a Data Set whose intervals do not share one frequency raises SemanticError 1-1-19-9, as the pandas engine does.

flow_to_stock/stock_to_flow needed no new SQL: for a TimeInterval identifier they partition by the other identifiers exactly as they do for Date, and the interval strings sort chronologically.

2. DuckDB Date path

  • timeshift snapped a month end to the month end of the target month. pd.DateOffset clamps but never snaps, so 2020-02-29 shifted by one month is 2020-03-29, not 2020-03-31.
  • flow_to_stock/stock_to_flow accumulated number attributes; only number measures accumulate (the Reference Manual types the operand as measure<number>, cf. Stock to flow weird error (DuckDb) #931).

3. pandas engine

defect was
fill_time_series(ds, all) dropped the Data Points that widen each series to the global range whenever the Data Set had an Attribute 6 Data Points where 8 were due
fill_time_series(ds, single) overlapping intervals were rewritten into intervals the operand never held 2020-01-01/2020-01-31 + 2020-01-31/2020-02-29 became 2020-01-01/2020-02-29 + 2020-01-31/2020-01-31
fill_time_series(ds, single) per-series limits looked up by the string representation of the other identifiers KeyError on any non-String identifier
flow_to_stock, stock_to_flow, fill_time_series a Data Set holding a single series has nothing to group by ValueError: No group keys passed!, on every time type
fill_time_series, timeshift an interval's duration was measured by reading the whole endpoint as a date ValueError: Invalid isoformat string on the T form the input format allows

The first two share a fix: the interval fill now adds only the Data Points whose key the operand is missing, each carrying no values at all. That removes the appended-boundary-Data-Point and duplicate-filter machinery the old code needed, so there is nothing left to deduplicate wrongly, and the operand's own Data Points are never touched. The intervals it adds keep the operand's representation, time of day included, on both engines.

Checklist

  • Code quality checks pass (ruff format, ruff check, mypy)
  • Tests pass (pytest)
  • Documentation updated (if applicable) — docs/duckdb_engine.rst does not enumerate per-operator/type support, so nothing to update

Impact / Risk

  • Behaviour changes, all of them towards the Reference Manual: time operators that raised TypeError, KeyError or ValueError now produce results, and the DuckDB Date path now matches the pandas engine.
  • The pandas fill_time_series change is the one to review closely: all mode over a Data Set with Attributes now returns more Data Points than before (the ones it should always have returned), and overlapping intervals are preserved rather than rewritten.
  • No API, schema or SDMX compatibility change.
  • Full suite green under both backends: VTL_ENGINE_BACKEND=duckdb 5507 passed / 27 skipped, VTL_ENGINE_BACKEND=pandas 4899 passed / 635 skipped.

Notes

Verified against the pandas engine throughout: the frequency macro matches all 24 cases of the existing test_classify_interval_period spec, and a 12-frequency × 6-operator matrix (canonical, month-end, mid-year and non-canonical P2Y/P7M/P14D series) matches row for row, as do nested calls such as flow_to_stock(fill_time_series(...)).

Tests, all file-based under tests/Bugs so CI pins both backends against the same references:

code covers
GH_918_1 all four operators over a Time identifier
GH_918_2 the two DuckDB Date-path fixes
GH_918_3 fill_time_series with an Attribute
GH_918_4 fill_time_series(single) with an Integer identifier
GH_918_5 a single series, Time identifier only
GH_918_6 a single series, Date identifier only
GH_918_7 overlapping intervals
GH_918_8 intervals carrying a time of day

Plus timeshift-on-interval and mixed-frequency cases in tests/DateTime, and macro unit tests in tests/duckdb_transpiler. test_fill_time_series_interval_uniform_frequency now also runs on DuckDB — it never did, which is why the original bug went unnoticed.

Fixes #918

…ators

The DuckDB transpiler only recognised Date and Time_Period identifiers, so
_resolve_time_identifier returned None for a Time (TimeInterval) identifier
and flow_to_stock, stock_to_flow, timeshift and fill_time_series all failed
with "cannot unpack non-iterable NoneType object".

A TimeInterval series is characterised by the duration of one interval, which
the pandas engine reports as Y/S/Q/M/W/D or a P<n>Y<n>M<n>D fallback. Both
forms reduce to a (months, days) offset, so the new SQL macros carry the
frequency as a STRUCT and materialise it as a native INTERVAL for
generate_series and for date arithmetic. A STRUCT rather than an INTERVAL
because DuckDB normalises a month to 30 days when comparing intervals.

Two DuckDB-vs-pandas divergences on the Date path are fixed alongside:
timeshift no longer snaps a month end to the month end of the target month,
and flow_to_stock/stock_to_flow no longer accumulate number attributes.
All five surfaced while building the cross-backend parity fixtures for the
TimeInterval work, and all five are in the pandas engine:

- fill_time_series with the all limits method lost the Data Points that widen
  each series to the global range whenever the Data Set had an Attribute: the
  boundary Data Point it appended carried row 0's Attribute values, so it no
  longer collapsed onto the grid Data Point and the duplicate filter removed
  both. The interval fill now adds only the Data Points whose key is missing,
  with no values at all, so there is nothing to deduplicate.

- That same rewrite keeps the operand's own intervals when they overlap. The
  two endpoint grids then come out different lengths, and joining them
  positionally used to rewrite the operand into intervals it never held.

- fill_time_series with the single limits method looked its per-series limits
  up by the string representation of the other identifiers, so any identifier
  that was not a String raised KeyError.

- A Data Set may hold a single series, with the time identifier as its only
  identifier. flow_to_stock, stock_to_flow and fill_time_series then have
  nothing to group by, which raised a Pandas ValueError on every time type.

- A Time value may carry a time component, which the input format allows.
  Measuring an interval's duration read the whole endpoint as a date and
  raised ValueError; the duration now reads the date part, and the intervals
  fill_time_series adds keep the operand's representation, time of day
  included, on both engines.
@javihern98
javihern98 marked this pull request as ready for review August 3, 2026 10:13
@javihern98
javihern98 requested review from a team, albertohernandez1995 and mla2001 and removed request for mla2001 August 3, 2026 10:13
@javihern98 javihern98 linked an issue Aug 3, 2026 that may be closed by this pull request
2 tasks
@albertohernandez1995
albertohernandez1995 merged commit 86f3cfb into 1.9.X Aug 4, 2026
17 checks passed
@albertohernandez1995
albertohernandez1995 deleted the cr-918 branch August 4, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error flow_to_stock() with time id (Duckdb)

2 participants