Fix #937: count accepts any Measure type and counts Data Points - #957
Conversation
javihern98
left a comment
There was a problem hiding this comment.
Need to review this one in a call
count no longer rejecting a Time Measure is the fix #937 asked for. Changing what count returns for a Data Point with a null Measure is a separate, breaking change and is tracked on its own; it is reverted here together with the reference Data Sets it required. What remains is the guard exemption, its regression test, and the two type-checking tests that asserted the error and now assert a result against their original references.
count no longer rejecting a Time Measure is the fix #937 asked for. Changing what count returns for a Data Point with a null Measure is a separate, breaking change and is tracked on its own; it is reverted here together with the reference Data Sets it required. What remains is the guard exemption, its regression test, and the two type-checking tests that asserted the error and now assert a result against their original references.
Agreed, and I have made that call smaller. This PR bundled two changes of very different
The second is now split out into #NNN and reverted here, together with every reference Data Worth knowing before merge: this fixes the error in #937, but the issue also asks for "the |
javihern98
left a comment
There was a problem hiding this comment.
Looks good now, thanks!😊
Summary
countrejected a Data Set with aTimeMeasure, raising1-1-19-12. The guardagainst aggregating a
TimeIntervalMeasure ran before thecountbranch thatdiscards every Measure and replaces it with
int_var, so count was refused a Measureit never aggregates. The reference manual
types count's operand as a plain
dataset, states "Additional Constraints: None", andcounts a String Measure in its own example. count is now exempt from that guard,
alongside the two exemptions the same method already made for it.
Fixing that exposed a second defect in the same operator, agreed as in scope because
the reporter asked for "the number of rows": count skipped any Data Point that had a
null in one of its Measures. Three sites did this — a
dropnaand aCOUNT(<first Measure>)on the pandas engine, and aCOUNT(CASE WHEN ... IS NOT NULL)on DuckDb — so count reported fewer Data Points than the operand held, andfrequently reported null instead of a number. The manual says only that count
"returns the number of the input Data Points".
Several existing test docstrings already recorded this as a defect: "Measure Date with
null, doesn't count the null, we think that should", and "Example that takes the most
left measure".
The two invocation forms have to stay distinct, which is why the manual gives them
separate syntaxes:
count ( dataset )count ( )count ( component )A
component_operandflag threaded from the interpreter keeps the Component form onits own semantics; without it the Component form would duplicate
count().Checklist
ruff format,ruff check,mypy)pytest)Impact / Risk
countover a Data Set, andcount(), now return larger numberswherever the operand holds Data Points with null Measures, and return a number where
they previously returned null. This is a breaking change for anyone relying on the
old counts, and worth a changelog entry.
countno longer raises1-1-19-12; a Data Set with aTimeMeasure is accepted.countover a Component is unchanged.Bugs and Additional. Each was checked rather than regenerated wholesale: the
reference-override helper also rewrites unaffected Data Sets, so five files whose only
change was column order or TimePeriod rendering (
2023-01to2023-M01) werereverted, and two more had their original
monthformatting restored. Every retainedchange is a count value, spot-checked against the input Data Points.
Notes
tests/Aggregate/GL_466_1is the clearest illustration: counting per month afterfill_time_seriesreturned null for every month except two, and now returns12, 28, 31, 30, ...— January 2023 counted from the 20th is 12 days.tests/UDO/GL_442_1previously had an
int_varcolumn that was entirely null.test_GH_937_1covers the reported case and fails against the previous code with thereported
1-1-19-12. Two type-checking tests that asserted that error became resulttests; both already had correct reference Data Sets.
Fixes #937