summaryStatistics and box plots report swapped quartiles for a column inside mathjs's comparison tolerance #1895

quartile1 and quartile3 come from math-expressions' quantileSeq, which selects the k-th value through mathjs's tolerant comparison — relTol 1e-12, absTol 1e-15. When a column's values fall inside that tolerance, the comparison calls them equal, the selection picks the wrong one, and the two quartiles can come back swapped.

utils/summaryStatistics.ts is the one definition <summaryStatistics> and <chart type="box"> share, so both report the same wrong pair.

utils/summaryStatistics.ts

export function quartile1(column: number[]): number {
    return quantileSeq(column, 0.25);
}

export function quartile3(column: number[]): number {
    return quantileSeq(column, 0.75);
}

What it does today

A box plot of 1E-16 2E-16 3E-16 4E-16 — quartiles that should be 1.75e-16 and 3.25e-16:

<chart type="box"><series>1E-16 2E-16 3E-16 4E-16</series></chart>

draws its box from 2.5e-16 to 3.5e-16, draws one of the four observations as an outlier that is not one, and tells a screen reader

minimum 1e-16, first quartile 3.5e-16, median 2.5e-16, third quartile 2.5e-16, maximum 4e-16

— a first quartile above the median, and a third quartile equal to it. The median is right because it no longer goes through quantileSeq (#1893); the two quartiles are the last of the five that do.

The same column read as a table reports the same pair.

How reachable it is

Two separate bands, because the tolerance has a relative part and an absolute one.

absTol 1e-15 — values close in absolute terms. This is the one that draws visibly wrong pictures, because the values survive to the drawing. Measured over 20,000 random columns per band, against a reference that uses mathjs's own interpolation so that only the selection differs:

relTol 1e-12 — values close in relative terms, at any magnitude. Four readings agreeing to thirteen significant figures at an ordinary magnitude invert:

quantileSeq([100, 100.00000000001, 100.000000000005, 100.0000000000075], 0.25)
// 100.00000000000813 — which is the true *third* quartile
quantileSeq([100, 100.00000000001, 100.000000000005, 100.0000000000075], 0.75)
// 100.00000000000375 — the true first

Over 20,000 random columns of readings near 100 agreeing to about twelve digits: 16,235 wrong quartile1, 4,627 swapped. The drawn and spoken consequences are mild here — snapNumber's twelve significant digits collapse the differences before anything is written, so the box draws as a flat line at 100, which is a fair picture of four readings that agree that closely — but $series.quartile1 and $summaryStatistics.quartile1 still hold the swapped values for anything that reads them programmatically.

The fix

Compute the quartiles here from an exact sort, the way median in the same file already is:

const s = [...column].sort((a, b) => (a < b ? -1 : a > b ? 1 : 0));
const index = p * (s.length - 1);
const lo = Math.floor(index), f = index % 1;
return Number.isInteger(index) ? s[index] : (1 - f) * s[lo] + f * s[lo + 1];

(1 - f) * left + f * right is mathjs's own interpolation, kept deliberately, so the only thing that changes is which values are selected — every column outside the tolerance keeps the number it reports today.

#1880 asked that the quartiles come from the same definition as <summaryStatistics>' so that a table and a box plot of one column cannot disagree on the page. That is preserved: both consumers still read one function in one file. What stops being shared is quantileSeq's definition of it, which is what #1880's "must come from the same definition" was protecting against in the first place.

Changing the mathjs config is not the lever, though it does prove the cause: an instance configured relTol: 1e-300, absTol: 0 takes the 1e-16 band from 17,534 wrong and 9,956 swapped to zero of each. But that config is global to math-expressions, and tolerant equality is what compares student answers — equal(0.1 + 0.2, 0.3) becomes false under it. (relTol: 0 is rejected outright.)

Fixing it upstream instead would mean a quantileSeq that sorts exactly regardless of config, which is a reasonable request to make of mathjs but not something to wait on.

Not changed by the Rust engine

Worth recording, since it is the obvious question: #1622 does not affect this. Its math-expressions-js-compat layer builds me.math as mathjs.create(mathjs.all) at ^15.2.0 with numeric.js imported on top — the same construction lib/mathjs.js uses on 2.x today, and neither calls math.config(), so the defaults are identical. quantileSeq after that switch is the same mathjs function with the same tolerance.

Related, not part of this

me.math.mean overflows the way me.math.median did before #1893 replaced it — (a + b + …) / n sums before it divides, so a column near the top of the double range reports Infinity. Same file's neighborhood, different defect; worth its own issue if anyone wants it filed.

Where it came from

Found while reviewing #1893 (<chart type="box">), which guards against the consequences in three places rather than the cause: the outlier fences are taken from the ordered quartile pair, the box's rectangle is written from the ordered corner, and a column with no observation between its own fences falls back to drawing its quartiles as the whisker ends. Those guards are what keep a tolerance-band column drawable; they do not make its quartiles right, and they can come out once these do.

🤖 Generated with Claude Code