Skip to content

[ON HOLD] Bug 2032243: Use the confidence interval for the significance column #1056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
kala-moz wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

[ON HOLD] Bug 2032243: Use the confidence interval for the significance column #1056

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
13259e7
mannWhitney: drive Sig column from bootstrap CI
kala-moz Jun 18, 2026
cb2f1cb
mannWhitney: render Sig column as S/NS text instead of arrow icon
kala-moz Jun 24, 2026
1f4ca56
mannWhitney: precompute bootstrap CI on replicates when available
kala-moz Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
306 changes: 153 additions & 153 deletions src/__tests__/CompareResults/ResultsTable.test.tsx
View file
Open in desktop

Large diffs are not rendered by default.

43 changes: 0 additions & 43 deletions src/__tests__/CompareResults/RevisionRow.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,49 +281,6 @@ describe('Expanded row', () => {
expect(cliffsDeltaHeader).toBeInTheDocument();
});

it('should display mann_whitney_test.interpretation for significance for mann-whitney-u testVersion', async () => {
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime });
const { testCompareMannWhitneyData: rowData } = getTestData();

renderWithRoute(
<RevisionRow
result={rowData[0]}
view={compareView}
gridTemplateColumns='none'
replicates={false}
testVersion='mann-whitney-u'
expandAll={false}
/>,
);

const expandRowButton = await screen.findByTestId(/ExpandMoreIcon/);
await user.click(expandRowButton);

const notSignificant = await screen.findAllByText(/Not significant/);
expect(notSignificant[0]).toBeInTheDocument();
});

it('should handle empty mann_whitney_test.interpretation for significance for mann-whitney-u testVersion', async () => {
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime });
const { testCompareMannWhitneyData: rowData } = getTestData();
const resultNoInterpretation = { ...rowData[0], mann_whitney_test: null };
renderWithRoute(
<RevisionRow
result={resultNoInterpretation}
view={compareView}
gridTemplateColumns='none'
replicates={false}
testVersion='mann-whitney-u'
expandAll={false}
/>,
);

const expandRowButton = await screen.findByTestId(/ExpandMoreIcon/);
await user.click(expandRowButton);
const emptySignificant = await screen.findAllByText(/-/);
expect(emptySignificant[0]).toBeInTheDocument();
});

it('should display median diff and 95% CI alerts when base/new runs are present', async () => {
const user = userEvent.setup({ advanceTimers: jest.advanceTimersByTime });
const { testCompareMannWhitneyData: rowData } = getTestData();
Expand Down
100 changes: 50 additions & 50 deletions src/__tests__/CompareResults/SubtestsResultsView.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,11 +527,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
await setupForSorting();
// Initial view (alphabetical ordered, even if "sort by subtests" isn't specified
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'browser.html: 0.963 %, -0.04, -, 15.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'improvement.html: 0.963 %, -0.05, , 50.00%',
'regression.html: 1.135 %, 0.12, , 25.00%',
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
'dhtml.html: 1.135 %, 0.02, S, 60.00%',
'improvement.html: 0.963 %, -0.05, S, 50.00%',
'regression.html: 1.135 %, 0.12, S, 25.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
]);

// Sort by Delta
Expand All @@ -542,11 +542,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
// Sort descending
await user.click(deltaButton);
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'regression.html: 1.135 %, 0.12, , 25.00%',
'improvement.html: 0.963 %, -0.05, , 50.00%',
'browser.html: 0.963 %, -0.04, -, 15.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'regression.html: 1.135 %, 0.12, S, 25.00%',
'improvement.html: 0.963 %, -0.05, S, 50.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
'dhtml.html: 1.135 %, 0.02, S, 60.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
]);

// It should have the "descending" SVG.
Expand All @@ -557,11 +557,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
// Sort ascending
await user.click(deltaButton);
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'browser.html: 0.963 %, -0.04, -, 15.00%',
'improvement.html: 0.963 %, -0.05, , 50.00%',
'regression.html: 1.135 %, 0.12, , 25.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
'dhtml.html: 1.135 %, 0.02, S, 60.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
'improvement.html: 0.963 %, -0.05, S, 50.00%',
'regression.html: 1.135 %, 0.12, S, 25.00%',
]);
// It should have the "ascending" SVG.
expect(deltaButton).toMatchSnapshot();
Expand All @@ -574,11 +574,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
});
await user.click(significanceButton);
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'browser.html: 0.963 %, -0.04, -, 15.00%',
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'regression.html: 1.135 %, 0.12, , 25.00%',
'improvement.html: 0.963 %, -0.05, , 50.00%',
'dhtml.html: 1.135 %, 0.02, NS, 60.00%',
'regression.html: 1.135 %, 0.12, NS, 25.00%',
'improvement.html: 0.963 %, -0.05, NS, 50.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
]);
// It should have the "no sort" SVG.
expect(deltaButton).toMatchSnapshot();
Expand All @@ -590,11 +590,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
// Sort by Significance ascending
await user.click(significanceButton);
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'improvement.html: 0.963 %, -0.05, , 50.00%',
'regression.html: 1.135 %, 0.12, , 25.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'browser.html: 0.963 %, -0.04, -, 15.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
'dhtml.html: 1.135 %, 0.02, NS, 60.00%',
'regression.html: 1.135 %, 0.12, NS, 25.00%',
'improvement.html: 0.963 %, -0.05, NS, 50.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
]);
expectParameterToHaveValue('sort', 'significance|asc');

Expand All @@ -604,11 +604,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
});
await user.click(effectButton);
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'browser.html: 0.963 %, -0.04, -, 15.00%',
'regression.html: 1.135 %, 0.12, , 25.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'improvement.html: 0.963 %, -0.05, , 50.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
'regression.html: 1.135 %, 0.12, NS, 25.00%',
'dhtml.html: 1.135 %, 0.02, NS, 60.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
'improvement.html: 0.963 %, -0.05, NS, 50.00%',
]);

// It should have the "descending" SVG.
Expand All @@ -619,11 +619,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
// Sort by Effect Size ascending
await user.click(effectButton);
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'improvement.html: 0.963 %, -0.05, , 50.00%',
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'regression.html: 1.135 %, 0.12, , 25.00%',
'browser.html: 0.963 %, -0.04, -, 15.00%',
'improvement.html: 0.963 %, -0.05, NS, 50.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
'dhtml.html: 1.135 %, 0.02, NS, 60.00%',
'regression.html: 1.135 %, 0.12, NS, 25.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
]);
expectParameterToHaveValue('sort', 'effects|asc');
});
Expand All @@ -632,11 +632,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
await setupForSorting({ extraParameters: 'sort=delta|asc' });
await screen.findByText('dhtml.html');
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'browser.html: 0.963 %, -0.04, -, 15.00%',
'improvement.html: 0.963 %, -0.05, , 50.00%',
'regression.html: 1.135 %, 0.12, , 25.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
'dhtml.html: 1.135 %, 0.02, S, 60.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
'improvement.html: 0.963 %, -0.05, S, 50.00%',
'regression.html: 1.135 %, 0.12, S, 25.00%',
]);
// It should have the "ascending" SVG.
expect(screen.getByRole('button', { name: /CD/ })).toMatchSnapshot();
Expand All @@ -646,11 +646,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
await setupForSorting({ extraParameters: 'sort=delta' });
await screen.findByText('dhtml.html');
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'regression.html: 1.135 %, 0.12, , 25.00%',
'improvement.html: 0.963 %, -0.05, , 50.00%',
'browser.html: 0.963 %, -0.04, -, 15.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'regression.html: 1.135 %, 0.12, S, 25.00%',
'improvement.html: 0.963 %, -0.05, S, 50.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
'dhtml.html: 1.135 %, 0.02, S, 60.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
]);
// It should have the "descending" SVG.
expect(screen.getByRole('button', { name: /CD/ })).toMatchSnapshot();
Expand All @@ -659,11 +659,11 @@ describe('SubtestsResultsView Component Tests for mann-whitney-u testVersion', (
it('initializes the sort from the URL at load time for a descending sort', async () => {
await setupForSorting({ extraParameters: 'sort=delta|desc' });
expect(summarizeVisibleRows('mann-whitney-u')).toEqual([
'regression.html: 1.135 %, 0.12, , 25.00%',
'improvement.html: 0.963 %, -0.05, , 50.00%',
'browser.html: 0.963 %, -0.04, -, 15.00%',
'dhtml.html: 1.135 %, 0.02, , 60.00%',
'tablemutation.html: 0.98 %, 0.01, -, 45.00%',
'regression.html: 1.135 %, 0.12, S, 25.00%',
'improvement.html: 0.963 %, -0.05, S, 50.00%',
'browser.html: 0.963 %, -0.04, NS, 15.00%',
'dhtml.html: 1.135 %, 0.02, S, 60.00%',
'tablemutation.html: 0.98 %, 0.01, NS, 45.00%',
]);
// It should have the "descending" SVG.
expect(screen.getByRole('button', { name: /CD/ })).toMatchSnapshot();
Expand Down
7 changes: 6 additions & 1 deletion src/__tests__/CompareResults/SubtestsRevisionRow.test.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,8 +193,13 @@ describe('SubtestsRevisionRow Component', () => {
const effects = roles[7]?.childNodes[0];
expect(effects).toHaveTextContent('60.00%');

// The Sig cell renders lazily: when no CI has been cached on the row
// yet, it falls back to `mann_whitney_test.interpretation` from the
// backend. The fixture's interpretation is "significant", so the cell
// shows "S" here. First click on the Sig column header would trigger
// BCa and could flip this if the CI includes zero.
const significanceCell = roles[8];
expect(significanceCell?.querySelector('svg')).not.toBeNull();
expect(significanceCell).toHaveTextContent('S');

const cliffs_delta = roles[6]?.childNodes[1];
expect(cliffs_delta).toHaveTextContent('0.02');
Expand Down
8 changes: 4 additions & 4 deletions src/__tests__/CompareResults/__snapshots__/OverTimeResultsView.test.tsx.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -824,7 +824,7 @@ exports[`Results View The table should match snapshot and other elements should
class="significance cell"
role="cell"
>
-
NS
</div>
<div
class="total-runs cell"
Expand Down Expand Up @@ -1023,7 +1023,7 @@ exports[`Results View The table should match snapshot and other elements should
class="significance cell"
role="cell"
>
-
NS
</div>
<div
class="total-runs cell"
Expand Down Expand Up @@ -1222,7 +1222,7 @@ exports[`Results View The table should match snapshot and other elements should
class="significance cell"
role="cell"
>
-
NS
</div>
<div
class="total-runs cell"
Expand Down Expand Up @@ -1421,7 +1421,7 @@ exports[`Results View The table should match snapshot and other elements should
class="significance cell"
role="cell"
>
-
NS
</div>
<div
class="total-runs cell"
Expand Down
Loading