diff --git a/src/content/docs/test-insights.mdx b/src/content/docs/test-insights.mdx index dc6fc1c918..1511cb5b9c 100644 --- a/src/content/docs/test-insights.mdx +++ b/src/content/docs/test-insights.mdx @@ -36,18 +36,19 @@ known flakiness. ## Key concepts -- **Flaky test**: A test that produces different results on the same commit. - For example, passing on one run and failing on the next with identical code. +- **Flaky test**: A test that both passes and fails on the same commit. For + example, passing on one run and failing on the next with identical code. A + test whose results differ without ever failing stays healthy. -- **Broken test**: A test that fails consistently, with recent runs weighted - more heavily. +- **Broken test**: A test that fails, with no passing run close in time to + contradict those failures. - **Health status**: A test's reliability classification: healthy, flaky, or broken. Based on results from multiple CI runs. -- **Confidence**: How much data is available to assess a test's health. Low - confidence means the status could still change significantly as more runs - are collected. +- **Confidence**: How much data is available to assess a test's health, + reported as low, medium, or high. Low confidence means the status could + still change as more runs are collected. - **Quarantine**: Isolating a test so its failures are ignored for merge decisions. The test still runs and results are still collected, preserving @@ -65,6 +66,7 @@ Test Insights is powered by the same CI integration as framework: - [GitHub Actions setup](/ci-insights/setup/github-actions) +- [Buildkite setup](/ci-insights/setup/buildkite) - [Jenkins setup](/ci-insights/setup/jenkins) - [Test framework configuration](#test-framework-configuration) diff --git a/src/content/docs/test-insights/detection.mdx b/src/content/docs/test-insights/detection.mdx index ff0f844653..2b5992f82e 100644 --- a/src/content/docs/test-insights/detection.mdx +++ b/src/content/docs/test-insights/detection.mdx @@ -31,29 +31,33 @@ Insights: it reports on tests running on pull request branches. ## How tests are classified -Mergify classifies tests based on their results across multiple CI runs, -with recent results weighted more heavily: +Mergify classifies a test from the results collected for it across CI runs: -- **Flaky**: The test produces inconsistent results on the same commit. It - passes on some runs and fails on others, without any code changes. +- **Flaky**: The test passed and failed close together on the same branch, + pipeline, and job, so its outcome does not follow from the code alone. -- **Broken**: The test fails consistently. Recent runs are weighted more - heavily, so a test that started failing recently will be classified as - broken even if it passed in earlier runs. +- **Broken**: The test failed and no result close in time contradicts those + failures. A test whose runs all fail is always broken. -Only unhealthy tests (flaky or broken) appear in Detection. Healthy tests -are not listed. +An isolated failure among a large number of passing runs is not enough to +make a test unhealthy. A test that has kept running, and passing, for a week +since its last failure goes back to healthy, so a test you fixed stops being +reported without waiting for its old failures to age out. + +Detection lists tests under a **Flaky**, **Broken**, or **Healthy** tab, so +you can work through the unhealthy ones and still look up a healthy test. ## Understanding confidence Confidence indicates how much data is available to assess a test's health. +It comes in three levels: + +- **High**: Enough runs have been collected to trust the health status. -- **High confidence**: Enough runs have been collected to make a reliable - assessment. The health status is unlikely to change significantly. +- **Medium**: A limited number of runs. The health status may still change. -- **Low confidence**: Limited data is available. The health status could - still shift as more runs are collected. Treat low-confidence results as - preliminary. +- **Low**: Very few runs, so the health status is unreliable. Treat + low-confidence results as preliminary. Confidence increases as more CI runs are collected for a given test. @@ -70,9 +74,10 @@ dashboard to enable it. ## Prioritizing with impact -The impact metric reflects how many failed executions a test causes. A -high-impact flaky test wastes more CI time and disrupts more workflows than -a low-impact one. +Impact is the share of a test's executions that failed, reported as low, +medium, or high. A test that fails on most of its runs has a higher impact +than one that fails once in a while, whatever the absolute number of +failures. Use impact to decide which tests to fix first: high-impact tests give you the most return on investment when fixed. @@ -81,8 +86,8 @@ the most return on investment when fixed. ### Finding your worst tests -Sort by impact to surface the tests causing the most CI disruption. These -are the best candidates for immediate attention. +Filter on high impact to surface the tests causing the most CI disruption. +These are the best candidates for immediate attention. ### Narrowing scope @@ -91,6 +96,8 @@ Use filters to focus on specific areas: - **Test name**: Search for a specific test or pattern - **Job name**: Focus on tests within a particular CI job - **Pipeline name**: Narrow to a specific CI pipeline +- **Confidence**: Keep only tests at a given confidence level +- **Impact**: Keep only tests at a given impact level ### Checking quarantine status @@ -104,4 +111,5 @@ Detection requires test metrics collection through repeated CI runs. See the CI setup guides for your platform: - [GitHub Actions setup](/ci-insights/setup/github-actions) +- [Buildkite setup](/ci-insights/setup/buildkite) - [Jenkins setup](/ci-insights/setup/jenkins) diff --git a/src/content/docs/test-insights/quarantine.mdx b/src/content/docs/test-insights/quarantine.mdx index 20f13438ce..12495e1c93 100644 --- a/src/content/docs/test-insights/quarantine.mdx +++ b/src/content/docs/test-insights/quarantine.mdx @@ -35,7 +35,7 @@ which is handy for scripting or for letting an AI coding agent quarantine a flak ```bash # Quarantine a test. mergify tests quarantines add -r owner/repo \ - --reason "flaky — tracked in MRGFY-1234" \ + --reason "flaky under load, fix in progress" \ "tests/auth/test_login.py::test_login_timeout" # Remove it from quarantine. diff --git a/src/content/docs/test-insights/test-frameworks/golang.mdx b/src/content/docs/test-insights/test-frameworks/golang.mdx index 04075e74dc..13e9758827 100644 --- a/src/content/docs/test-insights/test-frameworks/golang.mdx +++ b/src/content/docs/test-insights/test-frameworks/golang.mdx @@ -52,6 +52,7 @@ For example, in your workflow file: ```yaml - name: Run Go Tests and Generate JUnit Report + id: tests continue-on-error: true run: go test -v ./... 2>&1 | go-junit-report > junit.xml ``` @@ -82,7 +83,7 @@ After pushing these changes: You can then review your test results, including any failures or flaky tests, directly in the [Test Insights -dashboard](https://dashboard.mergify.com/test-insights/jobs). +dashboard](https://dashboard.mergify.com/test-insights/detection). ## Troubleshooting Tips diff --git a/src/content/docs/test-insights/test-frameworks/jest.mdx b/src/content/docs/test-insights/test-frameworks/jest.mdx index 8f4eaff7dc..ebefe6fb05 100644 --- a/src/content/docs/test-insights/test-frameworks/jest.mdx +++ b/src/content/docs/test-insights/test-frameworks/jest.mdx @@ -73,6 +73,7 @@ For example, in your workflow file: ```yaml - name: Run Jest Tests and Generate JUnit Report + id: tests continue-on-error: true env: JEST_JUNIT_OUTPUT_DIR: . @@ -106,7 +107,7 @@ After pushing these changes: You can then review your test results, including any failures or flaky tests, directly in the [Test Insights -dashboard](https://dashboard.mergify.com/test-insights/jobs). +dashboard](https://dashboard.mergify.com/test-insights/detection). ## Troubleshooting Tips diff --git a/src/content/docs/test-insights/test-frameworks/junit.mdx b/src/content/docs/test-insights/test-frameworks/junit.mdx index 6466e784af..85dfd7888b 100644 --- a/src/content/docs/test-insights/test-frameworks/junit.mdx +++ b/src/content/docs/test-insights/test-frameworks/junit.mdx @@ -80,6 +80,7 @@ For example, in your workflow file: ```yaml - name: Run JUnit Tests and Generate Report + id: tests continue-on-error: true run: mvn test ``` @@ -92,6 +93,8 @@ For Gradle projects: ```yaml - name: Run JUnit Tests and Generate Report + id: tests + continue-on-error: true run: ./gradlew test ``` @@ -134,7 +137,7 @@ After pushing these changes: You can then review your test results, including any failures or flaky tests, directly in the [Test Insights -dashboard](https://dashboard.mergify.com/test-insights/jobs). +dashboard](https://dashboard.mergify.com/test-insights/detection). ## Troubleshooting Tips diff --git a/src/content/docs/test-insights/test-frameworks/minitest.mdx b/src/content/docs/test-insights/test-frameworks/minitest.mdx index d96c03eb7b..a5f9d05eb4 100644 --- a/src/content/docs/test-insights/test-frameworks/minitest.mdx +++ b/src/content/docs/test-insights/test-frameworks/minitest.mdx @@ -128,6 +128,7 @@ For example, in your workflow file: ```yaml - name: Run Minitest Tests and Generate JUnit Report + id: tests continue-on-error: true env: JUNIT_OUTPUT_DIR: . @@ -140,6 +141,8 @@ Using `minitest-reporters` approach: ```yaml - name: Run Minitest Tests and Generate JUnit Report + id: tests + continue-on-error: true run: bundle exec rake test ``` @@ -169,7 +172,7 @@ After pushing these changes: You can then review your test results, including any failures or flaky tests, directly in the [Test Insights -dashboard](https://dashboard.mergify.com/test-insights/jobs). +dashboard](https://dashboard.mergify.com/test-insights/detection). ## Troubleshooting Tips