docs(agent): fix how detection describes test health, confidence and impact - #12411
Open
mergify-ci-bot wants to merge 1 commit into
Open
docs(agent): fix how detection describes test health, confidence and impact#12411mergify-ci-bot wants to merge 1 commit into
mergify-ci-bot wants to merge 1 commit into
Conversation
Contributor
Merge Protections🔴 2 of 6 protections blocking · waiting on 👀 reviews
🔴 👀 Review RequirementsWaiting for
This rule is failing.
🔴 🔎 ReviewsWaiting for
This rule is failing.
Show 4 satisfied protections🟢 🤖 Continuous Integration
🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
🟢 📕 PR description
🟢 🚦 Auto-queueWhen all merge protections are satisfied, this pull request will be queued automatically. |
jd
added a commit
that referenced
this pull request
Aug 17, 2026
…hat exists Eleven test framework recipes closed by sending the reader to the Test Insights dashboard. Ten of them linked `https://dashboard.mergify.com/test-insights/jobs`, which is not a page: Test Insights has Prevention, Detection and Mitigation, so that URL matched the router and rendered nothing. `jobs` is a CI Insights route, a different and org-scoped surface, which is where the copy-paste came from. The eleventh (Cypress) named the dashboard in the same sentence with no link at all. The paragraph promises "test results, including any failures or flaky tests", and Detection is the page listing flaky and broken tests, so that is where all eleven now point. It is also a partial now, included by the recipes rather than pasted into each. One stale URL became ten broken links precisely because it was ten copies of one sentence, and an eleventh copy had already drifted into a dead end. Separately, the recipes' inline GitHub Actions examples gained `id: tests` on the test-runner step. The quarantine partial every recipe includes passes `test_step_outcome: ${{ steps.tests.outcome }}` to the upload step, and that expression resolves to empty unless the test step carries that id, so the upload silently received no outcome and a runner that crashed while producing a clean-looking report could be treated as a pass. Two steps were also missing the `continue-on-error: true` the same partial requires. Settles Mergifyio/ci-bot#372, which asked for six of the ten. All eleven are fixed here: the six-versus-four split was an artifact of how that run was scoped, and leaving four behind keeps the corpus inconsistent. This does overlap #12411, which fixes the link line in four of them (golang, jest, junit, minitest), so whichever lands second should drop its overlapping lines. Part of MRGFY-8720 Change-Id: I482ec81d7710f37d83dcffdf65abc029b73437fc
JulianMaurin
left a comment
JulianMaurin
left a comment
Contributor
There was a problem hiding this comment.
I'm good for everything but the link and page name for test-insight detection
| You can then review your test results, including any failures or flaky tests, | ||
| directly in the [Test Insights | ||
| dashboard](https://dashboard.mergify.com/test-insights/jobs). | ||
| dashboard](https://dashboard.mergify.com/test-insights/detection). |
Contributor
There was a problem hiding this comment.
actually the new page link is: https://dashboard.mergify.com/orgs/{ORG}/repos/{REPO}/test-insights/detection
Should we name it Test Insights dashboard or the detection dashboard ?
jd
added a commit
that referenced
this pull request
Aug 18, 2026
…hat exists Eleven test framework recipes closed by sending the reader to the Test Insights dashboard. Ten of them linked `https://dashboard.mergify.com/test-insights/jobs`, which is not a page: Test Insights has Prevention, Detection and Mitigation, so that URL matched the router and rendered nothing. `jobs` is a CI Insights route, a different and org-scoped surface, which is where the copy-paste came from. The eleventh (Cypress) named the dashboard in the same sentence with no link at all. The paragraph promises "test results, including any failures or flaky tests", and Detection is the page listing flaky and broken tests, so that is where all eleven now point. It is also a partial now, included by the recipes rather than pasted into each. One stale URL became ten broken links precisely because it was ten copies of one sentence, and an eleventh copy had already drifted into a dead end. Separately, the recipes' inline GitHub Actions examples gained `id: tests` on the test-runner step. The quarantine partial every recipe includes passes `test_step_outcome: ${{ steps.tests.outcome }}` to the upload step, and that expression resolves to empty unless the test step carries that id, so the upload silently received no outcome and a runner that crashed while producing a clean-looking report could be treated as a pass. Two steps were also missing the `continue-on-error: true` the same partial requires. Settles Mergifyio/ci-bot#372, which asked for six of the ten. All eleven are fixed here: the six-versus-four split was an artifact of how that run was scoped, and leaving four behind keeps the corpus inconsistent. This does overlap #12411, which fixes the link line in four of them (golang, jest, junit, minitest), so whichever lands second should drop its overlapping lines. Part of MRGFY-8720 Change-Id: I482ec81d7710f37d83dcffdf65abc029b73437fc
jd
approved these changes
Aug 18, 2026
Contributor
|
@mergify-ci-bot this pull request is now in conflict 😩 |
mergify Bot
pushed a commit
that referenced
this pull request
Aug 20, 2026
…#12454) `pnpm check:internal-leaks` only ever saw a ticket as a *link*: its `internal-tracker` rule matches tracker hosts. A bare issue key in prose, in a code block or in an example value passed cleanly — and one did. The `mergify tests quarantines add` example on the quarantine page shipped with a `--reason` string naming an internal ticket, so it has been live on docs.mergify.com since #11681 and sits in this repository's public history. The scan reported that exact file clean, before and after. A bare key is also the case a reviewer is least likely to stop on: inside an example value it reads as a plausible sample rather than as a leak. What changed: - A `ticket-ref` rule for bare issue keys — the two live prefixes, plus the lowercase form a branch name carries. - Prefixes are listed explicitly instead of matching a generic `[A-Z]{3,}-\d+`. The generic shape fires on `AES-256`, `WCAG-2` and on a reader's own issue keys in a sample config, and that cost lands on every docs contributor; the miss cost of an explicit list falls only on a prefix somebody introduces deliberately, which is a deliberate act. - The allow directive now accepts several rule ids. One line can trip two rules — a tracker URL carries the key inside it — and such a line previously could not be allowed at all. - The corpus is swept: across the 143 scanned pages that example was the only occurrence. Its reason string is now "flaky under load, fix in progress", the same wording #12411 uses for the same line, so the two changes do not fight. That also removes the prose em dash the example carried inside a CLI string. - `AGENTS.md` and the `proofread-leaks` skill now list bare keys as well. The judgment layer had the same blind spot as the regex, which is why the line survived review too. Verified: the scan exits 1 against the pre-fix file and 0 against the tree, the full `pnpm test` suite and `pnpm check` are green, and the commit is green on its own as well as on top of the stack. One decision is deliberately left open: whether the already-published value needs scrubbing beyond this branch. It is a ticket prefix rather than a credential, and it is in the git history of a public repository as well as on the site, so removing it from the page is all this change claims to do. Reported as Mergifyio/ci-bot#368. MRGFY-8721
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Detection page described a classifier we do not have. It said health status is computed
with recent results weighted more heavily; there is no weighting. A test is flaky when it
passed and failed close together on the same branch, pipeline and job, and broken when
nothing close in time contradicts its failures. Recency enters only as a recovery rule: a
test that has kept running and passing for a week since its last failure goes back to
healthy. The page also said healthy tests are never listed, but Detection has Flaky, Broken
and Healthy tabs.
Confidence was documented as high or low. The API schema's
HealthConfidenceenum, and thevalue the dashboard renders, have three levels, so medium is now documented alongside the
other two, here and in the key concepts on the Test Insights hub page. Impact was described
as how many failed executions a test causes; it is a share, not a count, being the failure
ratio bucketed into low, medium and high. It is also a filter rather than a sort, and the
filter list gains the Confidence and Impact entries it was missing. The hub
page's glossary defined a flaky test as one producing different results on the
same commit, which is looser than the rule it stands for: a failure is
required, so a test skipped in one run and passed in another stays healthy.
The Go, Jest, JUnit and Minitest recipes closed by linking to a Test Insights dashboard URL
ending in
jobs. That is a CI Insights route; the dashboard registers only prevention,detection and mitigation under Test Insights, so the link rendered nothing. They now point
at Detection, the page that lists the failing and flaky tests the sentence promises. Their
workflow examples also gained the
id: teststhat the upload step'stest_step_outcomeinput refers to, plus the
continue-on-error: truequarantine needs, which two weremissing.