From 42d7aaeedaa247d7c8d36dd12ee12e7522cd0e5d Mon Sep 17 00:00:00 2001
From: Julien Danjou <julien@danjou.info>
Date: Mon, 17 Aug 2026 16:19:06 +0200
Subject: [PATCH] fix(ci-insights): name the job-name variable the uploader
 actually reads
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The CI Insights setup pages and the two matrix helpers disagreed with each other
about how to override the job name a test report is filed under, and none of
them said where the default comes from.

The three setup pages now name `MERGIFY_TEST_JOB_NAME` — the variable the CLI
reads, and the one every framework plugin (pytest, rspec, vitest) already
documents — say where the default comes from on each provider (`GITHUB_JOB` on
GitHub Actions, Jenkins' own `JOB_NAME`, the step label on Buildkite), and show
where to set it. They also state why it matters: the job name is part of a
test's identity, so a scheduled run that reports a different name splits its
results off from the pull request ones.

The GitHub Actions matrix case is stated the way it actually breaks: every leg
of a matrix reports the same job name because they share one `GITHUB_JOB`. The
old text left the reader to infer why an override was needed at all.

The two matrix helpers keep recommending the native override — the action's
`job_name` input and the Buildkite plugin's `job_name` property. Both currently
export `MERGIFY_JOB_NAME`, which no uploader reads, so neither takes effect
today; that is a bug in the three producers of that variable, not something the
docs should route readers around. Fixing them is tracked separately, and the
snippets pin moving tags (the action's `v25`, the plugin's `#v1`), so readers
pick the fix up when those tags move.

The pytest, rspec and vitest pages already agreed with this and are unchanged.

Settles Mergifyio/ci-bot#369.
Part of MRGFY-8720

Change-Id: I96e06bb4500bf9e7664d36d941ef0d962004e9a9
---
 .../BuildkiteCIUploadStepMatrix.astro         | 13 +++++-----
 .../MergifyCIUploadStepMatrix.astro           | 14 +++++------
 .../docs/ci-insights/setup/buildkite.mdx      | 15 ++++++++---
 .../docs/ci-insights/setup/github-actions.mdx | 25 ++++++++++++++++---
 .../docs/ci-insights/setup/jenkins.mdx        | 17 ++++++++++---
 5 files changed, 61 insertions(+), 23 deletions(-)

diff --git a/src/components/BuildkiteCIUploadStepMatrix.astro b/src/components/BuildkiteCIUploadStepMatrix.astro
index 724de1dd7f..92a12ed391 100644
--- a/src/components/BuildkiteCIUploadStepMatrix.astro
+++ b/src/components/BuildkiteCIUploadStepMatrix.astro
@@ -10,12 +10,13 @@ const { reportPath } = Astro.props;
 ---
 
 <p>
-	If you use a build matrix in your pipeline, set the <code>job_name</code> property so <strong>CI Insights</strong> can properly distinguish reports for each matrix variation.
+	If you use a build matrix in your pipeline, each variation reports its own step label as its job
+	name, so the variations are already distinct. Set the <code>job_name</code> plugin property when the
+	label is not the name you want <strong>CI Insights</strong> to file the reports under:
 </p>
-<p>For example:</p>
 <Code
 	code={`steps:
-  - label: "Tests ({{matrix}})"
+  - label: "Flaky detection ({{matrix}})"
     matrix:
       - "3.10"
       - "3.11"
@@ -29,9 +30,9 @@ const { reportPath } = Astro.props;
           token: "\${MERGIFY_TOKEN}"`}
 	lang="yaml"
 />
-<Aside type="tip">
+<Aside type="note">
 	<p>
-		Setting <code>job_name</code> helps <strong>CI Insights</strong> group and display reports correctly
-		for each matrix variation.
+		The job name is part of a test's identity, so a scheduled pipeline has to report the same name as
+		the step running those tests on your pull requests.
 	</p>
 </Aside>
diff --git a/src/components/MergifyCIUploadStepMatrix.astro b/src/components/MergifyCIUploadStepMatrix.astro
index 97fb1ef955..cb8acb823f 100644
--- a/src/components/MergifyCIUploadStepMatrix.astro
+++ b/src/components/MergifyCIUploadStepMatrix.astro
@@ -11,12 +11,10 @@ const { reportPath } = Astro.props;
 ---
 
 <p>
-	If you use a job matrix in your workflow (e.g., to test across multiple versions), ensure you set
-	the <code>job_name</code> input (or <code>MERGIFY_JOB_NAME</code> environment variable) so <strong
-		>CI Insights</strong
-	> can properly distinguish reports for each matrix job.
+	If you use a job matrix in your workflow (e.g., to test across multiple versions), every leg
+	reports the same job name, because GitHub Actions gives them all one <code>GITHUB_JOB</code>. Set
+	the <code>job_name</code> input so <strong>CI Insights</strong> can tell the reports apart:
 </p>
-<p>For example, with:</p>
 <Code
 	code={`jobs:
   example_matrix:
@@ -37,9 +35,9 @@ const { reportPath } = Astro.props;
     test_step_outcome: \${{ steps.tests.outcome }}`}
 	lang="yaml"
 />
-<Aside type="tip">
+<Aside type="note">
 	<p>
-		Setting <code>job_name</code> helps <strong>CI Insights</strong> group and display reports correctly
-		for each matrix variation.
+		The job name is part of a test's identity, so a scheduled workflow has to report the same name as
+		the job running those tests on your pull requests.
 	</p>
 </Aside>
diff --git a/src/content/docs/ci-insights/setup/buildkite.mdx b/src/content/docs/ci-insights/setup/buildkite.mdx
index 84e2a02659..dd56f317a0 100644
--- a/src/content/docs/ci-insights/setup/buildkite.mdx
+++ b/src/content/docs/ci-insights/setup/buildkite.mdx
@@ -139,9 +139,18 @@ every 12 hours on weekdays.
 - **Weekday Schedule**: Run Monday to Friday to avoid running when no changes
   are made to the code
 
-- **Job name**: Ensure the step label matches the one running tests on your
-  pull request, allowing CI Insights to identify them. It can be overridden
-  with the `job_name` plugin property if needed.
+- **Job name**: CI Insights records the job a test ran in as part of that
+  test's identity, so the scheduled step has to report the same job name as the
+  step running those tests on your pull requests. Otherwise the two sets of
+  results count as two different tests. By default that name is the step label.
+  Set `MERGIFY_TEST_JOB_NAME` in the step's `env` to report a different one:
+
+  ```yaml
+  steps:
+    - label: "Flaky test detection"
+      env:
+        MERGIFY_TEST_JOB_NAME: "Run tests"
+  ```
 
 ## Test Framework Configuration
 
diff --git a/src/content/docs/ci-insights/setup/github-actions.mdx b/src/content/docs/ci-insights/setup/github-actions.mdx
index c39674c8c5..e0cc5f5011 100644
--- a/src/content/docs/ci-insights/setup/github-actions.mdx
+++ b/src/content/docs/ci-insights/setup/github-actions.mdx
@@ -155,11 +155,30 @@ jobs:
 - **Weekday Schedule**: The example runs Monday to Friday (`1-5`) to avoid
   running when no changes are made to the code
 
-- **Job name**: Ensure the job name is the same as the one running tests on your pull request,
-  allowing CI Insights to identify them. It can be overridden with `MERGIFY_TEST_JOB_NAME` if needed.
+- **Job name**: CI Insights records the job a test ran in as part of that
+  test's identity, so the scheduled run has to report the same job name as the
+  job running those tests on your pull requests. Otherwise the two sets of
+  results count as two different tests. By default that name comes from
+  `GITHUB_JOB`, the workflow job's identifier. Set `MERGIFY_TEST_JOB_NAME` on
+  the job to report a different one. Matrix builds need it, because every
+  matrix job reports the same `GITHUB_JOB`:
+
+  ```yaml
+  jobs:
+    test:
+      strategy:
+        matrix:
+          version: [10, 12, 14]
+      env:
+        MERGIFY_TEST_JOB_NAME: test (${{ matrix.version }})
+  ```
+
+  Set it on the job rather than on a single step, so every step in the job
+  reports the same name. A step's `env` applies to that step alone, so a name
+  set on the step that runs your tests never reaches the upload step.
 
 ### Key Environment Variables
 
 - `MERGIFY_TOKEN`: Application key with `ci` scope for uploading test results to CI Insights
 - `RUN_COUNT`: Number of times to execute each test within a single job
-- `MERGIFY_TEST_JOB_NAME`: The job name reported to CI Insights
+- `MERGIFY_TEST_JOB_NAME`: Override for the job name reported to CI Insights
diff --git a/src/content/docs/ci-insights/setup/jenkins.mdx b/src/content/docs/ci-insights/setup/jenkins.mdx
index 9bc41c73f4..e4c9561b05 100644
--- a/src/content/docs/ci-insights/setup/jenkins.mdx
+++ b/src/content/docs/ci-insights/setup/jenkins.mdx
@@ -167,11 +167,22 @@ pipeline {
 - **Weekday Schedule**: The example runs Monday to Friday (`1-5`) to avoid
   running when no changes are made to the code
 
-- **Job name**: Ensure the job name is the same as the one running tests on your pull request,
-  allowing CI Insights to identify them. It can be overridden with `MERGIFY_TEST_JOB_NAME` if needed.
+- **Job name**: CI Insights records the job a test ran in as part of that
+  test's identity, so the scheduled job has to report the same job name as the
+  job running those tests on your pull requests. Otherwise the two sets of
+  results count as two different tests. By default that name comes from
+  Jenkins' `JOB_NAME`. Set `MERGIFY_TEST_JOB_NAME` in the pipeline's
+  `environment` block to report a different one. You need this when the
+  scheduled job and the pull request job have different Jenkins names:
+
+  ```groovy
+  environment {
+      MERGIFY_TEST_JOB_NAME = 'unit-tests'
+  }
+  ```
 
 ### Key Environment Variables
 
 - `MERGIFY_TOKEN`: Application key with `ci` scope for uploading test results to CI Insights
 - `RUN_COUNT`: Number of times to execute each test within a single job
-- `MERGIFY_TEST_JOB_NAME`: The job name reported to CI Insights
+- `MERGIFY_TEST_JOB_NAME`: Override for the job name reported to CI Insights