Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions src/components/BuildkiteCIUploadStepMatrix.astro
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,13 @@ const { reportPath } = Astro.props;
---

<p>
If you use a build matrix in your pipeline, set the <code>job_name</code> property so <strong>CI Insights</strong> can properly distinguish reports for each matrix variation.
If you use a build matrix in your pipeline, each variation reports its own step label as its job
name, so the variations are already distinct. Set the <code>job_name</code> plugin property when the
label is not the name you want <strong>CI Insights</strong> to file the reports under:
</p>
<p>For example:</p>
<Code
code={`steps:
- label: "Tests ({{matrix}})"
- label: "Flaky detection ({{matrix}})"
matrix:
- "3.10"
- "3.11"
Expand All @@ -29,9 +30,9 @@ const { reportPath } = Astro.props;
token: "\${MERGIFY_TOKEN}"`}
lang="yaml"
/>
<Aside type="tip">
<Aside type="note">
<p>
Setting <code>job_name</code> helps <strong>CI Insights</strong> group and display reports correctly
for each matrix variation.
The job name is part of a test's identity, so a scheduled pipeline has to report the same name as
the step running those tests on your pull requests.
</p>
</Aside>
14 changes: 6 additions & 8 deletions src/components/MergifyCIUploadStepMatrix.astro
Original file line number Diff line number Diff line change
Expand Up @@ -11,12 +11,10 @@ const { reportPath } = Astro.props;
---

<p>
If you use a job matrix in your workflow (e.g., to test across multiple versions), ensure you set
the <code>job_name</code> input (or <code>MERGIFY_JOB_NAME</code> environment variable) so <strong
>CI Insights</strong
> can properly distinguish reports for each matrix job.
If you use a job matrix in your workflow (e.g., to test across multiple versions), every leg
reports the same job name, because GitHub Actions gives them all one <code>GITHUB_JOB</code>. Set
the <code>job_name</code> input so <strong>CI Insights</strong> can tell the reports apart:
</p>
<p>For example, with:</p>
<Code
code={`jobs:
example_matrix:
Expand All @@ -37,9 +35,9 @@ const { reportPath } = Astro.props;
test_step_outcome: \${{ steps.tests.outcome }}`}
lang="yaml"
/>
<Aside type="tip">
<Aside type="note">
<p>
Setting <code>job_name</code> helps <strong>CI Insights</strong> group and display reports correctly
for each matrix variation.
The job name is part of a test's identity, so a scheduled workflow has to report the same name as
the job running those tests on your pull requests.
</p>
</Aside>
15 changes: 12 additions & 3 deletions src/content/docs/ci-insights/setup/buildkite.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,18 @@ every 12 hours on weekdays.
- **Weekday Schedule**: Run Monday to Friday to avoid running when no changes
are made to the code

- **Job name**: Ensure the step label matches the one running tests on your
pull request, allowing CI Insights to identify them. It can be overridden
with the `job_name` plugin property if needed.
- **Job name**: CI Insights records the job a test ran in as part of that
test's identity, so the scheduled step has to report the same job name as the
step running those tests on your pull requests. Otherwise the two sets of
results count as two different tests. By default that name is the step label.
Set `MERGIFY_TEST_JOB_NAME` in the step's `env` to report a different one:

```yaml
steps:
- label: "Flaky test detection"
env:
MERGIFY_TEST_JOB_NAME: "Run tests"
```

## Test Framework Configuration

Expand Down
25 changes: 22 additions & 3 deletions src/content/docs/ci-insights/setup/github-actions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -155,11 +155,30 @@ jobs:
- **Weekday Schedule**: The example runs Monday to Friday (`1-5`) to avoid
running when no changes are made to the code

- **Job name**: Ensure the job name is the same as the one running tests on your pull request,
allowing CI Insights to identify them. It can be overridden with `MERGIFY_TEST_JOB_NAME` if needed.
- **Job name**: CI Insights records the job a test ran in as part of that
test's identity, so the scheduled run has to report the same job name as the
job running those tests on your pull requests. Otherwise the two sets of
results count as two different tests. By default that name comes from
`GITHUB_JOB`, the workflow job's identifier. Set `MERGIFY_TEST_JOB_NAME` on
the job to report a different one. Matrix builds need it, because every
matrix job reports the same `GITHUB_JOB`:

```yaml
jobs:
test:
strategy:
matrix:
version: [10, 12, 14]
env:
MERGIFY_TEST_JOB_NAME: test (${{ matrix.version }})
```

Set it on the job rather than on a single step, so every step in the job
reports the same name. A step's `env` applies to that step alone, so a name
set on the step that runs your tests never reaches the upload step.

### Key Environment Variables

- `MERGIFY_TOKEN`: Application key with `ci` scope for uploading test results to CI Insights
- `RUN_COUNT`: Number of times to execute each test within a single job
- `MERGIFY_TEST_JOB_NAME`: The job name reported to CI Insights
- `MERGIFY_TEST_JOB_NAME`: Override for the job name reported to CI Insights
17 changes: 14 additions & 3 deletions src/content/docs/ci-insights/setup/jenkins.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -167,11 +167,22 @@ pipeline {
- **Weekday Schedule**: The example runs Monday to Friday (`1-5`) to avoid
running when no changes are made to the code

- **Job name**: Ensure the job name is the same as the one running tests on your pull request,
allowing CI Insights to identify them. It can be overridden with `MERGIFY_TEST_JOB_NAME` if needed.
- **Job name**: CI Insights records the job a test ran in as part of that
test's identity, so the scheduled job has to report the same job name as the
job running those tests on your pull requests. Otherwise the two sets of
results count as two different tests. By default that name comes from
Jenkins' `JOB_NAME`. Set `MERGIFY_TEST_JOB_NAME` in the pipeline's
`environment` block to report a different one. You need this when the
scheduled job and the pull request job have different Jenkins names:

```groovy
environment {
MERGIFY_TEST_JOB_NAME = 'unit-tests'
}
```

### Key Environment Variables

- `MERGIFY_TOKEN`: Application key with `ci` scope for uploading test results to CI Insights
- `RUN_COUNT`: Number of times to execute each test within a single job
- `MERGIFY_TEST_JOB_NAME`: The job name reported to CI Insights
- `MERGIFY_TEST_JOB_NAME`: Override for the job name reported to CI Insights
7 changes: 2 additions & 5 deletions src/content/docs/test-insights/test-frameworks/junit.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,12 @@ The JUnit XML reports will be generated in `target/surefire-reports/`.

### Using Gradle

Configure the test task in your `build.gradle`:
Gradle's `test` task writes JUnit XML reports without any extra configuration.
If your tests run on JUnit 5, enable the platform in your `build.gradle`:

```groovy
test {
useJUnitPlatform()
reports {
junitXml.enabled = true
junitXml.destination = file("$buildDir/test-results/test")
}
}
```

Expand Down
9 changes: 4 additions & 5 deletions src/content/docs/test-insights/test-frameworks/testng.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -55,15 +55,12 @@ mvn test

### Using Gradle

Configure the test task in your `build.gradle`:
Gradle's `test` task writes JUnit XML reports without any extra configuration.
Point the task at TestNG in your `build.gradle`:

```groovy
test {
useTestNG()
reports {
junitXml.enabled = true
junitXml.destination = file("$buildDir/test-results/test")
}
}
```

Expand All @@ -73,6 +70,8 @@ Run tests with:
./gradlew test
```

The JUnit XML reports will be generated in `build/test-results/test/`.

### Using TestNG Directly

You can also run TestNG directly with JUnit listener:
Expand Down
Loading