[CI-6519] Document linking test attachments by file name

CLAUDE.md

@@ -215,7 +215,7 @@ the same `slug:`).
 ### Code
 
 - Inline: backticks for filenames, command names, env var names. ``Open `bitrise.yml`.``
-- Blocks: triple backticks with a language hint (one of `yaml`, `bash`, `json`, `swift`, `kotlin`, `groovy`, `ruby`, `dart`, `diff` — those are the languages our Prism config loads).
+- Blocks: triple backticks with a language hint. `docusaurus.config.ts` adds `yaml`, `bash`, `json`, `swift`, `kotlin`, `groovy`, `ruby`, `dart`, `diff` on top of the languages `prism-react-renderer` bundles (among others `javascript`, `typescript`, `xml`, `go`, `python`). Use `text` for plain text, such as file names.
 - **Don't put code blocks inside admonitions.** Render the admonition first, then the code block as a sibling.
 
 **Showing a change to an existing file**

docs/bitrise-ci/testing/deploying-and-viewing-test-results.mdx

@@ -35,7 +35,7 @@ If you use any of these Steps, make sure you have the **Deploy to Bitrise.io** S
 
 :::important[Attachments and flaky tests]
 
-If your tests generate attachments or you want Bitrise to show flaky tests, the **Deploy to Bitrise.io** Step must be of version 2.19.1 or newer.
+If your tests generate attachments or you want Bitrise to show flaky tests, the **Deploy to Bitrise.io** Step must be of version 2.19.1 or newer. To link attachments to test cases by filename, it must be version 2.26.0 or newer: [Naming attachments after their test case](#naming-attachments-after-their-test-case).
 
 :::
 
@@ -50,7 +50,7 @@ If you run your tests using other Steps (for example, you can use [Script](https
 
    :::important[Attachments and flaky tests]
 
-   If your tests generate attachments or you want Bitrise to show flaky tests, the **Deploy to Bitrise.io** Step must be of version 2.19.1 or newer.
+   If your tests generate attachments or you want Bitrise to show flaky tests, the **Deploy to Bitrise.io** Step must be of version 2.19.1 or newer. To link attachments to test cases by filename, it must be version 2.26.0 or newer: [Naming attachments after their test case](#naming-attachments-after-their-test-case).
 
    :::
 
@@ -92,7 +92,12 @@ Test reports also support the use of the `system-out` tag in JUnit. If a test ca
 
 ## Collating test attachments with test results
 
-Many tests generate image files (screenshot tests, XCUITests, etc). Bitrise test reports automatically collate image files with associated test cases for Xcode tests (using the `xcresult` file).
+Many tests generate image files: screenshot tests, XCUITests, and so on. Bitrise can show these attachments under their associated test cases, so you can view them on the **Tests** tab of your build page. How to collate the test attachments with test results depends on your test tool and whether it generates an `xcresult` file:
+
+- If your tests generate an `xcresult` file, Bitrise automatically collates the attachments with their test cases. This requires no further configuration.
+- If your tests don't generate an `xcresult` file, you can:
+    - Name each file after its test case: [Naming attachments after their test case](#naming-attachments-after-their-test-case).
+    - Reference each file in the JUnit XML: [Referencing attachments in the JUnit XML](#referencing-attachments-in-the-junit-xml).
 
 :::note[Supported file formats]
 
@@ -102,7 +107,266 @@ Currently, we support the following file formats for attachments:
 
 :::
 
-However, if your test doesn't generate an `xcresult` file, you can still achieve the same thing by generating a JUnit XML file and using Bitrise Steps:
+### Naming attachments after their test case
+
+Name a screenshot, recording, or log after the test case that produced it, and Bitrise shows it under that test on the **Tests** tab.
+
+:::important[Step versions]
+
+This requires the **Deploy to Bitrise.io** Step version 2.26.0 or newer. If you use the **Export test results to Test Reports** Step, it must be version 1.3.0 or newer.
+
+:::
+
+You need to:
+
+- [Use the correct filename structure](#about-the-filename-structure).
+- [Set up your test tool to name the attachments correctly](#setting-up-your-test-tool).
+- [Export the attachments with a Bitrise Step](#exporting-the-attachments).
+
+#### About the filename structure
+
+The filename should have the following structure:
+
+```text
+<classname>__<name>[__run<k>]__<label>.<ext>
+```
+
+For example, `com.example.LoginTest__emptyState__1.png`.
+
+To get the correct filename:
+
+- Copy `classname` and `name` from the `<testcase>` element of your JUnit XML, and replace any of the `" * / : < > ? \ |` characters with `_`. Bitrise applies the same replacement to the names in the JUnit XML, so either spelling matches.
+- Always add a label, such as a counter. The label can't contain `__` or start with `_`.
+- If the same tests run on more than one device, put the device in the label, for example `api34_1`. Otherwise, files with the same name are skipped, and the attachments don't appear.
+- To attach a file to a later run of the same test case, add `__run2`, `__run3`, and so on before the label.
+
+#### Setting up your test tool
+
+Set up the naming once, in your test configuration. The following examples do this for some popular test tools:
+
+<Tabs>
+<TabItem value="espresso" label="Espresso" default>
+
+Register a `RunListener` that takes a screenshot when an Espresso check fails:
+
+```kotlin
+// app/build.gradle.kts
+android {
+    defaultConfig {
+        testInstrumentationRunnerArguments["listener"] = "com.example.testing.BitriseScreenshotListener"
+        testInstrumentationRunnerArguments["useTestStorageService"] = "true"
+    }
+}
+
+dependencies {
+    androidTestUtil("androidx.test.services:test-services:1.6.0")
+}
+```
+
+```kotlin
+// app/src/androidTest/java/com/example/testing/BitriseScreenshotListener.kt
+package com.example.testing
+
+import android.content.res.Resources
+import android.os.Build
+import androidx.test.core.app.takeScreenshot
+import androidx.test.core.graphics.writeToTestStorage
+import androidx.test.espresso.Espresso
+import androidx.test.espresso.base.DefaultFailureHandler
+import androidx.test.platform.app.InstrumentationRegistry
+import org.junit.runner.Description
+import org.junit.runner.notification.RunListener
+
+private fun clean(value: String) = value.replace(Regex("""["*/:<>?\\|]"""), "_")
+
+class BitriseScreenshotListener : RunListener() {
+    override fun testStarted(test: Description) {
+        val default = DefaultFailureHandler(InstrumentationRegistry.getInstrumentation().targetContext)
+        Espresso.setFailureHandler { error, viewMatcher ->
+            val screen = Resources.getSystem().displayMetrics.run { "${widthPixels}x$heightPixels" }
+            val name = "${clean(test.className)}__${clean(test.methodName)}__api${Build.VERSION.SDK_INT}_${screen}_failure"
+            takeScreenshot().writeToTestStorage(name)
+            default.handle(error, viewMatcher)
+        }
+    }
+}
+```
+
+Plain JUnit assertions don't get a screenshot.
+
+</TabItem>
+<TabItem value="paparazzi" label="Paparazzi">
+
+Create your Paparazzi rules through a shared function that adds a snapshot handler. The handler saves every snapshot under the attachment name as well:
+
+```kotlin
+// src/test/java/com/example/testing/BitrisePaparazzi.kt
+package com.example.testing
+
+import app.cash.paparazzi.DeviceConfig
+import app.cash.paparazzi.HtmlReportWriter
+import app.cash.paparazzi.Paparazzi
+import app.cash.paparazzi.Snapshot
+import app.cash.paparazzi.SnapshotHandler
+import app.cash.paparazzi.SnapshotVerifier
+import java.awt.image.BufferedImage
+import java.io.File
+import javax.imageio.ImageIO
+
+private fun clean(value: String) = value.replace(Regex("""["*/:<>?\\|]"""), "_")
+
+fun bitrisePaparazzi(deviceConfig: DeviceConfig = DeviceConfig.PIXEL_5) =
+    Paparazzi(deviceConfig = deviceConfig, snapshotHandler = BitriseSnapshotHandler())
+
+class BitriseSnapshotHandler(
+    maxPercentDifference: Double = 0.1,
+    private val delegate: SnapshotHandler =
+        if (System.getProperty("paparazzi.test.verify") == "true") SnapshotVerifier(maxPercentDifference)
+        else HtmlReportWriter(),
+) : SnapshotHandler by delegate {
+    override fun newFrameHandler(snapshot: Snapshot, frameCount: Int, fps: Int): SnapshotHandler.FrameHandler {
+        val frames = delegate.newFrameHandler(snapshot, frameCount, fps)
+        val test = snapshot.testName
+        val label = snapshot.name?.replace(Regex("_+"), "_")?.removePrefix("_") ?: "snapshot"
+        val name = "${clean("${test.packageName}.${test.className}")}__${clean(test.methodName)}__$label"
+        val file = File(System.getProperty("paparazzi.build.dir"), "bitrise-snapshots/${System.getProperty("bitrise.test.task")}/$name.png")
+        return object : SnapshotHandler.FrameHandler by frames {
+            override fun handle(image: BufferedImage) {
+                file.parentFile.mkdirs()
+                ImageIO.write(image, "png", file)
+                frames.handle(image)
+            }
+        }
+    }
+}
+```
+
+In your tests, replace `Paparazzi(...)` with `bitrisePaparazzi(...)`. Then add the following to the module's `build.gradle.kts`. The first block lets the handler use `java.awt` on Android Gradle plugin 8, and the second one gives each test task its own folder, removes the snapshots of the previous run, and keeps the current ones in the build cache:
+
+```kotlin
+tasks.withType<org.jetbrains.kotlin.gradle.tasks.KotlinJvmCompile>().configureEach {
+    if ("UnitTest" in name) compilerOptions.noJdk.set(false)
+}
+
+tasks.withType<Test>().configureEach {
+    val snapshots = layout.buildDirectory.dir("bitrise-snapshots/$name")
+    systemProperty("bitrise.test.task", name)
+    outputs.dir(snapshots)
+    doFirst { snapshots.get().asFile.deleteRecursively() }
+}
+```
+
+The example is for Paparazzi 1.3.5. On Paparazzi 2.0, create the report writer with `HtmlReportWriter(maxPercentDifference = maxPercentDifference)`.
+
+</TabItem>
+<TabItem value="roborazzi" label="Roborazzi">
+
+Copy Roborazzi's screenshots under the attachment name after each Roborazzi run:
+
+```kotlin
+// app/build.gradle.kts
+val bitriseScreenshots by tasks.registering(Sync::class) {
+    from(layout.buildDirectory.dir("outputs/roborazzi")) { include("*.png") }
+    into(layout.buildDirectory.dir("bitrise-screenshots"))
+    rename("""(.+)\.([^.]+?)(?:_(\d+))?(_actual|_compare)?\.png""", "$1__$2__screenshot$3$4.png")
+}
+tasks.named { it == "finalizeTestRoborazziDebug" }.configureEach { finalizedBy(bitriseScreenshots) }
+```
+
+Roborazzi writes screenshots only when you run `recordRoborazziDebug`, `verifyRoborazziDebug`, or `compareRoborazziDebug`. Parameterized tests don't get attachments.
+
+</TabItem>
+<TabItem value="detox" label="Detox">
+
+Point `artifacts.pathBuilder` in your Detox configuration to this file. It names every screenshot, video, and log the way jest-junit names the test with its default settings:
+
+```javascript
+// e2e/bitrisePathBuilder.js
+const path = require('path');
+
+const clean = (value) => value.replace(/["*/:<>?\\|]/g, '_');
+
+module.exports = class BitrisePathBuilder {
+  constructor({ rootDir }) {
+    this.rootDir = rootDir;
+  }
+
+  buildPathForTestArtifact(artifactName, testSummary) {
+    if (!testSummary) return path.join(this.rootDir, artifactName);
+
+    const { title, fullName } = testSummary;
+    const key = clean(`${fullName.slice(0, -title.length).trim()} ${title}`);
+    const ext = path.extname(artifactName);
+    const label = path.basename(artifactName, ext).replace(/_+/g, '_').replace(/^_/, '');
+    return path.join(this.rootDir, `${key}__${key}__${label}${ext}`);
+  }
+};
+```
+
+</TabItem>
+<TabItem value="playwright" label="Playwright">
+
+Turn on `includeProjectInTestName` for the junit reporter, and add a reporter that copies the screenshots and videos of each test under the attachment name:
+
+```typescript
+// playwright.config.ts
+reporter: [
+  ['junit', { outputFile: 'test-results/junit.xml', includeProjectInTestName: true }],
+  ['./bitrise-reporter.ts'],
+],
+use: { screenshot: 'only-on-failure', video: 'retain-on-failure' },
+```
+
+```typescript
+// bitrise-reporter.ts
+import fs from 'node:fs';
+import path from 'node:path';
+import type { Reporter, TestCase, TestResult } from '@playwright/test/reporter';
+
+const clean = (value: string) => value.replace(/["*/:<>?\\|]/g, '_');
+const supported = /\.(png|jpe?g|mp4|webm|ogg|txt|log)$/i;
+
+export default class BitriseAttachments implements Reporter {
+  onTestEnd(test: TestCase, result: TestResult) {
+    const [, project, file, ...titles] = test.titlePath();
+    const key = `${clean(file)}__${clean((project ? `[${project}] ` : '') + titles.join(' › '))}`;
+    const files = result.attachments.filter((a) => a.path && supported.test(a.path));
+    files.forEach((a, i) => {
+      const label = (result.retry ? `retry${result.retry}_` : '') + (i + 1);
+      const target = path.join('test-attachments', `${key}__${label}${path.extname(a.path!)}`);
+      fs.mkdirSync(path.dirname(target), { recursive: true });
+      fs.copyFileSync(a.path!, target);
+    });
+  }
+}
+```
+
+</TabItem>
+</Tabs>
+
+#### Exporting the attachments
+
+:::important[Step versions]
+
+This requires the **Deploy to Bitrise.io** Step version 2.26.0 or newer. If you use the **Export test results to Test Reports** Step, it must be version 1.3.0 or newer.
+
+:::
+
+The **Deploy to Bitrise.io** Step is always required: it deploys the correctly named attachments alongside your test results. What else you need depends on the Step that runs your tests:
+
+- **Official Bitrise testing Steps**: No further configuration is needed. The **Deploy to Bitrise.io** Step exports the attachments together with the test results. For the list of Steps, see [Deploying results from the official Bitrise testing Steps](#deploying-results-from-the-official-bitrise-testing-steps).
+- **Other Steps**, such as Script or npm Steps: Add the **Export test results to Test Reports** Step after the Step that runs your tests. It finds the attachments and moves them to the location the **Deploy to Bitrise.io** Step deploys from. Set its **Test result base path** input to a folder that contains both your test results and your attachments, for example `app/build/`. See [Using the Export test results to Test Reports Step](#using-the-export-test-results-to-test-reports-step).
+
+#### Troubleshooting missing attachments
+
+If an attachment doesn't show up, check the build log:
+
+- The **Export test results to Test Reports** Step logs a warning for names that match more than one test or file, or that have an invalid `__run` number or no label. Other skipped files appear only with verbose logging: set its **Enable verbose logging?** input to `yes`.
+- The **Deploy to Bitrise.io** Step logs a warning for every file in the report that it can't link to a test case.
+
+### Referencing attachments in the JUnit XML
+
+If you can't name the files after their test cases, reference them in the JUnit XML instead:
 
 <Tabs>
 <TabItem value="workflow-editor" label="Workflow Editor" default>
@@ -249,6 +513,8 @@ With the correct configuration, the Step finds the test results in your project
 
    - If your project’s root directory is `app`: `app/build/test-results/testDemoDebugUnitTest/`
    - If your test results are within a project folder but `app` is not the root directory: ./app/build/test-results/testDemoDebugUnitTest/
+
+   To export [attachments named after their test case](/bitrise-ci/testing/deploying-and-viewing-test-results#naming-attachments-after-their-test-case), set a folder that contains both your test results and your attachments, for example `app/build/`.
 1. In the **Test result search pattern** input, set a pattern that matches your test result file.
 
    This search pattern is used to search every file and folder of the path that was set in the **Path where custom test results reside** input.
@@ -259,6 +525,7 @@ With the correct configuration, the Step finds the test results in your project
 
    - Matching all files within the base path: `*`
    - Matching all files within a given directory of the base path: `*/build/test-results/testDemoDebugUnitTest/*`
+   - Matching a single JUnit XML when the base path is a wider folder, such as `app/build/`: `*/test-results/testDemoDebugUnitTest/TEST-com.example.ScreenshotTest.xml`
 1. In the **Step’s test result directory** input, make sure the path is correct.
 
    Do NOT modify this input’s value: this is the folder where the **Deploy to Bitrise.io** Step will look for the test results to export them. It should be set to the `$BITRISE_TEST_RESULT_DIR` Env Var.