From 8c79ca69d60ee7c9c9b4db2448bfb53673f968b3 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 13:18:04 +0200 Subject: [PATCH 01/24] Add app-testing contract RFC --- app-testing-contract/README.md | 170 +++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 app-testing-contract/README.md diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md new file mode 100644 index 0000000..07957df --- /dev/null +++ b/app-testing-contract/README.md @@ -0,0 +1,170 @@ +--- +creation_date: 2026-07-07 +issues: [] +owners: +- https://github.com/orgs/giantswarm/teams/team-bumblebee +- https://github.com/orgs/giantswarm/teams/team-honeybadger +- https://github.com/orgs/giantswarm/teams/team-tenet +state: review +summary: Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (chart tests on kind) and apptest-framework (e2e on workload clusters). One conventional directory per repo, test types via build tags or pytest markers, inputs via KUBECONFIG and ATS_* env vars. +--- + +# The app-testing contract + +## Problem + +We test managed apps with two harnesses. app-test-suite (ATS) deploys the +chart on a kind cluster in the PR pipeline. apptest-framework (atf) stands +up a workload cluster on a management cluster and installs an App CR in the +e2e pipeline. Both need "does the deployed app actually work" checks, and +today each repo writes them twice, in two idioms (pytest or plain Go for +ATS, Ginkgo suites for atf). In practice one side is usually empty or +stale. + +ATS already publishes a testing contract +([docs/TEST_CONTRACT.md](https://github.com/giantswarm/app-test-suite/blob/master/docs/TEST_CONTRACT.md)), +but it lives in one harness's repo and only that harness implements it. +This RFC extracts the contract, makes it harness-neutral, and extends +apptest-framework to implement it too. The contract is deliberately not +tied to a test framework or language. + +## Decision + +### The conventional directory + +Each app repo has one conventional test directory: `tests/app/`. It +contains the app's tests, written against the contract below. Any harness +that deploys the app runs this directory the same way. Presence of the +directory is the opt-in; there is no per-repo wiring. + +The directory is one Go module or one Python project, never both. The +executor is detected from its contents: + +- `go.mod` present: `go test -tags=` +- `pyproject.toml` present: `uv sync && uv run pytest -m ` +- both present: configuration error, fail fast + +### Test types + +A test declares its types via Go build tags or pytest markers. A test may +carry several types. + +| Type | Meaning | +|---|---| +| `smoke` | fast, fail-fast sanity checks, run first | +| `functional` | full feature tests | +| `pre_upgrade` | runs before the app is upgraded (for example: seed a workload) | +| `post_upgrade` | runs after the app is upgraded (for example: verify the workload survived) | + +`upgrade` remains a deprecated alias meaning both `pre_upgrade` and +`post_upgrade`, so existing ATS tests keep working during migration. +Distinct pre/post types replace the previous single `upgrade` type because +the canonical upgrade test is asymmetric, which one type run twice cannot +express. + +### Inputs + +Tests receive everything through the environment. They never provision: +no cluster creation, no chart install, no App CRs. + +| Variable | Required | Meaning | +|---|---|---| +| `KUBECONFIG` | yes | kubeconfig of the cluster the app is deployed on (never the MC) | +| `ATS_TEST_TYPE` | yes | the type currently being run | +| `ATS_RELEASE_NAME` | yes | Helm release name of the app under test | +| `ATS_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | +| `ATS_CHART_VERSION` | yes | version of the chart under test | +| `ATS_CLUSTER_TYPE` | yes | `kind`, `external`, or `capi` | +| `ATS_CLUSTER_VERSION` | optional | Kubernetes server version | +| `ATS_APP_CONFIG_FILE_PATH` | optional | values file the app was deployed with | +| `ATS_UPGRADE_FROM_VERSION` / `ATS_UPGRADE_TO_VERSION` | upgrade runs only | versions on either side of the upgrade | +| `ATS_EXTRA_*` | optional | harness extras, for example `ATS_EXTRA_GITOPS_ENGINE` | + +The `ATS_` prefix is kept as-is: it is the published contract of the +existing implementation, and renaming would break every current test for +cosmetic gain. Read it as "app-testing", not as the harness's name. + +### Runner guarantees + +Before invoking the executor, a conforming runner guarantees: + +1. the app is deployed and settled (ATS: chart installed via Helm or a + GitOps engine; atf: App CR reconciled to `deployed`), +2. all required variables above are exported, +3. the executor is invoked once per applicable test type, in order: + `smoke`, then `functional`; for upgrade flows: `pre_upgrade`, then the + upgrade is performed, then `post_upgrade`. + +"No tests for this type" is a pass, not a failure (Go: build constraints +exclude all files; pytest: exit code 5). Test results are emitted as junit +XML: `gotestsum --junitfile` for Go, `pytest --junitxml` for Python. + +### Shared configuration + +`tests/app/config.yaml` carries only the keys both harnesses need: + +```yaml +installNamespace: kube-system +upgrade: true # whether an upgrade flow applies to this app +``` + +Everything harness-specific stays in the harness's own config: +`.ats/main.yaml` (cluster types, catalogs, executor options) and +`tests/e2e/config.yaml` (appCatalog, providers, MC test options). Values +files also stay per harness: values legitimately differ between a kind +cluster and a workload cluster, and each provisioner consumes them through +its own mechanism. + +### Harness-specific tests + +The contract covers the default case, not everything. Where a test goes: + +1. Asserts on the deployed app and works on any cluster: `tests/app/`, + no gate. This should be the bulk. +2. Asserts on the deployed app but is only meaningful in one environment: + `tests/app/` plus a runtime skip on contract environment, for example + `if os.Getenv("ATS_CLUSTER_TYPE") != "kind" { t.Skip(...) }`. Skips + stay visible by name in both runners' output. +3. Needs harness machinery (MC access, bundle installs, AWS/IRSA, cluster + manipulation): a regular in-process apptest-framework suite under + `tests/e2e/suites/`, unchanged by this RFC. + +Tests may gate on environment properties the contract exposes, never on +which harness is running them. A test that needs the harness's name +belongs in category 3. + +## Implementation + +- **apptest-framework** gains a convention-runner: after provisioning the + workload cluster and App CR, it fetches the WC kubeconfig + (`Framework.GetClusterKubeConfig`), writes it to a file, exports the env + contract, detects the executor, and runs it per test type. The image + gains `uv` and `gotestsum`. Existing in-process suites are unaffected. + Enabling facts: Ginkgo runs under plain `go test`, and pytest tests + built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS + tests of both languages are immediately reusable on workload clusters. +- **app-test-suite** implements the `pre_upgrade` / `post_upgrade` types + in its upgrade scenario (keeping `upgrade` as the alias), searches + `tests/app/` in addition to its current `tests/ats/` default, reads the + shared config keys, and emits junit via gotestsum. Its + TEST_CONTRACT.md becomes a pointer to this RFC plus ATS-specific detail. +- **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so + non-portable suites share the same readiness vocabulary. +- **devctl `gen apptest` and template-app** scaffold the conventional + layout for new repos. +- Migration is opt-in as repos get touched; there is no flag day. + Pilot: [giantswarm/muster#954](https://github.com/giantswarm/muster/pull/954). + +## Alternatives considered + +- **A shared assertions library consumed by both harnesses per repo.** + Prototyped in muster; the module, replace directives, and adapter + helpers protected a dozen lines of predicate logic per repo. Rejected in + favor of aligning the runners so the test files themselves are shared. +- **Standardizing on Ginkgo as the contract.** Ginkgo runs under + `go test`, so it is allowed, but mandating it would exclude the pytest + repos and couple the contract to a framework for no gain. The contract + standardizes selection and inputs, not the test framework. +- **A `values: {ats: ..., e2e: ...}` map in the shared config.** Rejected: + it bakes harness names into the neutral file, the configuration + equivalent of a test gating on the harness's name. From 6abd9de3fa4b7495dac807c1687a7e75cbd3db69 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 13:38:47 +0200 Subject: [PATCH 02/24] Model upgrade stage as env var, not test types Types answer what kind of test; lifecycle position is exposed as ATS_UPGRADE_STAGE=pre|post, consistent with how tests gate on all other environment properties. Keeps the type set identical to the published ATS contract. --- app-testing-contract/README.md | 39 ++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 07957df..11719d6 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -53,14 +53,29 @@ carry several types. |---|---| | `smoke` | fast, fail-fast sanity checks, run first | | `functional` | full feature tests | -| `pre_upgrade` | runs before the app is upgraded (for example: seed a workload) | -| `post_upgrade` | runs after the app is upgraded (for example: verify the workload survived) | +| `upgrade` | runs twice during an upgrade flow: before and after the upgrade | + +Types answer "what kind of test"; lifecycle position is not a type. The +upgrade flow runs `upgrade`-typed tests twice and tells them where they +are via `ATS_UPGRADE_STAGE` (`pre` or `post`). The canonical asymmetric +upgrade test (seed a workload before, verify it survived after) branches +or skips on the stage: + +```go +if os.Getenv("ATS_UPGRADE_STAGE") != "post" { + t.Skip("verification runs after the upgrade") +} +``` + +This keeps the type set identical to ATS's published contract (no +migration for existing tests) and matches the principle used everywhere +else in this contract: tests gate on environment properties. Pre-only +tests appear as named skips in the post run and vice versa, which is +accepted for the simpler taxonomy. -`upgrade` remains a deprecated alias meaning both `pre_upgrade` and -`post_upgrade`, so existing ATS tests keep working during migration. -Distinct pre/post types replace the previous single `upgrade` type because -the canonical upgrade test is asymmetric, which one type run twice cannot -express. +Runner hooks (ATS's pre/post-hook executables, atf's `BeforeUpgrade` +callback) remain harness-side machinery for setup and teardown around the +flow; they are not part of this contract. Assertions live in tests. ### Inputs @@ -77,6 +92,7 @@ no cluster creation, no chart install, no App CRs. | `ATS_CLUSTER_TYPE` | yes | `kind`, `external`, or `capi` | | `ATS_CLUSTER_VERSION` | optional | Kubernetes server version | | `ATS_APP_CONFIG_FILE_PATH` | optional | values file the app was deployed with | +| `ATS_UPGRADE_STAGE` | upgrade runs only | `pre` or `post`: which side of the upgrade this run is on | | `ATS_UPGRADE_FROM_VERSION` / `ATS_UPGRADE_TO_VERSION` | upgrade runs only | versions on either side of the upgrade | | `ATS_EXTRA_*` | optional | harness extras, for example `ATS_EXTRA_GITOPS_ENGINE` | @@ -92,8 +108,9 @@ Before invoking the executor, a conforming runner guarantees: GitOps engine; atf: App CR reconciled to `deployed`), 2. all required variables above are exported, 3. the executor is invoked once per applicable test type, in order: - `smoke`, then `functional`; for upgrade flows: `pre_upgrade`, then the - upgrade is performed, then `post_upgrade`. + `smoke`, then `functional`; for upgrade flows: `upgrade` with + `ATS_UPGRADE_STAGE=pre`, then the upgrade is performed, then `upgrade` + with `ATS_UPGRADE_STAGE=post`. "No tests for this type" is a pass, not a failure (Go: build constraints exclude all files; pytest: exit code 5). Test results are emitted as junit @@ -143,8 +160,8 @@ belongs in category 3. Enabling facts: Ginkgo runs under plain `go test`, and pytest tests built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS tests of both languages are immediately reusable on workload clusters. -- **app-test-suite** implements the `pre_upgrade` / `post_upgrade` types - in its upgrade scenario (keeping `upgrade` as the alias), searches +- **app-test-suite** exports `ATS_UPGRADE_STAGE` to test processes in its + upgrade scenario (it already sets the equivalent for hooks), searches `tests/app/` in addition to its current `tests/ats/` default, reads the shared config keys, and emits junit via gotestsum. Its TEST_CONTRACT.md becomes a pointer to this RFC plus ATS-specific detail. From 97f89839491ecc8b8ec5d5af634159e05d135cbb Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 13:44:04 +0200 Subject: [PATCH 03/24] Add portable hooks to the contract Setup and teardown are app-specific like assertions and duplicate across harnesses the same way. Conventional executables get the same env as tests; the portable boundary stays the app cluster's KUBECONFIG, harness-native hooks remain for MC-side work. --- app-testing-contract/README.md | 49 +++++++++++++++++++++++++++------- 1 file changed, 39 insertions(+), 10 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 11719d6..64f470b 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -73,9 +73,33 @@ else in this contract: tests gate on environment properties. Pre-only tests appear as named skips in the post run and vice versa, which is accepted for the simpler taxonomy. -Runner hooks (ATS's pre/post-hook executables, atf's `BeforeUpgrade` -callback) remain harness-side machinery for setup and teardown around the -flow; they are not part of this contract. Assertions live in tests. +Assertions live in tests; setup and teardown live in hooks (next +section). + +### Hooks + +Setup and teardown are app-specific just like assertions, and duplicate +across harnesses the same way. The contract therefore defines portable +hooks: optional executables in the conventional directory, invoked by the +runner with the same environment as tests, plus `ATS_HOOK_STAGE` naming +the point. + +| Hook | Runs | +|---|---| +| `tests/app/hooks/setup` | after the cluster is ready, before the app is deployed (for example: install prerequisites) | +| `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | + +A missing hook is a no-op. A non-zero exit fails the run. Hooks gate on +environment properties exactly like category-2 tests (`ATS_CLUSTER_TYPE` +and friends); during upgrade flows they additionally receive +`ATS_UPGRADE_STAGE` and the from/to versions. + +The boundary is the same as for tests: a portable hook only gets the app +cluster's `KUBECONFIG`. Work that needs the harness's own machinery (MC +access, App CR manipulation, framework state) stays in harness-native +hooks: ATS's config-wired hook executables and atf's suite callbacks +(`AfterClusterReady`, `BeforeUpgrade`), which remain available and are +not part of this contract. ### Inputs @@ -104,13 +128,17 @@ cosmetic gain. Read it as "app-testing", not as the harness's name. Before invoking the executor, a conforming runner guarantees: -1. the app is deployed and settled (ATS: chart installed via Helm or a +1. the `setup` hook, if present, ran after the cluster was ready and + before the app was deployed, +2. the app is deployed and settled (ATS: chart installed via Helm or a GitOps engine; atf: App CR reconciled to `deployed`), -2. all required variables above are exported, -3. the executor is invoked once per applicable test type, in order: +3. all required variables above are exported, +4. the executor is invoked once per applicable test type, in order: `smoke`, then `functional`; for upgrade flows: `upgrade` with `ATS_UPGRADE_STAGE=pre`, then the upgrade is performed, then `upgrade` - with `ATS_UPGRADE_STAGE=post`. + with `ATS_UPGRADE_STAGE=post`, +5. the `teardown` hook, if present, runs after the last test type, before + the harness's own teardown. "No tests for this type" is a pass, not a failure (Go: build constraints exclude all files; pytest: exit code 5). Test results are emitted as junit @@ -161,9 +189,10 @@ belongs in category 3. built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS tests of both languages are immediately reusable on workload clusters. - **app-test-suite** exports `ATS_UPGRADE_STAGE` to test processes in its - upgrade scenario (it already sets the equivalent for hooks), searches - `tests/app/` in addition to its current `tests/ats/` default, reads the - shared config keys, and emits junit via gotestsum. Its + upgrade scenario (it already sets the equivalent for hooks), discovers + the conventional hooks by path in addition to its config-wired ones, + searches `tests/app/` in addition to its current `tests/ats/` default, + reads the shared config keys, and emits junit via gotestsum. Its TEST_CONTRACT.md becomes a pointer to this RFC plus ATS-specific detail. - **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so non-portable suites share the same readiness vocabulary. From cbc3bf3c879fea03627027ad651f027d1e255940 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 13:49:04 +0200 Subject: [PATCH 04/24] Rename env prefix to APP_TEST_, fix two variable names Canonical prefix is harness-neutral; runners dual-export the legacy ATS_ names so nothing breaks. ATS_APP_CONFIG_FILE_PATH becomes APP_TEST_VALUES_FILE and ATS_CLUSTER_VERSION becomes APP_TEST_KUBERNETES_VERSION. --- app-testing-contract/README.md | 61 ++++++++++++++++++++-------------- 1 file changed, 36 insertions(+), 25 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 64f470b..e33c7fb 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -6,7 +6,7 @@ owners: - https://github.com/orgs/giantswarm/teams/team-honeybadger - https://github.com/orgs/giantswarm/teams/team-tenet state: review -summary: Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (chart tests on kind) and apptest-framework (e2e on workload clusters). One conventional directory per repo, test types via build tags or pytest markers, inputs via KUBECONFIG and ATS_* env vars. +summary: Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (chart tests on kind) and apptest-framework (e2e on workload clusters). One conventional directory per repo, test types via build tags or pytest markers, inputs via KUBECONFIG and APP_TEST_* env vars. --- # The app-testing contract @@ -57,12 +57,12 @@ carry several types. Types answer "what kind of test"; lifecycle position is not a type. The upgrade flow runs `upgrade`-typed tests twice and tells them where they -are via `ATS_UPGRADE_STAGE` (`pre` or `post`). The canonical asymmetric +are via `APP_TEST_UPGRADE_STAGE` (`pre` or `post`). The canonical asymmetric upgrade test (seed a workload before, verify it survived after) branches or skips on the stage: ```go -if os.Getenv("ATS_UPGRADE_STAGE") != "post" { +if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { t.Skip("verification runs after the upgrade") } ``` @@ -81,7 +81,7 @@ section). Setup and teardown are app-specific just like assertions, and duplicate across harnesses the same way. The contract therefore defines portable hooks: optional executables in the conventional directory, invoked by the -runner with the same environment as tests, plus `ATS_HOOK_STAGE` naming +runner with the same environment as tests, plus `APP_TEST_HOOK_STAGE` naming the point. | Hook | Runs | @@ -90,9 +90,9 @@ the point. | `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | A missing hook is a no-op. A non-zero exit fails the run. Hooks gate on -environment properties exactly like category-2 tests (`ATS_CLUSTER_TYPE` +environment properties exactly like category-2 tests (`APP_TEST_CLUSTER_TYPE` and friends); during upgrade flows they additionally receive -`ATS_UPGRADE_STAGE` and the from/to versions. +`APP_TEST_UPGRADE_STAGE` and the from/to versions. The boundary is the same as for tests: a portable hook only gets the app cluster's `KUBECONFIG`. Work that needs the harness's own machinery (MC @@ -109,20 +109,30 @@ no cluster creation, no chart install, no App CRs. | Variable | Required | Meaning | |---|---|---| | `KUBECONFIG` | yes | kubeconfig of the cluster the app is deployed on (never the MC) | -| `ATS_TEST_TYPE` | yes | the type currently being run | -| `ATS_RELEASE_NAME` | yes | Helm release name of the app under test | -| `ATS_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | -| `ATS_CHART_VERSION` | yes | version of the chart under test | -| `ATS_CLUSTER_TYPE` | yes | `kind`, `external`, or `capi` | -| `ATS_CLUSTER_VERSION` | optional | Kubernetes server version | -| `ATS_APP_CONFIG_FILE_PATH` | optional | values file the app was deployed with | -| `ATS_UPGRADE_STAGE` | upgrade runs only | `pre` or `post`: which side of the upgrade this run is on | -| `ATS_UPGRADE_FROM_VERSION` / `ATS_UPGRADE_TO_VERSION` | upgrade runs only | versions on either side of the upgrade | -| `ATS_EXTRA_*` | optional | harness extras, for example `ATS_EXTRA_GITOPS_ENGINE` | - -The `ATS_` prefix is kept as-is: it is the published contract of the -existing implementation, and renaming would break every current test for -cosmetic gain. Read it as "app-testing", not as the harness's name. +| `APP_TEST_TEST_TYPE` | yes | the type currently being run | +| `APP_TEST_RELEASE_NAME` | yes | Helm release name of the app under test | +| `APP_TEST_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | +| `APP_TEST_CHART_VERSION` | yes | version of the chart under test | +| `APP_TEST_CLUSTER_TYPE` | yes | `kind`, `external`, or `capi` | +| `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | +| `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | +| `APP_TEST_UPGRADE_STAGE` | upgrade runs only | `pre` or `post`: which side of the upgrade this run is on | +| `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade runs only | versions on either side of the upgrade | +| `APP_TEST_EXTRA_*` | optional | harness extras, for example `APP_TEST_EXTRA_GITOPS_ENGINE` | + +The canonical prefix is `APP_TEST_`, neutral to both harnesses. The +existing implementation publishes these variables under the legacy `ATS_` +prefix; conforming runners export both, so no existing test breaks and +dual export costs nothing ongoing. New and scaffolded tests use +`APP_TEST_`. The mapping is mechanical (`ATS_X` becomes `APP_TEST_X`) +with two exceptions renamed for clarity: + +| Legacy | Canonical | +|---|---| +| `ATS_APP_CONFIG_FILE_PATH` | `APP_TEST_VALUES_FILE` | +| `ATS_CLUSTER_VERSION` | `APP_TEST_KUBERNETES_VERSION` | + +`KUBECONFIG` is unchanged: it is the Kubernetes-wide convention, not ours. ### Runner guarantees @@ -135,8 +145,8 @@ Before invoking the executor, a conforming runner guarantees: 3. all required variables above are exported, 4. the executor is invoked once per applicable test type, in order: `smoke`, then `functional`; for upgrade flows: `upgrade` with - `ATS_UPGRADE_STAGE=pre`, then the upgrade is performed, then `upgrade` - with `ATS_UPGRADE_STAGE=post`, + `APP_TEST_UPGRADE_STAGE=pre`, then the upgrade is performed, then `upgrade` + with `APP_TEST_UPGRADE_STAGE=post`, 5. the `teardown` hook, if present, runs after the last test type, before the harness's own teardown. @@ -168,7 +178,7 @@ The contract covers the default case, not everything. Where a test goes: no gate. This should be the bulk. 2. Asserts on the deployed app but is only meaningful in one environment: `tests/app/` plus a runtime skip on contract environment, for example - `if os.Getenv("ATS_CLUSTER_TYPE") != "kind" { t.Skip(...) }`. Skips + `if os.Getenv("APP_TEST_CLUSTER_TYPE") != "kind" { t.Skip(...) }`. Skips stay visible by name in both runners' output. 3. Needs harness machinery (MC access, bundle installs, AWS/IRSA, cluster manipulation): a regular in-process apptest-framework suite under @@ -188,8 +198,9 @@ belongs in category 3. Enabling facts: Ginkgo runs under plain `go test`, and pytest tests built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS tests of both languages are immediately reusable on workload clusters. -- **app-test-suite** exports `ATS_UPGRADE_STAGE` to test processes in its - upgrade scenario (it already sets the equivalent for hooks), discovers +- **app-test-suite** exports the canonical `APP_TEST_*` names alongside + its legacy `ATS_*` ones, adds the upgrade stage variable for test + processes (hooks already get the equivalent), discovers the conventional hooks by path in addition to its config-wired ones, searches `tests/app/` in addition to its current `tests/ats/` default, reads the shared config keys, and emits junit via gotestsum. Its From 24a56e2b693e10a07845bd5d57af3b5e355209d8 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 13:49:50 +0200 Subject: [PATCH 05/24] Rename APP_TEST_TEST_TYPE to APP_TEST_TYPE --- app-testing-contract/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index e33c7fb..155dfd8 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -109,7 +109,7 @@ no cluster creation, no chart install, no App CRs. | Variable | Required | Meaning | |---|---|---| | `KUBECONFIG` | yes | kubeconfig of the cluster the app is deployed on (never the MC) | -| `APP_TEST_TEST_TYPE` | yes | the type currently being run | +| `APP_TEST_TYPE` | yes | the type currently being run | | `APP_TEST_RELEASE_NAME` | yes | Helm release name of the app under test | | `APP_TEST_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | | `APP_TEST_CHART_VERSION` | yes | version of the chart under test | @@ -125,10 +125,11 @@ existing implementation publishes these variables under the legacy `ATS_` prefix; conforming runners export both, so no existing test breaks and dual export costs nothing ongoing. New and scaffolded tests use `APP_TEST_`. The mapping is mechanical (`ATS_X` becomes `APP_TEST_X`) -with two exceptions renamed for clarity: +with three exceptions renamed for clarity: | Legacy | Canonical | |---|---| +| `ATS_TEST_TYPE` | `APP_TEST_TYPE` | | `ATS_APP_CONFIG_FILE_PATH` | `APP_TEST_VALUES_FILE` | | `ATS_CLUSTER_VERSION` | `APP_TEST_KUBERNETES_VERSION` | From df64747314d0763ba260c30e01cfba54dbf02b64 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 13:55:20 +0200 Subject: [PATCH 06/24] Clarify the upgrade config key It drives the harnesses' existing upgrade primitives and doubles as a lint against typo'd upgrade tags. --- app-testing-contract/README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 155dfd8..c3d6241 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -164,6 +164,14 @@ installNamespace: kube-system upgrade: true # whether an upgrade flow applies to this app ``` +`upgrade` is the declarative form of each harness's existing upgrade +primitive: atf's `WithIsUpgrade(true)` (install the latest release, +upgrade to the version under test) and ATS's upgrade scenario. It is +explicit rather than inferred from the presence of `upgrade`-typed tests +because the upgrade flow is the expensive one, and the combination is a +lint: `upgrade: true` with zero `upgrade`-typed tests collected fails the +run, catching typo'd tags and markers instead of silently passing. + Everything harness-specific stays in the harness's own config: `.ats/main.yaml` (cluster types, catalogs, executor options) and `tests/e2e/config.yaml` (appCatalog, providers, MC test options). Values From 98b6622f0f6436fe121b01bf839cdcdccb577633 Mon Sep 17 00:00:00 2001 From: Quentin Bisson Date: Tue, 7 Jul 2026 14:07:09 +0200 Subject: [PATCH 07/24] Resolve review gaps: taxonomy, lints, conformance, supply chain - split test types into orthogonal depth (smoke/functional) and flow (upgrade) axes; define how a multi-tagged test executes - require committed lockfiles and frozen/offline installs - define cluster types by capability and warn against using them as a harness proxy; define external and the neither/empty-directory cases - specify hooks as out-of-process executables, exempt from one-language - add converse upgrade lint and optional expectedTypes collection check - add contractVersion, a conformance suite, and a named steward - note the two-Go-module layout and go.work --- app-testing-contract/README.md | 132 +++++++++++++++++++++++++++------ 1 file changed, 110 insertions(+), 22 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index c3d6241..cef3dbb 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -40,26 +40,49 @@ directory is the opt-in; there is no per-repo wiring. The directory is one Go module or one Python project, never both. The executor is detected from its contents: -- `go.mod` present: `go test -tags=` -- `pyproject.toml` present: `uv sync && uv run pytest -m ` -- both present: configuration error, fail fast +- `go.mod` present: `go test -mod=readonly -tags=` +- `pyproject.toml` present: `uv sync --frozen && uv run pytest -m ` +- both present, or neither present in a non-empty directory: + configuration error, fail fast + +Dependencies are pinned and installed offline: Go reads a committed +`go.sum` under `-mod=readonly`, Python a committed `uv.lock` under +`--frozen`. A runner never resolves versions from the network at test +time; a missing or stale lockfile is a failure, not a silent fetch. This +keeps the dependency closure identical across harnesses and auditable. + +An empty `tests/app/` (no module, no project) is not an opt-in and is +ignored. ### Test types -A test declares its types via Go build tags or pytest markers. A test may -carry several types. +A test declares its types via Go build tags or pytest markers. Types live +on two independent axes; a test carries at most one from each. + +Depth (what kind of check), selected in every flow: | Type | Meaning | |---|---| | `smoke` | fast, fail-fast sanity checks, run first | | `functional` | full feature tests | -| `upgrade` | runs twice during an upgrade flow: before and after the upgrade | -Types answer "what kind of test"; lifecycle position is not a type. The -upgrade flow runs `upgrade`-typed tests twice and tells them where they -are via `APP_TEST_UPGRADE_STAGE` (`pre` or `post`). The canonical asymmetric -upgrade test (seed a workload before, verify it survived after) branches -or skips on the stage: +Flow (which lifecycle the test participates in): + +| Type | Meaning | +|---|---| +| `upgrade` | also run during the upgrade flow, once before and once after the upgrade | + +The axes compose. `upgrade` selects tests *into* the upgrade flow; it does +not replace their depth. A test tagged `smoke, upgrade` is a smoke check +that also runs on both sides of an upgrade; a test with no `upgrade` tag +never runs in the upgrade flow. The runner selects by the type of the +current pass and never executes the same test twice within one pass, so +a multi-tagged test runs once per pass it matches (in the normal flow, and +each upgrade stage if tagged `upgrade`), never redundantly. + +The upgrade flow tells a test where it is via `APP_TEST_UPGRADE_STAGE` +(`pre` or `post`). The canonical asymmetric upgrade test (seed a workload +before, verify it survived after) branches or skips on the stage: ```go if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { @@ -67,11 +90,11 @@ if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { } ``` -This keeps the type set identical to ATS's published contract (no +This keeps the depth set identical to ATS's published contract (no migration for existing tests) and matches the principle used everywhere -else in this contract: tests gate on environment properties. Pre-only -tests appear as named skips in the post run and vice versa, which is -accepted for the simpler taxonomy. +else in this contract: tests gate on environment properties, not on a +lifecycle-specific type name. Pre-only tests appear as named skips in the +post run and vice versa, which is accepted for the simpler taxonomy. Assertions live in tests; setup and teardown live in hooks (next section). @@ -89,6 +112,13 @@ the point. | `tests/app/hooks/setup` | after the cluster is ready, before the app is deployed (for example: install prerequisites) | | `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | +A hook is any executable file at that path: a script with a shebang or a +built binary, invoked directly (not sourced, not run through a language +toolchain). Because it runs out of process, it is exempt from the +directory's one-language rule and may be written in whatever suits it; +keep it thin, since anything substantial belongs in a test or a +harness-native hook. + A missing hook is a no-op. A non-zero exit fails the run. Hooks gate on environment properties exactly like category-2 tests (`APP_TEST_CLUSTER_TYPE` and friends); during upgrade flows they additionally receive @@ -113,7 +143,7 @@ no cluster creation, no chart install, no App CRs. | `APP_TEST_RELEASE_NAME` | yes | Helm release name of the app under test | | `APP_TEST_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | | `APP_TEST_CHART_VERSION` | yes | version of the chart under test | -| `APP_TEST_CLUSTER_TYPE` | yes | `kind`, `external`, or `capi` | +| `APP_TEST_CLUSTER_TYPE` | yes | cluster the app runs on: `kind` (local single-node, no cloud), `capi` (a CAPI workload cluster with cloud identity), or `external` (a pre-existing cluster the runner did not provision) | | `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | | `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | | `APP_TEST_UPGRADE_STAGE` | upgrade runs only | `pre` or `post`: which side of the upgrade this run is on | @@ -152,25 +182,41 @@ Before invoking the executor, a conforming runner guarantees: the harness's own teardown. "No tests for this type" is a pass, not a failure (Go: build constraints -exclude all files; pytest: exit code 5). Test results are emitted as junit -XML: `gotestsum --junitfile` for Go, `pytest --junitxml` for Python. +exclude all files; pytest: exit code 5), because a repo may legitimately +carry only some types. Zero collection is never silent, though: the runner +records the collected count per type, so a typo'd tag or marker (which also +collects zero) is visible in the output rather than a green run. Repos that +want it enforced list the types they expect in `config.yaml` (see below); +a listed type collecting zero fails the run. Test results are emitted as +junit XML: `gotestsum --junitfile` for Go, `pytest --junitxml` for Python. ### Shared configuration `tests/app/config.yaml` carries only the keys both harnesses need: ```yaml +contractVersion: 1 # contract version this directory targets installNamespace: kube-system -upgrade: true # whether an upgrade flow applies to this app +upgrade: true # whether an upgrade flow applies to this app +expectedTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test ``` `upgrade` is the declarative form of each harness's existing upgrade primitive: atf's `WithIsUpgrade(true)` (install the latest release, upgrade to the version under test) and ATS's upgrade scenario. It is explicit rather than inferred from the presence of `upgrade`-typed tests -because the upgrade flow is the expensive one, and the combination is a -lint: `upgrade: true` with zero `upgrade`-typed tests collected fails the -run, catching typo'd tags and markers instead of silently passing. +because the upgrade flow is the expensive one. + +The two settings lint against each other and against what is collected, so +a mistake fails the run instead of passing green: + +- `upgrade: true` with zero `upgrade`-typed tests collected fails (dead + flow, or a typo'd tag). +- `upgrade`-typed tests present with `upgrade` unset or false fails (tests + that would never run). +- any type in `expectedTypes` collecting zero fails; `expectedTypes` is + optional, and omitting it keeps the permissive "no tests is a pass" + default for repos that do not want the check. Everything harness-specific stays in the harness's own config: `.ats/main.yaml` (cluster types, catalogs, executor options) and @@ -193,10 +239,49 @@ The contract covers the default case, not everything. Where a test goes: manipulation): a regular in-process apptest-framework suite under `tests/e2e/suites/`, unchanged by this RFC. +A repo may therefore hold two Go modules, `tests/app/` (portable) and +`tests/e2e/` (atf-native), each with its own `go.mod`. They stay separate +modules on purpose: the portable one must build without the atf +dependency tree. Repos that want unified tooling across them add a +`go.work` at the repo root; it is not required and is never committed as a +contract artifact. + Tests may gate on environment properties the contract exposes, never on which harness is running them. A test that needs the harness's name belongs in category 3. +`APP_TEST_CLUSTER_TYPE` is a capability axis, not a harness label. That +`kind` tends to mean ATS and `capi` tends to mean atf today is +incidental, and a gate written as "am I really asking about the harness?" +is a category error even when it happens to work. Gate on the property you +actually depend on: if a test needs cloud identity, express that (and let +`external` clusters that also provide it pass the same gate) rather than +hard-coding `!= "kind"`. If the property you need is not on any contract +variable, the test needs harness machinery and belongs in category 3. + +### Conformance, versioning, and ownership + +Two independent runners implement one contract, so drift is the default +failure mode unless something mechanically checks them. The contract ships +with a conformance suite: a fixture `tests/app/` (a trivial app, one test +of each type, one hook, a lockfile) plus a set of assertions on the +env-var, ordering, exit-code, and lint guarantees above. A runner is +conforming only if it passes the suite in its CI; both ATS and atf wire it +in. New guarantees land in the suite in the same change that adds them +here. + +The contract is versioned. This document is `v1`; the version is declared +in `tests/app/config.yaml` as `contractVersion: 1`. A runner refuses a +directory whose declared version it does not implement rather than +guessing. Breaking changes bump the integer and the conformance suite +carries a fixture per supported version. + +team-tenet stewards the contract (owns this document and the conformance +suite, arbitrates when the two runners disagree). team-honeybadger and +team-bumblebee own the ATS and atf implementations respectively. A change +to the contract is a PR here that updates the suite; a runner falling +behind is a bug against that runner, not a licence to fork the contract. + ## Implementation - **apptest-framework** gains a convention-runner: after provisioning the @@ -216,6 +301,9 @@ belongs in category 3. TEST_CONTRACT.md becomes a pointer to this RFC plus ATS-specific detail. - **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so non-portable suites share the same readiness vocabulary. +- **the conformance suite** lives in this repo alongside the RFC: the + fixture app plus the guarantee assertions. Both runners run it in CI; + it is the acceptance gate for "implements the contract." - **devctl `gen apptest` and template-app** scaffold the conventional layout for new repos. - Migration is opt-in as repos get touched; there is no flag day. From 47369d50885f63c6e35abe8d74f6dc7899148379 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 14:21:06 +0200 Subject: [PATCH 08/24] Reframe motivation as cadence split; add cross-runner parity and feedback-latency handling Problem section now leads with the deliberate fast-CI-on-kind vs nightly-e2e-on-WC split and the two-idiom authoring cost that suppressed adoption, rather than test duplication. Adds a cross-runner parity assertion to the conformance suite (settled semantics: Helm-installed vs App-CR-deployed), a Cadence and feedback latency section with an on-demand /run escape hatch for nightly-only cloud paths, and a single-runner alternative. --- app-testing-contract/README.md | 98 ++++++++++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 11 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index cef3dbb..386a0da 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -13,13 +13,30 @@ summary: Defines a harness-neutral contract for app tests so the same test files ## Problem -We test managed apps with two harnesses. app-test-suite (ATS) deploys the -chart on a kind cluster in the PR pipeline. apptest-framework (atf) stands -up a workload cluster on a management cluster and installs an App CR in the -e2e pipeline. Both need "does the deployed app actually work" checks, and -today each repo writes them twice, in two idioms (pytest or plain Go for -ATS, Ginkgo suites for atf). In practice one side is usually empty or -stale. +We run "does the deployed app actually work" checks at two cadences, on +purpose: + +- **app-test-suite (ATS)** deploys the chart on a kind cluster and runs + fast checks per pull request. +- **apptest-framework (atf)** stands up a workload cluster on a management + cluster, installs an App CR, and runs the full suite nightly. + +Both cadences earn their keep and we are keeping both: kind gives quick +per-PR signal, a real workload cluster catches what kind cannot (cloud +identity, real storage, upgrades). The split is not the problem. The +problem is that the two harnesses have divergent authoring models: ATS +expects pytest or plain Go, atf expects Ginkgo suites. Covering both +cadences means writing the same check twice in two idioms, so in practice +a repo writes it for one harness, or for neither. Adoption of both is +close to zero, and teams that did adopt a single harness are often +unhappy living in two worlds. + +So this RFC is not a dedup exercise; there is little duplication to +remove, because the double-idiom cost suppressed writing the tests in the +first place. It unifies the *authoring* model: one directory, one idiom +per repo, discovered and run the same way by both harnesses. A test +written once runs per-PR on kind (the subset kind can support) and nightly +on a workload cluster (everything), with no second copy to keep in sync. ATS already publishes a testing contract ([docs/TEST_CONTRACT.md](https://github.com/giantswarm/app-test-suite/blob/master/docs/TEST_CONTRACT.md)), @@ -190,6 +207,35 @@ want it enforced list the types they expect in `config.yaml` (see below); a listed type collecting zero fails the run. Test results are emitted as junit XML: `gotestsum --junitfile` for Go, `pytest --junitxml` for Python. +### Cadence and feedback latency + +The two cadences buy quick per-PR signal at the cost of a gap: the fast +runner on kind cannot exercise what kind lacks (cloud identity, real +storage, load balancers), so a test gated to those environments runs only +in the nightly workload-cluster flow. Its signal is then detached from the +change that broke it. A pull request that breaks a cloud-only path passes +per-PR CI green and fails nightly, hours later, against a batch of +unrelated commits. + +This is an accepted property of the split, not a defect the contract +introduces, but the contract must not let it be silent or unescapable: + +1. **A nightly-only test is a conscious choice, never an accident.** A + test that gates itself off kind (category 2) is by construction + per-PR-invisible. Reviewers see that in the diff; the collected-count + output makes "ran nowhere per-PR" legible rather than looking like + coverage. +2. **There is an on-demand full-flow trigger.** A change that touches a + cloud-only path can request the workload-cluster flow against the pull + request instead of waiting for the scheduled run, via the existing + `/run` pipeline convention. Catching a cloud regression a day late is + the default; paying for it on the PR is one comment away. + +Prefer expressing a real capability need over a hard environment gate (see +`APP_TEST_CLUSTER_TYPE` below): a test that only needs cloud identity, not +kind-vs-WC specifically, will also run per-PR on any `external` cluster +that provides it, which shrinks the nightly-only set. + ### Shared configuration `tests/app/config.yaml` carries only the keys both harnesses need: @@ -270,6 +316,21 @@ conforming only if it passes the suite in its CI; both ATS and atf wire it in. New guarantees land in the suite in the same change that adds them here. +Per-runner conformance is necessary but not sufficient: two runners can +each satisfy the letter of the contract and still disagree on what a test +observes, which is the failure that actually bites (a smoke test that +passes fast-CI and flakes nightly). The suite therefore also asserts +*parity*: the same fixture run through both runners must yield the same +collected-per-type counts and the same pass/fail outcome, and any +divergence fails the suite. The known sharp edge is guarantee 2, "the app +is settled": ATS reaches it via a Helm release reporting installed, atf +via an App CR reconciled to `deployed`, and those are not the same instant. +The contract fixes the observable, not the mechanism: settled means the +app's own readiness (its Deployments/StatefulSets Available) holds, and the +parity fixture asserts a runner does not hand off to tests before it does. +`clustertest.wait.IsDeploymentReady` is the shared vocabulary for that +check so both runners and the tests mean the same thing by "ready". + The contract is versioned. This document is `v1`; the version is declared in `tests/app/config.yaml` as `contractVersion: 1`. A runner refuses a directory whose declared version it does not implement rather than @@ -299,11 +360,17 @@ behind is a bug against that runner, not a licence to fork the contract. searches `tests/app/` in addition to its current `tests/ats/` default, reads the shared config keys, and emits junit via gotestsum. Its TEST_CONTRACT.md becomes a pointer to this RFC plus ATS-specific detail. -- **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so - non-portable suites share the same readiness vocabulary. +- **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so both + runners and non-portable suites share the same readiness vocabulary, and + the settled-parity assertion has one definition to check against. +- **the on-demand full-flow trigger**: the workload-cluster pipeline runs + on the `/run` convention against a pull request, not only on the nightly + schedule, so a change touching a cloud-only path can pull its signal + forward without waiting for the batch. - **the conformance suite** lives in this repo alongside the RFC: the - fixture app plus the guarantee assertions. Both runners run it in CI; - it is the acceptance gate for "implements the contract." + fixture app plus the guarantee assertions, including the cross-runner + parity check. Both runners run it in CI; it is the acceptance gate for + "implements the contract." - **devctl `gen apptest` and template-app** scaffold the conventional layout for new repos. - Migration is opt-in as repos get touched; there is no flag day. @@ -322,3 +389,12 @@ behind is a bug against that runner, not a licence to fork the contract. - **A `values: {ats: ..., e2e: ...}` map in the shared config.** Rejected: it bakes harness names into the neutral file, the configuration equivalent of a test gating on the harness's name. +- **One runner with two modes instead of two runners behind a contract.** + A single runner covering both fast-kind and workload-cluster modes would + need no contract to police, since there would be nothing to keep in step. + Rejected because the two cadences map onto two mature codebases owned by + two teams (ATS by team-honeybadger, atf by team-tenet), each carrying + provisioning and pipeline machinery the other does not want. Collapsing + them is a larger, riskier rewrite than aligning their edges, and the + parity check gives most of the anti-drift benefit at a fraction of the + cost. If the two runners keep diverging in practice, revisit this. From e6190169d2e2521133325546c1725e2d50e1db5a Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 14:29:38 +0200 Subject: [PATCH 09/24] Correct upgrade model: upgrade is a peer test type run once post-upgrade The two-axes/compose taxonomy did not match either harness. ATS's UpgradeTestScenario runs only the upgrade StepType; atf's upgrade suite runs its test function once after upgrading. Neither re-runs smoke and functional around the upgrade. Makes upgrade a third peer type (smoke, functional, upgrade) that runs once after the upgrade, moves pre-upgrade state seeding into an imperative pre-upgrade hook, drops APP_TEST_UPGRADE_STAGE and the pre/post double run. Hooks section now states hooks do side-effecting work and tests assert, which is what lets the upgrade flow seed without re-running a suite. --- app-testing-contract/README.md | 115 ++++++++++++++++----------------- 1 file changed, 54 insertions(+), 61 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 386a0da..354e71c 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -73,60 +73,47 @@ ignored. ### Test types -A test declares its types via Go build tags or pytest markers. Types live -on two independent axes; a test carries at most one from each. +A test declares its type via a Go build tag or pytest marker. There are +three peer types, matching ATS's existing `StepType`s; a test carries one. -Depth (what kind of check), selected in every flow: - -| Type | Meaning | -|---|---| -| `smoke` | fast, fail-fast sanity checks, run first | -| `functional` | full feature tests | - -Flow (which lifecycle the test participates in): - -| Type | Meaning | -|---|---| -| `upgrade` | also run during the upgrade flow, once before and once after the upgrade | - -The axes compose. `upgrade` selects tests *into* the upgrade flow; it does -not replace their depth. A test tagged `smoke, upgrade` is a smoke check -that also runs on both sides of an upgrade; a test with no `upgrade` tag -never runs in the upgrade flow. The runner selects by the type of the -current pass and never executes the same test twice within one pass, so -a multi-tagged test runs once per pass it matches (in the normal flow, and -each upgrade stage if tagged `upgrade`), never redundantly. - -The upgrade flow tells a test where it is via `APP_TEST_UPGRADE_STAGE` -(`pre` or `post`). The canonical asymmetric upgrade test (seed a workload -before, verify it survived after) branches or skips on the stage: - -```go -if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { - t.Skip("verification runs after the upgrade") -} -``` - -This keeps the depth set identical to ATS's published contract (no -migration for existing tests) and matches the principle used everywhere -else in this contract: tests gate on environment properties, not on a -lifecycle-specific type name. Pre-only tests appear as named skips in the -post run and vice versa, which is accepted for the simpler taxonomy. - -Assertions live in tests; setup and teardown live in hooks (next -section). +| Type | Runs | Meaning | +|---|---|---| +| `smoke` | normal flow, first | fast, fail-fast sanity checks | +| `functional` | normal flow, after smoke | full feature tests | +| `upgrade` | upgrade flow, once after the upgrade | verifies the app still works, and seeded state survived, across an upgrade | + +`upgrade` is a peer type, not a modifier on the others. The upgrade flow +does not re-run the `smoke` and `functional` suites before and after the +upgrade; it runs only the `upgrade`-typed tests, and it runs them once, +after the upgrade. This matches both harnesses: ATS's upgrade scenario +runs the `upgrade` type only, and atf's upgrade suite runs its test +function once after upgrading. + +The asymmetric case (state that must survive the upgrade) is split by +concern: an imperative **pre-upgrade hook** seeds the state on the old +version (create a pod, write a record), and an `upgrade`-typed test +verifies it after the upgrade. The seeding is a side effect, so it is a +hook, not a test that reruns; the verification is an assertion, so it is a +test. The `upgrade` test can read `APP_TEST_UPGRADE_FROM_VERSION` / +`APP_TEST_UPGRADE_TO_VERSION` if it needs the version pair. See Hooks below. + +Assertions live in tests; setup, teardown, and upgrade seeding live in +hooks (next section). ### Hooks -Setup and teardown are app-specific just like assertions, and duplicate -across harnesses the same way. The contract therefore defines portable -hooks: optional executables in the conventional directory, invoked by the -runner with the same environment as tests, plus `APP_TEST_HOOK_STAGE` naming -the point. +Hooks do imperative work with a side effect (install a prerequisite, seed +a pod, clean up an external resource); tests assert. Keeping the two +separate is what lets the upgrade flow seed state without re-running a test +suite. Setup, seeding, and teardown are app-specific just like assertions +and duplicate across harnesses the same way, so the contract defines +portable hooks: optional executables in the conventional directory, invoked +by the runner with the same environment as tests. | Hook | Runs | |---|---| | `tests/app/hooks/setup` | after the cluster is ready, before the app is deployed (for example: install prerequisites) | +| `tests/app/hooks/pre-upgrade` | upgrade flow only: after the previous version is deployed, before the upgrade (for example: create a pod or write a record whose survival an `upgrade` test then verifies) | | `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | A hook is any executable file at that path: a script with a shebang or a @@ -138,8 +125,8 @@ harness-native hook. A missing hook is a no-op. A non-zero exit fails the run. Hooks gate on environment properties exactly like category-2 tests (`APP_TEST_CLUSTER_TYPE` -and friends); during upgrade flows they additionally receive -`APP_TEST_UPGRADE_STAGE` and the from/to versions. +and friends); the `pre-upgrade` hook additionally receives +`APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. The boundary is the same as for tests: a portable hook only gets the app cluster's `KUBECONFIG`. Work that needs the harness's own machinery (MC @@ -163,8 +150,7 @@ no cluster creation, no chart install, no App CRs. | `APP_TEST_CLUSTER_TYPE` | yes | cluster the app runs on: `kind` (local single-node, no cloud), `capi` (a CAPI workload cluster with cloud identity), or `external` (a pre-existing cluster the runner did not provision) | | `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | | `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | -| `APP_TEST_UPGRADE_STAGE` | upgrade runs only | `pre` or `post`: which side of the upgrade this run is on | -| `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade runs only | versions on either side of the upgrade | +| `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade flow only | versions on either side of the upgrade; available to the `pre-upgrade` hook and `upgrade`-typed tests | | `APP_TEST_EXTRA_*` | optional | harness extras, for example `APP_TEST_EXTRA_GITOPS_ENGINE` | The canonical prefix is `APP_TEST_`, neutral to both harnesses. The @@ -184,20 +170,26 @@ with three exceptions renamed for clarity: ### Runner guarantees -Before invoking the executor, a conforming runner guarantees: +A conforming runner guarantees, before invoking the executor: 1. the `setup` hook, if present, ran after the cluster was ready and before the app was deployed, 2. the app is deployed and settled (ATS: chart installed via Helm or a GitOps engine; atf: App CR reconciled to `deployed`), 3. all required variables above are exported, -4. the executor is invoked once per applicable test type, in order: - `smoke`, then `functional`; for upgrade flows: `upgrade` with - `APP_TEST_UPGRADE_STAGE=pre`, then the upgrade is performed, then `upgrade` - with `APP_TEST_UPGRADE_STAGE=post`, -5. the `teardown` hook, if present, runs after the last test type, before +4. the `teardown` hook, if present, runs after the last test type, before the harness's own teardown. +In the **normal flow**, the executor is invoked once per applicable test +type, in order: `smoke`, then `functional`. `upgrade`-typed tests do not +run here. + +In the **upgrade flow** (`upgrade: true`), the runner instead: deploys the +previous version, runs the `pre-upgrade` hook if present, upgrades to the +version under test, waits for it to settle, then invokes the executor once +for the `upgrade` type. The `smoke` and `functional` suites are not +re-run; the upgrade tests run once, after the upgrade. + "No tests for this type" is a pass, not a failure (Go: build constraints exclude all files; pytest: exit code 5), because a repo may legitimately carry only some types. Zero collection is never silent, though: the runner @@ -354,11 +346,12 @@ behind is a bug against that runner, not a licence to fork the contract. built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS tests of both languages are immediately reusable on workload clusters. - **app-test-suite** exports the canonical `APP_TEST_*` names alongside - its legacy `ATS_*` ones, adds the upgrade stage variable for test - processes (hooks already get the equivalent), discovers - the conventional hooks by path in addition to its config-wired ones, - searches `tests/app/` in addition to its current `tests/ats/` default, - reads the shared config keys, and emits junit via gotestsum. Its + its legacy `ATS_*` ones, stops running the `upgrade` type in a pre-upgrade + pass and runs it once post-upgrade (its existing `pre_upgrade`/ + `post_upgrade` config hooks map onto the conventional `pre-upgrade` hook), + discovers the conventional hooks by path in addition to its config-wired + ones, searches `tests/app/` in addition to its current `tests/ats/` + default, reads the shared config keys, and emits junit via gotestsum. Its TEST_CONTRACT.md becomes a pointer to this RFC plus ATS-specific detail. - **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so both runners and non-portable suites share the same readiness vocabulary, and From 699ade2f5b5bc01f893c0550cc02bbd5b34ef7df Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 14:34:12 +0200 Subject: [PATCH 10/24] Keep the upgrade pre run: upgrade type runs pre and post Post-only dropped the baseline. Running the upgrade-typed tests on the old version before upgrading is what makes a post failure attributable to the upgrade rather than a pre-existing break, and it is cheap. Restores pre+post for the upgrade type (with APP_TEST_UPGRADE_STAGE) while keeping the two real fixes: upgrade is a peer type, and smoke/functional never run in the upgrade flow. ATS is unchanged (it already runs pre+post); atf's convention-runner gains the pre run. pre-upgrade hook stays for pure side-effect seeding. --- app-testing-contract/README.md | 80 +++++++++++++++++++++------------- 1 file changed, 49 insertions(+), 31 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 354e71c..9a37492 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -80,22 +80,33 @@ three peer types, matching ATS's existing `StepType`s; a test carries one. |---|---|---| | `smoke` | normal flow, first | fast, fail-fast sanity checks | | `functional` | normal flow, after smoke | full feature tests | -| `upgrade` | upgrade flow, once after the upgrade | verifies the app still works, and seeded state survived, across an upgrade | +| `upgrade` | upgrade flow, before and after the upgrade | verifies the app still works across an upgrade; the pre run is the baseline | `upgrade` is a peer type, not a modifier on the others. The upgrade flow -does not re-run the `smoke` and `functional` suites before and after the -upgrade; it runs only the `upgrade`-typed tests, and it runs them once, -after the upgrade. This matches both harnesses: ATS's upgrade scenario -runs the `upgrade` type only, and atf's upgrade suite runs its test -function once after upgrading. - -The asymmetric case (state that must survive the upgrade) is split by -concern: an imperative **pre-upgrade hook** seeds the state on the old -version (create a pod, write a record), and an `upgrade`-typed test -verifies it after the upgrade. The seeding is a side effect, so it is a -hook, not a test that reruns; the verification is an assertion, so it is a -test. The `upgrade` test can read `APP_TEST_UPGRADE_FROM_VERSION` / -`APP_TEST_UPGRADE_TO_VERSION` if it needs the version pair. See Hooks below. +does not re-run the `smoke` and `functional` suites; it runs only the +`upgrade`-typed tests. It runs them twice: once on the old version before +the upgrade (`APP_TEST_UPGRADE_STAGE=pre`) and once after +(`APP_TEST_UPGRADE_STAGE=post`). The pre run is the baseline that makes a +post failure attributable to the upgrade rather than to a pre-existing +break. This is ATS's existing behavior; atf gains the pre run. + +A symmetric invariant ("the app answers") is just an `upgrade` test that +asserts the same thing on both sides, and gets the baseline for free. The +asymmetric case (state that must survive the upgrade) has two shapes. When +the "before" step is a pure side effect, seed it in the `pre-upgrade` hook +and verify in the `post` run. When it is easier to keep in one file, a +single test branches on the stage: + +```go +if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { + t.Skip("verification runs after the upgrade") +} +``` + +Seeding-as-side-effect belongs in a hook, not a test that reruns; +verification is an assertion, so it is a test. Upgrade tests and the +`pre-upgrade` hook also receive `APP_TEST_UPGRADE_FROM_VERSION` / +`APP_TEST_UPGRADE_TO_VERSION`. Assertions live in tests; setup, teardown, and upgrade seeding live in hooks (next section). @@ -150,6 +161,7 @@ no cluster creation, no chart install, no App CRs. | `APP_TEST_CLUSTER_TYPE` | yes | cluster the app runs on: `kind` (local single-node, no cloud), `capi` (a CAPI workload cluster with cloud identity), or `external` (a pre-existing cluster the runner did not provision) | | `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | | `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | +| `APP_TEST_UPGRADE_STAGE` | upgrade flow only | `pre` or `post`: which side of the upgrade this `upgrade`-type run is on | | `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade flow only | versions on either side of the upgrade; available to the `pre-upgrade` hook and `upgrade`-typed tests | | `APP_TEST_EXTRA_*` | optional | harness extras, for example `APP_TEST_EXTRA_GITOPS_ENGINE` | @@ -184,11 +196,13 @@ In the **normal flow**, the executor is invoked once per applicable test type, in order: `smoke`, then `functional`. `upgrade`-typed tests do not run here. -In the **upgrade flow** (`upgrade: true`), the runner instead: deploys the -previous version, runs the `pre-upgrade` hook if present, upgrades to the -version under test, waits for it to settle, then invokes the executor once -for the `upgrade` type. The `smoke` and `functional` suites are not -re-run; the upgrade tests run once, after the upgrade. +In the **upgrade flow** (`upgrade: true`), the runner: deploys the previous +version and waits for it to settle, invokes the executor for the `upgrade` +type with `APP_TEST_UPGRADE_STAGE=pre` (the baseline), runs the +`pre-upgrade` hook if present, upgrades to the version under test and waits +for it to settle, then invokes the executor for the `upgrade` type with +`APP_TEST_UPGRADE_STAGE=post`. The `smoke` and `functional` suites are not +part of this flow. "No tests for this type" is a pass, not a failure (Go: build constraints exclude all files; pytest: exit code 5), because a repo may legitimately @@ -340,18 +354,22 @@ behind is a bug against that runner, not a licence to fork the contract. - **apptest-framework** gains a convention-runner: after provisioning the workload cluster and App CR, it fetches the WC kubeconfig (`Framework.GetClusterKubeConfig`), writes it to a file, exports the env - contract, detects the executor, and runs it per test type. The image - gains `uv` and `gotestsum`. Existing in-process suites are unaffected. - Enabling facts: Ginkgo runs under plain `go test`, and pytest tests - built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS - tests of both languages are immediately reusable on workload clusters. -- **app-test-suite** exports the canonical `APP_TEST_*` names alongside - its legacy `ATS_*` ones, stops running the `upgrade` type in a pre-upgrade - pass and runs it once post-upgrade (its existing `pre_upgrade`/ - `post_upgrade` config hooks map onto the conventional `pre-upgrade` hook), - discovers the conventional hooks by path in addition to its config-wired - ones, searches `tests/app/` in addition to its current `tests/ats/` - default, reads the shared config keys, and emits junit via gotestsum. Its + contract, detects the executor, and runs it per test type. For the + upgrade flow it gains the pre run: it runs the `upgrade` type against the + previous version (`APP_TEST_UPGRADE_STAGE=pre`) before upgrading, where + today it runs the suite once after. Its `BeforeUpgrade` callback maps onto + the conventional `pre-upgrade` hook. The image gains `uv` and `gotestsum`. + Existing in-process suites are unaffected. Enabling facts: Ginkgo runs + under plain `go test`, and pytest tests built on pytest-helm-charts + already read `KUBECONFIG`, so existing ATS tests of both languages are + immediately reusable on workload clusters. +- **app-test-suite** exports the canonical `APP_TEST_*` names alongside its + legacy `ATS_*` ones (including `APP_TEST_UPGRADE_STAGE` for the pre/post + runs it already performs), discovers the conventional hooks by path in + addition to its config-wired ones (its `pre_upgrade` config hook maps onto + the conventional `pre-upgrade` hook), searches `tests/app/` in addition to + its current `tests/ats/` default, reads the shared config keys, and emits + junit via gotestsum. Its upgrade pre/post behavior is unchanged. Its TEST_CONTRACT.md becomes a pointer to this RFC plus ATS-specific detail. - **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so both runners and non-portable suites share the same readiness vocabulary, and From af8e82256dcf531d7a316bb2ca23f2c9dca97a99 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 14:49:56 +0200 Subject: [PATCH 11/24] Support both hook mechanisms: convention discovery and config flags Convention discovery of tests/app/hooks/{setup,pre-upgrade,teardown} is the portable zero-wiring default every runner must implement; harnesses also keep their existing hook flags, so existing repos need no immediate move and harness-specific points stay available. If a flag and a convention hook target the same contract point the runner fails fast, so migration is drop-file-remove-flag in one change, not a silent double-run. Adds a per-harness mapping table and records that ATS needs a new pre-deploy point for setup (its --app-tests-pre-hook fires after deploy). --- app-testing-contract/README.md | 46 +++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 9 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 9a37492..0ba406e 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -139,12 +139,36 @@ environment properties exactly like category-2 tests (`APP_TEST_CLUSTER_TYPE` and friends); the `pre-upgrade` hook additionally receives `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. +Convention discovery of these three paths is the portable, zero-wiring +default a conforming runner must implement. A harness may *also* keep its +own config-wired hook flags, so existing repos need no immediate move and +harness-specific points stay available. Where a flag targets one of the +three contract points, it and the convention hook are alternatives: if both +are set for the same point, the runner fails fast (as with a directory that +has both `go.mod` and `pyproject.toml`), so migration is "drop the file, +remove the flag" in one change rather than a silent double-run. Only the +convention path is a contract guarantee and exercised by the conformance +suite; the flags are harness-native. + +How each harness supplies the three contract points today: + +| Contract hook | ATS | atf | +|---|---|---| +| `setup` (before deploy) | new pre-deploy point (its `--app-tests-pre-hook` fires after deploy) | `AfterClusterReady` (runs before install) | +| `pre-upgrade` | `--upgrade-tests-upgrade-hook` at `PRE_UPGRADE` | `BeforeUpgrade` | +| `teardown` (after tests) | `--app-tests-post-hook` | suite callback | + +Each runner satisfies a point either by discovering the convention file or +through the mapped flag, not both at once. Points outside this table +(ATS's `POST_UPGRADE` stage, its pre/post test hooks) stay harness-native. + The boundary is the same as for tests: a portable hook only gets the app cluster's `KUBECONFIG`. Work that needs the harness's own machinery (MC access, App CR manipulation, framework state) stays in harness-native -hooks: ATS's config-wired hook executables and atf's suite callbacks -(`AfterClusterReady`, `BeforeUpgrade`), which remain available and are -not part of this contract. +hooks: ATS's config-wired hooks at points the contract does not cover (its +pre/post *test* hooks, the `post-upgrade` stage) and atf's suite callbacks +(`AfterClusterReady`, `BeforeUpgrade`), which remain available and are not +part of this contract. ### Inputs @@ -365,12 +389,16 @@ behind is a bug against that runner, not a licence to fork the contract. immediately reusable on workload clusters. - **app-test-suite** exports the canonical `APP_TEST_*` names alongside its legacy `ATS_*` ones (including `APP_TEST_UPGRADE_STAGE` for the pre/post - runs it already performs), discovers the conventional hooks by path in - addition to its config-wired ones (its `pre_upgrade` config hook maps onto - the conventional `pre-upgrade` hook), searches `tests/app/` in addition to - its current `tests/ats/` default, reads the shared config keys, and emits - junit via gotestsum. Its upgrade pre/post behavior is unchanged. Its - TEST_CONTRACT.md becomes a pointer to this RFC plus ATS-specific detail. + runs it already performs), keeps its existing hook flags and additionally + discovers the conventional hooks by path (failing fast if a flag and a + convention hook target the same point), and gains a pre-deploy hook point + for `setup`: its `--app-tests-pre-hook` fires after deploy, so `setup` + (before deploy, for prerequisites) is a new call between + `_ensure_cluster_prerequisites` and the chart install. It searches + `tests/app/` in addition to its current `tests/ats/` default, reads the + shared config keys, and emits junit via gotestsum. Its upgrade pre/post + behavior is unchanged. Its TEST_CONTRACT.md becomes a pointer to this RFC + plus ATS-specific detail. - **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so both runners and non-portable suites share the same readiness vocabulary, and the settled-parity assertion has one definition to check against. From 3ca6ec2a086de30b65ede0487d3d545eb20687c4 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 15:04:35 +0200 Subject: [PATCH 12/24] Plainer prose; infer upgrade flow instead of configuring it Rewrites the doc in plainer language, cuts repetition and the argues-with-a-reviewer asides, and stops claiming the ATS_ dual export is free (it is deprecated, dropped at the next contract version). No decisions changed by the rewrite. Drops config key upgrade: true. The upgrade flow is now inferred from the presence of upgrade-typed tests, matching presence-is-the-opt-in used everywhere else. Removes the two cross-lints that only existed to guard the flag; expectedTypes still makes a type mandatory. --- app-testing-contract/README.md | 582 +++++++++++++++------------------ 1 file changed, 260 insertions(+), 322 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 0ba406e..659ac12 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -13,89 +13,77 @@ summary: Defines a harness-neutral contract for app tests so the same test files ## Problem -We run "does the deployed app actually work" checks at two cadences, on -purpose: - -- **app-test-suite (ATS)** deploys the chart on a kind cluster and runs - fast checks per pull request. -- **apptest-framework (atf)** stands up a workload cluster on a management - cluster, installs an App CR, and runs the full suite nightly. - -Both cadences earn their keep and we are keeping both: kind gives quick -per-PR signal, a real workload cluster catches what kind cannot (cloud -identity, real storage, upgrades). The split is not the problem. The -problem is that the two harnesses have divergent authoring models: ATS -expects pytest or plain Go, atf expects Ginkgo suites. Covering both -cadences means writing the same check twice in two idioms, so in practice -a repo writes it for one harness, or for neither. Adoption of both is -close to zero, and teams that did adopt a single harness are often -unhappy living in two worlds. - -So this RFC is not a dedup exercise; there is little duplication to -remove, because the double-idiom cost suppressed writing the tests in the -first place. It unifies the *authoring* model: one directory, one idiom -per repo, discovered and run the same way by both harnesses. A test -written once runs per-PR on kind (the subset kind can support) and nightly -on a workload cluster (everything), with no second copy to keep in sync. - -ATS already publishes a testing contract +We test managed apps two ways, and we want to keep both: + +- **app-test-suite (ATS)** installs the chart on a kind cluster and runs + quick checks on every PR. +- **apptest-framework (atf)** creates a real workload cluster, installs the + App CR, and runs the full suite nightly. + +kind is fast but can't do cloud identity, real storage, or upgrades; the +workload cluster can. So the two are a fast/slow pair, not duplicates. + +The trouble is writing the tests. ATS wants pytest or plain Go; atf wants +Ginkgo. To cover both you write the same check twice in two styles, so most +repos write it for one harness or skip it. Almost nobody has both, and the +people stuck on one aren't happy about it. + +This RFC doesn't dedupe existing tests; there aren't many to dedupe, for +the reason above. It makes the two harnesses agree on how tests are written +and run, so you write a check once and both run it: the kind-compatible +part on every PR, everything nightly. + +ATS already has a test contract ([docs/TEST_CONTRACT.md](https://github.com/giantswarm/app-test-suite/blob/master/docs/TEST_CONTRACT.md)), -but it lives in one harness's repo and only that harness implements it. -This RFC extracts the contract, makes it harness-neutral, and extends -apptest-framework to implement it too. The contract is deliberately not -tied to a test framework or language. +but it lives in the ATS repo and only ATS follows it. We lift it out, make +it harness-neutral, and make atf follow it too. It stays independent of any +test framework or language. ## Decision ### The conventional directory -Each app repo has one conventional test directory: `tests/app/`. It -contains the app's tests, written against the contract below. Any harness -that deploys the app runs this directory the same way. Presence of the -directory is the opt-in; there is no per-repo wiring. +Tests live in one directory: `tests/app/`. Any harness that deploys the app +runs that directory the same way. Having the directory is the opt-in; +there's nothing else to wire up. -The directory is one Go module or one Python project, never both. The -executor is detected from its contents: +It's either one Go module or one Python project, not both. The runner picks +the executor from what's there: -- `go.mod` present: `go test -mod=readonly -tags=` -- `pyproject.toml` present: `uv sync --frozen && uv run pytest -m ` -- both present, or neither present in a non-empty directory: - configuration error, fail fast +- `go.mod`: `go test -mod=readonly -tags=` +- `pyproject.toml`: `uv sync --frozen && uv run pytest -m ` +- both, or neither in a non-empty directory: config error, stop. -Dependencies are pinned and installed offline: Go reads a committed -`go.sum` under `-mod=readonly`, Python a committed `uv.lock` under -`--frozen`. A runner never resolves versions from the network at test -time; a missing or stale lockfile is a failure, not a silent fetch. This -keeps the dependency closure identical across harnesses and auditable. +Dependencies are pinned and installed offline: Go from a committed `go.sum` +(`-mod=readonly`), Python from a committed `uv.lock` (`--frozen`). No runner +resolves versions from the network at test time; a missing or stale +lockfile fails instead of quietly fetching. Same dependency set everywhere, +and you can audit it. -An empty `tests/app/` (no module, no project) is not an opt-in and is -ignored. +An empty `tests/app/` (no module, no project) isn't an opt-in and is +skipped. ### Test types -A test declares its type via a Go build tag or pytest marker. There are -three peer types, matching ATS's existing `StepType`s; a test carries one. +Each test carries one type, set with a Go build tag or a pytest marker. +There are three, the same ones ATS already has: -| Type | Runs | Meaning | +| Type | Runs | What it is | |---|---|---| -| `smoke` | normal flow, first | fast, fail-fast sanity checks | +| `smoke` | normal flow, first | quick sanity checks | | `functional` | normal flow, after smoke | full feature tests | -| `upgrade` | upgrade flow, before and after the upgrade | verifies the app still works across an upgrade; the pre run is the baseline | - -`upgrade` is a peer type, not a modifier on the others. The upgrade flow -does not re-run the `smoke` and `functional` suites; it runs only the -`upgrade`-typed tests. It runs them twice: once on the old version before -the upgrade (`APP_TEST_UPGRADE_STAGE=pre`) and once after -(`APP_TEST_UPGRADE_STAGE=post`). The pre run is the baseline that makes a -post failure attributable to the upgrade rather than to a pre-existing -break. This is ATS's existing behavior; atf gains the pre run. - -A symmetric invariant ("the app answers") is just an `upgrade` test that -asserts the same thing on both sides, and gets the baseline for free. The -asymmetric case (state that must survive the upgrade) has two shapes. When -the "before" step is a pure side effect, seed it in the `pre-upgrade` hook -and verify in the `post` run. When it is easier to keep in one file, a -single test branches on the stage: +| `upgrade` | upgrade flow, before and after | checks the app still works across an upgrade | + +`upgrade` is its own type, not a flag on the others. The upgrade flow +doesn't re-run smoke and functional; it runs the `upgrade` tests, once on +the old version (`APP_TEST_UPGRADE_STAGE=pre`) and once after upgrading +(`=post`). The pre run is the baseline: if it passes and post fails, the +upgrade caused it, not something that was already broken. + +Most upgrade checks are symmetric ("the app answers") and assert the same +thing both times. When some state has to survive the upgrade, either seed it +in the `pre-upgrade` hook and check it in the post run, or keep it in one +test that branches on the stage: ```go if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { @@ -103,54 +91,44 @@ if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { } ``` -Seeding-as-side-effect belongs in a hook, not a test that reruns; -verification is an assertion, so it is a test. Upgrade tests and the -`pre-upgrade` hook also receive `APP_TEST_UPGRADE_FROM_VERSION` / -`APP_TEST_UPGRADE_TO_VERSION`. - -Assertions live in tests; setup, teardown, and upgrade seeding live in -hooks (next section). +Seeding is a side effect, so it's a hook; checking is an assertion, so it's +a test. Upgrade tests and the `pre-upgrade` hook also get +`APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. ### Hooks -Hooks do imperative work with a side effect (install a prerequisite, seed -a pod, clean up an external resource); tests assert. Keeping the two -separate is what lets the upgrade flow seed state without re-running a test -suite. Setup, seeding, and teardown are app-specific just like assertions -and duplicate across harnesses the same way, so the contract defines -portable hooks: optional executables in the conventional directory, invoked -by the runner with the same environment as tests. +Hooks do things with side effects (install a prerequisite, create a pod, +clean up); tests check things. Splitting them is what lets the upgrade flow +seed state without re-running a suite. Like tests, setup and teardown are +per-app and get duplicated across harnesses, so the contract makes them +portable too: optional executables in `tests/app/`, run with the same +environment as tests. | Hook | Runs | |---|---| | `tests/app/hooks/setup` | after the cluster is ready, before the app is deployed (for example: install prerequisites) | -| `tests/app/hooks/pre-upgrade` | upgrade flow only: after the previous version is deployed, before the upgrade (for example: create a pod or write a record whose survival an `upgrade` test then verifies) | +| `tests/app/hooks/pre-upgrade` | upgrade flow only: after the previous version is deployed, before the upgrade (for example: create a pod or write a record an `upgrade` test then checks survived) | | `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | -A hook is any executable file at that path: a script with a shebang or a -built binary, invoked directly (not sourced, not run through a language -toolchain). Because it runs out of process, it is exempt from the -directory's one-language rule and may be written in whatever suits it; -keep it thin, since anything substantial belongs in a test or a -harness-native hook. +A hook is any executable at that path: a script with a shebang or a built +binary, run directly (not sourced). It runs out of process, so the +one-language rule doesn't apply and you can write it in whatever fits. Keep +it small; anything bigger is a test or a harness-native hook. -A missing hook is a no-op. A non-zero exit fails the run. Hooks gate on -environment properties exactly like category-2 tests (`APP_TEST_CLUSTER_TYPE` -and friends); the `pre-upgrade` hook additionally receives -`APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. +A missing hook does nothing. A non-zero exit fails the run. Hooks gate on +the environment like category-2 tests do; `pre-upgrade` also gets the +from/to versions. -Convention discovery of these three paths is the portable, zero-wiring -default a conforming runner must implement. A harness may *also* keep its -own config-wired hook flags, so existing repos need no immediate move and -harness-specific points stay available. Where a flag targets one of the -three contract points, it and the convention hook are alternatives: if both -are set for the same point, the runner fails fast (as with a directory that -has both `go.mod` and `pyproject.toml`), so migration is "drop the file, -remove the flag" in one change rather than a silent double-run. Only the -convention path is a contract guarantee and exercised by the conformance -suite; the flags are harness-native. +Convention discovery of those three paths is the default, and every runner +has to implement it. That's the zero-wiring part. A harness can also keep +its own hook flags, so existing repos don't have to move and +harness-specific hooks still work. If a flag and a convention file point at +the same contract hook, the runner stops (same as finding both `go.mod` and +`pyproject.toml`), so migrating is "add the file, drop the flag" in one +commit rather than running both. Only the convention path is guaranteed and +checked by the conformance suite; the flags are each harness's own business. -How each harness supplies the three contract points today: +How the three points map today: | Contract hook | ATS | atf | |---|---|---| @@ -158,22 +136,19 @@ How each harness supplies the three contract points today: | `pre-upgrade` | `--upgrade-tests-upgrade-hook` at `PRE_UPGRADE` | `BeforeUpgrade` | | `teardown` (after tests) | `--app-tests-post-hook` | suite callback | -Each runner satisfies a point either by discovering the convention file or -through the mapped flag, not both at once. Points outside this table -(ATS's `POST_UPGRADE` stage, its pre/post test hooks) stay harness-native. +A runner covers each point with either the file or the flag, not both. +Anything not in that table (ATS's `POST_UPGRADE`, its pre/post test hooks) +stays harness-native. -The boundary is the same as for tests: a portable hook only gets the app -cluster's `KUBECONFIG`. Work that needs the harness's own machinery (MC -access, App CR manipulation, framework state) stays in harness-native -hooks: ATS's config-wired hooks at points the contract does not cover (its -pre/post *test* hooks, the `post-upgrade` stage) and atf's suite callbacks -(`AfterClusterReady`, `BeforeUpgrade`), which remain available and are not -part of this contract. +Same boundary as tests: a hook only gets the app cluster's `KUBECONFIG`. +Anything that needs harness internals (MC access, the App CR, framework +state) stays in a harness-native hook: ATS's config hooks for points we +don't cover, or atf's `AfterClusterReady` / `BeforeUpgrade`. ### Inputs -Tests receive everything through the environment. They never provision: -no cluster creation, no chart install, no App CRs. +Tests get everything from the environment. They don't provision anything: +no clusters, no chart installs, no App CRs. | Variable | Required | Meaning | |---|---|---| @@ -189,12 +164,12 @@ no cluster creation, no chart install, no App CRs. | `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade flow only | versions on either side of the upgrade; available to the `pre-upgrade` hook and `upgrade`-typed tests | | `APP_TEST_EXTRA_*` | optional | harness extras, for example `APP_TEST_EXTRA_GITOPS_ENGINE` | -The canonical prefix is `APP_TEST_`, neutral to both harnesses. The -existing implementation publishes these variables under the legacy `ATS_` -prefix; conforming runners export both, so no existing test breaks and -dual export costs nothing ongoing. New and scaffolded tests use -`APP_TEST_`. The mapping is mechanical (`ATS_X` becomes `APP_TEST_X`) -with three exceptions renamed for clarity: +The prefix is `APP_TEST_`, which reads the same under either harness. ATS +publishes these under the old `ATS_` prefix today; runners export both so +nothing breaks, and new tests use `APP_TEST_`. Dual export isn't free (two +names to know and grep for), so `ATS_` is deprecated and drops on the next +contract version. Most names map straight across (`ATS_X` to `APP_TEST_X`); +three are renamed because the old names were unclear: | Legacy | Canonical | |---|---| @@ -202,238 +177,201 @@ with three exceptions renamed for clarity: | `ATS_APP_CONFIG_FILE_PATH` | `APP_TEST_VALUES_FILE` | | `ATS_CLUSTER_VERSION` | `APP_TEST_KUBERNETES_VERSION` | -`KUBECONFIG` is unchanged: it is the Kubernetes-wide convention, not ours. +`KUBECONFIG` stays as-is; it's the standard name, not ours. ### Runner guarantees -A conforming runner guarantees, before invoking the executor: +Before the tests run, a conforming runner makes sure: -1. the `setup` hook, if present, ran after the cluster was ready and - before the app was deployed, -2. the app is deployed and settled (ATS: chart installed via Helm or a - GitOps engine; atf: App CR reconciled to `deployed`), -3. all required variables above are exported, -4. the `teardown` hook, if present, runs after the last test type, before +1. the `setup` hook ran, if present, after the cluster was ready and before + the app was deployed, +2. the app is deployed and settled (ATS: Helm release installed, or via a + GitOps engine; atf: App CR at `deployed`), +3. the required variables are exported, +4. the `teardown` hook runs, if present, after the last test type and before the harness's own teardown. -In the **normal flow**, the executor is invoked once per applicable test -type, in order: `smoke`, then `functional`. `upgrade`-typed tests do not -run here. - -In the **upgrade flow** (`upgrade: true`), the runner: deploys the previous -version and waits for it to settle, invokes the executor for the `upgrade` -type with `APP_TEST_UPGRADE_STAGE=pre` (the baseline), runs the -`pre-upgrade` hook if present, upgrades to the version under test and waits -for it to settle, then invokes the executor for the `upgrade` type with -`APP_TEST_UPGRADE_STAGE=post`. The `smoke` and `functional` suites are not -part of this flow. - -"No tests for this type" is a pass, not a failure (Go: build constraints -exclude all files; pytest: exit code 5), because a repo may legitimately -carry only some types. Zero collection is never silent, though: the runner -records the collected count per type, so a typo'd tag or marker (which also -collects zero) is visible in the output rather than a green run. Repos that -want it enforced list the types they expect in `config.yaml` (see below); -a listed type collecting zero fails the run. Test results are emitted as -junit XML: `gotestsum --junitfile` for Go, `pytest --junitxml` for Python. +Normal flow: run `smoke`, then `functional`. Upgrade tests don't run here. + +Upgrade flow (any `upgrade` tests collected): install the previous version +and let it settle, run `upgrade` tests with `APP_TEST_UPGRADE_STAGE=pre`, run the +`pre-upgrade` hook, upgrade and let it settle, run `upgrade` tests with +`=post`. smoke and functional don't run here. + +"No tests of this type" passes rather than fails (Go excludes all files via +build tags; pytest exits 5), since a repo may only have some types. It isn't +silent, though: the runner reports how many tests it collected per type, so +a mistyped tag (also zero) shows up instead of going green. Repos that want +it strict list their expected types in `config.yaml`; a listed type with +zero tests fails. Results come out as junit XML (`gotestsum --junitfile`, +`pytest --junitxml`). ### Cadence and feedback latency -The two cadences buy quick per-PR signal at the cost of a gap: the fast -runner on kind cannot exercise what kind lacks (cloud identity, real -storage, load balancers), so a test gated to those environments runs only -in the nightly workload-cluster flow. Its signal is then detached from the -change that broke it. A pull request that breaks a cloud-only path passes -per-PR CI green and fails nightly, hours later, against a batch of -unrelated commits. - -This is an accepted property of the split, not a defect the contract -introduces, but the contract must not let it be silent or unescapable: - -1. **A nightly-only test is a conscious choice, never an accident.** A - test that gates itself off kind (category 2) is by construction - per-PR-invisible. Reviewers see that in the diff; the collected-count - output makes "ran nowhere per-PR" legible rather than looking like - coverage. -2. **There is an on-demand full-flow trigger.** A change that touches a - cloud-only path can request the workload-cluster flow against the pull - request instead of waiting for the scheduled run, via the existing - `/run` pipeline convention. Catching a cloud regression a day late is - the default; paying for it on the PR is one comment away. - -Prefer expressing a real capability need over a hard environment gate (see -`APP_TEST_CLUSTER_TYPE` below): a test that only needs cloud identity, not -kind-vs-WC specifically, will also run per-PR on any `external` cluster -that provides it, which shrinks the nightly-only set. +kind can't do cloud identity, storage, or load balancers, so tests that +need those only run nightly. Their result isn't tied to the PR that caused +it: a PR can break a cloud-only path, pass PR CI, and fail that night +against a batch of other commits. + +We accept that, but two things keep it from being a silent trap: + +1. A nightly-only test is a choice you can see. Gating a test off kind + (category 2) makes it invisible per-PR by design; the skip shows by name + and the collected counts show it didn't run, so it doesn't read as + coverage it isn't. +2. You can pull the nightly flow forward. `/run` triggers the + workload-cluster flow on a PR, so a cloud-path change can get its result + now instead of that night. + +Where you can, gate on the capability you need rather than kind-vs-WC (see +Harness-specific tests): a test that needs cloud identity also runs on an +`external` cluster that has it. ### Shared configuration -`tests/app/config.yaml` carries only the keys both harnesses need: +`tests/app/config.yaml` holds only what both harnesses need: ```yaml contractVersion: 1 # contract version this directory targets installNamespace: kube-system -upgrade: true # whether an upgrade flow applies to this app expectedTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test ``` -`upgrade` is the declarative form of each harness's existing upgrade -primitive: atf's `WithIsUpgrade(true)` (install the latest release, -upgrade to the version under test) and ATS's upgrade scenario. It is -explicit rather than inferred from the presence of `upgrade`-typed tests -because the upgrade flow is the expensive one. - -The two settings lint against each other and against what is collected, so -a mistake fails the run instead of passing green: - -- `upgrade: true` with zero `upgrade`-typed tests collected fails (dead - flow, or a typo'd tag). -- `upgrade`-typed tests present with `upgrade` unset or false fails (tests - that would never run). -- any type in `expectedTypes` collecting zero fails; `expectedTypes` is - optional, and omitting it keeps the permissive "no tests is a pass" - default for repos that do not want the check. - -Everything harness-specific stays in the harness's own config: -`.ats/main.yaml` (cluster types, catalogs, executor options) and -`tests/e2e/config.yaml` (appCatalog, providers, MC test options). Values -files also stay per harness: values legitimately differ between a kind -cluster and a workload cluster, and each provisioner consumes them through -its own mechanism. +The upgrade flow is inferred, not configured: if the runner collects any +`upgrade`-typed tests it runs the upgrade flow, otherwise it doesn't. Same +presence-is-the-opt-in rule as the directory and the other types, so there's +no separate switch to keep in sync. Each harness still learns which version +to upgrade from through its own config (ATS's stable-app settings, atf's +latest published release); that part is harness-specific, not contract. + +The one lint: a type in `expectedTypes` that collects zero tests fails the +run. `expectedTypes` is optional; leave it out to keep the "no tests is +fine" default. It's also how you make a type mandatory. List `upgrade`, and +a typo'd tag (which collects zero) fails instead of quietly skipping the +flow. + +Everything harness-specific stays in that harness's config: `.ats/main.yaml` +(cluster types, catalogs, executor options) and `tests/e2e/config.yaml` +(appCatalog, providers, MC options). Values files stay per-harness too; a +kind cluster and a workload cluster legitimately want different values, and +each harness loads them its own way. ### Harness-specific tests -The contract covers the default case, not everything. Where a test goes: +The contract is for the common case. Where a test goes: -1. Asserts on the deployed app and works on any cluster: `tests/app/`, - no gate. This should be the bulk. -2. Asserts on the deployed app but is only meaningful in one environment: - `tests/app/` plus a runtime skip on contract environment, for example +1. Checks the deployed app, works anywhere: `tests/app/`, no gate. Most + tests. +2. Checks the deployed app but only makes sense in one environment: + `tests/app/` with a runtime skip, for example `if os.Getenv("APP_TEST_CLUSTER_TYPE") != "kind" { t.Skip(...) }`. Skips - stay visible by name in both runners' output. + still show by name. 3. Needs harness machinery (MC access, bundle installs, AWS/IRSA, cluster - manipulation): a regular in-process apptest-framework suite under - `tests/e2e/suites/`, unchanged by this RFC. - -A repo may therefore hold two Go modules, `tests/app/` (portable) and -`tests/e2e/` (atf-native), each with its own `go.mod`. They stay separate -modules on purpose: the portable one must build without the atf -dependency tree. Repos that want unified tooling across them add a -`go.work` at the repo root; it is not required and is never committed as a -contract artifact. - -Tests may gate on environment properties the contract exposes, never on -which harness is running them. A test that needs the harness's name -belongs in category 3. - -`APP_TEST_CLUSTER_TYPE` is a capability axis, not a harness label. That -`kind` tends to mean ATS and `capi` tends to mean atf today is -incidental, and a gate written as "am I really asking about the harness?" -is a category error even when it happens to work. Gate on the property you -actually depend on: if a test needs cloud identity, express that (and let -`external` clusters that also provide it pass the same gate) rather than -hard-coding `!= "kind"`. If the property you need is not on any contract -variable, the test needs harness machinery and belongs in category 3. + manipulation): a normal atf suite under `tests/e2e/suites/`, which this + RFC doesn't touch. + +So a repo can hold two Go modules: `tests/app/` (portable) and `tests/e2e/` +(atf-native), each with its own `go.mod`. They're separate on purpose, since +the portable one has to build without atf's dependencies. If you want one +toolchain over both, add a `go.work` at the repo root; it's optional and +never part of the contract. + +Gate on what the contract tells you about the environment, never on which +harness is running. If a test needs to know the harness name, it's +category 3. + +`APP_TEST_CLUSTER_TYPE` is about capability, not which harness. `kind` +usually being ATS and `capi` usually being atf is a coincidence, and gating +on it as a stand-in for the harness is wrong even when it happens to work. +Gate on what you actually need: if a test needs cloud identity, check for +that, so an `external` cluster with cloud identity passes the same gate. If +what you need isn't in any contract variable, the test needs harness +machinery, which is category 3. ### Conformance, versioning, and ownership -Two independent runners implement one contract, so drift is the default -failure mode unless something mechanically checks them. The contract ships -with a conformance suite: a fixture `tests/app/` (a trivial app, one test -of each type, one hook, a lockfile) plus a set of assertions on the -env-var, ordering, exit-code, and lint guarantees above. A runner is -conforming only if it passes the suite in its CI; both ATS and atf wire it -in. New guarantees land in the suite in the same change that adds them -here. - -Per-runner conformance is necessary but not sufficient: two runners can -each satisfy the letter of the contract and still disagree on what a test -observes, which is the failure that actually bites (a smoke test that -passes fast-CI and flakes nightly). The suite therefore also asserts -*parity*: the same fixture run through both runners must yield the same -collected-per-type counts and the same pass/fail outcome, and any -divergence fails the suite. The known sharp edge is guarantee 2, "the app -is settled": ATS reaches it via a Helm release reporting installed, atf -via an App CR reconciled to `deployed`, and those are not the same instant. -The contract fixes the observable, not the mechanism: settled means the -app's own readiness (its Deployments/StatefulSets Available) holds, and the -parity fixture asserts a runner does not hand off to tests before it does. -`clustertest.wait.IsDeploymentReady` is the shared vocabulary for that -check so both runners and the tests mean the same thing by "ready". - -The contract is versioned. This document is `v1`; the version is declared -in `tests/app/config.yaml` as `contractVersion: 1`. A runner refuses a -directory whose declared version it does not implement rather than -guessing. Breaking changes bump the integer and the conformance suite -carries a fixture per supported version. - -team-tenet stewards the contract (owns this document and the conformance -suite, arbitrates when the two runners disagree). team-honeybadger and -team-bumblebee own the ATS and atf implementations respectively. A change -to the contract is a PR here that updates the suite; a runner falling -behind is a bug against that runner, not a licence to fork the contract. +Two runners, one contract, so they'll drift unless something checks. The +contract ships a conformance suite: a fixture `tests/app/` (trivial app, one +test per type, a hook, a lockfile) and assertions on the env vars, ordering, +exit codes, and lints above. A runner conforms only if it passes the suite +in CI; ATS and atf both wire it in. New guarantees go into the suite in the +same PR that adds them here. + +Passing per runner isn't enough: both can pass and still disagree on what a +test sees, which is the drift that hurts (a smoke test that's green on PR +and flaky at night). So the suite also checks parity: the same fixture +through both runners has to collect the same counts and end with the same +result, or the suite fails. The known trap is guarantee 2, "settled": ATS +gets there when the Helm release reports installed, atf when the App CR +reads `deployed`, and those aren't the same moment. The contract pins the +observable, not the mechanism: settled means the app's own workloads are +Available, and the parity fixture checks that neither runner starts tests +early. `clustertest.wait.IsDeploymentReady` is the shared definition of +ready. + +The contract is versioned. This is `v1`, declared as `contractVersion: 1`. +A runner refuses a version it doesn't implement instead of guessing. A +breaking change bumps the number, and the suite keeps a fixture per version. + +team-tenet owns the contract: this doc, the suite, and the call when the +runners disagree. team-honeybadger owns ATS, team-bumblebee owns atf. +Changing the contract is a PR here that updates the suite. A runner lagging +is a bug in that runner, not a reason to fork. ## Implementation -- **apptest-framework** gains a convention-runner: after provisioning the - workload cluster and App CR, it fetches the WC kubeconfig - (`Framework.GetClusterKubeConfig`), writes it to a file, exports the env - contract, detects the executor, and runs it per test type. For the - upgrade flow it gains the pre run: it runs the `upgrade` type against the - previous version (`APP_TEST_UPGRADE_STAGE=pre`) before upgrading, where - today it runs the suite once after. Its `BeforeUpgrade` callback maps onto - the conventional `pre-upgrade` hook. The image gains `uv` and `gotestsum`. - Existing in-process suites are unaffected. Enabling facts: Ginkgo runs - under plain `go test`, and pytest tests built on pytest-helm-charts - already read `KUBECONFIG`, so existing ATS tests of both languages are - immediately reusable on workload clusters. -- **app-test-suite** exports the canonical `APP_TEST_*` names alongside its - legacy `ATS_*` ones (including `APP_TEST_UPGRADE_STAGE` for the pre/post - runs it already performs), keeps its existing hook flags and additionally - discovers the conventional hooks by path (failing fast if a flag and a - convention hook target the same point), and gains a pre-deploy hook point - for `setup`: its `--app-tests-pre-hook` fires after deploy, so `setup` - (before deploy, for prerequisites) is a new call between - `_ensure_cluster_prerequisites` and the chart install. It searches - `tests/app/` in addition to its current `tests/ats/` default, reads the - shared config keys, and emits junit via gotestsum. Its upgrade pre/post - behavior is unchanged. Its TEST_CONTRACT.md becomes a pointer to this RFC - plus ATS-specific detail. -- **clustertest** gains `wait.IsDeploymentReady(name, namespace)` so both - runners and non-portable suites share the same readiness vocabulary, and - the settled-parity assertion has one definition to check against. -- **the on-demand full-flow trigger**: the workload-cluster pipeline runs - on the `/run` convention against a pull request, not only on the nightly - schedule, so a change touching a cloud-only path can pull its signal - forward without waiting for the batch. -- **the conformance suite** lives in this repo alongside the RFC: the - fixture app plus the guarantee assertions, including the cross-runner - parity check. Both runners run it in CI; it is the acceptance gate for - "implements the contract." -- **devctl `gen apptest` and template-app** scaffold the conventional - layout for new repos. -- Migration is opt-in as repos get touched; there is no flag day. - Pilot: [giantswarm/muster#954](https://github.com/giantswarm/muster/pull/954). +- **apptest-framework** gets a convention-runner: after the workload cluster + and App CR are up, it grabs the WC kubeconfig + (`Framework.GetClusterKubeConfig`), writes it out, exports the env + contract, picks the executor, and runs it per type. For upgrades it adds + the pre run. Today it runs the suite once after the upgrade; now it also + runs `upgrade` tests against the old version first + (`APP_TEST_UPGRADE_STAGE=pre`). `BeforeUpgrade` maps to the `pre-upgrade` + hook. The image adds `uv` and `gotestsum`. In-process suites are + untouched. Two things make this cheap: Ginkgo runs under plain `go test`, + and pytest tests built on pytest-helm-charts already read `KUBECONFIG`, so + existing ATS tests in either language run on workload clusters as-is. +- **app-test-suite** exports the `APP_TEST_*` names next to the old `ATS_*` + ones (including `APP_TEST_UPGRADE_STAGE`, which it already has pre/post + runs for), keeps its hook flags and also discovers the convention hooks by + path (stopping if a flag and a file point at the same one), and adds a + pre-deploy point for `setup`. Its `--app-tests-pre-hook` runs after + deploy, so `setup` is a new call between `_ensure_cluster_prerequisites` + and the install. It looks in `tests/app/` as well as today's `tests/ats/`, + reads the shared config, and emits junit via gotestsum. Its upgrade + pre/post behavior doesn't change. Its TEST_CONTRACT.md becomes a pointer + here plus ATS-specific detail. +- **clustertest** gets `wait.IsDeploymentReady(name, namespace)` so both + runners and the atf-native suites share one definition of ready, and the + parity check has one thing to assert against. +- **the on-demand trigger**: the workload-cluster pipeline runs on `/run` + against a PR, not just nightly, so a cloud-path change can get its result + without waiting. +- **the conformance suite** lives here with the RFC: the fixture app and the + assertions, parity check included. Both runners run it in CI; it's what + "conforms" means. +- **devctl `gen apptest` and template-app** scaffold the layout for new + repos. +- Migration happens as repos get touched; no flag day. Pilot: + [giantswarm/muster#954](https://github.com/giantswarm/muster/pull/954). ## Alternatives considered -- **A shared assertions library consumed by both harnesses per repo.** - Prototyped in muster; the module, replace directives, and adapter - helpers protected a dozen lines of predicate logic per repo. Rejected in - favor of aligning the runners so the test files themselves are shared. -- **Standardizing on Ginkgo as the contract.** Ginkgo runs under - `go test`, so it is allowed, but mandating it would exclude the pytest - repos and couple the contract to a framework for no gain. The contract - standardizes selection and inputs, not the test framework. -- **A `values: {ats: ..., e2e: ...}` map in the shared config.** Rejected: - it bakes harness names into the neutral file, the configuration - equivalent of a test gating on the harness's name. -- **One runner with two modes instead of two runners behind a contract.** - A single runner covering both fast-kind and workload-cluster modes would - need no contract to police, since there would be nothing to keep in step. - Rejected because the two cadences map onto two mature codebases owned by - two teams (ATS by team-honeybadger, atf by team-tenet), each carrying - provisioning and pipeline machinery the other does not want. Collapsing - them is a larger, riskier rewrite than aligning their edges, and the - parity check gives most of the anti-drift benefit at a fraction of the - cost. If the two runners keep diverging in practice, revisit this. +- **A shared assertions library both harnesses import per repo.** Tried it + in muster; the module, replace directives, and adapter code guarded about + a dozen lines of predicate per repo. Not worth it; better to align the + runners so the test files themselves are shared. +- **Standardize on Ginkgo.** Ginkgo runs under `go test`, so it's allowed, + but requiring it would shut out the pytest repos and tie the contract to a + framework for nothing. We standardize selection and inputs, not the + framework. +- **A `values: {ats: ..., e2e: ...}` map in the shared config.** No: it puts + harness names in the neutral file, which is the config version of gating a + test on the harness. +- **One runner with two modes instead of two behind a contract.** One runner + covering kind and workload clusters wouldn't need a contract at all. But + the two modes line up with two mature codebases owned by two teams + (ATS/honeybadger, atf/tenet), each with provisioning and pipeline code the + other doesn't want. Merging them is a bigger, riskier job than aligning + their edges, and the parity check gets most of the anti-drift value for + far less. Revisit if the runners keep drifting. From d03da147a37fceb24fd9a93e6273eeafd927f08f Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 15:18:38 +0200 Subject: [PATCH 13/24] Drop contractVersion from the shared config Version negotiation is overkill for two in-house runners that both track this doc and are gated by the same conformance suite. The contract evolves by changing the suite in the same PR; a lagging runner is a bug, not a version mismatch to arbitrate. Removes the config key and the versioning paragraph, renames the section to Conformance and ownership, and reworks the ATS_ deprecation to not reference a next version. --- app-testing-contract/README.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 659ac12..b3ea913 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -167,8 +167,9 @@ no clusters, no chart installs, no App CRs. The prefix is `APP_TEST_`, which reads the same under either harness. ATS publishes these under the old `ATS_` prefix today; runners export both so nothing breaks, and new tests use `APP_TEST_`. Dual export isn't free (two -names to know and grep for), so `ATS_` is deprecated and drops on the next -contract version. Most names map straight across (`ATS_X` to `APP_TEST_X`); +names to know and grep for), so `ATS_` is deprecated and will be removed in +a later change once repos have migrated. Most names map straight across +(`ATS_X` to `APP_TEST_X`); three are renamed because the old names were unclear: | Legacy | Canonical | @@ -232,7 +233,6 @@ Harness-specific tests): a test that needs cloud identity also runs on an `tests/app/config.yaml` holds only what both harnesses need: ```yaml -contractVersion: 1 # contract version this directory targets installNamespace: kube-system expectedTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test ``` @@ -288,7 +288,7 @@ that, so an `external` cluster with cloud identity passes the same gate. If what you need isn't in any contract variable, the test needs harness machinery, which is category 3. -### Conformance, versioning, and ownership +### Conformance and ownership Two runners, one contract, so they'll drift unless something checks. The contract ships a conformance suite: a fixture `tests/app/` (trivial app, one @@ -309,10 +309,6 @@ Available, and the parity fixture checks that neither runner starts tests early. `clustertest.wait.IsDeploymentReady` is the shared definition of ready. -The contract is versioned. This is `v1`, declared as `contractVersion: 1`. -A runner refuses a version it doesn't implement instead of guessing. A -breaking change bumps the number, and the suite keeps a fixture per version. - team-tenet owns the contract: this doc, the suite, and the call when the runners disagree. team-honeybadger owns ATS, team-bumblebee owns atf. Changing the contract is a PR here that updates the suite. A runner lagging From c0ee1b2b72d612d72e162c8db023c320f6d019b5 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 16:16:54 +0200 Subject: [PATCH 14/24] Fix factual errors and gate on capability not cluster type - correct broken TEST_CONTRACT.md link (master -> main) - fix atf kubeconfig call: framework.MC().GetClusterKubeConfig - pin settled on IsReleaseReady (all workload kinds) not IsDeploymentReady - add APP_TEST_CAPABILITIES; gate on capability, never cluster type/harness - correct upgrade-stage mapping (ATS_EXTRA_UPGRADE_TEST_STAGE, value change) - correct offline install guarantee (GOPROXY=off, uv --offline; atf too) - reflect app-test-suite#675 (ATS runs on a provided cluster) --- app-testing-contract/README.md | 164 ++++++++++++++++++++------------- 1 file changed, 99 insertions(+), 65 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index b3ea913..e66dd89 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -6,7 +6,7 @@ owners: - https://github.com/orgs/giantswarm/teams/team-honeybadger - https://github.com/orgs/giantswarm/teams/team-tenet state: review -summary: Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (chart tests on kind) and apptest-framework (e2e on workload clusters). One conventional directory per repo, test types via build tags or pytest markers, inputs via KUBECONFIG and APP_TEST_* env vars. +summary: Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (fast chart tests per PR) and apptest-framework (e2e on workload clusters). One conventional directory per repo, test types via build tags or pytest markers, inputs via KUBECONFIG and APP_TEST_* env vars. --- # The app-testing contract @@ -15,13 +15,17 @@ summary: Defines a harness-neutral contract for app tests so the same test files We test managed apps two ways, and we want to keep both: -- **app-test-suite (ATS)** installs the chart on a kind cluster and runs - quick checks on every PR. +- **app-test-suite (ATS)** installs the chart on a provided cluster (today + usually a local kind cluster) and runs quick checks on every PR. ATS is + dropping its built-in kind lifecycle + ([giantswarm/app-test-suite#675](https://github.com/giantswarm/app-test-suite/pull/675)), + so the cluster it runs against is increasingly whatever CI hands it. - **apptest-framework (atf)** creates a real workload cluster, installs the App CR, and runs the full suite nightly. -kind is fast but can't do cloud identity, real storage, or upgrades; the -workload cluster can. So the two are a fast/slow pair, not duplicates. +A local kind cluster is fast but can't do cloud identity, real storage, or +upgrades; a workload cluster can. So the two are a fast/slow pair, not +duplicates. The trouble is writing the tests. ATS wants pytest or plain Go; atf wants Ginkgo. To cover both you write the same check twice in two styles, so most @@ -30,11 +34,11 @@ people stuck on one aren't happy about it. This RFC doesn't dedupe existing tests; there aren't many to dedupe, for the reason above. It makes the two harnesses agree on how tests are written -and run, so you write a check once and both run it: the kind-compatible -part on every PR, everything nightly. +and run, so you write a check once and both run it: what the PR cluster +supports on every PR, everything nightly. ATS already has a test contract -([docs/TEST_CONTRACT.md](https://github.com/giantswarm/app-test-suite/blob/master/docs/TEST_CONTRACT.md)), +([docs/TEST_CONTRACT.md](https://github.com/giantswarm/app-test-suite/blob/main/docs/TEST_CONTRACT.md)), but it lives in the ATS repo and only ATS follows it. We lift it out, make it harness-neutral, and make atf follow it too. It stays independent of any test framework or language. @@ -54,11 +58,16 @@ the executor from what's there: - `pyproject.toml`: `uv sync --frozen && uv run pytest -m ` - both, or neither in a non-empty directory: config error, stop. -Dependencies are pinned and installed offline: Go from a committed `go.sum` -(`-mod=readonly`), Python from a committed `uv.lock` (`--frozen`). No runner -resolves versions from the network at test time; a missing or stale -lockfile fails instead of quietly fetching. Same dependency set everywhere, -and you can audit it. +Dependencies are pinned from committed lockfiles (Go `go.sum`, Python +`uv.lock`) and installed with the network off: the runner sets `GOPROXY=off` +alongside `-mod=readonly` (or vendors `tests/app/vendor/`) and runs +`uv sync --frozen --offline`, backed by a pre-warmed module/uv cache. The +lockfile flags alone (`-mod=readonly`, `--frozen`) only freeze resolution, +not fetching, so cutting the network is what makes a missing or stale entry +fail loudly instead of quietly downloading. Both runners implement this: +under this contract `tests/app/` is a separate module that atf builds at +test time, so atf gains the same fetch point ATS already has. Same +dependency set everywhere, and you can audit it. An empty `tests/app/` (no module, no project) isn't an opt-in and is skipped. @@ -133,12 +142,12 @@ How the three points map today: | Contract hook | ATS | atf | |---|---|---| | `setup` (before deploy) | new pre-deploy point (its `--app-tests-pre-hook` fires after deploy) | `AfterClusterReady` (runs before install) | -| `pre-upgrade` | `--upgrade-tests-upgrade-hook` at `PRE_UPGRADE` | `BeforeUpgrade` | +| `pre-upgrade` | `--upgrade-tests-upgrade-hook` at `ATS_HOOK_STAGE=pre_upgrade` | `BeforeUpgrade` | | `teardown` (after tests) | `--app-tests-post-hook` | suite callback | A runner covers each point with either the file or the flag, not both. -Anything not in that table (ATS's `POST_UPGRADE`, its pre/post test hooks) -stays harness-native. +Anything not in that table (ATS's `post_upgrade` stage, its pre/post test +hooks) stays harness-native. Same boundary as tests: a hook only gets the app cluster's `KUBECONFIG`. Anything that needs harness internals (MC access, the App CR, framework @@ -157,7 +166,8 @@ no clusters, no chart installs, no App CRs. | `APP_TEST_RELEASE_NAME` | yes | Helm release name of the app under test | | `APP_TEST_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | | `APP_TEST_CHART_VERSION` | yes | version of the chart under test | -| `APP_TEST_CLUSTER_TYPE` | yes | cluster the app runs on: `kind` (local single-node, no cloud), `capi` (a CAPI workload cluster with cloud identity), or `external` (a pre-existing cluster the runner did not provision) | +| `APP_TEST_CLUSTER_TYPE` | yes | topology of the cluster the app runs on: `kind` (local single-node), `capi` (a CAPI workload cluster), or `external` (a cluster the runner did not provision). Describes shape, not capability; gate on `APP_TEST_CAPABILITIES` instead | +| `APP_TEST_CAPABILITIES` | yes | comma-separated capabilities the cluster actually provides, e.g. `cloud-identity,persistent-storage,load-balancer`; empty is valid. The runner sets it from what it provisioned or was handed. This is what a test gates on | | `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | | `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | | `APP_TEST_UPGRADE_STAGE` | upgrade flow only | `pre` or `post`: which side of the upgrade this `upgrade`-type run is on | @@ -165,20 +175,27 @@ no clusters, no chart installs, no App CRs. | `APP_TEST_EXTRA_*` | optional | harness extras, for example `APP_TEST_EXTRA_GITOPS_ENGINE` | The prefix is `APP_TEST_`, which reads the same under either harness. ATS -publishes these under the old `ATS_` prefix today; runners export both so -nothing breaks, and new tests use `APP_TEST_`. Dual export isn't free (two -names to know and grep for), so `ATS_` is deprecated and will be removed in -a later change once repos have migrated. Most names map straight across -(`ATS_X` to `APP_TEST_X`); -three are renamed because the old names were unclear: - -| Legacy | Canonical | -|---|---| -| `ATS_TEST_TYPE` | `APP_TEST_TYPE` | -| `ATS_APP_CONFIG_FILE_PATH` | `APP_TEST_VALUES_FILE` | -| `ATS_CLUSTER_VERSION` | `APP_TEST_KUBERNETES_VERSION` | +publishes these under the old `ATS_` prefix today. Where a name maps +straight across (`ATS_X` to `APP_TEST_X`, e.g. `ATS_RELEASE_NAME`, +`ATS_CHART_VERSION`, `ATS_CLUSTER_TYPE`, `ATS_EXTRA_*`) the runner exports +both, so nothing breaks and new tests use `APP_TEST_`. Dual export isn't +free (two names to know and grep for), so `ATS_` is deprecated and will be +removed in a later change once repos have migrated. + +Four names are renamed, because the old ones were unclear or, for the +upgrade stage, were never test-facing under one name to begin with: + +| Legacy | Canonical | Note | +|---|---|---| +| `ATS_TEST_TYPE` | `APP_TEST_TYPE` | straight rename | +| `ATS_APP_CONFIG_FILE_PATH` | `APP_TEST_VALUES_FILE` | straight rename | +| `ATS_CLUSTER_VERSION` | `APP_TEST_KUBERNETES_VERSION` | straight rename | +| `ATS_EXTRA_UPGRADE_TEST_STAGE` (tests) / `ATS_HOOK_STAGE` (hooks) | `APP_TEST_UPGRADE_STAGE` | value also changes: `pre_upgrade`/`post_upgrade` become `pre`/`post`. The runner does not alias the value, so existing upgrade tests reading the old one must update | -`KUBECONFIG` stays as-is; it's the standard name, not ours. +`ATS_CHART_PATH` and `ATS_TEST_DIR` have no `APP_TEST_` equivalent; the +contract doesn't expose them and they stay ATS-only. `APP_TEST_CAPABILITIES` +is new, computed by the runner, with no `ATS_` predecessor. `KUBECONFIG` +stays as-is; it's the standard name, not ours. ### Runner guarantees @@ -186,8 +203,10 @@ Before the tests run, a conforming runner makes sure: 1. the `setup` hook ran, if present, after the cluster was ready and before the app was deployed, -2. the app is deployed and settled (ATS: Helm release installed, or via a - GitOps engine; atf: App CR at `deployed`), +2. the app is deployed and settled: each runner first waits on its own + mechanism signal (ATS: Helm release installed, or via a GitOps engine; + atf: App CR at `deployed`), then on the shared gate `IsReleaseReady` (the + release's workloads Available) before any test runs, 3. the required variables are exported, 4. the `teardown` hook runs, if present, after the last test type and before the harness's own teardown. @@ -209,24 +228,26 @@ zero tests fails. Results come out as junit XML (`gotestsum --junitfile`, ### Cadence and feedback latency -kind can't do cloud identity, storage, or load balancers, so tests that -need those only run nightly. Their result isn't tied to the PR that caused -it: a PR can break a cloud-only path, pass PR CI, and fail that night -against a batch of other commits. +The PR cluster often lacks cloud identity, real storage, or load balancers, +so tests that need those capabilities only run nightly on a workload +cluster. Their result isn't tied to the PR that caused it: a PR can break a +cloud-only path, pass PR CI, and fail that night against a batch of other +commits. We accept that, but two things keep it from being a silent trap: -1. A nightly-only test is a choice you can see. Gating a test off kind - (category 2) makes it invisible per-PR by design; the skip shows by name - and the collected counts show it didn't run, so it doesn't read as - coverage it isn't. +1. A nightly-only test is a choice you can see. A test that gates on a + capability the PR cluster lacks (category 2) is invisible per-PR by + design; the skip shows by name and the collected counts show it didn't + run, so it doesn't read as coverage it isn't. 2. You can pull the nightly flow forward. `/run` triggers the workload-cluster flow on a PR, so a cloud-path change can get its result now instead of that night. -Where you can, gate on the capability you need rather than kind-vs-WC (see -Harness-specific tests): a test that needs cloud identity also runs on an -`external` cluster that has it. +Gate on the capability you need (`APP_TEST_CAPABILITIES`), never on cluster +type or harness (see Harness-specific tests): a test that needs cloud +identity runs anywhere advertising `cloud-identity`, whether that's the +nightly WC or a provided cluster that happens to have it. ### Shared configuration @@ -262,10 +283,10 @@ The contract is for the common case. Where a test goes: 1. Checks the deployed app, works anywhere: `tests/app/`, no gate. Most tests. -2. Checks the deployed app but only makes sense in one environment: - `tests/app/` with a runtime skip, for example - `if os.Getenv("APP_TEST_CLUSTER_TYPE") != "kind" { t.Skip(...) }`. Skips - still show by name. +2. Checks the deployed app but needs a capability not present everywhere: + `tests/app/` with a runtime skip on the capability, for example + `if !slices.Contains(caps, "cloud-identity") { t.Skip(...) }` where `caps` + comes from `APP_TEST_CAPABILITIES`. Skips still show by name. 3. Needs harness machinery (MC access, bundle installs, AWS/IRSA, cluster manipulation): a normal atf suite under `tests/e2e/suites/`, which this RFC doesn't touch. @@ -280,13 +301,16 @@ Gate on what the contract tells you about the environment, never on which harness is running. If a test needs to know the harness name, it's category 3. -`APP_TEST_CLUSTER_TYPE` is about capability, not which harness. `kind` -usually being ATS and `capi` usually being atf is a coincidence, and gating -on it as a stand-in for the harness is wrong even when it happens to work. -Gate on what you actually need: if a test needs cloud identity, check for -that, so an `external` cluster with cloud identity passes the same gate. If -what you need isn't in any contract variable, the test needs harness -machinery, which is category 3. +Gate on `APP_TEST_CAPABILITIES`, not on `APP_TEST_CLUSTER_TYPE`, and never +on the harness. Cluster type is topology, not capability: `kind` usually +being the PR runner and `capi` usually being the nightly one is a +coincidence, and ATS moving to provided clusters +([app-test-suite#675](https://github.com/giantswarm/app-test-suite/pull/675)) +breaks even that, since the PR runner can then be handed a cluster with +cloud identity. Check the capability you actually need, so any cluster that +advertises it, including an `external` one, passes the same gate. If what +you need isn't a declared capability, the test needs harness machinery, +which is category 3. ### Conformance and ownership @@ -304,10 +328,15 @@ through both runners has to collect the same counts and end with the same result, or the suite fails. The known trap is guarantee 2, "settled": ATS gets there when the Helm release reports installed, atf when the App CR reads `deployed`, and those aren't the same moment. The contract pins the -observable, not the mechanism: settled means the app's own workloads are -Available, and the parity fixture checks that neither runner starts tests -early. `clustertest.wait.IsDeploymentReady` is the shared definition of -ready. +observable, not the mechanism: settled means every workload the release +created is ready, not that a status field flipped. +`clustertest.wait.IsReleaseReady(name, namespace)` is the shared definition: +it selects the release's objects by `app.kubernetes.io/instance` and waits +for Deployments, StatefulSets and DaemonSets to be Available and Jobs to +have succeeded. Each runner still waits on its own mechanism signal (Helm +`installed`, App CR `deployed`) first; `IsReleaseReady` is the common gate +on top, and the parity fixture checks that neither runner starts tests +before it holds. team-tenet owns the contract: this doc, the suite, and the call when the runners disagree. team-honeybadger owns ATS, team-bumblebee owns atf. @@ -318,7 +347,8 @@ is a bug in that runner, not a reason to fork. - **apptest-framework** gets a convention-runner: after the workload cluster and App CR are up, it grabs the WC kubeconfig - (`Framework.GetClusterKubeConfig`), writes it out, exports the env + (`framework.MC().GetClusterKubeConfig(ctx, name, namespace)` on the + clustertest MC client), writes it out, exports the env contract, picks the executor, and runs it per type. For upgrades it adds the pre run. Today it runs the suite once after the upgrade; now it also runs `upgrade` tests against the old version first @@ -328,18 +358,22 @@ is a bug in that runner, not a reason to fork. and pytest tests built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS tests in either language run on workload clusters as-is. - **app-test-suite** exports the `APP_TEST_*` names next to the old `ATS_*` - ones (including `APP_TEST_UPGRADE_STAGE`, which it already has pre/post - runs for), keeps its hook flags and also discovers the convention hooks by - path (stopping if a flag and a file point at the same one), and adds a + ones. It already runs `upgrade` tests both before and after the upgrade, + so `APP_TEST_UPGRADE_STAGE` is a rename of the stage it already tracks + (`ATS_EXTRA_UPGRADE_TEST_STAGE`), with the value normalized to + `pre`/`post`. It keeps its hook flags and also discovers the convention + hooks by path (stopping if a flag and a file point at the same one), and + adds a pre-deploy point for `setup`. Its `--app-tests-pre-hook` runs after deploy, so `setup` is a new call between `_ensure_cluster_prerequisites` and the install. It looks in `tests/app/` as well as today's `tests/ats/`, reads the shared config, and emits junit via gotestsum. Its upgrade pre/post behavior doesn't change. Its TEST_CONTRACT.md becomes a pointer here plus ATS-specific detail. -- **clustertest** gets `wait.IsDeploymentReady(name, namespace)` so both - runners and the atf-native suites share one definition of ready, and the - parity check has one thing to assert against. +- **clustertest** gets `wait.IsReleaseReady(name, namespace)` (release + objects selected by `app.kubernetes.io/instance`, all workload kinds + ready) so both runners and the atf-native suites share one definition of + ready, and the parity check has one thing to assert against. - **the on-demand trigger**: the workload-cluster pipeline runs on `/run` against a PR, not just nightly, so a cloud-path change can get its result without waiting. From f4ca8944a8e19d4e35c5b67aa565a99c16ebe0d7 Mon Sep 17 00:00:00 2001 From: QuentinBisson Date: Tue, 7 Jul 2026 16:22:40 +0200 Subject: [PATCH 15/24] Reuse existing AreAll*Ready for IsReleaseReady, add label scope --- app-testing-contract/README.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index e66dd89..098358c 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -330,10 +330,16 @@ gets there when the Helm release reports installed, atf when the App CR reads `deployed`, and those aren't the same moment. The contract pins the observable, not the mechanism: settled means every workload the release created is ready, not that a status field flipped. -`clustertest.wait.IsReleaseReady(name, namespace)` is the shared definition: -it selects the release's objects by `app.kubernetes.io/instance` and waits -for Deployments, StatefulSets and DaemonSets to be Available and Jobs to -have succeeded. Each runner still waits on its own mechanism signal (Helm +`clustertest.wait.IsReleaseReady(name, namespace)` is the shared definition. +It reuses clustertest's existing `AreAll*Ready` conditions rather than +reimplementing readiness; the only thing missing today is scope, since those +list cluster-wide, so they gain a label-selector argument and +`IsReleaseReady` ANDs them over the release's objects +(`app.kubernetes.io/instance=`): Deployments, StatefulSets and +DaemonSets Available, Jobs succeeded. Scoping matters because a workload +cluster runs far more than the app under test, so an unscoped "all ready" +would both stall on unrelated workloads and make the two runners observe +different sets. Each runner still waits on its own mechanism signal (Helm `installed`, App CR `deployed`) first; `IsReleaseReady` is the common gate on top, and the parity fixture checks that neither runner starts tests before it holds. @@ -370,10 +376,12 @@ is a bug in that runner, not a reason to fork. reads the shared config, and emits junit via gotestsum. Its upgrade pre/post behavior doesn't change. Its TEST_CONTRACT.md becomes a pointer here plus ATS-specific detail. -- **clustertest** gets `wait.IsReleaseReady(name, namespace)` (release - objects selected by `app.kubernetes.io/instance`, all workload kinds - ready) so both runners and the atf-native suites share one definition of - ready, and the parity check has one thing to assert against. +- **clustertest**: the existing `AreAll*Ready` conditions gain an optional + label-selector argument (matching the style of `AreNumNodesReady`, which + already takes `listOptions`), and a thin `wait.IsReleaseReady(name, + namespace)` ANDs them over `app.kubernetes.io/instance=`. No new + readiness logic; both runners and the atf-native suites share one + definition of ready, and the parity check has one thing to assert against. - **the on-demand trigger**: the workload-cluster pipeline runs on `/run` against a PR, not just nightly, so a cloud-path change can get its result without waiting. From d72db9f36c7c5ee9f6cda0dc5ba6834f7bd8d0ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Pi=C4=85tkowski?= Date: Mon, 13 Jul 2026 18:14:30 +0200 Subject: [PATCH 16/24] Add prerequisite controllers to the app-testing contract Fold in the test-time controllers contract so charts that need a running controller (Flux, Argo, External Secrets) to reconcile the CRs they create can declare them once and have any conforming harness bootstrap them. - New "Prerequisite controllers" section: declarative, named, versioned controllers the runner installs before deploy (the declarative sibling of the setup hook); semver via Masterminds/semver v3; per-harness values files listed via a `harness[]` list (name as a field, not a map key); significant install order; reuse-if-present with a version check; install-only lifecycle. - Shared config moves to `.apptest/config.yaml` (code in tests/app/, declarations in .apptest/); controllers + their values files live there. - Runner guarantees: controllers bootstrapped and ready before the setup hook and the app deploy. - Implementation + conformance: both runners parse and bootstrap controllers; the conformance fixture declares one and the parity check covers it. - Adjust the rejected `values:{ats,e2e}` alternative to allow the bounded `harness[]` exception for controller values. Co-Authored-By: Claude Opus 4.8 --- app-testing-contract/README.md | 119 +++++++++++++++++++++++++++------ 1 file changed, 100 insertions(+), 19 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 098358c..7a97ff8 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -6,7 +6,7 @@ owners: - https://github.com/orgs/giantswarm/teams/team-honeybadger - https://github.com/orgs/giantswarm/teams/team-tenet state: review -summary: Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (fast chart tests per PR) and apptest-framework (e2e on workload clusters). One conventional directory per repo, test types via build tags or pytest markers, inputs via KUBECONFIG and APP_TEST_* env vars. +summary: Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (fast chart tests per PR) and apptest-framework (e2e on workload clusters). One conventional directory per repo, test types via build tags or pytest markers, inputs via KUBECONFIG and APP_TEST_* env vars, prerequisite controllers declared in .apptest/config.yaml. --- # The app-testing contract @@ -154,6 +154,69 @@ Anything that needs harness internals (MC access, the App CR, framework state) stays in a harness-native hook: ATS's config hooks for points we don't cover, or atf's `AfterClusterReady` / `BeforeUpgrade`. +### Prerequisite controllers + +Some apps under test create custom resources — a Flux `Kustomization`, an +Argo `Application`, an `ExternalSecret` — that do nothing until a controller +is running to reconcile them. Only some apps need any given controller, and +installing one is expensive, so the app declares the controllers it needs and +the runner bootstraps exactly those before the app is deployed. There's no +auto-detection: declaring is the opt-in, the same rule as everything else +here. + +This is the declarative sibling of the `setup` hook. `setup` runs an +app-specific script; a controller names something the runner already knows how +to install. The declaration is shared, but the provider code that installs a +named controller is each harness's own — a harness targeting kind and one +targeting a workload cluster install it differently — so a controller only +works on a harness that has a provider registered for that name. + +Controllers are declared in the shared `.apptest/config.yaml` (see Shared +configuration): + +```yaml +controllers: + - name: flux + semver: ">=2.0.0 <3.0.0" + harness: + - name: ats + valuesFile: flux-small.yaml + - name: atf + valuesFile: flux-full.yaml + - name: external-secrets + semver: "0.x" +``` + +- `name` (required): the controller's harness-neutral id. If the running + harness has no provider registered for it, the run fails. +- `semver` (required): a version range with Masterminds/semver v3 semantics + (the same Flux `OCIRepository` and Helm `--version` use), resolved to the + highest version that satisfies it. +- `harness` (optional): per-harness install values files, listed by harness + name rather than keyed by it, so the neutral file stays a list you extend, + not a map with harness names baked into its shape. Each `valuesFile` ends in + `.yaml`, sits beside `config.yaml` under `.apptest/`, and layers over the + controller's defaults. No entry for the running harness means defaults; a + named file that's missing or not `.yaml` fails the run. + +Order matters: controllers install in list order, each fully ready before the +next, so one that depends on another goes after it. They install once per run +and are shared across every type and both flows — a prerequisite is run +infrastructure, not something per test. + +An already-present controller is reused. The runner checks the installed +version: absent, it installs; present and within `semver`, it leaves it alone; +present but outside `semver`, the run fails. The runner never upgrades, +downgrades, or removes a controller it finds — the cluster may not be ours, +and leaving controllers in place is also what makes the next run on the same +cluster faster. A provider may run its own pre-install and post-install steps +(create RBAC, wait for a webhook or a CRD to establish) around the install. + +Controllers are not capabilities. `APP_TEST_CAPABILITIES` is what a cluster +already provides and a test gates on; a controller is something the runner +adds to any cluster. When a bootstrapped controller is a gitops engine, the +runner surfaces which one through `APP_TEST_EXTRA_GITOPS_ENGINE`. + ### Inputs Tests get everything from the environment. They don't provision anything: @@ -201,14 +264,16 @@ stays as-is; it's the standard name, not ours. Before the tests run, a conforming runner makes sure: -1. the `setup` hook ran, if present, after the cluster was ready and before - the app was deployed, -2. the app is deployed and settled: each runner first waits on its own +1. the controllers declared in `.apptest/config.yaml`, if any, are + bootstrapped and ready, once per run, before anything is deployed, +2. the `setup` hook ran, if present, after the controllers were ready and + before the app was deployed, +3. the app is deployed and settled: each runner first waits on its own mechanism signal (ATS: Helm release installed, or via a GitOps engine; atf: App CR at `deployed`), then on the shared gate `IsReleaseReady` (the release's workloads Available) before any test runs, -3. the required variables are exported, -4. the `teardown` hook runs, if present, after the last test type and before +4. the required variables are exported, +5. the `teardown` hook runs, if present, after the last test type and before the harness's own teardown. Normal flow: run `smoke`, then `functional`. Upgrade tests don't run here. @@ -222,7 +287,7 @@ and let it settle, run `upgrade` tests with `APP_TEST_UPGRADE_STAGE=pre`, run th build tags; pytest exits 5), since a repo may only have some types. It isn't silent, though: the runner reports how many tests it collected per type, so a mistyped tag (also zero) shows up instead of going green. Repos that want -it strict list their expected types in `config.yaml`; a listed type with +it strict list their expected types in `.apptest/config.yaml`; a listed type with zero tests fails. Results come out as junit XML (`gotestsum --junitfile`, `pytest --junitxml`). @@ -251,11 +316,14 @@ nightly WC or a provided cluster that happens to have it. ### Shared configuration -`tests/app/config.yaml` holds only what both harnesses need: +Test *code* lives in `tests/app/`; shared *declarations* live in `.apptest/`. +`.apptest/config.yaml` holds only what both harnesses need, and the controller +values files (see Prerequisite controllers) sit beside it: ```yaml installNamespace: kube-system expectedTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test +controllers: [ ... ] # optional: see Prerequisite controllers ``` The upgrade flow is inferred, not configured: if the runner collects any @@ -315,17 +383,18 @@ which is category 3. ### Conformance and ownership Two runners, one contract, so they'll drift unless something checks. The -contract ships a conformance suite: a fixture `tests/app/` (trivial app, one -test per type, a hook, a lockfile) and assertions on the env vars, ordering, -exit codes, and lints above. A runner conforms only if it passes the suite -in CI; ATS and atf both wire it in. New guarantees go into the suite in the +contract ships a conformance suite: a fixture (trivial app, one test per +type, a hook, a declared controller, a lockfile) and assertions on the env +vars, ordering, exit codes, controller bootstrap, and lints above. A runner conforms only +if it passes the suite in CI; ATS and atf both wire it in. New guarantees go into the suite in the same PR that adds them here. Passing per runner isn't enough: both can pass and still disagree on what a test sees, which is the drift that hurts (a smoke test that's green on PR and flaky at night). So the suite also checks parity: the same fixture -through both runners has to collect the same counts and end with the same -result, or the suite fails. The known trap is guarantee 2, "settled": ATS +through both runners has to bootstrap the same controllers, collect the same +counts, and end with the same result, or the suite fails. The known trap is +guarantee 2, "settled": ATS gets there when the Helm release reports installed, atf when the App CR reads `deployed`, and those aren't the same moment. The contract pins the observable, not the mechanism: settled means every workload the release @@ -373,8 +442,8 @@ is a bug in that runner, not a reason to fork. pre-deploy point for `setup`. Its `--app-tests-pre-hook` runs after deploy, so `setup` is a new call between `_ensure_cluster_prerequisites` and the install. It looks in `tests/app/` as well as today's `tests/ats/`, - reads the shared config, and emits junit via gotestsum. Its upgrade - pre/post behavior doesn't change. Its TEST_CONTRACT.md becomes a pointer + reads the shared `.apptest/config.yaml`, and emits junit via gotestsum. Its + upgrade pre/post behavior doesn't change. Its TEST_CONTRACT.md becomes a pointer here plus ATS-specific detail. - **clustertest**: the existing `AreAll*Ready` conditions gain an optional label-selector argument (matching the style of `AreNumNodesReady`, which @@ -382,6 +451,14 @@ is a bug in that runner, not a reason to fork. namespace)` ANDs them over `app.kubernetes.io/instance=`. No new readiness logic; both runners and the atf-native suites share one definition of ready, and the parity check has one thing to assert against. +- **controllers**: both runners parse `.apptest/config.yaml`'s `controllers` + and bootstrap them before deploy — detect, install the `semver`-selected + version if absent, fail if a present one is out of range, wait until ready. + The declaration is shared; the provider that installs a given controller + name is per-harness. ATS lands the provider framework first and syncs its + existing providers in after, so until then a declared controller fails as + "unknown controller", which is the contract's behaviour for an unregistered + name. - **the on-demand trigger**: the workload-cluster pipeline runs on `/run` against a PR, not just nightly, so a cloud-path change can get its result without waiting. @@ -403,9 +480,13 @@ is a bug in that runner, not a reason to fork. but requiring it would shut out the pytest repos and tie the contract to a framework for nothing. We standardize selection and inputs, not the framework. -- **A `values: {ats: ..., e2e: ...}` map in the shared config.** No: it puts - harness names in the neutral file, which is the config version of gating a - test on the harness. +- **A `values: {ats: ..., e2e: ...}` map for the app's deploy values.** No: + the app's values are harness-specific (a kind and a workload cluster want + different ones) and stay in each harness's own config, not the neutral file + — that would be the config version of gating a test on the harness. The + `controllers` section is the bounded exception: it names per-harness values + files, but as a `harness` list (the name is a field, not a map key) and only + for runner-bootstrapped infrastructure, not the app. - **One runner with two modes instead of two behind a contract.** One runner covering kind and workload clusters wouldn't need a contract at all. But the two modes line up with two mature codebases owned by two teams From ae465addc9856fa331393c80fa238765b1ce4b86 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Pi=C4=85tkowski?= Date: Tue, 14 Jul 2026 16:46:15 +0200 Subject: [PATCH 17/24] intro rewritten --- app-testing-contract/README.md | 676 ++++++++++++++------------------- 1 file changed, 295 insertions(+), 381 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 7a97ff8..0556e33 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -2,97 +2,91 @@ creation_date: 2026-07-07 issues: [] owners: -- https://github.com/orgs/giantswarm/teams/team-bumblebee -- https://github.com/orgs/giantswarm/teams/team-honeybadger -- https://github.com/orgs/giantswarm/teams/team-tenet + - https://github.com/orgs/giantswarm/teams/team-bumblebee + - https://github.com/orgs/giantswarm/teams/team-honeybadger + - https://github.com/orgs/giantswarm/teams/team-tenet state: review -summary: Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (fast chart tests per PR) and apptest-framework (e2e on workload clusters). One conventional directory per repo, test types via build tags or pytest markers, inputs via KUBECONFIG and APP_TEST_* env vars, prerequisite controllers declared in .apptest/config.yaml. +summary: + Defines a harness-neutral contract for app tests so the same test files run under both app-test-suite (fast + chart tests per PR) and apptest-framework (e2e on workload clusters). One conventional directory per repo, + test types via build tags or pytest markers, inputs via KUBECONFIG and APP_TEST_* env vars, prerequisite + controllers declared in .apptest/config.yaml. --- # The app-testing contract ## Problem -We test managed apps two ways, and we want to keep both: - -- **app-test-suite (ATS)** installs the chart on a provided cluster (today - usually a local kind cluster) and runs quick checks on every PR. ATS is - dropping its built-in kind lifecycle - ([giantswarm/app-test-suite#675](https://github.com/giantswarm/app-test-suite/pull/675)), - so the cluster it runs against is increasingly whatever CI hands it. -- **apptest-framework (atf)** creates a real workload cluster, installs the - App CR, and runs the full suite nightly. - -A local kind cluster is fast but can't do cloud identity, real storage, or -upgrades; a workload cluster can. So the two are a fast/slow pair, not -duplicates. - -The trouble is writing the tests. ATS wants pytest or plain Go; atf wants -Ginkgo. To cover both you write the same check twice in two styles, so most -repos write it for one harness or skip it. Almost nobody has both, and the -people stuck on one aren't happy about it. - -This RFC doesn't dedupe existing tests; there aren't many to dedupe, for -the reason above. It makes the two harnesses agree on how tests are written -and run, so you write a check once and both run it: what the PR cluster -supports on every PR, everything nightly. - -ATS already has a test contract -([docs/TEST_CONTRACT.md](https://github.com/giantswarm/app-test-suite/blob/main/docs/TEST_CONTRACT.md)), -but it lives in the ATS repo and only ATS follows it. We lift it out, make -it harness-neutral, and make atf follow it too. It stays independent of any -test framework or language. +We test our helm charts delivering managed apps using two frameworks: `app-test-suite` (ATS) and +`apptest-framework` (ATF). So far, it was not clear which framework teams should use and how to write the +tests. We propose to keep both frameworks, but specialize them to the two most frequent use cases: + +- **app-test-suite (ATS)** - should provide rapid local feedback on PRs and in local dev environments. Its + goal is to deploy the helm chart under tests as fast as possible and start testing the chart's + functionality. This means that it will sacrifice all the possible real cluster features to achieve this + goal. The cluster environment it is meant to run is `kind`, although it doesn't make any assumption about + the cluster type. +- **apptest-framework (ATF)** takes the opposite approach: it chooses environment realism over the time needed + to execute the tests. It creates a real workload cluster, installs the chart using the App Platform, and + runs the full suite, preferably nightly. + +With this in mind, it's clear that to provide a comprehensive test coverage and to follow the "fail fast" +principle, we need to use both frameworks. + +The trouble is writing the tests. The tests author doesn't want to implement the same or very similar set of +tests twice, once for each framework. The goal of this doc is to propose a test implementation spec that will +allow both ATS and ATF to run the same test suites in their respective environments. Full parity of tests +might not be possible, as test itself might depend on the environment, but we want to get as close as +possible. A local kind cluster is fast but can't do cloud identity, real storage, or upgrades; a workload +cluster can. If a test suite needs to be aware of these differences, the author needs to get the possibility +to gate on the capabilities it needs. + +This goal of this RFC is not to dedupe existing tests. It's to define how the two frameworks are different, +what environments they provide and what is the convention that, when respected, can allow to fit both +frameworks with the same test code. ## Decision ### The conventional directory -Tests live in one directory: `tests/app/`. Any harness that deploys the app -runs that directory the same way. Having the directory is the opt-in; -there's nothing else to wire up. +Tests live in one directory: `tests/app/`. Any harness that deploys the app runs that directory the same way. +Having the directory is the opt-in; there's nothing else to wire up. -It's either one Go module or one Python project, not both. The runner picks -the executor from what's there: +It's either one Go module or one Python project, not both. The runner picks the executor from what's there: - `go.mod`: `go test -mod=readonly -tags=` - `pyproject.toml`: `uv sync --frozen && uv run pytest -m ` - both, or neither in a non-empty directory: config error, stop. -Dependencies are pinned from committed lockfiles (Go `go.sum`, Python -`uv.lock`) and installed with the network off: the runner sets `GOPROXY=off` -alongside `-mod=readonly` (or vendors `tests/app/vendor/`) and runs -`uv sync --frozen --offline`, backed by a pre-warmed module/uv cache. The -lockfile flags alone (`-mod=readonly`, `--frozen`) only freeze resolution, -not fetching, so cutting the network is what makes a missing or stale entry -fail loudly instead of quietly downloading. Both runners implement this: -under this contract `tests/app/` is a separate module that atf builds at -test time, so atf gains the same fetch point ATS already has. Same -dependency set everywhere, and you can audit it. +Dependencies are pinned from committed lockfiles (Go `go.sum`, Python `uv.lock`) and installed with the +network off: the runner sets `GOPROXY=off` alongside `-mod=readonly` (or vendors `tests/app/vendor/`) and runs +`uv sync --frozen --offline`, backed by a pre-warmed module/uv cache. The lockfile flags alone +(`-mod=readonly`, `--frozen`) only freeze resolution, not fetching, so cutting the network is what makes a +missing or stale entry fail loudly instead of quietly downloading. Both runners implement this: under this +contract `tests/app/` is a separate module that atf builds at test time, so atf gains the same fetch point ATS +already has. Same dependency set everywhere, and you can audit it. -An empty `tests/app/` (no module, no project) isn't an opt-in and is -skipped. +An empty `tests/app/` (no module, no project) isn't an opt-in and is skipped. ### Test types -Each test carries one type, set with a Go build tag or a pytest marker. -There are three, the same ones ATS already has: +Each test carries one type, set with a Go build tag or a pytest marker. There are three, the same ones ATS +already has: -| Type | Runs | What it is | -|---|---|---| -| `smoke` | normal flow, first | quick sanity checks | -| `functional` | normal flow, after smoke | full feature tests | -| `upgrade` | upgrade flow, before and after | checks the app still works across an upgrade | +| Type | Runs | What it is | +| ------------ | ------------------------------ | -------------------------------------------- | +| `smoke` | normal flow, first | quick sanity checks | +| `functional` | normal flow, after smoke | full feature tests | +| `upgrade` | upgrade flow, before and after | checks the app still works across an upgrade | -`upgrade` is its own type, not a flag on the others. The upgrade flow -doesn't re-run smoke and functional; it runs the `upgrade` tests, once on -the old version (`APP_TEST_UPGRADE_STAGE=pre`) and once after upgrading -(`=post`). The pre run is the baseline: if it passes and post fails, the -upgrade caused it, not something that was already broken. +`upgrade` is its own type, not a flag on the others. The upgrade flow doesn't re-run smoke and functional; it +runs the `upgrade` tests, once on the old version (`APP_TEST_UPGRADE_STAGE=pre`) and once after upgrading +(`=post`). The pre run is the baseline: if it passes and post fails, the upgrade caused it, not something that +was already broken. -Most upgrade checks are symmetric ("the app answers") and assert the same -thing both times. When some state has to survive the upgrade, either seed it -in the `pre-upgrade` hook and check it in the post run, or keep it in one -test that branches on the stage: +Most upgrade checks are symmetric ("the app answers") and assert the same thing both times. When some state +has to survive the upgrade, either seed it in the `pre-upgrade` hook and check it in the post run, or keep it +in one test that branches on the stage: ```go if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { @@ -100,79 +94,66 @@ if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { } ``` -Seeding is a side effect, so it's a hook; checking is an assertion, so it's -a test. Upgrade tests and the `pre-upgrade` hook also get -`APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. +Seeding is a side effect, so it's a hook; checking is an assertion, so it's a test. Upgrade tests and the +`pre-upgrade` hook also get `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. ### Hooks -Hooks do things with side effects (install a prerequisite, create a pod, -clean up); tests check things. Splitting them is what lets the upgrade flow -seed state without re-running a suite. Like tests, setup and teardown are -per-app and get duplicated across harnesses, so the contract makes them -portable too: optional executables in `tests/app/`, run with the same -environment as tests. +Hooks do things with side effects (install a prerequisite, create a pod, clean up); tests check things. +Splitting them is what lets the upgrade flow seed state without re-running a suite. Like tests, setup and +teardown are per-app and get duplicated across harnesses, so the contract makes them portable too: optional +executables in `tests/app/`, run with the same environment as tests. -| Hook | Runs | -|---|---| -| `tests/app/hooks/setup` | after the cluster is ready, before the app is deployed (for example: install prerequisites) | +| Hook | Runs | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `tests/app/hooks/setup` | after the cluster is ready, before the app is deployed (for example: install prerequisites) | | `tests/app/hooks/pre-upgrade` | upgrade flow only: after the previous version is deployed, before the upgrade (for example: create a pod or write a record an `upgrade` test then checks survived) | -| `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | - -A hook is any executable at that path: a script with a shebang or a built -binary, run directly (not sourced). It runs out of process, so the -one-language rule doesn't apply and you can write it in whatever fits. Keep -it small; anything bigger is a test or a harness-native hook. - -A missing hook does nothing. A non-zero exit fails the run. Hooks gate on -the environment like category-2 tests do; `pre-upgrade` also gets the -from/to versions. - -Convention discovery of those three paths is the default, and every runner -has to implement it. That's the zero-wiring part. A harness can also keep -its own hook flags, so existing repos don't have to move and -harness-specific hooks still work. If a flag and a convention file point at -the same contract hook, the runner stops (same as finding both `go.mod` and -`pyproject.toml`), so migrating is "add the file, drop the flag" in one -commit rather than running both. Only the convention path is guaranteed and -checked by the conformance suite; the flags are each harness's own business. +| `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | + +A hook is any executable at that path: a script with a shebang or a built binary, run directly (not sourced). +It runs out of process, so the one-language rule doesn't apply and you can write it in whatever fits. Keep it +small; anything bigger is a test or a harness-native hook. + +A missing hook does nothing. A non-zero exit fails the run. Hooks gate on the environment like category-2 +tests do; `pre-upgrade` also gets the from/to versions. + +Convention discovery of those three paths is the default, and every runner has to implement it. That's the +zero-wiring part. A harness can also keep its own hook flags, so existing repos don't have to move and +harness-specific hooks still work. If a flag and a convention file point at the same contract hook, the runner +stops (same as finding both `go.mod` and `pyproject.toml`), so migrating is "add the file, drop the flag" in +one commit rather than running both. Only the convention path is guaranteed and checked by the conformance +suite; the flags are each harness's own business. How the three points map today: -| Contract hook | ATS | atf | -|---|---|---| -| `setup` (before deploy) | new pre-deploy point (its `--app-tests-pre-hook` fires after deploy) | `AfterClusterReady` (runs before install) | -| `pre-upgrade` | `--upgrade-tests-upgrade-hook` at `ATS_HOOK_STAGE=pre_upgrade` | `BeforeUpgrade` | -| `teardown` (after tests) | `--app-tests-post-hook` | suite callback | +| Contract hook | ATS | atf | +| ------------------------ | -------------------------------------------------------------------- | ----------------------------------------- | +| `setup` (before deploy) | new pre-deploy point (its `--app-tests-pre-hook` fires after deploy) | `AfterClusterReady` (runs before install) | +| `pre-upgrade` | `--upgrade-tests-upgrade-hook` at `ATS_HOOK_STAGE=pre_upgrade` | `BeforeUpgrade` | +| `teardown` (after tests) | `--app-tests-post-hook` | suite callback | -A runner covers each point with either the file or the flag, not both. -Anything not in that table (ATS's `post_upgrade` stage, its pre/post test -hooks) stays harness-native. +A runner covers each point with either the file or the flag, not both. Anything not in that table (ATS's +`post_upgrade` stage, its pre/post test hooks) stays harness-native. -Same boundary as tests: a hook only gets the app cluster's `KUBECONFIG`. -Anything that needs harness internals (MC access, the App CR, framework -state) stays in a harness-native hook: ATS's config hooks for points we +Same boundary as tests: a hook only gets the app cluster's `KUBECONFIG`. Anything that needs harness internals +(MC access, the App CR, framework state) stays in a harness-native hook: ATS's config hooks for points we don't cover, or atf's `AfterClusterReady` / `BeforeUpgrade`. ### Prerequisite controllers -Some apps under test create custom resources — a Flux `Kustomization`, an -Argo `Application`, an `ExternalSecret` — that do nothing until a controller -is running to reconcile them. Only some apps need any given controller, and -installing one is expensive, so the app declares the controllers it needs and -the runner bootstraps exactly those before the app is deployed. There's no -auto-detection: declaring is the opt-in, the same rule as everything else -here. +Some apps under test create custom resources — a Flux `Kustomization`, an Argo `Application`, an +`ExternalSecret` — that do nothing until a controller is running to reconcile them. Only some apps need any +given controller, and installing one is expensive, so the app declares the controllers it needs and the runner +bootstraps exactly those before the app is deployed. There's no auto-detection: declaring is the opt-in, the +same rule as everything else here. -This is the declarative sibling of the `setup` hook. `setup` runs an -app-specific script; a controller names something the runner already knows how -to install. The declaration is shared, but the provider code that installs a -named controller is each harness's own — a harness targeting kind and one -targeting a workload cluster install it differently — so a controller only -works on a harness that has a provider registered for that name. +This is the declarative sibling of the `setup` hook. `setup` runs an app-specific script; a controller names +something the runner already knows how to install. The declaration is shared, but the provider code that +installs a named controller is each harness's own — a harness targeting kind and one targeting a workload +cluster install it differently — so a controller only works on a harness that has a provider registered for +that name. -Controllers are declared in the shared `.apptest/config.yaml` (see Shared -configuration): +Controllers are declared in the shared `.apptest/config.yaml` (see Shared configuration): ```yaml controllers: @@ -187,310 +168,243 @@ controllers: semver: "0.x" ``` -- `name` (required): the controller's harness-neutral id. If the running - harness has no provider registered for it, the run fails. -- `semver` (required): a version range with Masterminds/semver v3 semantics - (the same Flux `OCIRepository` and Helm `--version` use), resolved to the - highest version that satisfies it. -- `harness` (optional): per-harness install values files, listed by harness - name rather than keyed by it, so the neutral file stays a list you extend, - not a map with harness names baked into its shape. Each `valuesFile` ends in - `.yaml`, sits beside `config.yaml` under `.apptest/`, and layers over the - controller's defaults. No entry for the running harness means defaults; a - named file that's missing or not `.yaml` fails the run. - -Order matters: controllers install in list order, each fully ready before the -next, so one that depends on another goes after it. They install once per run -and are shared across every type and both flows — a prerequisite is run -infrastructure, not something per test. - -An already-present controller is reused. The runner checks the installed -version: absent, it installs; present and within `semver`, it leaves it alone; -present but outside `semver`, the run fails. The runner never upgrades, -downgrades, or removes a controller it finds — the cluster may not be ours, -and leaving controllers in place is also what makes the next run on the same -cluster faster. A provider may run its own pre-install and post-install steps -(create RBAC, wait for a webhook or a CRD to establish) around the install. - -Controllers are not capabilities. `APP_TEST_CAPABILITIES` is what a cluster -already provides and a test gates on; a controller is something the runner -adds to any cluster. When a bootstrapped controller is a gitops engine, the -runner surfaces which one through `APP_TEST_EXTRA_GITOPS_ENGINE`. +- `name` (required): the controller's harness-neutral id. If the running harness has no provider registered + for it, the run fails. +- `semver` (required): a version range with Masterminds/semver v3 semantics (the same Flux `OCIRepository` and + Helm `--version` use), resolved to the highest version that satisfies it. +- `harness` (optional): per-harness install values files, listed by harness name rather than keyed by it, so + the neutral file stays a list you extend, not a map with harness names baked into its shape. Each + `valuesFile` ends in `.yaml`, sits beside `config.yaml` under `.apptest/`, and layers over the controller's + defaults. No entry for the running harness means defaults; a named file that's missing or not `.yaml` fails + the run. + +Order matters: controllers install in list order, each fully ready before the next, so one that depends on +another goes after it. They install once per run and are shared across every type and both flows — a +prerequisite is run infrastructure, not something per test. + +An already-present controller is reused. The runner checks the installed version: absent, it installs; present +and within `semver`, it leaves it alone; present but outside `semver`, the run fails. The runner never +upgrades, downgrades, or removes a controller it finds — the cluster may not be ours, and leaving controllers +in place is also what makes the next run on the same cluster faster. A provider may run its own pre-install +and post-install steps (create RBAC, wait for a webhook or a CRD to establish) around the install. + +Controllers are not capabilities. `APP_TEST_CAPABILITIES` is what a cluster already provides and a test gates +on; a controller is something the runner adds to any cluster. When a bootstrapped controller is a gitops +engine, the runner surfaces which one through `APP_TEST_EXTRA_GITOPS_ENGINE`. ### Inputs -Tests get everything from the environment. They don't provision anything: -no clusters, no chart installs, no App CRs. - -| Variable | Required | Meaning | -|---|---|---| -| `KUBECONFIG` | yes | kubeconfig of the cluster the app is deployed on (never the MC) | -| `APP_TEST_TYPE` | yes | the type currently being run | -| `APP_TEST_RELEASE_NAME` | yes | Helm release name of the app under test | -| `APP_TEST_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | -| `APP_TEST_CHART_VERSION` | yes | version of the chart under test | -| `APP_TEST_CLUSTER_TYPE` | yes | topology of the cluster the app runs on: `kind` (local single-node), `capi` (a CAPI workload cluster), or `external` (a cluster the runner did not provision). Describes shape, not capability; gate on `APP_TEST_CAPABILITIES` instead | -| `APP_TEST_CAPABILITIES` | yes | comma-separated capabilities the cluster actually provides, e.g. `cloud-identity,persistent-storage,load-balancer`; empty is valid. The runner sets it from what it provisioned or was handed. This is what a test gates on | -| `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | -| `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | -| `APP_TEST_UPGRADE_STAGE` | upgrade flow only | `pre` or `post`: which side of the upgrade this `upgrade`-type run is on | -| `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade flow only | versions on either side of the upgrade; available to the `pre-upgrade` hook and `upgrade`-typed tests | -| `APP_TEST_EXTRA_*` | optional | harness extras, for example `APP_TEST_EXTRA_GITOPS_ENGINE` | - -The prefix is `APP_TEST_`, which reads the same under either harness. ATS -publishes these under the old `ATS_` prefix today. Where a name maps -straight across (`ATS_X` to `APP_TEST_X`, e.g. `ATS_RELEASE_NAME`, -`ATS_CHART_VERSION`, `ATS_CLUSTER_TYPE`, `ATS_EXTRA_*`) the runner exports -both, so nothing breaks and new tests use `APP_TEST_`. Dual export isn't -free (two names to know and grep for), so `ATS_` is deprecated and will be -removed in a later change once repos have migrated. - -Four names are renamed, because the old ones were unclear or, for the -upgrade stage, were never test-facing under one name to begin with: - -| Legacy | Canonical | Note | -|---|---|---| -| `ATS_TEST_TYPE` | `APP_TEST_TYPE` | straight rename | -| `ATS_APP_CONFIG_FILE_PATH` | `APP_TEST_VALUES_FILE` | straight rename | -| `ATS_CLUSTER_VERSION` | `APP_TEST_KUBERNETES_VERSION` | straight rename | -| `ATS_EXTRA_UPGRADE_TEST_STAGE` (tests) / `ATS_HOOK_STAGE` (hooks) | `APP_TEST_UPGRADE_STAGE` | value also changes: `pre_upgrade`/`post_upgrade` become `pre`/`post`. The runner does not alias the value, so existing upgrade tests reading the old one must update | - -`ATS_CHART_PATH` and `ATS_TEST_DIR` have no `APP_TEST_` equivalent; the -contract doesn't expose them and they stay ATS-only. `APP_TEST_CAPABILITIES` -is new, computed by the runner, with no `ATS_` predecessor. `KUBECONFIG` -stays as-is; it's the standard name, not ours. +Tests get everything from the environment. They don't provision anything: no clusters, no chart installs, no +App CRs. + +| Variable | Required | Meaning | +| --------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `KUBECONFIG` | yes | kubeconfig of the cluster the app is deployed on (never the MC) | +| `APP_TEST_TYPE` | yes | the type currently being run | +| `APP_TEST_RELEASE_NAME` | yes | Helm release name of the app under test | +| `APP_TEST_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | +| `APP_TEST_CHART_VERSION` | yes | version of the chart under test | +| `APP_TEST_CLUSTER_TYPE` | yes | topology of the cluster the app runs on: `kind` (local single-node), `capi` (a CAPI workload cluster), or `external` (a cluster the runner did not provision). Describes shape, not capability; gate on `APP_TEST_CAPABILITIES` instead | +| `APP_TEST_CAPABILITIES` | yes | comma-separated capabilities the cluster actually provides, e.g. `cloud-identity,persistent-storage,load-balancer`; empty is valid. The runner sets it from what it provisioned or was handed. This is what a test gates on | +| `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | +| `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | +| `APP_TEST_UPGRADE_STAGE` | upgrade flow only | `pre` or `post`: which side of the upgrade this `upgrade`-type run is on | +| `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade flow only | versions on either side of the upgrade; available to the `pre-upgrade` hook and `upgrade`-typed tests | +| `APP_TEST_EXTRA_*` | optional | harness extras, for example `APP_TEST_EXTRA_GITOPS_ENGINE` | + +The prefix is `APP_TEST_`, which reads the same under either harness. ATS publishes these under the old `ATS_` +prefix today. Where a name maps straight across (`ATS_X` to `APP_TEST_X`, e.g. `ATS_RELEASE_NAME`, +`ATS_CHART_VERSION`, `ATS_CLUSTER_TYPE`, `ATS_EXTRA_*`) the runner exports both, so nothing breaks and new +tests use `APP_TEST_`. Dual export isn't free (two names to know and grep for), so `ATS_` is deprecated and +will be removed in a later change once repos have migrated. + +Four names are renamed, because the old ones were unclear or, for the upgrade stage, were never test-facing +under one name to begin with: + +| Legacy | Canonical | Note | +| ----------------------------------------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ATS_TEST_TYPE` | `APP_TEST_TYPE` | straight rename | +| `ATS_APP_CONFIG_FILE_PATH` | `APP_TEST_VALUES_FILE` | straight rename | +| `ATS_CLUSTER_VERSION` | `APP_TEST_KUBERNETES_VERSION` | straight rename | +| `ATS_EXTRA_UPGRADE_TEST_STAGE` (tests) / `ATS_HOOK_STAGE` (hooks) | `APP_TEST_UPGRADE_STAGE` | value also changes: `pre_upgrade`/`post_upgrade` become `pre`/`post`. The runner does not alias the value, so existing upgrade tests reading the old one must update | + +`ATS_CHART_PATH` and `ATS_TEST_DIR` have no `APP_TEST_` equivalent; the contract doesn't expose them and they +stay ATS-only. `APP_TEST_CAPABILITIES` is new, computed by the runner, with no `ATS_` predecessor. +`KUBECONFIG` stays as-is; it's the standard name, not ours. ### Runner guarantees Before the tests run, a conforming runner makes sure: -1. the controllers declared in `.apptest/config.yaml`, if any, are - bootstrapped and ready, once per run, before anything is deployed, -2. the `setup` hook ran, if present, after the controllers were ready and - before the app was deployed, -3. the app is deployed and settled: each runner first waits on its own - mechanism signal (ATS: Helm release installed, or via a GitOps engine; - atf: App CR at `deployed`), then on the shared gate `IsReleaseReady` (the - release's workloads Available) before any test runs, +1. the controllers declared in `.apptest/config.yaml`, if any, are bootstrapped and ready, once per run, + before anything is deployed, +2. the `setup` hook ran, if present, after the controllers were ready and before the app was deployed, +3. the app is deployed and settled: each runner first waits on its own mechanism signal (ATS: Helm release + installed, or via a GitOps engine; atf: App CR at `deployed`), then on the shared gate `IsReleaseReady` + (the release's workloads Available) before any test runs, 4. the required variables are exported, -5. the `teardown` hook runs, if present, after the last test type and before - the harness's own teardown. +5. the `teardown` hook runs, if present, after the last test type and before the harness's own teardown. Normal flow: run `smoke`, then `functional`. Upgrade tests don't run here. -Upgrade flow (any `upgrade` tests collected): install the previous version -and let it settle, run `upgrade` tests with `APP_TEST_UPGRADE_STAGE=pre`, run the -`pre-upgrade` hook, upgrade and let it settle, run `upgrade` tests with -`=post`. smoke and functional don't run here. +Upgrade flow (any `upgrade` tests collected): install the previous version and let it settle, run `upgrade` +tests with `APP_TEST_UPGRADE_STAGE=pre`, run the `pre-upgrade` hook, upgrade and let it settle, run `upgrade` +tests with `=post`. smoke and functional don't run here. -"No tests of this type" passes rather than fails (Go excludes all files via -build tags; pytest exits 5), since a repo may only have some types. It isn't -silent, though: the runner reports how many tests it collected per type, so -a mistyped tag (also zero) shows up instead of going green. Repos that want -it strict list their expected types in `.apptest/config.yaml`; a listed type with -zero tests fails. Results come out as junit XML (`gotestsum --junitfile`, -`pytest --junitxml`). +"No tests of this type" passes rather than fails (Go excludes all files via build tags; pytest exits 5), since +a repo may only have some types. It isn't silent, though: the runner reports how many tests it collected per +type, so a mistyped tag (also zero) shows up instead of going green. Repos that want it strict list their +expected types in `.apptest/config.yaml`; a listed type with zero tests fails. Results come out as junit XML +(`gotestsum --junitfile`, `pytest --junitxml`). ### Cadence and feedback latency -The PR cluster often lacks cloud identity, real storage, or load balancers, -so tests that need those capabilities only run nightly on a workload -cluster. Their result isn't tied to the PR that caused it: a PR can break a -cloud-only path, pass PR CI, and fail that night against a batch of other -commits. +The PR cluster often lacks cloud identity, real storage, or load balancers, so tests that need those +capabilities only run nightly on a workload cluster. Their result isn't tied to the PR that caused it: a PR +can break a cloud-only path, pass PR CI, and fail that night against a batch of other commits. We accept that, but two things keep it from being a silent trap: -1. A nightly-only test is a choice you can see. A test that gates on a - capability the PR cluster lacks (category 2) is invisible per-PR by - design; the skip shows by name and the collected counts show it didn't +1. A nightly-only test is a choice you can see. A test that gates on a capability the PR cluster lacks + (category 2) is invisible per-PR by design; the skip shows by name and the collected counts show it didn't run, so it doesn't read as coverage it isn't. -2. You can pull the nightly flow forward. `/run` triggers the - workload-cluster flow on a PR, so a cloud-path change can get its result - now instead of that night. +2. You can pull the nightly flow forward. `/run` triggers the workload-cluster flow on a PR, so a cloud-path + change can get its result now instead of that night. -Gate on the capability you need (`APP_TEST_CAPABILITIES`), never on cluster -type or harness (see Harness-specific tests): a test that needs cloud -identity runs anywhere advertising `cloud-identity`, whether that's the -nightly WC or a provided cluster that happens to have it. +Gate on the capability you need (`APP_TEST_CAPABILITIES`), never on cluster type or harness (see +Harness-specific tests): a test that needs cloud identity runs anywhere advertising `cloud-identity`, whether +that's the nightly WC or a provided cluster that happens to have it. ### Shared configuration -Test *code* lives in `tests/app/`; shared *declarations* live in `.apptest/`. -`.apptest/config.yaml` holds only what both harnesses need, and the controller -values files (see Prerequisite controllers) sit beside it: +Test _code_ lives in `tests/app/`; shared _declarations_ live in `.apptest/`. `.apptest/config.yaml` holds +only what both harnesses need, and the controller values files (see Prerequisite controllers) sit beside it: ```yaml installNamespace: kube-system -expectedTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test -controllers: [ ... ] # optional: see Prerequisite controllers +expectedTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test +controllers: [...] # optional: see Prerequisite controllers ``` -The upgrade flow is inferred, not configured: if the runner collects any -`upgrade`-typed tests it runs the upgrade flow, otherwise it doesn't. Same -presence-is-the-opt-in rule as the directory and the other types, so there's -no separate switch to keep in sync. Each harness still learns which version -to upgrade from through its own config (ATS's stable-app settings, atf's -latest published release); that part is harness-specific, not contract. - -The one lint: a type in `expectedTypes` that collects zero tests fails the -run. `expectedTypes` is optional; leave it out to keep the "no tests is -fine" default. It's also how you make a type mandatory. List `upgrade`, and -a typo'd tag (which collects zero) fails instead of quietly skipping the -flow. - -Everything harness-specific stays in that harness's config: `.ats/main.yaml` -(cluster types, catalogs, executor options) and `tests/e2e/config.yaml` -(appCatalog, providers, MC options). Values files stay per-harness too; a -kind cluster and a workload cluster legitimately want different values, and -each harness loads them its own way. +The upgrade flow is inferred, not configured: if the runner collects any `upgrade`-typed tests it runs the +upgrade flow, otherwise it doesn't. Same presence-is-the-opt-in rule as the directory and the other types, so +there's no separate switch to keep in sync. Each harness still learns which version to upgrade from through +its own config (ATS's stable-app settings, atf's latest published release); that part is harness-specific, not +contract. + +The one lint: a type in `expectedTypes` that collects zero tests fails the run. `expectedTypes` is optional; +leave it out to keep the "no tests is fine" default. It's also how you make a type mandatory. List `upgrade`, +and a typo'd tag (which collects zero) fails instead of quietly skipping the flow. + +Everything harness-specific stays in that harness's config: `.ats/main.yaml` (cluster types, catalogs, +executor options) and `tests/e2e/config.yaml` (appCatalog, providers, MC options). Values files stay +per-harness too; a kind cluster and a workload cluster legitimately want different values, and each harness +loads them its own way. ### Harness-specific tests The contract is for the common case. Where a test goes: -1. Checks the deployed app, works anywhere: `tests/app/`, no gate. Most - tests. -2. Checks the deployed app but needs a capability not present everywhere: - `tests/app/` with a runtime skip on the capability, for example - `if !slices.Contains(caps, "cloud-identity") { t.Skip(...) }` where `caps` +1. Checks the deployed app, works anywhere: `tests/app/`, no gate. Most tests. +2. Checks the deployed app but needs a capability not present everywhere: `tests/app/` with a runtime skip on + the capability, for example `if !slices.Contains(caps, "cloud-identity") { t.Skip(...) }` where `caps` comes from `APP_TEST_CAPABILITIES`. Skips still show by name. -3. Needs harness machinery (MC access, bundle installs, AWS/IRSA, cluster - manipulation): a normal atf suite under `tests/e2e/suites/`, which this - RFC doesn't touch. - -So a repo can hold two Go modules: `tests/app/` (portable) and `tests/e2e/` -(atf-native), each with its own `go.mod`. They're separate on purpose, since -the portable one has to build without atf's dependencies. If you want one -toolchain over both, add a `go.work` at the repo root; it's optional and -never part of the contract. - -Gate on what the contract tells you about the environment, never on which -harness is running. If a test needs to know the harness name, it's -category 3. - -Gate on `APP_TEST_CAPABILITIES`, not on `APP_TEST_CLUSTER_TYPE`, and never -on the harness. Cluster type is topology, not capability: `kind` usually -being the PR runner and `capi` usually being the nightly one is a +3. Needs harness machinery (MC access, bundle installs, AWS/IRSA, cluster manipulation): a normal atf suite + under `tests/e2e/suites/`, which this RFC doesn't touch. + +So a repo can hold two Go modules: `tests/app/` (portable) and `tests/e2e/` (atf-native), each with its own +`go.mod`. They're separate on purpose, since the portable one has to build without atf's dependencies. If you +want one toolchain over both, add a `go.work` at the repo root; it's optional and never part of the contract. + +Gate on what the contract tells you about the environment, never on which harness is running. If a test needs +to know the harness name, it's category 3. + +Gate on `APP_TEST_CAPABILITIES`, not on `APP_TEST_CLUSTER_TYPE`, and never on the harness. Cluster type is +topology, not capability: `kind` usually being the PR runner and `capi` usually being the nightly one is a coincidence, and ATS moving to provided clusters -([app-test-suite#675](https://github.com/giantswarm/app-test-suite/pull/675)) -breaks even that, since the PR runner can then be handed a cluster with -cloud identity. Check the capability you actually need, so any cluster that -advertises it, including an `external` one, passes the same gate. If what -you need isn't a declared capability, the test needs harness machinery, -which is category 3. +([app-test-suite#675](https://github.com/giantswarm/app-test-suite/pull/675)) breaks even that, since the PR +runner can then be handed a cluster with cloud identity. Check the capability you actually need, so any +cluster that advertises it, including an `external` one, passes the same gate. If what you need isn't a +declared capability, the test needs harness machinery, which is category 3. ### Conformance and ownership -Two runners, one contract, so they'll drift unless something checks. The -contract ships a conformance suite: a fixture (trivial app, one test per -type, a hook, a declared controller, a lockfile) and assertions on the env -vars, ordering, exit codes, controller bootstrap, and lints above. A runner conforms only -if it passes the suite in CI; ATS and atf both wire it in. New guarantees go into the suite in the -same PR that adds them here. - -Passing per runner isn't enough: both can pass and still disagree on what a -test sees, which is the drift that hurts (a smoke test that's green on PR -and flaky at night). So the suite also checks parity: the same fixture -through both runners has to bootstrap the same controllers, collect the same -counts, and end with the same result, or the suite fails. The known trap is -guarantee 2, "settled": ATS -gets there when the Helm release reports installed, atf when the App CR -reads `deployed`, and those aren't the same moment. The contract pins the -observable, not the mechanism: settled means every workload the release -created is ready, not that a status field flipped. -`clustertest.wait.IsReleaseReady(name, namespace)` is the shared definition. -It reuses clustertest's existing `AreAll*Ready` conditions rather than -reimplementing readiness; the only thing missing today is scope, since those -list cluster-wide, so they gain a label-selector argument and -`IsReleaseReady` ANDs them over the release's objects -(`app.kubernetes.io/instance=`): Deployments, StatefulSets and -DaemonSets Available, Jobs succeeded. Scoping matters because a workload -cluster runs far more than the app under test, so an unscoped "all ready" -would both stall on unrelated workloads and make the two runners observe -different sets. Each runner still waits on its own mechanism signal (Helm -`installed`, App CR `deployed`) first; `IsReleaseReady` is the common gate -on top, and the parity fixture checks that neither runner starts tests -before it holds. - -team-tenet owns the contract: this doc, the suite, and the call when the -runners disagree. team-honeybadger owns ATS, team-bumblebee owns atf. -Changing the contract is a PR here that updates the suite. A runner lagging +Two runners, one contract, so they'll drift unless something checks. The contract ships a conformance suite: a +fixture (trivial app, one test per type, a hook, a declared controller, a lockfile) and assertions on the env +vars, ordering, exit codes, controller bootstrap, and lints above. A runner conforms only if it passes the +suite in CI; ATS and atf both wire it in. New guarantees go into the suite in the same PR that adds them here. + +Passing per runner isn't enough: both can pass and still disagree on what a test sees, which is the drift that +hurts (a smoke test that's green on PR and flaky at night). So the suite also checks parity: the same fixture +through both runners has to bootstrap the same controllers, collect the same counts, and end with the same +result, or the suite fails. The known trap is guarantee 2, "settled": ATS gets there when the Helm release +reports installed, atf when the App CR reads `deployed`, and those aren't the same moment. The contract pins +the observable, not the mechanism: settled means every workload the release created is ready, not that a +status field flipped. `clustertest.wait.IsReleaseReady(name, namespace)` is the shared definition. It reuses +clustertest's existing `AreAll*Ready` conditions rather than reimplementing readiness; the only thing missing +today is scope, since those list cluster-wide, so they gain a label-selector argument and `IsReleaseReady` +ANDs them over the release's objects (`app.kubernetes.io/instance=`): Deployments, StatefulSets and +DaemonSets Available, Jobs succeeded. Scoping matters because a workload cluster runs far more than the app +under test, so an unscoped "all ready" would both stall on unrelated workloads and make the two runners +observe different sets. Each runner still waits on its own mechanism signal (Helm `installed`, App CR +`deployed`) first; `IsReleaseReady` is the common gate on top, and the parity fixture checks that neither +runner starts tests before it holds. + +team-tenet owns the contract: this doc, the suite, and the call when the runners disagree. team-honeybadger +owns ATS, team-bumblebee owns atf. Changing the contract is a PR here that updates the suite. A runner lagging is a bug in that runner, not a reason to fork. ## Implementation -- **apptest-framework** gets a convention-runner: after the workload cluster - and App CR are up, it grabs the WC kubeconfig - (`framework.MC().GetClusterKubeConfig(ctx, name, namespace)` on the - clustertest MC client), writes it out, exports the env - contract, picks the executor, and runs it per type. For upgrades it adds - the pre run. Today it runs the suite once after the upgrade; now it also - runs `upgrade` tests against the old version first - (`APP_TEST_UPGRADE_STAGE=pre`). `BeforeUpgrade` maps to the `pre-upgrade` - hook. The image adds `uv` and `gotestsum`. In-process suites are - untouched. Two things make this cheap: Ginkgo runs under plain `go test`, - and pytest tests built on pytest-helm-charts already read `KUBECONFIG`, so - existing ATS tests in either language run on workload clusters as-is. -- **app-test-suite** exports the `APP_TEST_*` names next to the old `ATS_*` - ones. It already runs `upgrade` tests both before and after the upgrade, - so `APP_TEST_UPGRADE_STAGE` is a rename of the stage it already tracks - (`ATS_EXTRA_UPGRADE_TEST_STAGE`), with the value normalized to - `pre`/`post`. It keeps its hook flags and also discovers the convention - hooks by path (stopping if a flag and a file point at the same one), and - adds a - pre-deploy point for `setup`. Its `--app-tests-pre-hook` runs after - deploy, so `setup` is a new call between `_ensure_cluster_prerequisites` - and the install. It looks in `tests/app/` as well as today's `tests/ats/`, - reads the shared `.apptest/config.yaml`, and emits junit via gotestsum. Its - upgrade pre/post behavior doesn't change. Its TEST_CONTRACT.md becomes a pointer - here plus ATS-specific detail. -- **clustertest**: the existing `AreAll*Ready` conditions gain an optional - label-selector argument (matching the style of `AreNumNodesReady`, which - already takes `listOptions`), and a thin `wait.IsReleaseReady(name, - namespace)` ANDs them over `app.kubernetes.io/instance=`. No new - readiness logic; both runners and the atf-native suites share one - definition of ready, and the parity check has one thing to assert against. -- **controllers**: both runners parse `.apptest/config.yaml`'s `controllers` - and bootstrap them before deploy — detect, install the `semver`-selected - version if absent, fail if a present one is out of range, wait until ready. - The declaration is shared; the provider that installs a given controller - name is per-harness. ATS lands the provider framework first and syncs its - existing providers in after, so until then a declared controller fails as - "unknown controller", which is the contract's behaviour for an unregistered - name. -- **the on-demand trigger**: the workload-cluster pipeline runs on `/run` - against a PR, not just nightly, so a cloud-path change can get its result - without waiting. -- **the conformance suite** lives here with the RFC: the fixture app and the - assertions, parity check included. Both runners run it in CI; it's what - "conforms" means. -- **devctl `gen apptest` and template-app** scaffold the layout for new - repos. +- **apptest-framework** gets a convention-runner: after the workload cluster and App CR are up, it grabs the + WC kubeconfig (`framework.MC().GetClusterKubeConfig(ctx, name, namespace)` on the clustertest MC client), + writes it out, exports the env contract, picks the executor, and runs it per type. For upgrades it adds the + pre run. Today it runs the suite once after the upgrade; now it also runs `upgrade` tests against the old + version first (`APP_TEST_UPGRADE_STAGE=pre`). `BeforeUpgrade` maps to the `pre-upgrade` hook. The image adds + `uv` and `gotestsum`. In-process suites are untouched. Two things make this cheap: Ginkgo runs under plain + `go test`, and pytest tests built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS tests in + either language run on workload clusters as-is. +- **app-test-suite** exports the `APP_TEST_*` names next to the old `ATS_*` ones. It already runs `upgrade` + tests both before and after the upgrade, so `APP_TEST_UPGRADE_STAGE` is a rename of the stage it already + tracks (`ATS_EXTRA_UPGRADE_TEST_STAGE`), with the value normalized to `pre`/`post`. It keeps its hook flags + and also discovers the convention hooks by path (stopping if a flag and a file point at the same one), and + adds a pre-deploy point for `setup`. Its `--app-tests-pre-hook` runs after deploy, so `setup` is a new call + between `_ensure_cluster_prerequisites` and the install. It looks in `tests/app/` as well as today's + `tests/ats/`, reads the shared `.apptest/config.yaml`, and emits junit via gotestsum. Its upgrade pre/post + behavior doesn't change. Its TEST_CONTRACT.md becomes a pointer here plus ATS-specific detail. +- **clustertest**: the existing `AreAll*Ready` conditions gain an optional label-selector argument (matching + the style of `AreNumNodesReady`, which already takes `listOptions`), and a thin + `wait.IsReleaseReady(name, namespace)` ANDs them over `app.kubernetes.io/instance=`. No new readiness + logic; both runners and the atf-native suites share one definition of ready, and the parity check has one + thing to assert against. +- **controllers**: both runners parse `.apptest/config.yaml`'s `controllers` and bootstrap them before deploy + — detect, install the `semver`-selected version if absent, fail if a present one is out of range, wait until + ready. The declaration is shared; the provider that installs a given controller name is per-harness. ATS + lands the provider framework first and syncs its existing providers in after, so until then a declared + controller fails as "unknown controller", which is the contract's behaviour for an unregistered name. +- **the on-demand trigger**: the workload-cluster pipeline runs on `/run` against a PR, not just nightly, so a + cloud-path change can get its result without waiting. +- **the conformance suite** lives here with the RFC: the fixture app and the assertions, parity check + included. Both runners run it in CI; it's what "conforms" means. +- **devctl `gen apptest` and template-app** scaffold the layout for new repos. - Migration happens as repos get touched; no flag day. Pilot: [giantswarm/muster#954](https://github.com/giantswarm/muster/pull/954). ## Alternatives considered -- **A shared assertions library both harnesses import per repo.** Tried it - in muster; the module, replace directives, and adapter code guarded about - a dozen lines of predicate per repo. Not worth it; better to align the - runners so the test files themselves are shared. -- **Standardize on Ginkgo.** Ginkgo runs under `go test`, so it's allowed, - but requiring it would shut out the pytest repos and tie the contract to a - framework for nothing. We standardize selection and inputs, not the +- **A shared assertions library both harnesses import per repo.** Tried it in muster; the module, replace + directives, and adapter code guarded about a dozen lines of predicate per repo. Not worth it; better to + align the runners so the test files themselves are shared. +- **Standardize on Ginkgo.** Ginkgo runs under `go test`, so it's allowed, but requiring it would shut out the + pytest repos and tie the contract to a framework for nothing. We standardize selection and inputs, not the framework. -- **A `values: {ats: ..., e2e: ...}` map for the app's deploy values.** No: - the app's values are harness-specific (a kind and a workload cluster want - different ones) and stay in each harness's own config, not the neutral file - — that would be the config version of gating a test on the harness. The - `controllers` section is the bounded exception: it names per-harness values - files, but as a `harness` list (the name is a field, not a map key) and only - for runner-bootstrapped infrastructure, not the app. -- **One runner with two modes instead of two behind a contract.** One runner - covering kind and workload clusters wouldn't need a contract at all. But - the two modes line up with two mature codebases owned by two teams - (ATS/honeybadger, atf/tenet), each with provisioning and pipeline code the - other doesn't want. Merging them is a bigger, riskier job than aligning - their edges, and the parity check gets most of the anti-drift value for - far less. Revisit if the runners keep drifting. +- **A `values: {ats: ..., e2e: ...}` map for the app's deploy values.** No: the app's values are + harness-specific (a kind and a workload cluster want different ones) and stay in each harness's own config, + not the neutral file — that would be the config version of gating a test on the harness. The `controllers` + section is the bounded exception: it names per-harness values files, but as a `harness` list (the name is a + field, not a map key) and only for runner-bootstrapped infrastructure, not the app. +- **One runner with two modes instead of two behind a contract.** One runner covering kind and workload + clusters wouldn't need a contract at all. But the two modes line up with two mature codebases owned by two + teams (ATS/honeybadger, atf/tenet), each with provisioning and pipeline code the other doesn't want. Merging + them is a bigger, riskier job than aligning their edges, and the parity check gets most of the anti-drift + value for far less. Revisit if the runners keep drifting. From 37ff03c672eb737a9c7d34372deb7269520b6cb8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Pi=C4=85tkowski?= Date: Wed, 15 Jul 2026 16:58:12 +0200 Subject: [PATCH 18/24] reviewed to 'full test flow' --- app-testing-contract/README.md | 196 +++++++++++++++------------------ 1 file changed, 86 insertions(+), 110 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 0556e33..55998cb 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -45,12 +45,52 @@ This goal of this RFC is not to dedupe existing tests. It's to define how the tw what environments they provide and what is the convention that, when respected, can allow to fit both frameworks with the same test code. -## Decision - -### The conventional directory +## Decisions + +### ATS and ATF + +We keep both frameworks, but specialize them to the two most frequent use cases: + +- `ats` is meant for rapid feedback testing on PRs and in local dev environments. It deploys the helm chart + under tests as fast as possible and starts testing the chart's functionality. It sacrifices all the possible + real cluster features to achieve this goal. +- `atf` takes the opposite approach: it chooses environment realism over the time needed to execute the tests. + It creates a real workload cluster on a GS installation, installs the chart using the App Platform, and runs + the full suite as a batch run, preferably nightly. +- we want a test development convention that will allow app maintainers to write tests once and run them under + both frameworks, with the same test code. + +### Test Development Convention + +#### Assumptions + +To avoid forcing a specific test framework or technology on test authors, we decided to make a convention that +tests are executed as a separate process by the test frameworks. The only requirements are that: + +- we group all the tests into: + - **smoke**: very basic tests that check if the app is deployed and running, and if the main functionality + is working and worth even trying actual functional test. They should be fast, simple and reliable. + - **functional**: tests that check the actual functionality of the app, and that it behaves as expected. + They should be more complex and cover more scenarios than smoke tests. + - **upgrade**: tests that check if the app can be upgraded from a previous version, by default the last + stable version available in the OCI registry. +- a single test can be of multiple types, for example a test that checks if the app is deployed and running + can be both smoke and functional; a test that checks if the main page of an app loads can (and probably + should) be functional and upgrade. +- tests need to be runnable with `go test` (golang) or `pytest` (python), and are using test filtering to run + only the tests of a specific type (smoke, functional, upgrade). The test filtering is done with build tags + (golang) or markers (python). +- all the information about the test environmenrt is passed to the tests via environment variables, and the + tests should not depend on any other external information (like a config file or a specific cluster setup). + The test frameworks are responsible for setting up the environment and passing the information to the tests. +- exit code `0` means the test run passed, exit code `5` that the run was successful, but only because it + executed no test and any other non-zero exit code means the test run failed. +- test developer can deliver hooks that are executed by the test frameworks (see below). + +#### The conventional directory Tests live in one directory: `tests/app/`. Any harness that deploys the app runs that directory the same way. -Having the directory is the opt-in; there's nothing else to wire up. +An empty `tests/app/` (no module, no project) isn't an opt-in and is skipped. It's either one Go module or one Python project, not both. The runner picks the executor from what's there: @@ -58,17 +98,10 @@ It's either one Go module or one Python project, not both. The runner picks the - `pyproject.toml`: `uv sync --frozen && uv run pytest -m ` - both, or neither in a non-empty directory: config error, stop. -Dependencies are pinned from committed lockfiles (Go `go.sum`, Python `uv.lock`) and installed with the -network off: the runner sets `GOPROXY=off` alongside `-mod=readonly` (or vendors `tests/app/vendor/`) and runs -`uv sync --frozen --offline`, backed by a pre-warmed module/uv cache. The lockfile flags alone -(`-mod=readonly`, `--frozen`) only freeze resolution, not fetching, so cutting the network is what makes a -missing or stale entry fail loudly instead of quietly downloading. Both runners implement this: under this -contract `tests/app/` is a separate module that atf builds at test time, so atf gains the same fetch point ATS -already has. Same dependency set everywhere, and you can audit it. - -An empty `tests/app/` (no module, no project) isn't an opt-in and is skipped. +Dependencies are pinned from committed lockfiles (Go `go.sum`, Python `uv.lock`) and installed with +readonly/frozen flags. -### Test types +#### Test types Each test carries one type, set with a Go build tag or a pytest marker. There are three, the same ones ATS already has: @@ -97,102 +130,45 @@ if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { Seeding is a side effect, so it's a hook; checking is an assertion, so it's a test. Upgrade tests and the `pre-upgrade` hook also get `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. -### Hooks - -Hooks do things with side effects (install a prerequisite, create a pod, clean up); tests check things. -Splitting them is what lets the upgrade flow seed state without re-running a suite. Like tests, setup and -teardown are per-app and get duplicated across harnesses, so the contract makes them portable too: optional -executables in `tests/app/`, run with the same environment as tests. - -| Hook | Runs | -| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `tests/app/hooks/setup` | after the cluster is ready, before the app is deployed (for example: install prerequisites) | -| `tests/app/hooks/pre-upgrade` | upgrade flow only: after the previous version is deployed, before the upgrade (for example: create a pod or write a record an `upgrade` test then checks survived) | -| `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | - -A hook is any executable at that path: a script with a shebang or a built binary, run directly (not sourced). -It runs out of process, so the one-language rule doesn't apply and you can write it in whatever fits. Keep it -small; anything bigger is a test or a harness-native hook. - -A missing hook does nothing. A non-zero exit fails the run. Hooks gate on the environment like category-2 -tests do; `pre-upgrade` also gets the from/to versions. - -Convention discovery of those three paths is the default, and every runner has to implement it. That's the -zero-wiring part. A harness can also keep its own hook flags, so existing repos don't have to move and -harness-specific hooks still work. If a flag and a convention file point at the same contract hook, the runner -stops (same as finding both `go.mod` and `pyproject.toml`), so migrating is "add the file, drop the flag" in -one commit rather than running both. Only the convention path is guaranteed and checked by the conformance -suite; the flags are each harness's own business. - -How the three points map today: - -| Contract hook | ATS | atf | -| ------------------------ | -------------------------------------------------------------------- | ----------------------------------------- | -| `setup` (before deploy) | new pre-deploy point (its `--app-tests-pre-hook` fires after deploy) | `AfterClusterReady` (runs before install) | -| `pre-upgrade` | `--upgrade-tests-upgrade-hook` at `ATS_HOOK_STAGE=pre_upgrade` | `BeforeUpgrade` | -| `teardown` (after tests) | `--app-tests-post-hook` | suite callback | - -A runner covers each point with either the file or the flag, not both. Anything not in that table (ATS's -`post_upgrade` stage, its pre/post test hooks) stays harness-native. - -Same boundary as tests: a hook only gets the app cluster's `KUBECONFIG`. Anything that needs harness internals -(MC access, the App CR, framework state) stays in a harness-native hook: ATS's config hooks for points we -don't cover, or atf's `AfterClusterReady` / `BeforeUpgrade`. - -### Prerequisite controllers - -Some apps under test create custom resources — a Flux `Kustomization`, an Argo `Application`, an -`ExternalSecret` — that do nothing until a controller is running to reconcile them. Only some apps need any -given controller, and installing one is expensive, so the app declares the controllers it needs and the runner -bootstraps exactly those before the app is deployed. There's no auto-detection: declaring is the opt-in, the -same rule as everything else here. - -This is the declarative sibling of the `setup` hook. `setup` runs an app-specific script; a controller names -something the runner already knows how to install. The declaration is shared, but the provider code that -installs a named controller is each harness's own — a harness targeting kind and one targeting a workload -cluster install it differently — so a controller only works on a harness that has a provider registered for -that name. - -Controllers are declared in the shared `.apptest/config.yaml` (see Shared configuration): - -```yaml -controllers: - - name: flux - semver: ">=2.0.0 <3.0.0" - harness: - - name: ats - valuesFile: flux-small.yaml - - name: atf - valuesFile: flux-full.yaml - - name: external-secrets - semver: "0.x" -``` - -- `name` (required): the controller's harness-neutral id. If the running harness has no provider registered - for it, the run fails. -- `semver` (required): a version range with Masterminds/semver v3 semantics (the same Flux `OCIRepository` and - Helm `--version` use), resolved to the highest version that satisfies it. -- `harness` (optional): per-harness install values files, listed by harness name rather than keyed by it, so - the neutral file stays a list you extend, not a map with harness names baked into its shape. Each - `valuesFile` ends in `.yaml`, sits beside `config.yaml` under `.apptest/`, and layers over the controller's - defaults. No entry for the running harness means defaults; a named file that's missing or not `.yaml` fails - the run. - -Order matters: controllers install in list order, each fully ready before the next, so one that depends on -another goes after it. They install once per run and are shared across every type and both flows — a -prerequisite is run infrastructure, not something per test. - -An already-present controller is reused. The runner checks the installed version: absent, it installs; present -and within `semver`, it leaves it alone; present but outside `semver`, the run fails. The runner never -upgrades, downgrades, or removes a controller it finds — the cluster may not be ours, and leaving controllers -in place is also what makes the next run on the same cluster faster. A provider may run its own pre-install -and post-install steps (create RBAC, wait for a webhook or a CRD to establish) around the install. - -Controllers are not capabilities. `APP_TEST_CAPABILITIES` is what a cluster already provides and a test gates -on; a controller is something the runner adds to any cluster. When a bootstrapped controller is a gitops -engine, the runner surfaces which one through `APP_TEST_EXTRA_GITOPS_ENGINE`. - -### Inputs +#### Hooks + +Hooks do things with side effects (install a prerequisite, create a pod, clean up); tests check things. Hooks +are delivered as executables in the `tests/app/hooks/` directory, and are optional. They should be implemented +in platform-independent way, preferably in bash or python. Hooks are executed by the test framework. Splitting +them is what lets the upgrade flow seed state without re-running a suite. Hooks get the test information +through environment variables, the same as tests. The hooks are: + +| Hook | Runs | +| -------------------------- | -------------------------------------------------------------------------------------------------- | +| `tests/app/hooks/setup` | after the cluster is ready, before the app is deployed (for example: install prerequisites, CRDs) | +| `tests/app/hooks/pre-run` | Run before every test suite invocation, for each detected test types | +| `tests/app/hooks/post-run` | Run after every test suite invocation, for each detected test types | +| `tests/app/hooks/teardown` | after all tests, before the harness tears anything down (for example: clean up external resources) | + +A missing hook is ignored. A non-zero exit fails the test run. + +#### Full Test Flow + +The test framework (`ats` or `atf`) runs the tests using this flow: + +1. Test framework detects tests are present in `tests/app/`. +1. Test framework prepares the cluster used for testing (installs tools or dependencies it needs to execute + tests). +1. If present, the `setup` hook runs after the cluster is ready, before the app is deployed. +1. The app is deployed using the passed helm chart and the installation is settled (chart install exists + cleanly). +1. For each test type of `smoke`, `functional`, and `upgrade` (where `upgrade` tests are executed twice, first + for old version, then after the upgrade, for the new version), in this order: + 1. `pre-run` hook runs for `type` tests (if present). + 1. Tests are executed for `type` type, using either `go test` or `pytest`, depending on the detected module + type. + 1. `post-run` hook runs for `type` tests (if present). + 1. If it's an `upgrade` type test and the execution for `old` version succeeded, the app is upgraded to the + `new` version. +1. If present, the `teardown` hook runs after all tests, before the harness tears anything down. +1. The app is uninstalled. + +#### Inputs Tests get everything from the environment. They don't provision anything: no clusters, no chart installs, no App CRs. From df7f51597a4498b2190376986678dd67388128b2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Pi=C4=85tkowski?= Date: Thu, 16 Jul 2026 14:58:27 +0200 Subject: [PATCH 19/24] reviewed full text, added glossary and consistency checks --- app-testing-contract/README.md | 296 ++++++++++----------------------- 1 file changed, 92 insertions(+), 204 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 55998cb..2a9911f 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -1,6 +1,6 @@ --- creation_date: 2026-07-07 -issues: [] +issues: []j owners: - https://github.com/orgs/giantswarm/teams/team-bumblebee - https://github.com/orgs/giantswarm/teams/team-honeybadger @@ -15,11 +15,17 @@ summary: # The app-testing contract +## Glossary + +- test toolkit: the test runner and orchestrator, either `app-test-suite` (ATS) or `apptest-framework` (ATF) +- test framework: software framework used to implement the tests, e.g. Ginkgo or pytest +- test suite: a collection of tests, usually in one repo, that test a specific chart (app) + ## Problem -We test our helm charts delivering managed apps using two frameworks: `app-test-suite` (ATS) and -`apptest-framework` (ATF). So far, it was not clear which framework teams should use and how to write the -tests. We propose to keep both frameworks, but specialize them to the two most frequent use cases: +We test our helm charts delivering managed apps using two toolkits: `app-test-suite` (ATS) and +`apptest-framework` (ATF). So far, it was not clear which toolkit teams should use and how to write the tests. +We propose to keep both toolkits, but specialize them to the two most frequent use cases: - **app-test-suite (ATS)** - should provide rapid local feedback on PRs and in local dev environments. Its goal is to deploy the helm chart under tests as fast as possible and start testing the chart's @@ -31,25 +37,25 @@ tests. We propose to keep both frameworks, but specialize them to the two most f runs the full suite, preferably nightly. With this in mind, it's clear that to provide a comprehensive test coverage and to follow the "fail fast" -principle, we need to use both frameworks. +principle, we need to use both toolkits. -The trouble is writing the tests. The tests author doesn't want to implement the same or very similar set of -tests twice, once for each framework. The goal of this doc is to propose a test implementation spec that will -allow both ATS and ATF to run the same test suites in their respective environments. Full parity of tests -might not be possible, as test itself might depend on the environment, but we want to get as close as -possible. A local kind cluster is fast but can't do cloud identity, real storage, or upgrades; a workload -cluster can. If a test suite needs to be aware of these differences, the author needs to get the possibility -to gate on the capabilities it needs. +The trouble is writing the tests (test suites). The tests author doesn't want to implement the same or very +similar set of tests twice, once for each toolkit. The goal of this doc is to propose a test implementation +spec that will allow both ATS and ATF to run the same test suites in their respective environments. Full +parity of tests might not be possible, as test itself might depend on the environment, but we want to get as +close as possible. A local kind cluster is fast but can't do cloud identity, real storage, or upgrades; a +workload cluster can. If a test suite needs to be aware of these differences, the author needs to get the +possibility to gate on the capabilities it needs. -This goal of this RFC is not to dedupe existing tests. It's to define how the two frameworks are different, -what environments they provide and what is the convention that, when respected, can allow to fit both -frameworks with the same test code. +This goal of this RFC is not to dedupe existing tests. It's to define how the two toolkits are different, what +environments they provide and what is the convention that, when respected, can allow to fit both toolkits with +the same test code. ## Decisions ### ATS and ATF -We keep both frameworks, but specialize them to the two most frequent use cases: +We keep both toolkits, but specialize them to the two most frequent use cases: - `ats` is meant for rapid feedback testing on PRs and in local dev environments. It deploys the helm chart under tests as fast as possible and starts testing the chart's functionality. It sacrifices all the possible @@ -58,22 +64,24 @@ We keep both frameworks, but specialize them to the two most frequent use cases: It creates a real workload cluster on a GS installation, installs the chart using the App Platform, and runs the full suite as a batch run, preferably nightly. - we want a test development convention that will allow app maintainers to write tests once and run them under - both frameworks, with the same test code. + both toolkits, with the same test code. ### Test Development Convention #### Assumptions To avoid forcing a specific test framework or technology on test authors, we decided to make a convention that -tests are executed as a separate process by the test frameworks. The only requirements are that: +tests are executed as a separate process by the test toolkits. The only requirements are that: -- we group all the tests into: +- for each test suite, we group all the tests into: - **smoke**: very basic tests that check if the app is deployed and running, and if the main functionality is working and worth even trying actual functional test. They should be fast, simple and reliable. - **functional**: tests that check the actual functionality of the app, and that it behaves as expected. They should be more complex and cover more scenarios than smoke tests. - **upgrade**: tests that check if the app can be upgraded from a previous version, by default the last - stable version available in the OCI registry. + stable version available in the OCI registry. The tests from this groups are executed twice, once before + the upgrade (the "old" version, that we start the upgrade test with) and once after the upgrade (the "new" + version, under the test). - a single test can be of multiple types, for example a test that checks if the app is deployed and running can be both smoke and functional; a test that checks if the main page of an app loads can (and probably should) be functional and upgrade. @@ -82,10 +90,10 @@ tests are executed as a separate process by the test frameworks. The only requir (golang) or markers (python). - all the information about the test environmenrt is passed to the tests via environment variables, and the tests should not depend on any other external information (like a config file or a specific cluster setup). - The test frameworks are responsible for setting up the environment and passing the information to the tests. -- exit code `0` means the test run passed, exit code `5` that the run was successful, but only because it - executed no test and any other non-zero exit code means the test run failed. -- test developer can deliver hooks that are executed by the test frameworks (see below). + The test toolkits are responsible for setting up the environment and passing the information to the tests. +- exit code `0` means the test run passed, including the case where it executed no tests, and any other + non-zero exit code means the test run failed. +- test developer can deliver hooks that are executed by the test toolkits (see below). #### The conventional directory @@ -113,13 +121,13 @@ already has: | `upgrade` | upgrade flow, before and after | checks the app still works across an upgrade | `upgrade` is its own type, not a flag on the others. The upgrade flow doesn't re-run smoke and functional; it -runs the `upgrade` tests, once on the old version (`APP_TEST_UPGRADE_STAGE=pre`) and once after upgrading -(`=post`). The pre run is the baseline: if it passes and post fails, the upgrade caused it, not something that -was already broken. +runs the `upgrade` tests, once for the "old" version (the version to test the upgrade from, usually the last +stable version) and once after upgrading to the "new" version (the version under test). The first run is the +baseline: if it passes and post fails, the upgrade caused it, and the upgrade test fails. Most upgrade checks are symmetric ("the app answers") and assert the same thing both times. When some state -has to survive the upgrade, either seed it in the `pre-upgrade` hook and check it in the post run, or keep it -in one test that branches on the stage: +has to survive the upgrade, either seed it in the `pre-run` hook and check it in the post run, or keep it in +one test that branches on the stage: ```go if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { @@ -127,14 +135,14 @@ if os.Getenv("APP_TEST_UPGRADE_STAGE") != "post" { } ``` -Seeding is a side effect, so it's a hook; checking is an assertion, so it's a test. Upgrade tests and the -`pre-upgrade` hook also get `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. +Seeding is a side effect, so it's a hook; checking is an assertion, so it's a test. Upgrade tests and hooks +hook also get `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION`. #### Hooks Hooks do things with side effects (install a prerequisite, create a pod, clean up); tests check things. Hooks are delivered as executables in the `tests/app/hooks/` directory, and are optional. They should be implemented -in platform-independent way, preferably in bash or python. Hooks are executed by the test framework. Splitting +in platform-independent way, preferably in bash or python. Hooks are executed by the test toolkit. Splitting them is what lets the upgrade flow seed state without re-running a suite. Hooks get the test information through environment variables, the same as tests. The hooks are: @@ -149,53 +157,58 @@ A missing hook is ignored. A non-zero exit fails the test run. #### Full Test Flow -The test framework (`ats` or `atf`) runs the tests using this flow: +The test suite is always invoked by the test toolkit (`ats` or `atf`). The toolkit runs the tests using this +flow: -1. Test framework detects tests are present in `tests/app/`. -1. Test framework prepares the cluster used for testing (installs tools or dependencies it needs to execute - tests). +1. Test toolkit detects that tests are present in `tests/app/`. +1. Test toolkit installs software dependencies for the test suite (go or python) using the lockfile in + `tests/app/`. +1. Test toolkit prepares the cluster used for testing (installs tools or dependencies it needs to execute + tests; might be a no-op, depends on the test toolkit). 1. If present, the `setup` hook runs after the cluster is ready, before the app is deployed. 1. The app is deployed using the passed helm chart and the installation is settled (chart install exists cleanly). -1. For each test type of `smoke`, `functional`, and `upgrade` (where `upgrade` tests are executed twice, first - for old version, then after the upgrade, for the new version), in this order: +1. For each test `type` in `smoke`, `functional`: 1. `pre-run` hook runs for `type` tests (if present). 1. Tests are executed for `type` type, using either `go test` or `pytest`, depending on the detected module type. 1. `post-run` hook runs for `type` tests (if present). - 1. If it's an `upgrade` type test and the execution for `old` version succeeded, the app is upgraded to the - `new` version. -1. If present, the `teardown` hook runs after all tests, before the harness tears anything down. +1. If there are `upgrade` tests: + 1. If the version under test (new) is already installed in the cluster, it is uninstalled. + 1. The "old" stable version of the app is installed. + 1. `pre-run` hook runs for `upgrade` tests (if present). + 1. `upgrade` type tests are executed for the "old" version. + 1. `post-run` hook runs for `upgrade` tests (if present). + 1. The app is upgraded to the "new" version. + 1. `pre-run` hook runs for `upgrade` tests (if present). + 1. `upgrade` type tests are executed for the "new" version. + 1. `post-run` hook runs for `upgrade` tests (if present). 1. The app is uninstalled. +1. If present, the `teardown` hook runs after all tests, before the harness tears anything down. #### Inputs -Tests get everything from the environment. They don't provision anything: no clusters, no chart installs, no -App CRs. - -| Variable | Required | Meaning | -| --------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `KUBECONFIG` | yes | kubeconfig of the cluster the app is deployed on (never the MC) | -| `APP_TEST_TYPE` | yes | the type currently being run | -| `APP_TEST_RELEASE_NAME` | yes | Helm release name of the app under test | -| `APP_TEST_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | -| `APP_TEST_CHART_VERSION` | yes | version of the chart under test | -| `APP_TEST_CLUSTER_TYPE` | yes | topology of the cluster the app runs on: `kind` (local single-node), `capi` (a CAPI workload cluster), or `external` (a cluster the runner did not provision). Describes shape, not capability; gate on `APP_TEST_CAPABILITIES` instead | -| `APP_TEST_CAPABILITIES` | yes | comma-separated capabilities the cluster actually provides, e.g. `cloud-identity,persistent-storage,load-balancer`; empty is valid. The runner sets it from what it provisioned or was handed. This is what a test gates on | -| `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | -| `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | -| `APP_TEST_UPGRADE_STAGE` | upgrade flow only | `pre` or `post`: which side of the upgrade this `upgrade`-type run is on | -| `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade flow only | versions on either side of the upgrade; available to the `pre-upgrade` hook and `upgrade`-typed tests | -| `APP_TEST_EXTRA_*` | optional | harness extras, for example `APP_TEST_EXTRA_GITOPS_ENGINE` | - -The prefix is `APP_TEST_`, which reads the same under either harness. ATS publishes these under the old `ATS_` -prefix today. Where a name maps straight across (`ATS_X` to `APP_TEST_X`, e.g. `ATS_RELEASE_NAME`, -`ATS_CHART_VERSION`, `ATS_CLUSTER_TYPE`, `ATS_EXTRA_*`) the runner exports both, so nothing breaks and new -tests use `APP_TEST_`. Dual export isn't free (two names to know and grep for), so `ATS_` is deprecated and -will be removed in a later change once repos have migrated. - -Four names are renamed, because the old ones were unclear or, for the upgrade stage, were never test-facing -under one name to begin with: +Tests get all the information about what is being tested from the environment variables. The following +variables are guaranteed to be set by the test toolkit before the tests are executed: + +| Variable | Required | Meaning | +| --------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `KUBECONFIG` | yes | kubeconfig of the cluster the app is deployed on (never the MC) | +| `APP_TEST_TYPE` | yes | the type currently being run (`smoke`, `functional`, `upgrade`) | +| `APP_TEST_TOOLKIT` | yes | The name of the toolkit running the tests (`ats` or `atf`) | +| `APP_TEST_RELEASE_NAME` | yes | Helm release name of the app under test | +| `APP_TEST_RELEASE_NAMESPACE` | yes | namespace the app is deployed into | +| `APP_TEST_CHART_VERSION` | yes | version of the chart under test | +| `APP_TEST_CLUSTER_TYPE` | yes | type of the cluster the app runs on - a label: `kind` (local single-node), `capi` (a CAPI workload cluster), or `external` (a cluster the runner did not provision). Describes shape, not capability; gate on `APP_TEST_CAPABILITIES` instead | +| `APP_TEST_CAPABILITIES` | yes | comma-separated capabilities the cluster actually provides, e.g. `cloud-identity,persistent-storage,load-balancer`; empty is valid. The runner sets it from what it provisioned or was handed. This is what a test gates on | +| `APP_TEST_KUBERNETES_VERSION` | optional | Kubernetes server version | +| `APP_TEST_VALUES_FILE` | optional | values file the app was deployed with | +| `APP_TEST_UPGRADE_STAGE` | upgrade flow only | `pre` or `post`: which side of the upgrade this `upgrade`-type run is on | +| `APP_TEST_UPGRADE_FROM_VERSION` / `APP_TEST_UPGRADE_TO_VERSION` | upgrade flow only | versions on either side of the upgrade | + +**Note**: These values come originally from `ats`, but are renamed here to match the test framework +independence. Four names are renamed, because the old ones were unclear or, for the upgrade stage, were never +test-facing under one name to begin with: | Legacy | Canonical | Note | | ----------------------------------------------------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | @@ -208,66 +221,30 @@ under one name to begin with: stay ATS-only. `APP_TEST_CAPABILITIES` is new, computed by the runner, with no `ATS_` predecessor. `KUBECONFIG` stays as-is; it's the standard name, not ours. -### Runner guarantees - -Before the tests run, a conforming runner makes sure: - -1. the controllers declared in `.apptest/config.yaml`, if any, are bootstrapped and ready, once per run, - before anything is deployed, -2. the `setup` hook ran, if present, after the controllers were ready and before the app was deployed, -3. the app is deployed and settled: each runner first waits on its own mechanism signal (ATS: Helm release - installed, or via a GitOps engine; atf: App CR at `deployed`), then on the shared gate `IsReleaseReady` - (the release's workloads Available) before any test runs, -4. the required variables are exported, -5. the `teardown` hook runs, if present, after the last test type and before the harness's own teardown. +#### Capabilities -Normal flow: run `smoke`, then `functional`. Upgrade tests don't run here. +The cluster intended for PR testing is `kind`. As a simple cluster instance, it lacks features like cloud +identity, real storage, or load balancers. Tests that need those capabilities should gate on the +`APP_TEST_CAPABILITIES` and only run nightly on a real workload cluster. -Upgrade flow (any `upgrade` tests collected): install the previous version and let it settle, run `upgrade` -tests with `APP_TEST_UPGRADE_STAGE=pre`, run the `pre-upgrade` hook, upgrade and let it settle, run `upgrade` -tests with `=post`. smoke and functional don't run here. +We define the following capabilities, which the toolkit sets in `APP_TEST_CAPABILITIES`: -"No tests of this type" passes rather than fails (Go excludes all files via build tags; pytest exits 5), since -a repo may only have some types. It isn't silent, though: the runner reports how many tests it collected per -type, so a mistyped tag (also zero) shows up instead of going green. Repos that want it strict list their -expected types in `.apptest/config.yaml`; a listed type with zero tests fails. Results come out as junit XML -(`gotestsum --junitfile`, `pytest --junitxml`). +- cloud-identity +- persistent-storage +- load-balancer -### Cadence and feedback latency - -The PR cluster often lacks cloud identity, real storage, or load balancers, so tests that need those -capabilities only run nightly on a workload cluster. Their result isn't tied to the PR that caused it: a PR -can break a cloud-only path, pass PR CI, and fail that night against a batch of other commits. - -We accept that, but two things keep it from being a silent trap: - -1. A nightly-only test is a choice you can see. A test that gates on a capability the PR cluster lacks - (category 2) is invisible per-PR by design; the skip shows by name and the collected counts show it didn't - run, so it doesn't read as coverage it isn't. -2. You can pull the nightly flow forward. `/run` triggers the workload-cluster flow on a PR, so a cloud-path - change can get its result now instead of that night. - -Gate on the capability you need (`APP_TEST_CAPABILITIES`), never on cluster type or harness (see -Harness-specific tests): a test that needs cloud identity runs anywhere advertising `cloud-identity`, whether -that's the nightly WC or a provided cluster that happens to have it. +# TODO: define and complete the capabilites list ### Shared configuration -Test _code_ lives in `tests/app/`; shared _declarations_ live in `.apptest/`. `.apptest/config.yaml` holds -only what both harnesses need, and the controller values files (see Prerequisite controllers) sit beside it: +Test _code_ lives in `tests/app/`; shared _configuration_ lives in `.apptest/`. `.apptest/config.yaml` holds +only what both toolkits need, and the chart values files. The config schema is: ```yaml installNamespace: kube-system -expectedTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test -controllers: [...] # optional: see Prerequisite controllers +testTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test ``` -The upgrade flow is inferred, not configured: if the runner collects any `upgrade`-typed tests it runs the -upgrade flow, otherwise it doesn't. Same presence-is-the-opt-in rule as the directory and the other types, so -there's no separate switch to keep in sync. Each harness still learns which version to upgrade from through -its own config (ATS's stable-app settings, atf's latest published release); that part is harness-specific, not -contract. - The one lint: a type in `expectedTypes` that collects zero tests fails the run. `expectedTypes` is optional; leave it out to keep the "no tests is fine" default. It's also how you make a type mandatory. List `upgrade`, and a typo'd tag (which collects zero) fails instead of quietly skipping the flow. @@ -277,95 +254,6 @@ executor options) and `tests/e2e/config.yaml` (appCatalog, providers, MC options per-harness too; a kind cluster and a workload cluster legitimately want different values, and each harness loads them its own way. -### Harness-specific tests - -The contract is for the common case. Where a test goes: - -1. Checks the deployed app, works anywhere: `tests/app/`, no gate. Most tests. -2. Checks the deployed app but needs a capability not present everywhere: `tests/app/` with a runtime skip on - the capability, for example `if !slices.Contains(caps, "cloud-identity") { t.Skip(...) }` where `caps` - comes from `APP_TEST_CAPABILITIES`. Skips still show by name. -3. Needs harness machinery (MC access, bundle installs, AWS/IRSA, cluster manipulation): a normal atf suite - under `tests/e2e/suites/`, which this RFC doesn't touch. - -So a repo can hold two Go modules: `tests/app/` (portable) and `tests/e2e/` (atf-native), each with its own -`go.mod`. They're separate on purpose, since the portable one has to build without atf's dependencies. If you -want one toolchain over both, add a `go.work` at the repo root; it's optional and never part of the contract. - -Gate on what the contract tells you about the environment, never on which harness is running. If a test needs -to know the harness name, it's category 3. - -Gate on `APP_TEST_CAPABILITIES`, not on `APP_TEST_CLUSTER_TYPE`, and never on the harness. Cluster type is -topology, not capability: `kind` usually being the PR runner and `capi` usually being the nightly one is a -coincidence, and ATS moving to provided clusters -([app-test-suite#675](https://github.com/giantswarm/app-test-suite/pull/675)) breaks even that, since the PR -runner can then be handed a cluster with cloud identity. Check the capability you actually need, so any -cluster that advertises it, including an `external` one, passes the same gate. If what you need isn't a -declared capability, the test needs harness machinery, which is category 3. - -### Conformance and ownership - -Two runners, one contract, so they'll drift unless something checks. The contract ships a conformance suite: a -fixture (trivial app, one test per type, a hook, a declared controller, a lockfile) and assertions on the env -vars, ordering, exit codes, controller bootstrap, and lints above. A runner conforms only if it passes the -suite in CI; ATS and atf both wire it in. New guarantees go into the suite in the same PR that adds them here. - -Passing per runner isn't enough: both can pass and still disagree on what a test sees, which is the drift that -hurts (a smoke test that's green on PR and flaky at night). So the suite also checks parity: the same fixture -through both runners has to bootstrap the same controllers, collect the same counts, and end with the same -result, or the suite fails. The known trap is guarantee 2, "settled": ATS gets there when the Helm release -reports installed, atf when the App CR reads `deployed`, and those aren't the same moment. The contract pins -the observable, not the mechanism: settled means every workload the release created is ready, not that a -status field flipped. `clustertest.wait.IsReleaseReady(name, namespace)` is the shared definition. It reuses -clustertest's existing `AreAll*Ready` conditions rather than reimplementing readiness; the only thing missing -today is scope, since those list cluster-wide, so they gain a label-selector argument and `IsReleaseReady` -ANDs them over the release's objects (`app.kubernetes.io/instance=`): Deployments, StatefulSets and -DaemonSets Available, Jobs succeeded. Scoping matters because a workload cluster runs far more than the app -under test, so an unscoped "all ready" would both stall on unrelated workloads and make the two runners -observe different sets. Each runner still waits on its own mechanism signal (Helm `installed`, App CR -`deployed`) first; `IsReleaseReady` is the common gate on top, and the parity fixture checks that neither -runner starts tests before it holds. - -team-tenet owns the contract: this doc, the suite, and the call when the runners disagree. team-honeybadger -owns ATS, team-bumblebee owns atf. Changing the contract is a PR here that updates the suite. A runner lagging -is a bug in that runner, not a reason to fork. - -## Implementation - -- **apptest-framework** gets a convention-runner: after the workload cluster and App CR are up, it grabs the - WC kubeconfig (`framework.MC().GetClusterKubeConfig(ctx, name, namespace)` on the clustertest MC client), - writes it out, exports the env contract, picks the executor, and runs it per type. For upgrades it adds the - pre run. Today it runs the suite once after the upgrade; now it also runs `upgrade` tests against the old - version first (`APP_TEST_UPGRADE_STAGE=pre`). `BeforeUpgrade` maps to the `pre-upgrade` hook. The image adds - `uv` and `gotestsum`. In-process suites are untouched. Two things make this cheap: Ginkgo runs under plain - `go test`, and pytest tests built on pytest-helm-charts already read `KUBECONFIG`, so existing ATS tests in - either language run on workload clusters as-is. -- **app-test-suite** exports the `APP_TEST_*` names next to the old `ATS_*` ones. It already runs `upgrade` - tests both before and after the upgrade, so `APP_TEST_UPGRADE_STAGE` is a rename of the stage it already - tracks (`ATS_EXTRA_UPGRADE_TEST_STAGE`), with the value normalized to `pre`/`post`. It keeps its hook flags - and also discovers the convention hooks by path (stopping if a flag and a file point at the same one), and - adds a pre-deploy point for `setup`. Its `--app-tests-pre-hook` runs after deploy, so `setup` is a new call - between `_ensure_cluster_prerequisites` and the install. It looks in `tests/app/` as well as today's - `tests/ats/`, reads the shared `.apptest/config.yaml`, and emits junit via gotestsum. Its upgrade pre/post - behavior doesn't change. Its TEST_CONTRACT.md becomes a pointer here plus ATS-specific detail. -- **clustertest**: the existing `AreAll*Ready` conditions gain an optional label-selector argument (matching - the style of `AreNumNodesReady`, which already takes `listOptions`), and a thin - `wait.IsReleaseReady(name, namespace)` ANDs them over `app.kubernetes.io/instance=`. No new readiness - logic; both runners and the atf-native suites share one definition of ready, and the parity check has one - thing to assert against. -- **controllers**: both runners parse `.apptest/config.yaml`'s `controllers` and bootstrap them before deploy - — detect, install the `semver`-selected version if absent, fail if a present one is out of range, wait until - ready. The declaration is shared; the provider that installs a given controller name is per-harness. ATS - lands the provider framework first and syncs its existing providers in after, so until then a declared - controller fails as "unknown controller", which is the contract's behaviour for an unregistered name. -- **the on-demand trigger**: the workload-cluster pipeline runs on `/run` against a PR, not just nightly, so a - cloud-path change can get its result without waiting. -- **the conformance suite** lives here with the RFC: the fixture app and the assertions, parity check - included. Both runners run it in CI; it's what "conforms" means. -- **devctl `gen apptest` and template-app** scaffold the layout for new repos. -- Migration happens as repos get touched; no flag day. Pilot: - [giantswarm/muster#954](https://github.com/giantswarm/muster/pull/954). - ## Alternatives considered - **A shared assertions library both harnesses import per repo.** Tried it in muster; the module, replace From 3d9c9365fe24e8295fbb9a22d1c8d88b74cd7720 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Pi=C4=85tkowski?= Date: Thu, 16 Jul 2026 17:02:43 +0200 Subject: [PATCH 20/24] add config proposal --- app-testing-contract/README.md | 50 ++++++++++++++++++++++++---------- 1 file changed, 36 insertions(+), 14 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 2a9911f..e407cbe 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -91,8 +91,8 @@ tests are executed as a separate process by the test toolkits. The only requirem - all the information about the test environmenrt is passed to the tests via environment variables, and the tests should not depend on any other external information (like a config file or a specific cluster setup). The test toolkits are responsible for setting up the environment and passing the information to the tests. -- exit code `0` means the test run passed, including the case where it executed no tests, and any other - non-zero exit code means the test run failed. +- exit code `0` means the test run passed and at least 1 test was executed, exit code `5` means there was no + error, but no test was executed at all, and any other non-zero exit code means the test run failed. - test developer can deliver hooks that are executed by the test toolkits (see below). #### The conventional directory @@ -235,24 +235,46 @@ We define the following capabilities, which the toolkit sets in `APP_TEST_CAPABI # TODO: define and complete the capabilites list -### Shared configuration +### Toolkit outputs + +Toolkits should let the test frameworks they execute to log to stdout/stderr, and should not filter or +redirect the output. Test toolkit exit code `0` means the test run passed and at least 1 test was executed, +exit code `5` means there was no error, but no test was executed at all, and any other non-zero exit code +means the test run failed. + +### Shared toolkit configuration + +Test toolkits need to know some information about how to handle the helm chart under test, i.e. in which +namespace it should be installed or what `values.yaml` file should be used. This information has to be easily +set in CI/CD pipelines, where config files are not convenient when the configuration has to be dynamic. Thus, +we propose a shared optional config file that both toolkits should use. Each of the config options in the file +must accept environment variable overrides, as specified below. Each test toolkit should print the effective +configuration it is using at the start of the toolkit run. Test _code_ lives in `tests/app/`; shared _configuration_ lives in `.apptest/`. `.apptest/config.yaml` holds -only what both toolkits need, and the chart values files. The config schema is: +only what both toolkits need, and the chart values files. The config schema is (with default values and +respective env vars): ```yaml -installNamespace: kube-system -testTypes: [smoke, functional, upgrade] # optional: types that must collect at least one test +releaseNamespace: default # APP_TEST_RELEASE_NAMESPACE +testTypes: [smoke, functional, upgrade] # APP_TEST_TEST_TYPES="a,b,c" - normally autodetected +chartConfig: + shared: # optional configuration files, applied and merged in the list order, shared between both toolkits + valueFiles: + - file1.yaml + - file2.yaml + toolkitSpecific: # mutually exclusive with chartConfig.shared + - name: ats # ATS reads its own entry + valueFiles: + - file1.yaml + - file2.yaml + - name: atf # apptest-framework reads its own entry + valueFiles: + - file1.yaml + - file2.yaml ``` -The one lint: a type in `expectedTypes` that collects zero tests fails the run. `expectedTypes` is optional; -leave it out to keep the "no tests is fine" default. It's also how you make a type mandatory. List `upgrade`, -and a typo'd tag (which collects zero) fails instead of quietly skipping the flow. - -Everything harness-specific stays in that harness's config: `.ats/main.yaml` (cluster types, catalogs, -executor options) and `tests/e2e/config.yaml` (appCatalog, providers, MC options). Values files stay -per-harness too; a kind cluster and a workload cluster legitimately want different values, and each harness -loads them its own way. +# TODO: just a config proopsal, discuss ## Alternatives considered From 313c0dab47732cb7d7d8e9df0dc35c14bf3d3244 Mon Sep 17 00:00:00 2001 From: Quentin Bisson Date: Thu, 16 Jul 2026 18:08:41 +0200 Subject: [PATCH 21/24] Update README.md --- app-testing-contract/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index e407cbe..f93e8f8 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -1,6 +1,6 @@ --- creation_date: 2026-07-07 -issues: []j +issues: [] owners: - https://github.com/orgs/giantswarm/teams/team-bumblebee - https://github.com/orgs/giantswarm/teams/team-honeybadger From 6699084149953332bee8283977880bae4e90b7bc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Pi=C4=85tkowski?= Date: Mon, 20 Jul 2026 14:11:15 +0200 Subject: [PATCH 22/24] make better config section --- app-testing-contract/README.md | 59 ++++++++++++++++++++-------------- 1 file changed, 35 insertions(+), 24 deletions(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index f93e8f8..0a0bb64 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -223,11 +223,12 @@ stay ATS-only. `APP_TEST_CAPABILITIES` is new, computed by the runner, with no ` #### Capabilities -The cluster intended for PR testing is `kind`. As a simple cluster instance, it lacks features like cloud -identity, real storage, or load balancers. Tests that need those capabilities should gate on the -`APP_TEST_CAPABILITIES` and only run nightly on a real workload cluster. +The cluster intended for PR testing with `ats` is `kind`. As a simple cluster instance, it lacks features like +cloud identity, real storage, or load balancers. On the other hand, a "full" cluster, like created by `atf`, +will probably be able to deliver some of these capabilities. To cope with these differences, tests that need +those capabilities should gate on the `APP_TEST_CAPABILITIES` and only run nightly on a real workload cluster. -We define the following capabilities, which the toolkit sets in `APP_TEST_CAPABILITIES`: +We define the following capabilities, which a toolkit might set in `APP_TEST_CAPABILITIES`: - cloud-identity - persistent-storage @@ -239,39 +240,49 @@ We define the following capabilities, which the toolkit sets in `APP_TEST_CAPABI Toolkits should let the test frameworks they execute to log to stdout/stderr, and should not filter or redirect the output. Test toolkit exit code `0` means the test run passed and at least 1 test was executed, -exit code `5` means there was no error, but no test was executed at all, and any other non-zero exit code -means the test run failed. +exit code `5` means there was no error, but no test was executed at all (this is based on the `pytest` +convention), and any other non-zero exit code means the test run failed. ### Shared toolkit configuration Test toolkits need to know some information about how to handle the helm chart under test, i.e. in which -namespace it should be installed or what `values.yaml` file should be used. This information has to be easily -set in CI/CD pipelines, where config files are not convenient when the configuration has to be dynamic. Thus, -we propose a shared optional config file that both toolkits should use. Each of the config options in the file -must accept environment variable overrides, as specified below. Each test toolkit should print the effective -configuration it is using at the start of the toolkit run. +namespace it should be installed or what `values.yaml` file should be used to render it. The test developer +should be able to easily configure that and make sure that it's respected by toolkits. On the other hand, this +configuration has to be easily set in CI/CD pipelines, where config files are not convenient to work with, as +the configuration has to be dynamic. Thus, we propose a shared optional config file that both toolkits must +use, but with overrides possible to set using environment variables, so that an integration with CI pipelines +is easy (i.e. matrix builds). Each of the config options in the file must be loaded by a toolkit, but the +toolkit also has to accept environment variable overrides, as specified below. Each test toolkit should print +the effective configuration it is using at the start of the toolkit run. -Test _code_ lives in `tests/app/`; shared _configuration_ lives in `.apptest/`. `.apptest/config.yaml` holds -only what both toolkits need, and the chart values files. The config schema is (with default values and -respective env vars): +For a test suite, the test _code_ lives in `tests/app/`; the shared _configuration_ lives in +`.apptest/config.yaml`. The file provides only what both toolkits can interpret. For each test toolkit, an +override file might be created, with the name `.apptest/config.[TOOLKIT_NAME].yaml`, with the same schema as +the main file. The toolkit specific config file acts as an override over the values loaded from the shared +`config.yaml`. + +To sum up, the effective configuration is built by test toolkits in the following order, starting from the +lowest priority: + +- `.apptest/config.yaml` (shared config file, optional) +- `.apptest/config.[TOOLKIT_NAME].yaml` (toolkit specific config file, optional) +- environment variables (highest priority) + +The reserved toolkit specific names and related overrides are: + +- `ats` - `.apptest/config.ats.yaml` +- `atf` - `.apptest/config.atf.yaml` + +The proposed config schema is (including the default values and respective env vars for overrides): ```yaml releaseNamespace: default # APP_TEST_RELEASE_NAMESPACE testTypes: [smoke, functional, upgrade] # APP_TEST_TEST_TYPES="a,b,c" - normally autodetected chartConfig: - shared: # optional configuration files, applied and merged in the list order, shared between both toolkits + shared: # optional configuration files, applied by the toolkit in the list order valueFiles: - file1.yaml - file2.yaml - toolkitSpecific: # mutually exclusive with chartConfig.shared - - name: ats # ATS reads its own entry - valueFiles: - - file1.yaml - - file2.yaml - - name: atf # apptest-framework reads its own entry - valueFiles: - - file1.yaml - - file2.yaml ``` # TODO: just a config proopsal, discuss From e8d46e7641ace99b05176f34dab809d8feb94dcb Mon Sep 17 00:00:00 2001 From: Quentin Bisson Date: Thu, 23 Jul 2026 15:11:14 +0200 Subject: [PATCH 23/24] Update app-testing-contract/README.md --- app-testing-contract/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 0a0bb64..3a640db 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -34,7 +34,7 @@ We propose to keep both toolkits, but specialize them to the two most frequent u the cluster type. - **apptest-framework (ATF)** takes the opposite approach: it chooses environment realism over the time needed to execute the tests. It creates a real workload cluster, installs the chart using the App Platform, and - runs the full suite, preferably nightly. + runs the full suite. With this in mind, it's clear that to provide a comprehensive test coverage and to follow the "fail fast" principle, we need to use both toolkits. From 399d83c30b67de27bac79412ba482e67780a1eb3 Mon Sep 17 00:00:00 2001 From: Quentin Bisson Date: Thu, 23 Jul 2026 15:11:50 +0200 Subject: [PATCH 24/24] Update app-testing-contract/README.md --- app-testing-contract/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/app-testing-contract/README.md b/app-testing-contract/README.md index 3a640db..04a8565 100644 --- a/app-testing-contract/README.md +++ b/app-testing-contract/README.md @@ -226,7 +226,7 @@ stay ATS-only. `APP_TEST_CAPABILITIES` is new, computed by the runner, with no ` The cluster intended for PR testing with `ats` is `kind`. As a simple cluster instance, it lacks features like cloud identity, real storage, or load balancers. On the other hand, a "full" cluster, like created by `atf`, will probably be able to deliver some of these capabilities. To cope with these differences, tests that need -those capabilities should gate on the `APP_TEST_CAPABILITIES` and only run nightly on a real workload cluster. +those capabilities should gate on the `APP_TEST_CAPABILITIES`. We define the following capabilities, which a toolkit might set in `APP_TEST_CAPABILITIES`: