diff --git a/doc/contributing-konflux-testing-rhoai.md b/doc/contributing-konflux-testing-rhoai.md index a8af51400..3e257600a 100644 --- a/doc/contributing-konflux-testing-rhoai.md +++ b/doc/contributing-konflux-testing-rhoai.md @@ -13,7 +13,7 @@ Konflux/Tekton objects, OLM and operator vocabulary, and [**olminstall**](../int | **Snapshot** | A Konflux [CR](#cr) that records built component images. When a Snapshot matches an [ITS](#its), the Integration Service creates a [PipelineRun](#pipelinerun). | | **ITS** | IntegrationTestScenario: A Konflux [CR](#cr) that defines which pipeline to run and which applications or [Snapshots](#snapshot) should trigger it. | | **Integration Service** | Konflux subsystem that matches [Snapshots](#snapshot) to [ITS](#its) resources and creates [PipelineRuns](#pipelinerun) ([docs](https://konflux-ci.dev/docs/how-tos/testing/integration/)). | -| **BVT** | Build Verification Test — basic post-install checks to confirm the operator deployed and core behavior works. | +| **BVT** | Build Verification Test — basic post-install checks to confirm the operator deployed and core behavior works. The Konflux [olminstall](../integration-tests/olminstall/README.md) pipeline runs `opendatahub-tests` pytest markers `cluster_health` and `operator_health` after [CSV](#csv) success when `TESTS` includes `bvt` (aligned with Jenkins `validateHealth`). | | **CSV** | ClusterServiceVersion — the [OLM](#olm) object that represents an installed operator version; reaching `Succeeded` means the operator is running | | **CRI-O** | Container Runtime Interface for OCI — the container runtime used by OpenShift nodes | | **DBus** | D-Bus — Linux inter-process communication bus; used here by [HCCO](#hcco) to signal kubelet to restart | @@ -134,7 +134,7 @@ If the file uses a fixed `metadata.name`, delete the previous run before recreat ## Snapshot-driven example (olminstall) -The **olminstall** integration under `integration-tests/olminstall/` exercises [Snapshot](#snapshot) + [ITS](#its): Konflux [EaaS](#eaas) provisions a short-lived [HyperShift](#hypershift) cluster, installs the operator from the snapshot catalog, then destroys the cluster. For triggers (including [`olm_pipeline.py`](../integration-tests/olminstall/olm_pipeline.py)), parameters, pipeline behavior, and glossary cross-links, see [integration-tests/olminstall/README.md](../integration-tests/olminstall/README.md) (**Triggering**). Vocabulary is in [Terms and abbreviations](#terms-and-abbreviations) above. +The **olminstall** integration under `integration-tests/olminstall/` exercises [Snapshot](#snapshot) + [ITS](#its): Konflux [EaaS](#eaas) provisions a short-lived [HyperShift](#hypershift) cluster, installs the operator from the snapshot catalog, then destroys the cluster. For triggers (including [`olm_pipeline.py`](../integration-tests/olminstall/olm_pipeline.py)), parameters, pipeline behavior, and glossary cross-links, see [integration-tests/olminstall/README.md](../integration-tests/olminstall/README.md) (**Triggering**). To stream logs or replay pruned runs from KubeArchive, use **`python3 integration-tests/olminstall/olm_pipeline.py --watch`** (see that README). Vocabulary is in [Terms and abbreviations](#terms-and-abbreviations) above. diff --git a/integration-tests/olminstall/README.md b/integration-tests/olminstall/README.md index 3ec0c8eeb..dd2b984b5 100644 --- a/integration-tests/olminstall/README.md +++ b/integration-tests/olminstall/README.md @@ -1,6 +1,6 @@ # olminstall Integration Test Scenario -End-to-end Konflux integration test for [ODH](../../doc/contributing-konflux-testing-rhoai.md#odh)/[RHOAI](../../doc/contributing-konflux-testing-rhoai.md#rhoai) operator installation via [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm). Provisions an ephemeral [HyperShift](../../doc/contributing-konflux-testing-rhoai.md#hypershift) cluster using Konflux [EaaS](../../doc/contributing-konflux-testing-rhoai.md#eaas) ([provisioning docs](https://konflux.pages.redhat.com/docs/users/testing/cluster-provisioning.html#methods)), installs the operator from the [FBCF](../../doc/contributing-konflux-testing-rhoai.md#fbc--fbcf) catalog image in the Konflux [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot), and verifies the [CSV](../../doc/contributing-konflux-testing-rhoai.md#csv) reaches `Succeeded`. +End-to-end Konflux integration test for [ODH](../../doc/contributing-konflux-testing-rhoai.md#odh)/[RHOAI](../../doc/contributing-konflux-testing-rhoai.md#rhoai) operator installation via [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm). Provisions an ephemeral [HyperShift](../../doc/contributing-konflux-testing-rhoai.md#hypershift) cluster using Konflux [EaaS](../../doc/contributing-konflux-testing-rhoai.md#eaas) ([provisioning docs](https://konflux.pages.redhat.com/docs/users/testing/cluster-provisioning.html#methods)), installs the operator from the [FBCF](../../doc/contributing-konflux-testing-rhoai.md#fbc--fbcf) catalog image in the Konflux [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot), verifies the [CSV](../../doc/contributing-konflux-testing-rhoai.md#csv) reaches `Succeeded`, then runs [BVT](../../doc/contributing-konflux-testing-rhoai.md#bvt) (`opendatahub-tests` `cluster_health` and `operator_health` markers). **Terms and abbreviations:** [BVT](../../doc/contributing-konflux-testing-rhoai.md#bvt), [CSV](../../doc/contributing-konflux-testing-rhoai.md#csv), [EaaS](../../doc/contributing-konflux-testing-rhoai.md#eaas), [FBC / FBCF](../../doc/contributing-konflux-testing-rhoai.md#fbc--fbcf), [HyperShift](../../doc/contributing-konflux-testing-rhoai.md#hypershift), [IDMS](../../doc/contributing-konflux-testing-rhoai.md#idms), [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm), [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot), [ITS](../../doc/contributing-konflux-testing-rhoai.md#its), [full glossary](../../doc/contributing-konflux-testing-rhoai.md#terms-and-abbreviations) ([DBus](../../doc/contributing-konflux-testing-rhoai.md#dbus), [DSC](../../doc/contributing-konflux-testing-rhoai.md#dsc), [HCCO](../../doc/contributing-konflux-testing-rhoai.md#hcco), [MCO](../../doc/contributing-konflux-testing-rhoai.md#mco), …). @@ -22,6 +22,9 @@ flowchart TD HCCO["🤖 HCCO syncs kubelet creds to all nodes"]:::hcco OLM["📦 OLM: CatalogSource + Subscription + bundle-unpack + CSV"]:::olm PASS["✅ CSV Succeeded - operator version recorded"]:::pass + BVT_RESOLVE["🔎 Resolve opendatahub-tests image tag from CSV"]:::olm + BVT_RUN["🧪 BVT pytest cluster_health + operator_health"]:::pass + ALLPASS["✅ Install and BVT passed"]:::pass FAIL["❌ Failed - oc adm inspect + diagnostics collected"]:::fail BUILD -->|~20 min to provision| CLUSTER @@ -30,7 +33,12 @@ flowchart TD AUTH -->|rhoai-quay-pull linked to SA| OLM HCCO -->|nodes synced before Subscription| OLM OLM --> PASS + PASS --> BVT_RESOLVE + BVT_RESOLVE --> BVT_RUN + BVT_RUN --> ALLPASS OLM -.->|timeout / error| FAIL + BVT_RESOLVE -.->|resolve failure| FAIL + BVT_RUN -.->|pytest failure| FAIL ``` The `BUILD` node is the entry point for both **automatic** and **manual** runs (see [Triggering](#triggering) and the [contributing guide](../../doc/contributing-konflux-testing-rhoai.md)). @@ -39,27 +47,54 @@ The `BUILD` node is the entry point for both **automatic** and **manual** runs ( 1. **Parses the snapshot** — extracts the [FBCF](../../doc/contributing-konflux-testing-rhoai.md#fbc--fbcf) `containerImage` for the configured `FBCF_COMPONENT_NAME`. 2. **provision-eaas-space** — reserves an [EaaS](../../doc/contributing-konflux-testing-rhoai.md#eaas) environment using the `provision-eaas-space` step action from [konflux-ci/build-definitions](https://github.com/konflux-ci/build-definitions) (`main`). -3. **provision-cluster** — queries EaaS for supported versions, selects the latest patch release for the chosen prefix, and creates an ephemeral [HyperShift](../../doc/contributing-konflux-testing-rhoai.md#hypershift) cluster (AWS, `m5.2xlarge` by default) via `konflux-ci/build-definitions` step actions. Configures an [IDMS](../../doc/contributing-konflux-testing-rhoai.md#idms) mirror: `registry.redhat.io/rhoai` → `quay.io/rhoai`. +3. **provision-cluster** — shallow-clones `SCRIPTS_REPO_*` (so helpers are on disk), queries EaaS for supported versions via `konflux-ci/build-definitions` step actions, writes the version prefix with [`helpers/resolve_ocp_prefix.py`](helpers/resolve_ocp_prefix.py), resolves the latest patch, then provisions an ephemeral [HyperShift](../../doc/contributing-konflux-testing-rhoai.md#hypershift) cluster (AWS, `m5.2xlarge` by default) using [`helpers/create_eaas_cluster.py`](helpers/create_eaas_cluster.py). Configures an [IDMS](../../doc/contributing-konflux-testing-rhoai.md#idms) mirror: `registry.redhat.io/rhoai` → `quay.io/rhoai`. 4. **install-operator** — clones two repos and runs scripts against the provisioned cluster: - [opendatahub-io/odh-konflux-central](https://github.com/opendatahub-io/odh-konflux-central) (`SCRIPTS_REPO_URL` / `SCRIPTS_REPO_REVISION`): provides [`helpers/patch_cluster_pull_secret.py`](helpers/patch_cluster_pull_secret.py) and [`helpers/install_and_verify.py`](helpers/install_and_verify.py). - olminstall repo (`OLMINSTALL_REPO_URL` / `OLMINSTALL_REPO_REVISION`): provides the `resources/install-rhods-operator.yaml` template (Namespace + OperatorGroup + Subscription) and `utils/oc_wait.sh` / `utils/oc_approve.sh` utilities. This avoids re-implementing tested [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm) install logic. - `patch_cluster_pull_secret.py`: merges `quay.io/rhoai` credentials into the cluster pull secret, creates an `additional-pull-secret` in `kube-system` for [HCCO](../../doc/contributing-konflux-testing-rhoai.md#hcco) node sync (see [Auth strategy](#auth-strategy-for-idms-mirrors)), and creates `rhoai-quay-pull` in `openshift-marketplace`. - `install_and_verify.py`: creates the CatalogSource (using the [FBCF](../../doc/contributing-konflux-testing-rhoai.md#fbc--fbcf) image — the part not covered by olminstall), waits for [HCCO](../../doc/contributing-konflux-testing-rhoai.md#hcco) to sync credentials to all nodes, then delegates Subscription creation, InstallPlan approval, and [CSV](../../doc/contributing-konflux-testing-rhoai.md#csv) wait to olminstall's resources and utilities. -5. **post-results** — sends a Slack notification (if `SLACK_WEBHOOK_URL` is configured) and reports final status. `TEST_OUTPUT` is exposed from `install-operator` results. -6. **collect-diagnostics** _(on failure)_ — runs `oc adm inspect` on the operator namespace and relevant [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm) resources via a `konflux-ci/build-definitions` step action. +5. **parse-pipeline-tests** — runs in parallel with snapshot parsing / provisioning and **must finish before install-operator** (when **`PRODUCT`** is **`rhoai`** or **`odh`**); clones `SCRIPTS_REPO_*` to `/workspace/repo`, runs [`helpers/write_pipeline_test_flags.py`](helpers/write_pipeline_test_flags.py) with param `TESTS` (default `bvt,smoke`) and [`olminstall-tests-config.yaml`](olminstall-tests-config.yaml) to set Tekton results (`RUN_SMOKE`, `RUN_BVT`, `RUN_TIER1`, …). Any phase with `requiredInSelection: true` in that file must appear in `TESTS` (none are required in the default catalog). OLM install runs only when **`PRODUCT`** is not **`none`**; **`TESTS`** selects optional post-install phases (`smoke` placeholder, `bvt`, `tier1` placeholder) independently. +6. **smoke-placeholder** _(when `smoke` is in `TESTS`)_ — no-op step reserved for a future smoke pytest suite (`RUN_SMOKE` from config). +7. **resolve-opendatahub-tests-image-no-install** / **resolve-opendatahub-tests-image-after-install** _(when `bvt` is in `TESTS`)_ — mutually exclusive by **`PRODUCT`**: with **`PRODUCT=none`**, **`no-install`** clones `SCRIPTS_REPO_*` and runs [`helpers/resolve_opendatahub_tests_image.py`](helpers/resolve_opendatahub_tests_image.py) with an empty CSV version (→ **`opendatahub-tests:latest`**). With **`PRODUCT`** **`rhoai`**/**`odh`**, **`after-install`** runs the same helper after **`install-operator`** using the installed CSV version (`skopeo` probe, fallback `:latest`). +8. **bvt-health-checks-no-eaas** / **bvt-health-checks-with-eaas** _(when `bvt` is in `TESTS`)_ — **`no-eaas`** ( **`PRODUCT=none`** ) uses a stub kubeconfig so **`uv run pytest`** still runs and fails against no API. **`with-eaas`** fetches the EaaS kubeconfig and runs the same pytest markers (`cluster_health`, `operator_health`) as before. +9. **tier1-placeholder** _(when `tier1` is in `TESTS`)_ — no-op step reserved for a future tier1 suite. +10. **post-results** — sends a Slack notification (if `SLACK_WEBHOOK_URL` is configured). The Konflux **post-results → Details → Parameters** panel shows **`ARTIFACTS_URL`** (predicted browser path from pipeline params + run name; safe — no `$(tasks.*.results)` reference). **`patch-summary-annotations`** patches **`olminstall.*`** annotations when upload published a different URL, emits Tekton results for the PipelineRun **Results** panel, and prints the full summary in that step’s log. Pass/fail reflects aggregate pipeline task status (skipped optional tasks do not fail the run). **`TEST_OUTPUT`** maps to **`install-operator`** `INSTALL_STATUS` (see [Tekton pipeline results](https://tekton.dev/docs/pipelines/pipelines/#emitting-results-from-a-pipeline)). +11. **collect-diagnostics** _(on failure)_ — runs `oc adm inspect` on the operator namespace and relevant [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm) resources via a `konflux-ci/build-definitions` step action. + +### Konflux UI: PipelineRun results, task logs, and files on the step pod + +- **PipelineRun / TaskRun logs:** Open your application → **Pipeline runs** (or activity / pipelines for your tenant), select the run, then open individual **tasks** / **steps** to stream logs ([Konflux user flows vary slightly by deployment](https://konflux-ci.dev/docs/getting-started/)). +- **Tekton results surfaced in the UI:** Integration pipelines often expose standard **`TEST_OUTPUT`** and similar task results; Konflux [documents](https://konflux-ci.dev/docs/testing/integration/standardized-outputs/) that task-level results appear when you **click the task name** and inspect the details panel. +- **Pipeline-level results:** The **`PipelineRun` → Results / Summary** area shows values wired under `spec.results` (when emitted); some references are omitted when the source task was skipped. +- **post-results task Results (Konflux):** Open **post-results** (not only `bvt-health-checks-*`) → **Details** → **Results** for **`TEST_OUTPUT`** and **`ARTIFACTS_URL`**, same names as the BVT tasks. The pipeline definition comes from the ITS **`resolverRef` revision** (not only `SCRIPTS_REPO_REVISION`); use the same branch as your fork (e.g. `olminstall-bvt`) for both when testing UI changes. +- **BVT JUnit in the shared artifact browser:** After **`bvt-health-checks-no-eaas`** or **`bvt-health-checks-with-eaas`** completes successfully, JUnit XML and console logs are pushed to **`quay.io/opendatahub/odh-ci-artifacts`** with OCI tag **`-bvt`**. The Konflux QE **Artifacts Browser** lists that registry under a path segment (default **`odh-ci-artifacts`** via **`ARTIFACT_BROWSER_REPO_PATH`**). The BVT task **`ARTIFACTS_URL`** result and **`post-results`** / **`olm_pipeline.py`** run summary only show a browser link when **`upload-artifacts`** actually published that result; otherwise they omit the line (if BVT was not in **`TESTS`**) or explain why nothing was uploaded (BVT skipped/failed early). + +- **Large files on the pod filesystem:** Files written only inside the pod (**`/artifacts/*.xml`**, **`collect-diagnostics`** **`/diag/diagnostics-bundle.tgz`**, etc.) are **not automatically downloadable as blobs from the Konflux UI** unless a task publishes them as a Tekton **result** (size-limited) or copies them to external storage. In practice you rely on **step logs** (this pipeline **`tee`**s console output into the log) or copy from the pod while it still exists, e.g. **`oc cp /:/artifacts/cluster-health.xml .`** (exact pod name from the TaskRun’s Pod; completed pods may be **deleted after GC**). **Uncertainty:** Red Hat hosted Konflux builds evolve — confirm the exact panel labels (**Pipeline run**, **Task runs**, **Results**) in your tenant. ## Files | File | Purpose | |------|---------| -| [`olminstall-smoke-pipeline.yaml`](olminstall-smoke-pipeline.yaml) | Pipeline: [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot) → [EaaS](../../doc/contributing-konflux-testing-rhoai.md#eaas) cluster → install → verify | -| [`its-olminstall-open-data-hub-tenant.yaml`](its-olminstall-open-data-hub-tenant.yaml) | [ITS](../../doc/contributing-konflux-testing-rhoai.md#its) for [ODH](../../doc/contributing-konflux-testing-rhoai.md#odh) (`open-data-hub-tenant`, `odh-operator-catalog` component) | -| [`its-olminstall-rhoai-tenant.yaml`](its-olminstall-rhoai-tenant.yaml) | [ITS](../../doc/contributing-konflux-testing-rhoai.md#its) for [RHOAI](../../doc/contributing-konflux-testing-rhoai.md#rhoai) sandbox testing (`rhoai-tenant`, `rhoai-fbc-fragment-ocp-421`) | +| [`olminstall-pipeline.yaml`](olminstall-pipeline.yaml) | Pipeline: [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot) → [EaaS](../../doc/contributing-konflux-testing-rhoai.md#eaas) cluster → install → phases from **`TESTS`** (`smoke` placeholder, **`bvt`**, `tier1` placeholder) | +| [`olminstall-tests-config.yaml`](olminstall-tests-config.yaml) | Declarative **phases** (ids, defaults, Tekton `RUN_*` mapping); read by `olm_pipeline.py` and by `parse-pipeline-tests` after cloning `SCRIPTS_REPO` | +| [`helpers/tekton_util.py`](helpers/tekton_util.py) | Shared library: `require_env`, `write_result`, `git_clone` (with optional RH internal TLS workaround), `run`, `parse_junit_summary` | +| [`helpers/create_eaas_cluster.py`](helpers/create_eaas_cluster.py) | Tekton step: provision an ephemeral EaaS HyperShift cluster via `ClusterTemplateInstance` | +| [`helpers/resolve_ocp_prefix.py`](helpers/resolve_ocp_prefix.py) | Tekton step: derive `OCP_VERSION_PREFIX` / default-minor prefix string for EaaS `pick-version` | +| [`helpers/extract_fbcf_image.py`](helpers/extract_fbcf_image.py) | Tekton step: extract FBCF container image from a Konflux `ApplicationSnapshot` JSON | +| [`helpers/resolve_opendatahub_tests_image.py`](helpers/resolve_opendatahub_tests_image.py) | Tekton step: maps installed CSV version to `opendatahub-tests` image tag (`skopeo` probe, `:latest` fallback) | +| [`helpers/run_bvt_pytest.py`](helpers/run_bvt_pytest.py) | Tekton step: parameterised BVT pytest runner (`PYTEST_MARKER`, `PYTEST_EXTRA_ARGS`, `ARTIFACT_PREFIX`) with `uv`/pip fallback | +| [`helpers/emit_test_output.py`](helpers/emit_test_output.py) | Tekton step: parse JUnit XML files and write a Konflux-standardised `TEST_OUTPUT` result | +| [`helpers/collect_diagnostics.py`](helpers/collect_diagnostics.py) | Tekton step: `oc adm inspect` + marketplace diagnostics on failure | +| [`helpers/send_notification.py`](helpers/send_notification.py) | Tekton step: Slack notification summarising pipeline run results | | [`helpers/patch_cluster_pull_secret.py`](helpers/patch_cluster_pull_secret.py) | Tekton step: injects `quay.io/rhoai` credentials into the [EaaS](../../doc/contributing-konflux-testing-rhoai.md#eaas) cluster at all required levels | +| [`its-olminstall-open-data-hub-tenant.yaml`](its-olminstall-open-data-hub-tenant.yaml) | [ITS](../../doc/contributing-konflux-testing-rhoai.md#its) `odh-olminstall` for [ODH](../../doc/contributing-konflux-testing-rhoai.md#odh) (`open-data-hub-tenant`, `odh-operator-catalog` component) | +| [`its-olminstall-rhoai-tenant.yaml`](its-olminstall-rhoai-tenant.yaml) | [ITS](../../doc/contributing-konflux-testing-rhoai.md#its) `odh-olminstall-testops` for [RHOAI](../../doc/contributing-konflux-testing-rhoai.md#rhoai) sandbox (`rhoai-tenant`, `rhoai-fbc-fragment-ocp-421`) | +| [`helpers/tests_plan.py`](helpers/tests_plan.py) | Validates/normalizes `TESTS` strings using [`olminstall-tests-config.yaml`](olminstall-tests-config.yaml) (or `--tests-config`) | | [`helpers/install_and_verify.py`](helpers/install_and_verify.py) | Tekton step: creates [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm) resources, waits for [CSV](../../doc/contributing-konflux-testing-rhoai.md#csv) `Succeeded`, writes `INSTALL_STATUS` | -| [`olm_pipeline.py`](olm_pipeline.py) | Local CLI — apply ITS with optional overrides (`SCRIPTS_REPO_*`, `UPDATE_CHANNEL`, `OCP_VERSION_PREFIX` via `--ocp-version`), resolve an image, create a [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot), and stream logs | +| [`olm_pipeline.py`](olm_pipeline.py) | Local CLI — apply ITS with optional overrides, create a [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot), and stream logs. Default **`--product none`** injects pipeline param **`PRODUCT=none`**: EaaS cluster + operator install are skipped; **`--product rhoai`** / **`odh`** runs the full install path (optionally with **`--image`** to override the snapshot FBC/catalog digest). **`TESTS`** is independent (for example **`--tests bvt`** still selects BVT when install is skipped). | | [`requirements.txt`](requirements.txt) | Documents Python deps for this directory (stdlib-only for [`olm_pipeline.py`](olm_pipeline.py) and [`helpers/`](helpers/); no `pip install` required) | | [`test-snapshot.yaml`](test-snapshot.yaml) | Example [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot) for manual pipeline trigger | +| [`helpers/prune_stale_testops_its.py`](helpers/prune_stale_testops_its.py) | Optional: `oc delete` legacy `IntegrationTestScenario` names before raw `oc create -f test-snapshot.yaml` (same list as trigger-time prune) | | [`test-pipelinerun.yaml`](test-pipelinerun.yaml) | Example [PipelineRun](../../doc/contributing-konflux-testing-rhoai.md#pipelinerun) for local/manual execution | ## Tenant and application @@ -68,6 +103,12 @@ The `BUILD` node is the entry point for both **automatic** and **manual** runs ( [`its-olminstall-rhoai-tenant.yaml`](its-olminstall-rhoai-tenant.yaml) targets **`rhoai-tenant`**, application **`testops-playpen`**, used for development iteration and sandbox testing of the [RHOAI](../../doc/contributing-konflux-testing-rhoai.md#rhoai) FBC fragment builds. +**Why extra PipelineRuns appear:** A Konflux [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot) for an **Application** starts **one `PipelineRun` per `IntegrationTestScenario`** that matches that app. Old scenarios still **on the cluster** (for example `rhoai-test` → `testops-e2e-test`) are **not** removed when you update git; they keep firing until deleted. **`testops-playpen-enterprise-contract-*`** runs are **Enterprise Contract** policy checks — separate from olminstall; tune or disable them in Konflux application / release / EC settings for your tenant, not via `olminstall-pipeline.yaml`. + +**`olm_pipeline.py` default:** For default **`--namespace rhoai-tenant`** and **`--app testops-playpen`**, before `oc apply` + Snapshot the CLI runs **`oc delete integrationtestscenario`** for the legacy name `rhoai-test` so you typically only see **`odh-olminstall-testops-*`** from this repo’s ITS. Pass **`--no-prune-stale-its`** to skip that delete. Raw **`oc create -f test-snapshot.yaml`** does not prune — use the CLI or run [`helpers/prune_stale_testops_its.py`](helpers/prune_stale_testops_its.py) first. + +> **Pipeline name:** [`olminstall-pipeline.yaml`](olminstall-pipeline.yaml) defines `metadata.name: odh-olminstall-test`. In-tree [ITS](../../doc/contributing-konflux-testing-rhoai.md#its) objects are `odh-olminstall` (ODH) and `odh-olminstall-testops` (RHOAI sandbox). `--list` includes every `PipelineRun` for the selected application label (for example olminstall, enterprise-contract, or any other ITS graph). + The pipeline also needs a tenant secret with quay credentials. Each ITS sets `QUAY_PULL_SECRET_NAME`: - `its-olminstall-open-data-hub-tenant.yaml` uses `odh-quay-secret` - `its-olminstall-rhoai-tenant.yaml` uses `rhoai-quay-secret` @@ -109,22 +150,22 @@ LATEST=$(oc get snapshots -n rhoai-tenant \ sed "s|containerImage:.*|containerImage: $LATEST|" \ integration-tests/olminstall/test-snapshot.yaml | oc create -n rhoai-tenant -f - oc get pipelinerun -n rhoai-tenant -tkn pipelinerun logs -n rhoai-tenant --last -f +python3 integration-tests/olminstall/olm_pipeline.py --watch -n rhoai-tenant --app testops-playpen ``` For generic Konflux testing (login, namespaces, [PipelineRun](../../doc/contributing-konflux-testing-rhoai.md#pipelinerun) vs [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot)/[ITS](../../doc/contributing-konflux-testing-rhoai.md#its)), see [contributing guide](../../doc/contributing-konflux-testing-rhoai.md#terms-and-abbreviations). Tooling for local debug commands in this section: - `oc` (required) -- `python3` (required for [`olm_pipeline.py`](olm_pipeline.py)) -- `tkn` (recommended for logs; otherwise poll with `oc`) +- `python3` (required for [`olm_pipeline.py`](olm_pipeline.py); use **`--watch`** to stream logs or replay from KubeArchive) +- `tkn` (optional; trigger mode uses it when installed, otherwise polls with `oc`) - `skopeo` (optional; used by `--product odh` when Konflux snapshots are unavailable) -- `yq` (required only when using `olm_pipeline.py` overrides such as `--konflux-repo`, `--konflux-branch`, `--channel`, or `--ocp-version`) +- `yq` (required for `olm_pipeline.py` trigger/apply: the CLI always patches ITS param **`PRODUCT`** to match **`--product`**, plus any of `--konflux-repo`, `--konflux-branch`, `--channel`, `--ocp-version`, non-default **`--tests`**, **`--slack-channel-id`**, or **`--product odh`** operator overrides) -Quick watch after triggering: +Quick watch after triggering (newest olminstall run for the app; add a PipelineRun name after `--watch` to target one run): ```bash -tkn pipelinerun logs -n rhoai-tenant --last -f +python3 integration-tests/olminstall/olm_pipeline.py --watch -n rhoai-tenant --app testops-playpen ``` ## Parameters (Pipeline) @@ -134,7 +175,7 @@ tkn pipelinerun logs -n rhoai-tenant --last -f | `FBCF_COMPONENT_NAME` | `odh-operator-catalog` | [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot) component name for the [FBCF](../../doc/contributing-konflux-testing-rhoai.md#fbc--fbcf) catalog image ([ITS](../../doc/contributing-konflux-testing-rhoai.md#its) overrides to `rhoai-fbc-fragment-ocp-421` for [RHOAI](../../doc/contributing-konflux-testing-rhoai.md#rhoai)) | | `UPDATE_CHANNEL` | `stable` | [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm) subscription channel | | `OPERATOR_NAMESPACE` | `redhat-ods-operator` | Namespace for operator installation (must match [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm) package expectations; `install_and_verify.py` adapts olminstall manifests to this namespace) | -| `OPERATOR_NAME` | `rhods-operator` | [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm) package name (use `rhods-operator` for [RHOAI](../../doc/contributing-konflux-testing-rhoai.md#rhoai), `opendatahub-operator` for [ODH](../../doc/contributing-konflux-testing-rhoai.md#odh)) | +| `OPERATOR_NAME` | `rhods-operator` | [OLM](../../doc/contributing-konflux-testing-rhoai.md#olm) package via olminstall `install-operator.sh`. [RHOAI](../../doc/contributing-konflux-testing-rhoai.md#rhoai) and **ODH `odh-stable`** (Konflux catalog) both use **`rhods-operator`** (same as Jenkins `odhTestConfigOperator`). Upstream ODH channels outside `odh-stable` / `odh-nightlies` may use `opendatahub-operator` — see Jenkins `generateTestConfigFile.groovy`. | | `HYPERSHIFT_INSTANCE_TYPE` | `m5.2xlarge` | AWS worker instance type for the ephemeral [HyperShift](../../doc/contributing-konflux-testing-rhoai.md#hypershift) cluster | | `SCRIPTS_REPO_URL` | `https://github.com/opendatahub-io/odh-konflux-central.git` | Repo that provides `integration-tests/olminstall/helpers/` (`patch_cluster_pull_secret.py`, `install_and_verify.py`) | | `SCRIPTS_REPO_REVISION` | `main` | Branch/SHA of the scripts repo | @@ -142,6 +183,9 @@ tkn pipelinerun logs -n rhoai-tenant --last -f | `OLMINSTALL_REPO_REVISION` | `main` | Branch/SHA of the olminstall repo | | `OLMINSTALL_CATALOG_NAME` | `rhoai-catalog-dev` | CatalogSource name used by olminstall's `install-operator.sh` | | `QUAY_PULL_SECRET_NAME` | `rhoai-quay-secret` | Tenant secret mounted for `quay.io/rhoai` credentials (`its-olminstall-open-data-hub-tenant.yaml` overrides to `odh-quay-secret`) | +| `PRODUCT` | `rhoai` (pipeline default); ITS examples set `rhoai` / `odh` | When **`none`**, Tekton skips **`provision-eaas-space`**, **`provision-cluster`**, and **`install-operator`**. **`rhoai`** / **`odh`** runs the full EaaS + install path. The CLI injects this from **`--product`**. | +| `OPENDATAHUB_TESTS_REPO` | `quay.io/opendatahub/opendatahub-tests` | Image repository (no tag) for [BVT](../../doc/contributing-konflux-testing-rhoai.md#bvt); tag derived from installed CSV when install ran, else from **`resolve_opendatahub_tests_image.py`** (empty version → `latest`) | +| `TESTS` | `bvt,smoke` | Comma-separated **phase ids** from [`olminstall-tests-config.yaml`](olminstall-tests-config.yaml) in `SCRIPTS_REPO`. Include every phase marked `requiredInSelection: true` (the default file has none). **`smoke`** selects the **`smoke-placeholder`** task until a real smoke suite exists. **`bvt`** runs **`bvt-health-checks-with-eaas`** when **`PRODUCT`** is **`rhoai`** or **`odh`**, and **`bvt-health-checks-no-eaas`** when **`PRODUCT=none`** (pytest against a stub kubeconfig until external kube is wired). Override via ITS or `olm_pipeline.py --tests` / `--tests-config`. | Sandbox development may override `SCRIPTS_*` / `OLMINSTALL_*` (and the ITS `resolverRef` URL/revision) so Konflux runs a pipeline revision that is not yet on `main`; see [`its-olminstall-rhoai-tenant.yaml`](its-olminstall-rhoai-tenant.yaml). @@ -150,11 +194,13 @@ Sandbox development may override `SCRIPTS_*` / `OLMINSTALL_*` (and the ITS `reso From the repo root, invoke `python3 integration-tests/olminstall/olm_pipeline.py` (paths shown below assume that working directory). With **no arguments**, it prints the same usage as `--help`. Use it for local Konflux olminstall workflows (trigger a run, watch logs, list runs, or query supported OCP). It can: - List latest PipelineRuns for the selected app (`--list-pipelines [N]`, default `10`), including archived runs from KubeArchive - Apply the [ITS](../../doc/contributing-konflux-testing-rhoai.md#its) safely on repeated runs -- Resolve an image (auto/latest, explicit `--image`, or `--product rhoai --version x.y`) -- Inject ITS overrides (`SCRIPTS_REPO_*`, `UPDATE_CHANNEL`) +- Resolve an image (explicit `--image`, **`--product rhoai`** with optional `--version`, **`--product odh`**, or **`--product none`** to keep the snapshot’s pinned image) +- Inject ITS overrides (`PRODUCT` from **`--product`**, `SCRIPTS_REPO_*`, `UPDATE_CHANNEL`, `--tests` → ITS param `TESTS`) - Watch your latest owned PipelineRun or a specific one (`--watch [PIPELINERUN]`), with KubeArchive fallback for runs pruned from the cluster - Create a [Snapshot](../../doc/contributing-konflux-testing-rhoai.md#snapshot), stream logs, and print a Konflux URL summary +**Default `--product` is `none`:** the ITS gets **`PRODUCT=none`** — Konflux skips EaaS provisioning and operator install. If **`TESTS`** includes **`bvt`**, **`bvt-health-checks-no-eaas`** still runs pytest (against a stub kubeconfig) so results reflect test failures rather than skipped tasks. Omit **`bvt`** from **`TESTS`** if you only want **`extract-fbcf-image`** / snapshot validation without pytest. Pass **`--product rhoai`** (or **`odh`**) for a full install + EaaS BVT. **`--image`** overrides the snapshot FBC/catalog digest when install runs (still requires **`--product rhoai`** or **`odh`** so install is not skipped). A future option may accept an existing cluster API URL / kubeconfig from this CLI. + Examples: ```bash @@ -165,7 +211,7 @@ python3 integration-tests/olminstall/olm_pipeline.py --help python3 integration-tests/olminstall/olm_pipeline.py --watch # Watch a specific existing PipelineRun -python3 integration-tests/olminstall/olm_pipeline.py --watch odh-olminstall-smoke-testops-xxxxx +python3 integration-tests/olminstall/olm_pipeline.py --watch odh-olminstall-testops-xxxxx # List latest PipelineRuns for selected app (default 10) python3 integration-tests/olminstall/olm_pipeline.py --list-pipelines @@ -197,16 +243,31 @@ python3 integration-tests/olminstall/olm_pipeline.py --product rhoai --ocp-versi # Override OLM channel python3 integration-tests/olminstall/olm_pipeline.py --channel beta +# OLM install + smoke placeholder only (skip BVT and tier1) +python3 integration-tests/olminstall/olm_pipeline.py --product rhoai --tests smoke + +# Install + BVT health pytest only (omit smoke / tier1 tokens from TESTS) +python3 integration-tests/olminstall/olm_pipeline.py --product rhoai --tests bvt + +# Full default phases + tier1 no-op placeholder +python3 integration-tests/olminstall/olm_pipeline.py --product rhoai --tests bvt,smoke,tier1 + +# Custom phases file (same schema as olminstall-tests-config.yaml; needs PyYAML or yq) +python3 integration-tests/olminstall/olm_pipeline.py --product rhoai \ + --tests-config /path/to/my-olminstall-tests.yaml --tests bvt,smoke + # Trigger against ODH (uses sandbox ITS with ODH-specific pipeline params) python3 integration-tests/olminstall/olm_pipeline.py --product odh ``` -Omit `--konflux-repo`/`--konflux-branch` to keep pipeline defaults (`opendatahub-io` + `main` for scripts clone). +**Without a Konflux cluster / RHOAI pipeline context**, a full `olm_pipeline.py` trigger or watch (including **`--tests bvt`**) will fail at **`oc whoami`** or **`ensure_konflux_cluster`** — that is expected. The standalone checker **[`verify_cli_args.py`](verify_cli_args.py)** only validates argument parsing and **should pass** without `oc`. A **future** option may accept an existing cluster API URL (or equivalent) for workflows that do not use this Konflux tenant; that is not implemented yet. -> **Concurrent runs:** The CLI does not take a cluster-side lock. If two users run the script simultaneously against the same namespace, both may create Snapshots and trigger separate PipelineRuns. On startup, the helper now tries to re-attach as early as possible: it first prefers running PipelineRuns marked with your current `oc whoami` identity, and if owner metadata is unavailable it falls back to the latest running `olminstall` PipelineRun for the same app (to avoid spawning a duplicate run). Use `--watch` to follow your latest owned run, or `--watch ` for an explicit run. The cleanup trap deletes your Snapshot on exit, but the other run will continue; deletion of a Snapshot mid-run is non-fatal to the PipelineRun (which has already resolved the snapshot). To avoid confusion, coordinate with your team before triggering manually in a shared namespace. +Omit `--konflux-repo`/`--konflux-branch` to keep the ITS default Git source for the remote pipeline definition. If you see **`CouldntGetPipeline`** / **`olminstall-pipeline.yaml`: file does not exist**, the default revision does not ship that path yet—use **`--konflux-repo`** + **`--konflux-branch`** on a fork/branch where `integration-tests/olminstall/olminstall-pipeline.yaml` exists. Trigger mode prints a **WARN** when **`--konflux-repo` is set without `--konflux-branch`** (resolver revision may stay at the ITS YAML default, e.g. `main`); omitting both flags uses the committed ITS default with no such warning. -If a freshly-created Snapshot takes time to trigger, `olm_pipeline.py` waits up to `PR_APPEAR_TIMEOUT_SECONDS` (default `600`) for the corresponding PipelineRun before failing. On this timeout path, it keeps the test Snapshot so a delayed trigger can still be attached on the next invocation. +> **Concurrent runs:** The CLI does not take a cluster-side lock. If two users run the script simultaneously against the same namespace, both may create Snapshots and trigger separate PipelineRuns. **Trigger mode always starts a new run** (it does not attach to an already-running PipelineRun). If you still have a run in progress for the same app, the helper prints an **INFO** with a copy-pastable `--watch ` command so you can stream that run instead. Use `--watch` (no name) to follow your newest owned olminstall run for `--app`, or `--watch ` for an explicit run. On normal exit the helper deletes your trigger Snapshot; **Ctrl-C while streaming logs only detaches locally** — the PipelineRun keeps running and the Snapshot is left in place. Deleting a Snapshot mid-run is non-fatal to the PipelineRun (which has already resolved the snapshot). To avoid confusion, coordinate with your team before triggering manually in a shared namespace. + +If a freshly-created Snapshot takes time to trigger, `olm_pipeline.py` waits up to `PR_APPEAR_TIMEOUT_SECONDS` (default `600`) for the corresponding PipelineRun before failing. On this timeout path, it keeps the test Snapshot so a delayed run can still be followed with `--watch` (or `--watch ` once the PipelineRun name is visible in the Konflux UI). > **Archived runs (KubeArchive):** Completed PipelineRuns are pruned from the live cluster by Tekton Results / cluster GC shortly after completion. `--list-pipelines` and `--watch` automatically fall back to the [KubeArchive](https://konflux-ci.dev/architecture/core/pipeline-service/) REST API to retrieve pruned runs and replay their logs. The `KA_HOST` environment variable can override the KubeArchive endpoint if needed. If KubeArchive is unreachable, the script degrades gracefully to live-only data. @@ -239,9 +300,40 @@ python3 integration-tests/olminstall/olm_pipeline.py --channel stable-3.x python3 integration-tests/olminstall/olm_pipeline.py --channel beta ``` +## Test phases configuration (`olminstall-tests-config.yaml`) + +Phase ids, defaults, and which Tekton results (`RUN_SMOKE`, `RUN_BVT`, `RUN_TIER1`, …) each phase toggles live in [`olminstall-tests-config.yaml`](olminstall-tests-config.yaml), similar in spirit to Jenkins component lists such as `components-tests.yaml`. The Konflux pipeline clones `SCRIPTS_REPO` and evaluates that file at runtime, so changing phases does not require editing Python token lists. + +When you add a new phase that should drive pipeline `when:` branches, extend **both** the YAML (`setsPipelineResults`) and [`olminstall-pipeline.yaml`](olminstall-pipeline.yaml) (new `results`, `parse-pipeline-tests` wiring, and tasks). + +## Rebasing on upstream `main` + +[opendatahub-io/odh-konflux-central#362](https://github.com/opendatahub-io/odh-konflux-central/pull/362) is merged. For follow-up work, rebase (or branch) from current upstream `main`: + +```bash +git fetch upstream +git rebase upstream/main +# or: git switch -c my-feature upstream/main +``` + +To base a branch on another open upstream PR, fetch its head by number (`N`): + +```bash +git fetch upstream pull/N/head:pr-upstream-N +git switch -c my-follow-up pr-upstream-N +``` + +## BVT on an existing cluster (outside this CLI) + +BVT pytest (`cluster_health` / `operator_health`) runs when `bvt` is included in `TESTS`. With **`PRODUCT`** **`rhoai`** or **`odh`**, task **`bvt-health-checks-with-eaas`** uses the EaaS kubeconfig after install. With **`PRODUCT=none`**, task **`bvt-health-checks-no-eaas`** still runs pytest against a **stub kubeconfig** (no real API) so failures come from the test harness, not from skipped tasks. A future option may supply a real kubeconfig when **`PRODUCT=none`**. The [`opendatahub-io/opendatahub-tests`](https://github.com/opendatahub-io/opendatahub-tests) container is built with **`uv sync`** and **`ENTRYPOINT ["uv", "run", "pytest"]`**; the pipeline uses **`uv run pytest …`** from the repo root so the same dependency set applies (bare `python3 -m pytest` skips that venv). To re-run only those checks locally, use `uv run pytest …` from a clone, or see tasks **`bvt-health-checks-*`** in [`olminstall-pipeline.yaml`](olminstall-pipeline.yaml) for the exact flags. + +## Failure diagnostics vs BVT logs + +On pipeline **failure**, **collect-diagnostics** writes large `oc adm inspect` / YAML data under `/diag` on the TaskRun pod and a **small** `DIAGNOSTICS_MANIFEST` Tekton result (file listing + short excerpts). The main log stream stays readable; pull `/diag/diagnostics-bundle.tgz` from the completed **collect-diagnostics** pod with `oc cp` if you need the full bundle. + ## Slack notifications -The `post-results` task posts to Slack when **`SLACK_WEBHOOK_URL`** is set. Create an optional Secret in the tenant namespace: +The `post-results` task posts to Slack when **`SLACK_WEBHOOK_URL`** is set. The message uses aggregate pipeline task status (`Succeeded` when every non-finally task that ran has succeeded; optional phases skipped via `TESTS` do not count as failure). Create an optional Secret in the tenant namespace: ```text Name: slack-webhook @@ -252,5 +344,5 @@ If the Secret is absent, the step logs the message and exits without failing the ## Maintenance -- **Image digest pins** — Some steps in [`olminstall-smoke-pipeline.yaml`](olminstall-smoke-pipeline.yaml) pin tool images by digest (e.g. `konflux-test:stable@sha256:…`) so runs stay reproducible; refresh those digests on whatever cadence your team uses and re-run smoke after each bump. -- **Post-install [BVT](../../doc/contributing-konflux-testing-rhoai.md#bvt)** — not part of this pipeline yet (current scope ends at [CSV](../../doc/contributing-konflux-testing-rhoai.md#csv) `Succeeded`). +- **Image digest pins** — Some steps in [`olminstall-pipeline.yaml`](olminstall-pipeline.yaml) pin tool images by digest (e.g. `konflux-test:stable@sha256:…`) so runs stay reproducible; refresh those digests on whatever cadence your team uses and re-run the pipeline after each bump. +- **BVT image** — **`resolve-opendatahub-tests-image-*`** tasks use `quay.io/konflux-ci/konflux-test` (clone + `skopeo` in the resolve step) and a versioned or `:latest` [`opendatahub-tests`](https://quay.io/repository/opendatahub/opendatahub-tests) image. **`bvt-health-checks-*`** runs **`uv run pytest`** where available (see upstream [`Dockerfile`](https://github.com/opendatahub-io/opendatahub-tests/blob/main/Dockerfile) and [`pyproject.toml`](https://github.com/opendatahub-io/opendatahub-tests/blob/main/pyproject.toml): `pytest`, `openshift-python-wrapper`, `pytest-testconfig`, `structlog`, etc.). JUnit + console logs live under **`/artifacts`** on the TaskRun pod until pruned. diff --git a/integration-tests/olminstall/helpers/bvt_artifacts.py b/integration-tests/olminstall/helpers/bvt_artifacts.py new file mode 100644 index 000000000..8eb958bd0 --- /dev/null +++ b/integration-tests/olminstall/helpers/bvt_artifacts.py @@ -0,0 +1,71 @@ +"""Resolve BVT artifact browser URLs from PipelineRun / TaskRun state.""" + +from __future__ import annotations + +from typing import Any + +from .tekton_incluster import ( + list_taskruns_in_cluster, + namespace_from_env, + result_map, + task_name, + task_reason, +) + + +def tests_include_bvt(tests_csv: str) -> bool: + return "bvt" in {p.strip().lower() for p in (tests_csv or "").split(",") if p.strip()} + + +def published_artifacts_url_from_taskruns(taskruns: list[dict[str, Any]]) -> str: + for tr in taskruns: + task = task_name(tr).lower() + if "bvt-health-checks" not in task: + continue + url = result_map(tr).get("ARTIFACTS_URL", "").strip() + if url: + return url + return "" + + +def bvt_unpublished_reason(taskruns: list[dict[str, Any]]) -> str: + """Short explanation when BVT was requested but no ARTIFACTS_URL was published.""" + bvt_tasks = [task_name(tr) for tr in taskruns if "bvt-health-checks" in task_name(tr).lower()] + if not bvt_tasks: + return "BVT did not run (pipeline failed or was skipped before bvt-health-checks)" + for tr in taskruns: + task = task_name(tr) + if "bvt-health-checks" not in task.lower(): + continue + reason = task_reason(tr) + if reason == "Completed": + if "upload-artifacts" in task.lower() or task.endswith("upload-artifacts"): + return "upload-artifacts finished but did not publish ARTIFACTS_URL" + return "BVT finished without publishing artifacts (upload-artifacts may have been skipped)" + if reason in ("Failed", "PipelineRunFailed", "TaskRunFailed"): + return f"{task} failed — see TaskRun logs" + if reason in ("Cancelled", "TaskRunCancelled", "PipelineRunCancelled"): + return f"{task} was cancelled" + if reason in ("Skipped", "TaskRunSkipped", "PipelineRunSkipped"): + return f"{task} was skipped" + return "BVT did not publish JUnit to the artifact browser" + + +def resolve_artifacts_notification_line( + *, + tests_csv: str, + pipeline_run: str, + taskruns: list[dict[str, Any]] | None = None, +) -> str | None: + """Return a single notification line, or None to omit artifacts entirely.""" + if not tests_include_bvt(tests_csv): + return None + runs = taskruns if taskruns is not None else [] + if not runs and pipeline_run: + ns = namespace_from_env() + if ns: + runs = list_taskruns_in_cluster(pipeline_run, ns) + published = published_artifacts_url_from_taskruns(runs) + if published: + return f"Artifacts: {published}" + return f"Artifacts: (none — {bvt_unpublished_reason(runs)})" diff --git a/integration-tests/olminstall/helpers/bvt_product_none_placeholder_junit.py b/integration-tests/olminstall/helpers/bvt_product_none_placeholder_junit.py new file mode 100644 index 000000000..9aa5b6c8c --- /dev/null +++ b/integration-tests/olminstall/helpers/bvt_product_none_placeholder_junit.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +"""Emit skipped JUnit XML for PRODUCT=none BVT (no workload cluster / no ODH APIs on worker).""" + +from __future__ import annotations + +import sys +import xml.etree.ElementTree as ET +from pathlib import Path + +_SKIP = ( + "PRODUCT=none: full opendatahub-tests BVT needs a cluster with Open Data Hub APIs " + "(e.g. DataScienceCluster). Use PRODUCT rhoai or odh for EaaS-backed BVT." +) + +_ARTIFACTS_ROOT = Path("/artifacts").resolve() + + +def _safe_artifact_out(raw: Path) -> Path: + out = raw.resolve() if raw.is_absolute() else (_ARTIFACTS_ROOT / raw).resolve() + if out != _ARTIFACTS_ROOT and _ARTIFACTS_ROOT not in out.parents: + raise ValueError(f"output directory must be under {_ARTIFACTS_ROOT}, got {raw}") + return out + + +def _testsuite_xml(name: str, msg: str) -> str: + testsuite = ET.Element( + "testsuite", + { + "name": name, + "tests": "1", + "skipped": "1", + "failures": "0", + "errors": "0", + "time": "0.0", + }, + ) + testcase = ET.SubElement( + testsuite, + "testcase", + {"name": "bvt_skipped_no_odh_cluster", "classname": "olminstall"}, + ) + ET.SubElement(testcase, "skipped", {"message": msg}) + if hasattr(ET, "indent"): + ET.indent(testsuite, space=" ") + body = ET.tostring(testsuite, encoding="unicode", xml_declaration=False) + return '\n' + body + + +def main() -> int: + try: + out = _safe_artifact_out(Path(sys.argv[1] if len(sys.argv) > 1 else "/artifacts")) + except ValueError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + out.mkdir(parents=True, exist_ok=True) + for prefix in ("cluster-health", "operator-health"): + (out / f"{prefix}.xml").write_text( + _testsuite_xml(name=prefix, msg=_SKIP), + encoding="utf-8", + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/cli.py b/integration-tests/olminstall/helpers/cli.py index 3c90e329b..eb45d0ee4 100644 --- a/integration-tests/olminstall/helpers/cli.py +++ b/integration-tests/olminstall/helpers/cli.py @@ -20,8 +20,11 @@ DEFAULT_PRODUCT, LIST_SUPPORTED_OCP_MAX_PRS, PRODUCT_CHOICES, + default_tests_config_path, ) from .errors import AppError +from .tests_config import load_tests_catalog +from .tests_plan import validate_and_normalize_tests_csv # When user passes ``--ka-host`` with no URL, read KA_HOST from the environment. _KA_HOST_FROM_ENV = "__KA_HOST_FROM_ENV__" @@ -104,8 +107,26 @@ def make_parser(description: str = "", epilog: str | None = None) -> CliArgument default=os.environ.get("KONFLUX_SERVER", DEFAULT_KONFLUX_SERVER), help="Konflux API URL for oc login fallback (env KONFLUX_SERVER)", ) - parser.add_argument("--konflux-repo", metavar="URL", default="", help="ITS scripts repo URL (needs yq)") - parser.add_argument("--konflux-branch", metavar="REF", default="", help="ITS scripts revision (needs yq)") + parser.add_argument( + "--konflux-repo", + metavar="URL", + default="", + help=( + "Git URL for pipeline + scripts (must contain integration-tests/olminstall/). " + "Omit (with no --konflux-branch) = ITS default opendatahub-io/odh-konflux-central @ main. " + "Optional: set OLMINSTALL_PIPELINE_REPO and OLMINSTALL_PIPELINE_REVISION (or KONFLUX_PIPELINE_*) " + "when both CLI flags are omitted. Needs yq when this or --konflux-branch patches the ITS." + ), + ) + parser.add_argument( + "--konflux-branch", + metavar="REF", + default="", + help=( + "Git revision for --konflux-repo (branch/tag/SHA). Omit with --konflux-repo = resolver stays ``main`` " + "from the ITS YAML. Needs yq when patching." + ), + ) parser.add_argument( "--channel", metavar="NAME", @@ -116,7 +137,11 @@ def make_parser(description: str = "", epilog: str | None = None) -> CliArgument "--product", default=DEFAULT_PRODUCT, choices=PRODUCT_CHOICES, - help="rhoai or odh (catalog / ITS wiring)", + help=( + "none = no rhoai/odh auto image resolution (pinned test-snapshot.yaml unless --image); " + "rhoai or odh for catalog / ITS wiring. Omit = same as none (default). " + "Use rhoai/odh for full installs." + ), ) parser.add_argument( "--version", @@ -132,13 +157,32 @@ def make_parser(description: str = "", epilog: str | None = None) -> CliArgument default="", help="EaaS cluster minor (e.g. 4.19); with --list-supported-ocp, assert minor is listed", ) + parser.add_argument( + "--tests", + metavar="LIST", + default=None, + help=( + "Comma-separated test phases for ITS param TESTS (include every phase marked " + "requiredInSelection in olminstall-tests-config.yaml; the default file may have none). " + "Omit to use defaults from that file." + ), + ) + parser.add_argument( + "--tests-config", + metavar="PATH", + default="", + help=( + "Path to olminstall-tests-config.yaml (phase list + defaults). " + f"Default: {default_tests_config_path()}" + ), + ) parser.add_argument( "--watch", nargs="?", const="", default=None, metavar="PIPELINERUN", - help="Watch latest owned run or PIPELINERUN (archive if pruned)", + help="Watch newest olminstall run for --app (same order as --list), else owner/Snapshot match; or PIPELINERUN", ) parser.add_argument( "--list-pipelines", @@ -154,10 +198,32 @@ def make_parser(description: str = "", epilog: str | None = None) -> CliArgument action="store_true", help=f"Print supported OCP minors from logs (≤{LIST_SUPPORTED_OCP_MAX_PRS} runs); optional --ocp-version", ) + parser.add_argument( + "--slack-channel-id", + metavar="CHANNEL", + default="", + help=( + "Optional Slack channel ID (e.g. C01234ABCDE) to receive a run notification via the " + "slack-webhook secret in the namespace. Omit (the default) to suppress all Slack messages. " + "Requires the slack-webhook secret to be present when set." + ), + ) + parser.add_argument( + "--no-prune-stale-its", + dest="prune_stale_its", + action="store_false", + default=True, + help=( + "When creating a Snapshot with default --namespace rhoai-tenant and --app testops-playpen, " + "skip deleting legacy IntegrationTestScenario CRs (rhoai-test) " + "before oc apply + Snapshot. Default: prune so only odh-olminstall-testops runs for that app." + ), + ) return parser def parse_cli_args(parser: CliArgumentParser, argv: list[str]) -> argparse.Namespace: + tests_explicit = any(x == "--tests" or x.startswith("--tests=") for x in argv) args = parser.parse_args(argv) if args.version and args.product != "rhoai": @@ -182,6 +248,13 @@ def parse_cli_args(parser: CliArgumentParser, argv: list[str]) -> argparse.Names if args.konflux_server and not args.konflux_server.startswith("https://"): raise AppError("--konflux-server must use https://", 2) + cfg_arg = (args.tests_config or "").strip() + cfg_path = Path(cfg_arg).expanduser().resolve() if cfg_arg else default_tests_config_path() + catalog = load_tests_catalog(cfg_path) + args.tests_catalog_default_csv = catalog.default_csv + args.tests = validate_and_normalize_tests_csv(args.tests, catalog) + args.tests_explicit = tests_explicit + if args.list_pipelines is not None: try: lp = int(args.list_pipelines) @@ -217,6 +290,10 @@ def _trigger_options_incompatible_with_query() -> list[str]: bad.append("--konflux-branch") if args.ocp_version and not list_ocp_on: bad.append("--ocp-version") + if getattr(args, "tests_explicit", False): + bad.append("--tests") + if (args.tests_config or "").strip(): + bad.append("--tests-config") return bad if query_modes and (bad := _trigger_options_incompatible_with_query()): diff --git a/integration-tests/olminstall/helpers/collect_diagnostics.py b/integration-tests/olminstall/helpers/collect_diagnostics.py new file mode 100644 index 000000000..0ad5ccb38 --- /dev/null +++ b/integration-tests/olminstall/helpers/collect_diagnostics.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +"""Collect failure diagnostics from the target cluster. + +Env (required): + OPERATOR_NAMESPACE + DIAG_MANIFEST_RESULT -- Tekton result file path +Env (optional): + DIAG_DIR -- output directory (default /diag) +""" + +from __future__ import annotations + +import os +import shlex +import shutil +import subprocess +import sys +from pathlib import Path + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.tekton_util import require_env, run, write_result + +_OC = shutil.which("oc") or "oc" + + +def _oc(args: list[str], **kwargs: object) -> subprocess.CompletedProcess[str]: + return run([_OC, *args], check=False, capture=True, **kwargs) # type: ignore[arg-type] + + +def _oc_to_file(args: list[str], dest: Path) -> None: + r = _oc(args) + if r.returncode != 0: + invoc = " ".join(shlex.quote(x) for x in [_OC, *args]) + stderr = r.stderr or "" + stdout = r.stdout or "" + blob = ( + f"OC COMMAND FAILED: exitcode={r.returncode}\n" + f"COMMAND: {invoc}\n" + f"STDERR:\n{stderr}\n" + f"STDOUT:\n{stdout}\n" + ) + dest.write_text(blob, encoding="utf-8") + else: + dest.write_text(r.stdout or "", encoding="utf-8") + + +def main() -> int: + ns = require_env("OPERATOR_NAMESPACE") + result_path = require_env("DIAG_MANIFEST_RESULT") + diag_dir = Path(os.environ.get("DIAG_DIR", "/diag").strip()) + diag_dir.mkdir(parents=True, exist_ok=True) + + print(f"Writing heavy diagnostics under {diag_dir} (not streaming full YAML to pipeline log)...") + + inspect_dest = diag_dir / "inspect-ns-operator" + inspect_dest.mkdir(parents=True, exist_ok=True) + inspect_log = inspect_dest / "adm-inspect.log" + ir = _oc(["adm", "inspect", f"ns/{ns}", f"--dest-dir={inspect_dest}"]) + inspect_log.write_text( + f"exit={ir.returncode}\nSTDERR:\n{ir.stderr or ''}\nSTDOUT:\n{ir.stdout or ''}\n", + encoding="utf-8", + ) + inspect_failed = ir.returncode != 0 + if inspect_failed: + (inspect_dest / "FAILED").write_text( + f"oc adm inspect exited {ir.returncode}; see adm-inspect.log\n", + encoding="utf-8", + ) + print( + f"ERROR: oc adm inspect failed (exit {ir.returncode}); see {inspect_log}", + file=sys.stderr, + ) + _oc_to_file(["get", "csv", "-n", ns, "-o", "yaml"], diag_dir / "csv.yaml") + _oc_to_file(["describe", "sub", "-n", ns], diag_dir / "subscription-describe.txt") + + # Marketplace jobs summary + lines: list[str] = [] + lines.append("=== jobs openshift-marketplace (wide) ===") + r = _oc(["get", "jobs", "-n", "openshift-marketplace", "-o", "wide"]) + lines.append(r.stdout or "") + + lines.append("=== bundle-unpack job spec (image + SA) ===") + r = _oc(["get", "jobs", "-n", "openshift-marketplace", "-o", "yaml"]) + if r.stdout: + for line in r.stdout.splitlines(): + stripped = line.strip() + if any(stripped.startswith(k) for k in ("image:", "serviceAccountName:", "activeDeadlineSeconds:")): + lines.append(line) + + lines.append("=== bundle-unpack job events (trimmed) ===") + r = _oc(["get", "jobs", "-n", "openshift-marketplace", "-o", "jsonpath={.items[*].metadata.name}"]) + job_names = (r.stdout or "").split() + for job in job_names: + lines.append(f"Job: {job}") + desc = _oc(["describe", "job", job, "-n", "openshift-marketplace"]) + if desc.stdout: + for dl in desc.stdout.splitlines(): + if any(kw in dl for kw in ("Events", "Image", "Status", "Message", "Reason")): + lines.append(dl) + lines.append("Job logs (last 20 lines):") + logs = _oc(["logs", f"job/{job}", "-n", "openshift-marketplace", "--tail=20"]) + lines.append(logs.stdout or " (no logs)") + + lines.append("=== SAs openshift-marketplace (pull secrets) ===") + r = _oc(["get", "sa", "-n", "openshift-marketplace", + "-o", "custom-columns=NAME:.metadata.name,PULL_SECRETS:.imagePullSecrets"]) + lines.append(r.stdout or "") + + (diag_dir / "marketplace-jobs-summary.txt").write_text("\n".join(lines), encoding="utf-8") + + # Bundle beside diag_dir so "tar -C diag_dir ." never archives the growing output file. + bundle = diag_dir.parent / "diagnostics-bundle.tgz" + tar_r = run(["tar", "czf", str(bundle), "-C", str(diag_dir), "."], check=False, capture=True) + if tar_r.returncode != 0: + tail = ((tar_r.stderr or "") + (tar_r.stdout or "")).strip()[:2000] + print( + f"ERROR: tar diagnostics bundle failed (exit {tar_r.returncode}): {tail or '(no output)'}", + file=sys.stderr, + ) + elif bundle.is_file() and bundle.stat().st_size > 0: + run(["du", "-sh", str(diag_dir), str(bundle)], check=False) + print(f"Diagnostics bundle: {bundle} (copy from collect-diagnostics TaskRun pod if needed).") + else: + print(f"ERROR: tar did not produce a non-empty {bundle.name}", file=sys.stderr) + + # Build manifest result (truncated to 3584 bytes) + manifest_lines: list[str] = [] + if inspect_failed: + manifest_lines.append("=== DIAGNOSTICS PARTIAL FAILURE ===") + manifest_lines.append( + f"oc adm inspect ns/{ns} failed (exit {ir.returncode}); " + f"see {inspect_dest}/adm-inspect.log and {inspect_dest}/FAILED" + ) + manifest_lines.append(f"=== {diag_dir} file listing ===") + for f in sorted(diag_dir.rglob("*")): + if f.is_file(): + manifest_lines.append(f" {f} ({f.stat().st_size} bytes)") + + sub_desc = diag_dir / "subscription-describe.txt" + if sub_desc.exists(): + manifest_lines.append("=== subscription-describe (first 80 lines) ===") + manifest_lines.extend(sub_desc.read_text(encoding="utf-8", errors="replace").splitlines()[:80]) + + mkt = diag_dir / "marketplace-jobs-summary.txt" + if mkt.exists(): + manifest_lines.append("=== marketplace-jobs-summary (first 120 lines) ===") + manifest_lines.extend(mkt.read_text(encoding="utf-8", errors="replace").splitlines()[:120]) + + raw = "\n".join(manifest_lines).encode("utf-8", errors="replace")[:3584] + manifest = raw.decode("utf-8", errors="ignore") + write_result(result_path, manifest) + if inspect_failed: + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/constants.py b/integration-tests/olminstall/helpers/constants.py index 8bb32b76c..5d56f62ab 100644 --- a/integration-tests/olminstall/helpers/constants.py +++ b/integration-tests/olminstall/helpers/constants.py @@ -1,17 +1,64 @@ """Shared defaults for Konflux olminstall tooling.""" +from pathlib import Path + DEFAULT_NAMESPACE = "rhoai-tenant" DEFAULT_APP = "testops-playpen" -DEFAULT_PRODUCT = "rhoai" -PRODUCT_CHOICES = ("rhoai", "odh") +# Canonical sandbox ITS in its-olminstall-rhoai-tenant.yaml (resolver → olminstall-pipeline.yaml). +OLMINSTALL_TESTOPS_ITS_NAME = "odh-olminstall-testops" +# Legacy IntegrationTestScenario names still present on some tenants; Konflux starts one PipelineRun +# per ITS when a Snapshot is created for the same application. ``olm_pipeline.py`` deletes these +# before triggering when using default -n/--app unless ``--no-prune-stale-its`` is passed. +STALE_TESTOPS_PLAYPEN_ITS_NAMES: frozenset[str] = frozenset( + ( + "odh-olminstall-smoke-testops", + "rhoai-test", + ) +) +# ``none``: skip rhoai/odh catalog auto-pick; use ``test-snapshot.yaml`` image unless ``--image`` is set. +# Use ``--product rhoai`` or ``odh`` for full installs. +DEFAULT_PRODUCT = "none" +PRODUCT_CHOICES = ("none", "rhoai", "odh") + +# Tekton label `tekton.dev/pipeline` on PipelineRuns resolved from olminstall-pipeline.yaml. +OLMINSTALL_PIPELINE_LABEL_CURRENT = "odh-olminstall-test" +# Old smoke-only pipeline; runs with this label (or smoke in the name) are ignored by the CLI. +OLMINSTALL_PIPELINE_LABEL_SMOKE_ONLY = "odh-olminstall-smoke-test" + + +def olminstall_smoke_only_pipelinerun(name: str, pipeline_label: str = "") -> bool: + """True if this run is from the smoke-only pipeline — exclude from --list and owned-run selection.""" + if pipeline_label == OLMINSTALL_PIPELINE_LABEL_SMOKE_ONLY: + return True + return "olminstall-smoke" in name + + +def default_tests_config_path() -> Path: + """Path to olminstall-tests-config.yaml next to this package (integration-tests/olminstall).""" + return Path(__file__).resolve().parent.parent / "olminstall-tests-config.yaml" + + +# Value of ITS param TESTS in committed its-olminstall-*.yaml (patch when CLI selection differs). +ITS_TESTS_PARAM_DEFAULT = "bvt,smoke" + + DEFAULT_LIST_COUNT = 10 # How many recent PipelineRuns to scan for --list-supported-ocp (newest first). LIST_SUPPORTED_OCP_MAX_PRS = 40 DEFAULT_KONFLUX_UI = "" DEFAULT_KA_HOST = "" DEFAULT_KONFLUX_SERVER = "" + +# Must match ``ARTIFACT_BROWSER_URL`` / ``ARTIFACT_BROWSER_REPO_PATH`` defaults in olminstall-pipeline.yaml. +DEFAULT_ARTIFACT_BROWSER_URL = ( + "https://app-artifact-browser.apps.rosa.konflux-qe.zmr9.p3.openshiftapps.com" +) +DEFAULT_ARTIFACT_BROWSER_REPO_PATH = "odh-ci-artifacts" PENDING_REASONS = {"", "PipelineRunPending", "ResolvingPipelineRef"} +# Snapshot ``containerImage`` for RHOAI FBCF may be digest-pinned (``…@sha256:…``) or tag form (``…:tag``). +RHOAI_FBCF_IMAGE_REF_PATTERN = r"rhoai-fbc-fragment(?:@|:)" + # Non-secret CLI context stored on Snapshot / PipelineRun for watch and archive UX. OLMINSTALL_WRITE_ANNOTATION_KEYS = ( "olminstall.product", @@ -20,6 +67,22 @@ "olminstall.ocp-version", "olminstall.scripts-repo-url", "olminstall.scripts-repo-revision", + "olminstall.tests", + "olminstall.bvt-env-only", +) +# Written by patch_pipelinerun_summary (post-results) and read by CLI / Konflux UI. +OLMINSTALL_SUMMARY_ANNOTATION_KEYS = ( + "olminstall.fbcf-image", + "olminstall.operator-version", + "olminstall.ephemeral-cluster", + "olminstall.test-results-url", + "olminstall.artifacts-status", + "olminstall.pipeline-test-output", ) + # Order when printing from existing PipelineRuns (includes run-owner from annotate). -OLMINSTALL_CTX_PRINT_KEYS = OLMINSTALL_WRITE_ANNOTATION_KEYS + ("olminstall.run-owner",) +OLMINSTALL_CTX_PRINT_KEYS = ( + OLMINSTALL_WRITE_ANNOTATION_KEYS + + ("olminstall.run-owner",) + + OLMINSTALL_SUMMARY_ANNOTATION_KEYS +) diff --git a/integration-tests/olminstall/helpers/create_eaas_cluster.py b/integration-tests/olminstall/helpers/create_eaas_cluster.py new file mode 100644 index 000000000..86c838d6c --- /dev/null +++ b/integration-tests/olminstall/helpers/create_eaas_cluster.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 +"""Provision an ephemeral EaaS HyperShift cluster via ClusterTemplateInstance. + +Env: INSTANCE_TYPE, VERSION, KUBECONFIG_VALUE, TENANT, TIMEOUT (default 30m), + ICS_VALUE, CLUSTER_NAME_RESULT_PATH (Tekton step result file). +""" + +from __future__ import annotations + +import json +import os +import shutil +import sys +import tempfile +import time +from pathlib import Path + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.tekton_util import require_env, run, write_result + +# Extra seconds beyond CTI `timeout` for API/controller lag before we declare poll timeout. +_POLL_BUFFER_S = 90 + + +def _parse_timeout_to_seconds(raw: str, *, fallback: int = 1800) -> int: + """Parse TIMEOUT-style values: plain seconds, or ``60s`` / ``30m`` / ``1h`` (case-insensitive).""" + s = raw.strip() + if not s: + return fallback + lower = s.lower() + try: + if lower.endswith("h"): + return max(1, int(float(lower[:-1].strip()) * 3600)) + if lower.endswith("m"): + return max(1, int(float(lower[:-1].strip()) * 60)) + if lower.endswith("s"): + return max(1, int(float(lower[:-1].strip()))) + return max(1, int(float(s))) + except (ValueError, OverflowError): + return fallback + + +def _format_wait_hint(seconds: int) -> str: + if seconds >= 3600 and seconds % 3600 == 0: + return f"{seconds // 3600}h" + if seconds >= 60 and seconds % 60 == 0: + return f"{seconds // 60}m" + return f"{seconds}s" + + +def _emit_cti_yaml( + path: Path, + *, + tenant: str, + instance_type: str, + version: str, + timeout: str, + ics_value: str, +) -> None: + """Write ClusterTemplateInstance YAML without ``yq`` (``konflux-test`` ships kislyuk ``yq``, not Go ``yq``).""" + ics_body = ics_value.strip("\n") + ics_lines = "\n".join(" " + line for line in ics_body.splitlines()) if ics_body else "" + block = f" value: |\n{ics_lines}\n" if ics_lines else ' value: ""\n' + text = ( + "---\n" + "apiVersion: clustertemplate.openshift.io/v1alpha1\n" + "kind: ClusterTemplateInstance\n" + "metadata:\n" + " generateName: cluster-\n" + " labels:\n" + f" eaas.konflux-ci.dev/tenant: {json.dumps(tenant)}\n" + "spec:\n" + " clusterTemplateRef: hypershift-aws-cluster\n" + " parameters:\n" + f" - name: instanceType\n value: {json.dumps(instance_type)}\n" + f" - name: version\n value: {json.dumps(version)}\n" + f" - name: timeout\n value: {json.dumps(timeout)}\n" + " - name: imageContentSources\n" + f"{block}" + " - name: fips\n value: \"false\"\n" + ) + path.write_text(text, encoding="utf-8") + + +def _write_secure_kubeconfig(content: str) -> Path: + """Write kubeconfig to a unique temp file with mode 0o600 (avoids /tmp/kubeconfig races).""" + fd, path_str = tempfile.mkstemp(prefix="kubeconfig-", suffix=".yaml") + try: + os.write(fd, content.encode("utf-8")) + os.fchmod(fd, 0o600) + finally: + os.close(fd) + return Path(path_str) + + +def main() -> int: + instance_type = require_env("INSTANCE_TYPE") + version = require_env("VERSION") + kubeconfig_value = require_env("KUBECONFIG_VALUE") + tenant = require_env("TENANT") + result_path = require_env("CLUSTER_NAME_RESULT_PATH") + timeout = os.environ.get("TIMEOUT", "30m").strip() or "30m" + ics_value = os.environ.get("ICS_VALUE", "").strip() + + kubeconfig = _write_secure_kubeconfig(kubeconfig_value) + os.environ["KUBECONFIG"] = str(kubeconfig) + + # konflux-test image provides ``oc``; ``kubectl`` may be absent. + kube = shutil.which("kubectl") or shutil.which("oc") or "oc" + + try: + cti_path = Path("cti.yaml") + _emit_cti_yaml( + cti_path, + tenant=tenant, + instance_type=instance_type, + version=version, + timeout=timeout, + ics_value=ics_value, + ) + + print(f"Creating ephemeral cluster (OCP {version}, {instance_type})...") + print(cti_path.read_text(encoding="utf-8")) + + result = run([kube, "create", "-f", str(cti_path), "-o=jsonpath={.metadata.name}"], capture=True) + cti_name = result.stdout.strip() + if result.returncode != 0 or not cti_name: + tail = ((result.stderr or "") + (result.stdout or "")).strip()[:4000] + print( + f"❌ ClusterTemplateInstance create failed or name missing (exit {result.returncode}): " + f"{tail or '(no output)'}", + file=sys.stderr, + ) + return 1 + print(f"Created ClusterTemplateInstance {cti_name}") + write_result(result_path, cti_name) + + wait_s = _parse_timeout_to_seconds(timeout) + deadline = time.time() + wait_s + _POLL_BUFFER_S + print( + f"Waiting for cluster to be ready (CTI timeout≈{_format_wait_hint(wait_s)}, " + f"poll deadline +{_POLL_BUFFER_S}s buffer for controller overhead)..." + ) + while time.time() < deadline: + phase_r = run( + [kube, "get", "cti", cti_name, "-o", "jsonpath={.status.phase}"], + check=False, capture=True, + ) + phase = phase_r.stdout.strip() if phase_r.returncode == 0 else "Unknown" + msg_r = run( + [kube, "get", "cti", cti_name, "-o", "jsonpath={.status.message}"], + check=False, capture=True, + ) + msg = msg_r.stdout.strip() if msg_r.returncode == 0 else "" + suffix = f" ({msg})" if msg else "" + print(f" phase={phase}{suffix}") + + if phase == "Ready": + print(f"Successfully provisioned {cti_name}") + return 0 + if phase in ("Failed", "Error"): + print("Cluster provisioning failed") + run([kube, "get", "cti", cti_name, "-o", "yaml"], check=False) + return 1 + time.sleep(30) + + print("Timed out waiting for cluster") + run([kube, "get", "cti", cti_name, "-o", "yaml"], check=False) + return 1 + finally: + # Every ``run()`` above blocks until its subprocess exits; nothing keeps the file open. + kubeconfig.unlink(missing_ok=True) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/emit_test_output.py b/integration-tests/olminstall/helpers/emit_test_output.py new file mode 100644 index 000000000..55e918d9a --- /dev/null +++ b/integration-tests/olminstall/helpers/emit_test_output.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +"""Parse JUnit XML files and write a Konflux-standardised TEST_OUTPUT result. + +Env (required): + TEST_OUTPUT_PATH -- Tekton result file for the JSON summary + ARTIFACTS_URL_PATH -- Tekton result file for the artifact browser URL + ARTIFACT_BROWSER_BASE -- base URL for the artifact browser (no trailing slash) + PR_NAME -- PipelineRun name + OCI_TAG_SUFFIX -- tag suffix (e.g. "bvt" or "bvt-env") +Env (optional): + ARTIFACT_BROWSER_REPO_PATH -- path segment in browser (default odh-ci-artifacts) + ARTIFACTS_DIR -- directory containing JUnit XML (default /artifacts) + NOTE_PREFIX -- prefix for the note field (default "BVT") +""" + +from __future__ import annotations + +import json +import os +import sys +from datetime import datetime, timezone +from pathlib import Path + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.tekton_util import parse_junit_summary, require_env, write_result + + +def main() -> int: + test_output_path = require_env("TEST_OUTPUT_PATH") + artifacts_url_path = require_env("ARTIFACTS_URL_PATH") + browser_base = require_env("ARTIFACT_BROWSER_BASE") + repo_path = os.environ.get("ARTIFACT_BROWSER_REPO_PATH", "odh-ci-artifacts").strip().strip("/") + pr_name = require_env("PR_NAME") + oci_tag_suffix = require_env("OCI_TAG_SUFFIX") + artifacts_dir = os.environ.get("ARTIFACTS_DIR", "/artifacts").strip() + note_prefix = os.environ.get("NOTE_PREFIX", "BVT").strip() + + s = parse_junit_summary(artifacts_dir) + result = "SUCCESS" if (s["failures"] + s["errors"]) == 0 else "FAILURE" + note = f"{note_prefix}: {s['passed']}/{s['total']} passed, {s['failures']} failed, {s['errors']} errors, {s['skipped']} skipped" + ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + + output = json.dumps({ + "result": result, + "timestamp": ts, + "failures": s["failures"] + s["errors"], + "warnings": 0, + "successes": s["passed"], + "note": note, + }, separators=(",", ":")) + write_result(test_output_path, output) + print(note) + + base = browser_base.rstrip("/") + artifacts_url = f"{base}/{repo_path}/{pr_name}-{oci_tag_suffix}/" + write_result(artifacts_url_path, artifacts_url) + print(f"Artifacts: {artifacts_url}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/extract_fbcf_image.py b/integration-tests/olminstall/helpers/extract_fbcf_image.py new file mode 100644 index 000000000..b66088120 --- /dev/null +++ b/integration-tests/olminstall/helpers/extract_fbcf_image.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +"""Extract the FBCF container image from a Konflux ApplicationSnapshot JSON. + +Env: SNAPSHOT (JSON string), COMPONENT_NAME, RESULT_PATH (Tekton result file). +""" +from __future__ import annotations + +import json +import os +import sys +from pathlib import Path + +_TEKTON_RESULTS = Path("/tekton/results").resolve() + + +def _validated_result_path(raw: str) -> Path: + p = Path(raw).expanduser() + rp = p.resolve() + if rp != _TEKTON_RESULTS and _TEKTON_RESULTS not in rp.parents: + raise ValueError(f"RESULT_PATH must be under {_TEKTON_RESULTS}, got {raw!r}") + return rp + + +def main() -> int: + snapshot_raw = os.environ.get("SNAPSHOT", "").strip() + component = os.environ.get("COMPONENT_NAME", "").strip() + result_path = os.environ.get("RESULT_PATH", "").strip() + + if not snapshot_raw: + print("❌ SNAPSHOT env var is empty", file=sys.stderr) + return 1 + if not component: + print("❌ COMPONENT_NAME env var is empty", file=sys.stderr) + return 1 + if not result_path: + print("❌ RESULT_PATH env var is empty", file=sys.stderr) + return 1 + + print("Parsing Konflux snapshot...") + try: + snap = json.loads(snapshot_raw) + except json.JSONDecodeError as exc: + print(f"❌ Invalid SNAPSHOT JSON: {exc}", file=sys.stderr) + return 1 + + if not isinstance(snap, dict): + print("❌ SNAPSHOT JSON root must be an object", file=sys.stderr) + return 1 + components = snap.get("components") + if components is None: + print("❌ SNAPSHOT missing 'components' array", file=sys.stderr) + return 1 + if not isinstance(components, list): + print( + f"❌ SNAPSHOT 'components' must be a list, got {type(components).__name__}", + file=sys.stderr, + ) + return 1 + + fbcf_image = None + found_component = False + for i, comp in enumerate(components): + if not isinstance(comp, dict): + print( + f"❌ snapshot.components[{i}] must be an object, got {type(comp).__name__}", + file=sys.stderr, + ) + return 1 + if comp.get("name") == component: + found_component = True + fbcf_image = comp.get("containerImage") + break + + if not found_component: + print( + f"❌ SNAPSHOT has no component named {component!r}", + file=sys.stderr, + ) + return 1 + if fbcf_image is None: + print( + f"❌ Component {component!r} has no containerImage field (or it is null)", + file=sys.stderr, + ) + return 1 + if not isinstance(fbcf_image, str): + print( + f"❌ Component {component!r} containerImage must be a string, " + f"got {type(fbcf_image).__name__}", + file=sys.stderr, + ) + return 1 + if not fbcf_image.strip(): + print( + f"❌ Component {component!r} has empty or whitespace-only containerImage", + file=sys.stderr, + ) + return 1 + + print(f"✓ Extracted FBCF Image: {fbcf_image}") + try: + out = _validated_result_path(result_path) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(fbcf_image, encoding="utf-8") + except ValueError as exc: + print(f"❌ Invalid RESULT_PATH: {exc}", file=sys.stderr) + return 1 + except (FileNotFoundError, PermissionError, OSError) as exc: + print(f"❌ Failed to write RESULT_PATH {result_path!r}: {exc}", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/install_and_verify.py b/integration-tests/olminstall/helpers/install_and_verify.py index 0dd7846a6..bd1f79fe7 100644 --- a/integration-tests/olminstall/helpers/install_and_verify.py +++ b/integration-tests/olminstall/helpers/install_and_verify.py @@ -28,6 +28,58 @@ NS_PATCH_PATTERN = re.compile(r"^(\s*namespace:\s*)redhat-ods-operator\s*$", re.MULTILINE) OC_WAIT_NEEDLE = 'local namespace="${2:-default}"' +# Minimal DSCI + DSC for BVT. The operator must be installed first; these CRs +# activate RHOAI so the opendatahub-tests conftest can discover the cluster. +_DSCI_YAML = """\ +apiVersion: dscinitialization.opendatahub.io/v1 +kind: DSCInitialization +metadata: + name: default-dsci +spec: + applicationsNamespace: redhat-ods-applications + monitoring: + managementState: Managed + namespace: redhat-ods-monitoring + serviceMesh: + managementState: Removed + trustedCABundle: + customCABundle: "" + managementState: Removed +""" + +_DSC_YAML = """\ +apiVersion: datasciencecluster.opendatahub.io/v2 +kind: DataScienceCluster +metadata: + name: default-dsc +spec: + components: + dashboard: + managementState: Managed + workbenches: + managementState: Managed + modelmeshserving: + managementState: Removed + datasciencepipelines: + managementState: Removed + kserve: + managementState: Removed + codeflare: + managementState: Removed + ray: + managementState: Removed + kueue: + managementState: Removed + modelregistry: + managementState: Removed + trainingoperator: + managementState: Removed + trustyai: + managementState: Removed + modelcontroller: + managementState: Removed +""" + def fail(message: str = "") -> NoReturn: if message: @@ -42,6 +94,8 @@ def fail(message: str = "") -> NoReturn: def require_env(name: str) -> str: + # Local version that calls fail() to write INSTALL_STATUS on error. + # Other scripts use the shared helpers.tekton_util.require_env instead. v = os.environ.get(name, "").strip() if not v: fail(f"❌ Required environment variable is missing: {name}") @@ -107,16 +161,37 @@ def patch_manifest_namespace(manifest_path: Path, operator_namespace: str) -> No manifest_path.write_text(patched, encoding="utf-8") -def resolve_olminstall_manifest(olminstall_dir: Path, operator_name: str) -> tuple[Path, str]: +def normalize_odh_olm_targets(operator_name: str, operator_namespace: str, update_channel: str) -> tuple[str, str]: + """Align ODH catalog installs with Jenkins (odhTestConfigOperator / generateTestConfigFile). + + For ``odh-stable`` (Konflux ODH catalog) Jenkins uses the rhods-operator OLM package and + downstream (RHOAI) operator namespace — same as ``install-operator.sh rhods-operator`` in olminstall. + """ + if update_channel == "odh-stable": + if operator_name != "rhods-operator" or operator_namespace != "redhat-ods-operator": + print( + "ODH odh-stable: using Jenkins/olminstall targets " + f"rhods-operator / redhat-ods-operator " + f"(was {operator_name!r} / {operator_namespace!r})" + ) + return "rhods-operator", "redhat-ods-operator" + if update_channel == "odh-nightlies" and operator_name == "opendatahub-operator": + print("ODH odh-nightlies: using rhods-operator OLM package (Jenkins default)") + return "rhods-operator", operator_namespace + return operator_name, operator_namespace + + +def resolve_olminstall_manifest(olminstall_dir: Path, operator_name: str) -> Path: + """Return path to ``resources/install-.yaml`` in the cloned olminstall repo.""" resources_dir = (olminstall_dir / "resources").resolve() manifest = olminstall_dir / "resources" / f"install-{operator_name}.yaml" try: manifest.resolve().relative_to(resources_dir) except (ValueError, OSError): fail(f"❌ Resolved manifest path escapes olminstall dir: {manifest}") - if manifest.is_file(): - return manifest, operator_name - fail(f"❌ Missing olminstall manifest: {manifest}") + if not manifest.is_file(): + fail(f"❌ Missing olminstall manifest: {manifest}") + return manifest def apply_catalog_source(name: str, fbcf_image: str) -> None: @@ -327,6 +402,67 @@ def wait_global_pull_secret_syncer() -> None: print(pods_diag.stdout.rstrip()) +def _cr_exists(kind: str, name: str) -> bool: + r = oc_run(["get", kind, name], check=False, capture_output=True, timeout=30) + return r.returncode == 0 + + +def _apply_cr(kind: str, name: str, yaml_doc: str) -> None: + r = oc_run(["apply", "-f", "-"], stdin_text=yaml_doc, check=False, capture_output=True, timeout=60) + if r.returncode != 0: + err = (r.stderr or r.stdout or "").strip() + print(f"⚠ Could not apply {kind}/{name}: {err}", file=sys.stderr) + fail(f"oc apply failed for {kind}/{name}: {err or 'unknown error'}") + print(f"✓ Applied {kind}/{name}") + + +def setup_dsc_resources() -> None: + """Create DSCInitialization and DataScienceCluster if they don't already exist.""" + print("\nSetting up RHOAI DataScienceCluster resources for BVT testing...") + + if _cr_exists("dscinitialization", "default-dsci"): + print(" DSCInitialization/default-dsci already exists — skipping") + else: + _apply_cr("dscinitialization", "default-dsci", _DSCI_YAML) + + if _cr_exists("datasciencecluster", "default-dsc"): + print(" DataScienceCluster/default-dsc already exists — skipping") + else: + _apply_cr("datasciencecluster", "default-dsc", _DSC_YAML) + + +def wait_dsc_ready(timeout_s: int = 600) -> bool: + """Poll until DataScienceCluster/default-dsc has Ready==True or timeout expires.""" + print(f"Waiting for DataScienceCluster/default-dsc to be Ready (up to {timeout_s}s)...") + deadline = time.time() + timeout_s + iteration = 0 + while time.time() < deadline: + r = oc_run( + [ + "get", "datasciencecluster", "default-dsc", + "-o", "jsonpath={.status.conditions[?(@.type==\"Ready\")].status}", + ], + check=False, capture_output=True, text=True, timeout=30, + ) + status = (r.stdout or "").strip() + if status == "True": + print("✓ DataScienceCluster/default-dsc is Ready") + return True + iteration += 1 + print(f" DSC Ready status: {status or 'unknown'} (iter {iteration})") + if iteration % 4 == 0: + oc_run( + ["get", "datasciencecluster", "default-dsc", "-o", + "custom-columns=NAME:.metadata.name,PHASE:.status.phase," + "READY:.status.conditions[?(@.type==\"Ready\")].status"], + capture_output=False, check=False, timeout=60, + ) + time.sleep(15) + print(f"⚠ DataScienceCluster/default-dsc not Ready after {timeout_s}s — BVT tests may fail") + oc_run(["describe", "datasciencecluster", "default-dsc"], capture_output=False, check=False, timeout=120) + return False + + def pick_succeeded_csv_version(namespace: str, olminstall_operator: str) -> str | None: r = oc_run(["get", "csv", "-n", namespace, "-o", "json"], capture_output=True, text=True, check=False, timeout=120) if r.returncode != 0: @@ -430,6 +566,10 @@ def main() -> int: validate_dns_label(catalog_name, "OLMINSTALL_CATALOG_NAME") validate_dns_label(cluster_pull_secret, "CLUSTER_MARKETPLACE_PULL_SECRET_NAME") + operator_name, operator_namespace = normalize_odh_olm_targets( + operator_name, operator_namespace, update_channel + ) + print("=========================================") print(" ODH/RHOAI Operator Installation") print(f" FBCF: {fbcf_image}") @@ -527,14 +667,14 @@ def main() -> int: wait_global_pull_secret_syncer() patch_oc_wait_sh(olminstall_dir, operator_namespace) - manifest_path, olminstall_operator = resolve_olminstall_manifest(olminstall_dir, operator_name) + manifest_path = resolve_olminstall_manifest(olminstall_dir, operator_name) patch_manifest_namespace(manifest_path, operator_namespace) print( - f"Running olminstall (./install-operator.sh {olminstall_operator} {update_channel} {catalog_name})..." + f"Running olminstall (./install-operator.sh {operator_name} {update_channel} {catalog_name})..." ) r_install = subprocess.run( - ["./install-operator.sh", olminstall_operator, update_channel, catalog_name], + ["./install-operator.sh", operator_name, update_channel, catalog_name], cwd=olminstall_dir, timeout=7200, ) @@ -544,13 +684,18 @@ def main() -> int: oc_run(["describe", "sub", "-n", operator_namespace], capture_output=False, check=False, timeout=120) fail() - csv_version = pick_succeeded_csv_version(operator_namespace, olminstall_operator) + csv_version = pick_succeeded_csv_version(operator_namespace, operator_name) if not csv_version: print(f"❌ No CSV reached Succeeded phase in namespace {operator_namespace}") oc_run(["get", "csv", "-n", operator_namespace], capture_output=False, check=False, timeout=120) fail() Path(operator_version_path).write_text(csv_version, encoding="utf-8") + setup_dsc_resources() + if not wait_dsc_ready(timeout_s=600): + print("❌ DataScienceCluster/default-dsc did not become Ready within timeout", file=sys.stderr) + fail("DSC not Ready") + print("") print("=========================================") print(" Installation Results") diff --git a/integration-tests/olminstall/helpers/kubearchive.py b/integration-tests/olminstall/helpers/kubearchive.py index b4d28c113..b6fcea18d 100644 --- a/integration-tests/olminstall/helpers/kubearchive.py +++ b/integration-tests/olminstall/helpers/kubearchive.py @@ -3,7 +3,9 @@ from __future__ import annotations import json +import os import sys +import time from typing import Any from urllib.error import HTTPError, URLError @@ -11,6 +13,35 @@ from urllib.request import Request, urlopen +class KubeArchiveAuthError(Exception): + """Raised when KubeArchive rejects the bearer token (HTTP 401 / 403).""" + + def __init__(self, code: int, path: str) -> None: + self.code = code + self.path = path + super().__init__( + f"KubeArchive HTTP {code} for {path!r}: token may be expired or lack permission " + "to read archived Tekton resources." + ) + + +def _ka_ts() -> str: + return time.strftime("%H:%M:%S") + + +def _http_timeout_s() -> float: + """Per-request timeout for KubeArchive GET (connect + read body). + + Default 20s; override with ``OLMINSTALL_KA_HTTP_TIMEOUT`` (5–120). + """ + raw = os.environ.get("OLMINSTALL_KA_HTTP_TIMEOUT", "20").strip() + try: + v = float(raw) + except ValueError: + v = 20.0 + return max(5.0, min(v, 120.0)) + + class KubeArchiveClient: def __init__(self, host: str, token: str) -> None: self.host = host.rstrip("/") @@ -29,20 +60,54 @@ def _request(self, path: str) -> str: headers={"Authorization": f"Bearer {self.token}"}, method="GET", ) + t0 = time.monotonic() try: - with urlopen(req, timeout=20) as resp: + with urlopen(req, timeout=_http_timeout_s()) as resp: return resp.read().decode("utf-8") except HTTPError as exc: - print(f"WARN KubeArchive HTTP {exc.code} for GET {path}", file=sys.stderr) + elapsed = time.monotonic() - t0 + if exc.code in (401, 403): + raise KubeArchiveAuthError(exc.code, path) from exc + print( + f"[{_ka_ts()}] WARN KubeArchive HTTP {exc.code} for GET {path} (after {elapsed:.1f}s)", + file=sys.stderr, + ) return "" except URLError as exc: + elapsed = time.monotonic() - t0 reason = getattr(exc, "reason", exc) - print(f"WARN KubeArchive request failed for GET {path}: {reason}", file=sys.stderr) + print( + f"[{_ka_ts()}] WARN KubeArchive request failed for GET {path} (after {elapsed:.1f}s): {reason}", + file=sys.stderr, + ) + return "" + except OSError as exc: + # Catches IncompleteRead, RemoteDisconnected, and other socket/chunked-transfer errors + # that are not wrapped by urllib into URLError. + elapsed = time.monotonic() - t0 + print( + f"[{_ka_ts()}] WARN KubeArchive read error for GET {path} (after {elapsed:.1f}s): {exc}", + file=sys.stderr, + ) return "" def check(self) -> bool: + """Return whether KubeArchive ``/livez`` responds OK. + + On HTTP 401/403 returns ``False`` (warns) instead of raising; see + :meth:`get_json` / :meth:`get_text` which may still raise + :class:`KubeArchiveAuthError` for protected paths when the token is invalid. + """ if self.available is None: - raw = self._request("/livez") + try: + raw = self._request("/livez") + except KubeArchiveAuthError: + print( + f"[{_ka_ts()}] WARN KubeArchive auth failed for /livez; treating as unavailable.", + file=sys.stderr, + ) + self.available = False + return False try: self.available = bool(raw and json.loads(raw).get("code") == 200) except json.JSONDecodeError: @@ -50,6 +115,7 @@ def check(self) -> bool: return bool(self.available) def get_json(self, path: str) -> dict[str, Any]: + """GET *path* and parse JSON. May raise :class:`KubeArchiveAuthError` (401/403).""" raw = self._request(path) if not raw: return {} @@ -59,4 +125,5 @@ def get_json(self, path: str) -> dict[str, Any]: return {} def get_text(self, path: str) -> str: + """GET *path* and return body text. May raise :class:`KubeArchiveAuthError` (401/403).""" return self._request(path) diff --git a/integration-tests/olminstall/helpers/patch_pipelinerun_summary.py b/integration-tests/olminstall/helpers/patch_pipelinerun_summary.py new file mode 100644 index 000000000..a3632447f --- /dev/null +++ b/integration-tests/olminstall/helpers/patch_pipelinerun_summary.py @@ -0,0 +1,74 @@ +#!/usr/bin/env python3 +"""Patch olminstall summary onto the PipelineRun and print it for Konflux UI. + +Writes PipelineRun annotations, Tekton task results (Results panel), and a log block +visible under post-results → patch-summary-annotations in the Konflux UI. + +Env: + PIPELINE_RUN_NAME -- Tekton PipelineRun (default: /etc/tekton/pipelineRunName) + TESTS -- comma-separated test phases (optional; read from PipelineRun params) + ARTIFACTS_URL_PATH, TEST_OUTPUT_PATH, OPERATOR_VERSION_PATH, EPHEMERAL_CLUSTER_PATH + -- optional Tekton result file paths (set by the pipeline step) +""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.pipelinerun_summary import ( # noqa: E402 + collect_summary_annotations, + format_summary_log_block, + get_pipelinerun_json, + merge_patch_pipelinerun_annotations, + namespace_from_env, + pipeline_run_name_from_env, + pipelinerun_param_value, + write_summary_tekton_results, +) + + +def main() -> int: + pr_name = pipeline_run_name_from_env() + if not pr_name: + print("PIPELINE_RUN_NAME missing", file=sys.stderr) + return 1 + ns = namespace_from_env() + if not ns: + print("namespace missing", file=sys.stderr) + return 1 + + prj = get_pipelinerun_json(pr_name, ns) + tests_csv = os.environ.get("TESTS", "").strip() or pipelinerun_param_value(prj, "TESTS", "") + ann = collect_summary_annotations( + pipeline_run=pr_name, + namespace=ns, + tests_csv=tests_csv, + prj=prj, + ) + + predicted_url = os.environ.get("ARTIFACTS_URL", "").strip() + if predicted_url and not ann.get("olminstall.test-results-url"): + ann["olminstall.test-results-url"] = predicted_url + + if ann: + if not merge_patch_pipelinerun_annotations(pr_name, ns, ann): + return 1 + prj = get_pipelinerun_json(pr_name, ns) or prj + else: + print("No olminstall summary annotations collected from TaskRuns") + + write_summary_tekton_results(ann) + print(format_summary_log_block(pipeline_run=pr_name, prj=prj, summary_annotations=ann)) + if ann: + print(f"\nPatched {len(ann)} olminstall summary annotation(s) on PipelineRun/{pr_name}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/pipelinerun_summary.py b/integration-tests/olminstall/helpers/pipelinerun_summary.py new file mode 100644 index 000000000..68eab1e5d --- /dev/null +++ b/integration-tests/olminstall/helpers/pipelinerun_summary.py @@ -0,0 +1,289 @@ +"""Collect olminstall summary fields and patch them onto the PipelineRun for Konflux UI.""" + +from __future__ import annotations + +import json +import os +import ssl +import sys +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path +from typing import Any + +from helpers.bvt_artifacts import ( + bvt_unpublished_reason, + published_artifacts_url_from_taskruns, + tests_include_bvt, +) +from helpers.tekton_incluster import ( + in_cluster_get, + kubernetes_api_base_url, + list_taskruns_in_cluster, + namespace_from_env, + pipeline_run_name_from_env, + result_map, + task_name, + validate_kubernetes_api_url, +) +from helpers.constants import ( + DEFAULT_ARTIFACT_BROWSER_REPO_PATH, + DEFAULT_ARTIFACT_BROWSER_URL, + OLMINSTALL_CTX_PRINT_KEYS, +) + +SUMMARY_ANNOTATION_KEYS: tuple[str, ...] = ( + "olminstall.fbcf-image", + "olminstall.operator-version", + "olminstall.ephemeral-cluster", + "olminstall.test-results-url", + "olminstall.artifacts-status", + "olminstall.pipeline-test-output", +) + +SUMMARY_ANNOTATION_LABELS: dict[str, str] = { + "olminstall.fbcf-image": "FBCF image", + "olminstall.operator-version": "Operator version", + "olminstall.ephemeral-cluster": "Ephemeral CTI", + "olminstall.test-results-url": "Test Results", + "olminstall.artifacts-status": "Artifacts status", + "olminstall.pipeline-test-output": "Pipeline test output", + "olminstall.run-owner": "Run owner", + "olminstall.product": "Product", + "olminstall.update-channel": "Update channel", + "olminstall.rhoai-version": "RHOAI version", + "olminstall.ocp-version": "OCP version (ephemeral)", + "olminstall.scripts-repo-url": "Scripts repo", + "olminstall.scripts-repo-revision": "Scripts branch/revision", + "olminstall.tests": "Test phases (TESTS)", + "olminstall.slack-channel-id": "Slack channel ID", + "olminstall.bvt-env-only": "BVT env-only", +} + + +def _k8s_request( + method: str, + url: str, + token: str, + ca_path: Path, + *, + body: dict[str, Any] | None = None, +) -> dict[str, Any]: + validate_kubernetes_api_url(url) + ctx = ssl.create_default_context(cafile=str(ca_path)) + data = None + headers = {"Authorization": f"Bearer {token}"} + if body is not None: + data = json.dumps(body).encode("utf-8") + headers["Content-Type"] = "application/merge-patch+json" + req = urllib.request.Request(url, data=data, headers=headers, method=method) + with urllib.request.urlopen(req, context=ctx, timeout=60) as resp: + raw = resp.read() + if not raw: + return {} + parsed = json.loads(raw.decode("utf-8")) + return parsed if isinstance(parsed, dict) else {} + + +def get_pipelinerun_json(pipeline_run: str, namespace: str) -> dict[str, Any]: + token_path = Path("/var/run/secrets/kubernetes.io/serviceaccount/token") + ca_path = Path("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt") + base = kubernetes_api_base_url() + if not (pipeline_run and namespace and token_path.is_file() and ca_path.is_file() and base): + return {} + token = token_path.read_text(encoding="utf-8") + url = ( + f"{base}/apis/tekton.dev/v1/namespaces/" + f"{urllib.parse.quote(namespace)}/pipelineruns/{urllib.parse.quote(pipeline_run)}" + ) + try: + return in_cluster_get(url, token, ca_path) + except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, OSError, ValueError): + return {} + + +def pipelinerun_param_value(prj: dict[str, Any], name: str, default: str = "") -> str: + for p in prj.get("spec", {}).get("params", []) or []: + if p.get("name") != name: + continue + val = p.get("value") + if isinstance(val, str) and val.strip(): + return val.strip() + return default + + +def predicted_artifacts_browser_url(prj: dict[str, Any], pipeline_run: str) -> str: + base = pipelinerun_param_value(prj, "ARTIFACT_BROWSER_URL", DEFAULT_ARTIFACT_BROWSER_URL).rstrip("/") + repo = pipelinerun_param_value(prj, "ARTIFACT_BROWSER_REPO_PATH", DEFAULT_ARTIFACT_BROWSER_REPO_PATH).strip( + "/" + ) + pr_name = (pipeline_run or "").strip() + if not pr_name: + return f"{base}/{repo}/-bvt/" + return f"{base}/{repo}/{pr_name}-bvt/" + + +def task_result(taskruns: list[dict[str, Any]], task_substr: str, result_name: str) -> str: + needle = task_substr.lower() + for tr in taskruns: + if needle not in task_name(tr).lower(): + continue + val = result_map(tr).get(result_name, "").strip() + if val: + return val + return "" + + +def pick_pipeline_test_output(taskruns: list[dict[str, Any]]) -> str: + for prefer in ("bvt-health-checks-with-eaas", "bvt-health-checks-no-eaas"): + val = task_result(taskruns, prefer, "TEST_OUTPUT") + if val: + return val + return task_result(taskruns, "install-operator", "INSTALL_STATUS") + + +def collect_summary_annotations( + *, + pipeline_run: str, + namespace: str, + tests_csv: str, + prj: dict[str, Any] | None = None, + taskruns: list[dict[str, Any]] | None = None, +) -> dict[str, str]: + """Build ``olminstall.*`` annotations for Konflux UI and CLI summaries.""" + doc = prj if prj is not None else get_pipelinerun_json(pipeline_run, namespace) + runs = taskruns if taskruns is not None else list_taskruns_in_cluster(pipeline_run, namespace) + out: dict[str, str] = {} + + fbcf = task_result(runs, "extract-fbcf-image", "FBCF_IMAGE") + if fbcf: + out["olminstall.fbcf-image"] = fbcf + + op_ver = task_result(runs, "install-operator", "OPERATOR_VERSION") + if op_ver and op_ver not in ("(see pipeline run logs)", "(unknown)"): + out["olminstall.operator-version"] = op_ver + + cluster = task_result(runs, "provision-cluster", "clusterName") + if cluster: + out["olminstall.ephemeral-cluster"] = cluster + + if tests_include_bvt(tests_csv): + published = published_artifacts_url_from_taskruns(runs) + if published: + out["olminstall.test-results-url"] = published + else: + out["olminstall.test-results-url"] = predicted_artifacts_browser_url(doc, pipeline_run) + out["olminstall.artifacts-status"] = bvt_unpublished_reason(runs) + + test_output = pick_pipeline_test_output(runs) + if test_output: + out["olminstall.pipeline-test-output"] = test_output[:500] + + return out + + +def pipeline_succeeded_status_label(prj: dict[str, Any]) -> str: + conds = (prj.get("status") or {}).get("conditions", []) or [] + cond = next( + (c for c in conds if isinstance(c, dict) and c.get("type") == "Succeeded"), + {}, + ) + if not isinstance(cond, dict): + return "Unknown" + st = cond.get("status", "") + reason = (cond.get("reason") or "").strip() + if st == "True": + return "Succeeded" + if st == "False": + return reason or "Failed" + return reason or "Unknown" + + +def format_summary_log_block( + *, + pipeline_run: str, + prj: dict[str, Any], + summary_annotations: dict[str, str], +) -> str: + """Human-readable block for post-results step logs (Konflux task log panel).""" + merged = dict((prj.get("metadata") or {}).get("annotations") or {}) + merged.update(summary_annotations) + status = pipeline_succeeded_status_label(prj) + lines = [ + "===========================================================", + " Olminstall run summary (post-results)", + "===========================================================", + f" PipelineRun : {pipeline_run} [{status}]", + ] + for key in ( + "olminstall.operator-version", + "olminstall.fbcf-image", + "olminstall.ephemeral-cluster", + "olminstall.test-results-url", + "olminstall.artifacts-status", + ): + val = (merged.get(key) or "").strip() + if val: + label = SUMMARY_ANNOTATION_LABELS.get(key, key) + lines.append(f" {label + ':':16} {val}") + lines.append("") + lines.append("Trigger context (PipelineRun annotations):") + ctx_any = False + for key in OLMINSTALL_CTX_PRINT_KEYS: + val = (merged.get(key) or "").strip() + if not val: + continue + ctx_any = True + label = SUMMARY_ANNOTATION_LABELS.get(key, key) + lines.append(f" {label}: {val}") + if not ctx_any: + lines.append(" (no olminstall.* annotations on this PipelineRun)") + test_out = (merged.get("olminstall.pipeline-test-output") or "").strip() + if test_out: + lines.append("") + lines.append(f" Pipeline test output: {test_out}") + lines.append("===========================================================") + return "\n".join(lines) + + +def write_summary_tekton_results(summary_annotations: dict[str, str]) -> None: + """Expose key fields as Tekton task results (Konflux PipelineRun → Results panel).""" + from helpers.tekton_util import write_result + + mapping = { + "ARTIFACTS_URL_PATH": "olminstall.test-results-url", + "TEST_OUTPUT_PATH": "olminstall.pipeline-test-output", + "OPERATOR_VERSION_PATH": "olminstall.operator-version", + "EPHEMERAL_CLUSTER_PATH": "olminstall.ephemeral-cluster", + } + for env_key, ann_key in mapping.items(): + path = os.environ.get(env_key, "").strip() + if path: + write_result(path, summary_annotations.get(ann_key, "")) + + +def merge_patch_pipelinerun_annotations( + pipeline_run: str, + namespace: str, + annotations: dict[str, str], +) -> bool: + if not annotations: + return True + token_path = Path("/var/run/secrets/kubernetes.io/serviceaccount/token") + ca_path = Path("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt") + base = kubernetes_api_base_url() + if not (pipeline_run and namespace and token_path.is_file() and ca_path.is_file() and base): + return False + token = token_path.read_text(encoding="utf-8") + url = ( + f"{base}/apis/tekton.dev/v1/namespaces/" + f"{urllib.parse.quote(namespace)}/pipelineruns/{urllib.parse.quote(pipeline_run)}" + ) + body = {"metadata": {"annotations": annotations}} + try: + _k8s_request("PATCH", url, token, ca_path, body=body) + return True + except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, OSError, ValueError) as exc: + print(f"WARN: could not patch PipelineRun annotations: {exc}", file=sys.stderr) + return False diff --git a/integration-tests/olminstall/helpers/propagate_pipeline_test_output.py b/integration-tests/olminstall/helpers/propagate_pipeline_test_output.py new file mode 100644 index 000000000..99d2f13be --- /dev/null +++ b/integration-tests/olminstall/helpers/propagate_pipeline_test_output.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python3 +"""Emit pipeline-level TEST_OUTPUT from TaskRun results (Tekton finally task). + +Reads the current PipelineRun via the in-cluster API so pipeline results need not +reference $(tasks.install-operator.results.*) when that task was skipped. + +Priority: bvt-health-checks-with-eaas.TEST_OUTPUT, then bvt-health-checks-no-eaas.TEST_OUTPUT, +then install-operator.INSTALL_STATUS, else a short status fallback. + +Env: + RESULT_PATH -- Tekton result file path to write (required) +Optional: + PIPELINE_RUN_NAME -- default: /etc/tekton/pipelineRunName (Tekton-injected) +""" +from __future__ import annotations + +import os +import sys +import urllib.error +import urllib.parse +from pathlib import Path + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.pipelinerun_summary import pick_pipeline_test_output +from helpers.tekton_incluster import ( + in_cluster_get, + kubernetes_api_base_url, + list_taskruns_in_cluster, + namespace_from_env, + pipeline_run_name_from_env, +) +from helpers.tekton_util import write_result + + +def _pick_output(pr: dict[str, object]) -> str: + meta = pr.get("metadata") + if not isinstance(meta, dict): + return "ERROR: PipelineRun missing metadata" + pr_name = str(meta.get("name") or "") + + ns = namespace_from_env(required=True) + list_errors: list[str] = [] + taskruns = list_taskruns_in_cluster(pr_name, ns, error_out=list_errors) + if list_errors: + return list_errors[0] + if not taskruns: + return _pipeline_run_condition_fallback(pr, pr_name) + + picked = pick_pipeline_test_output(taskruns) + if picked: + return picked + return _pipeline_run_condition_fallback(pr, pr_name) + + +def _pipeline_run_condition_fallback(pr: dict[str, object], pr_name: str) -> str: + conds = pr.get("status", {}) + if isinstance(conds, dict): + c = conds.get("conditions") + if isinstance(c, list) and c: + first = c[0] + if isinstance(first, dict): + return ( + f"PipelineRun {pr_name}: {first.get('type', 'condition')}=" + f"{first.get('status', '')} ({first.get('reason', '')})" + ) + return f"PipelineRun {pr_name}: no TEST_OUTPUT/INSTALL_STATUS found on TaskRuns" + + +def main() -> int: + result_path = os.environ.get("RESULT_PATH", "").strip() + if not result_path: + print("RESULT_PATH is required", file=sys.stderr) + return 1 + + pr_name = pipeline_run_name_from_env(required=True) + ns = namespace_from_env(required=True) + try: + token = Path("/var/run/secrets/kubernetes.io/serviceaccount/token").read_text( + encoding="utf-8" + ) + ca = Path("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt") + except OSError as exc: + print(f"ERROR: cannot read in-cluster serviceaccount credentials: {exc}", file=sys.stderr) + return 1 + base = kubernetes_api_base_url() + if not base: + print( + "KUBERNETES_SERVICE_HOST is missing or not an allowed in-cluster API host", + file=sys.stderr, + ) + return 1 + pr_url = ( + f"{base}/apis/tekton.dev/v1/namespaces/{urllib.parse.quote(ns)}" + f"/pipelineruns/{urllib.parse.quote(pr_name)}" + ) + try: + pr = in_cluster_get(pr_url, token, ca) + except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, OSError, ValueError) as exc: + print(f"ERROR: get PipelineRun: {exc}", file=sys.stderr) + return 1 + + text = _pick_output(pr) + write_result(result_path, text) + print(f"Wrote pipeline TEST_OUTPUT ({len(text)} chars) to {result_path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/prune_stale_testops_its.py b/integration-tests/olminstall/helpers/prune_stale_testops_its.py new file mode 100644 index 000000000..cfd88e7f0 --- /dev/null +++ b/integration-tests/olminstall/helpers/prune_stale_testops_its.py @@ -0,0 +1,73 @@ +#!/usr/bin/env python3 +"""Delete legacy IntegrationTestScenario CRs before raw ``oc create -f test-snapshot.yaml``. + +Deletes ``STALE_TESTOPS_PLAYPEN_ITS_NAMES`` (``helpers/constants.py``) — the same set +``OLMInstallRunner.prune_stale_integration_test_scenarios`` removes before Snapshot create. + +Exit code 1 if any ``oc delete`` returned non-zero (best-effort per object; failures are surfaced). + +Usage: + python3 integration-tests/olminstall/helpers/prune_stale_testops_its.py + python3 integration-tests/olminstall/helpers/prune_stale_testops_its.py -n my-tenant +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +_OLMINSTALL_DIR = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL_DIR) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL_DIR)) + +from helpers.constants import DEFAULT_NAMESPACE, STALE_TESTOPS_PLAYPEN_ITS_NAMES +from helpers.oc_util import filter_warning_lines, run_cmd + + +def delete_stale_testops_playpen_integration_test_scenarios(namespace: str) -> int: + """``oc delete`` each name in ``STALE_TESTOPS_PLAYPEN_ITS_NAMES`` (ignore-not-found). + + Returns the number of ``oc delete`` invocations that exited non-zero (warnings printed). + """ + failures = 0 + for name in sorted(STALE_TESTOPS_PLAYPEN_ITS_NAMES): + proc = run_cmd( + ["oc", "delete", "integrationtestscenario", name, "-n", namespace, "--ignore-not-found"], + capture=True, + check=False, + ) + if proc.returncode != 0: + failures += 1 + msg = filter_warning_lines(f"{proc.stdout}\n{proc.stderr}").strip() + print(f"WARN oc delete integrationtestscenario/{name}: {msg or proc.returncode}", file=sys.stderr) + return failures + + +def main(argv: list[str] | None = None) -> int: + p = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + p.add_argument( + "-n", + "--namespace", + default=DEFAULT_NAMESPACE, + help=f"Kubernetes namespace (default: {DEFAULT_NAMESPACE})", + ) + args = p.parse_args(argv) + ns = args.namespace.strip() + if not ns: + print("ERROR: empty namespace", file=sys.stderr) + return 2 + stale = sorted(STALE_TESTOPS_PLAYPEN_ITS_NAMES) + if not stale: + print("Nothing to prune (STALE_TESTOPS_PLAYPEN_ITS_NAMES is empty).") + return 0 + print(f"Deleting legacy IntegrationTestScenario objects in namespace {ns!r}: {', '.join(stale)}") + failures = delete_stale_testops_playpen_integration_test_scenarios(ns) + if failures: + print(f"ERROR: {failures} oc delete command(s) failed (see stderr above).", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/resolve_ocp_prefix.py b/integration-tests/olminstall/helpers/resolve_ocp_prefix.py new file mode 100644 index 000000000..97d6c6cf9 --- /dev/null +++ b/integration-tests/olminstall/helpers/resolve_ocp_prefix.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +"""Write the OCP minor prefix step result for EaaS pick-version. + +Env: + OVERRIDE — optional ``OCP_VERSION_PREFIX`` from the pipeline (whitespace stripped). + DEFAULT_MINOR — first supported minor from ``eaas-get-supported-versions`` (e.g. ``4.20``). + PREFIX_RESULT_PATH — Tekton ``step.results.prefix`` file path. +""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.tekton_util import require_env, write_result + + +def compute_prefix(override_raw: str, default_minor: str) -> str: + """Match legacy bash: empty override → ``{default_minor}.``; else strip spaces and ensure trailing ``.``.""" + override = "".join(override_raw.split()) + if not override: + return f"{default_minor}." + return override if override.endswith(".") else f"{override}." + + +def main() -> int: + override_raw = os.environ.get("OVERRIDE", "") + default_minor = require_env("DEFAULT_MINOR").strip() + if not default_minor: + print("DEFAULT_MINOR is empty after trim", file=sys.stderr) + return 1 + result_path = require_env("PREFIX_RESULT_PATH") + write_result(result_path, compute_prefix(override_raw, default_minor)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/resolve_opendatahub_tests_image.py b/integration-tests/olminstall/helpers/resolve_opendatahub_tests_image.py new file mode 100644 index 000000000..b765edeeb --- /dev/null +++ b/integration-tests/olminstall/helpers/resolve_opendatahub_tests_image.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""Resolve quay.io/opendatahub/opendatahub-tests image reference for BVT. + +Tag rules mirror Jenkins vars/validateHealth.groovy (RHOAI_VERSION -> image tag). +Writes the resolved image reference to RESULT_PATH for Tekton results. + +Env: OPERATOR_VERSION, OPENDATAHUB_TESTS_REPO (default quay.io/opendatahub/opendatahub-tests), + RESULT_PATH (required -- Tekton result file). +""" + +from __future__ import annotations + +import os +import re +import shutil +import sys +from pathlib import Path + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.tekton_util import require_env, run, write_result + +_EA_RE = re.compile(r"^(\d+)\.(\d+)\.\d+-ea\.(\d+)$") +_MAJOR_MINOR_RE = re.compile(r"^(\d+)\.(\d+)\.") + + +def main() -> int: + result_path = require_env("RESULT_PATH") + repo = os.environ.get("OPENDATAHUB_TESTS_REPO", "").strip() or "quay.io/opendatahub/opendatahub-tests" + csv_version = os.environ.get("OPERATOR_VERSION", "").strip() + latest_img = f"{repo}:latest" + + if not csv_version or csv_version == "latest": + print(f"No CSV version (empty or latest) -- using {latest_img}") + write_result(result_path, latest_img) + return 0 + + tag = "" + m = _EA_RE.match(csv_version) + if m: + tag = f"{m.group(1)}.{m.group(2)}ea{m.group(3)}" + print(f"EA CSV version {csv_version} -> image tag {tag}") + else: + m = _MAJOR_MINOR_RE.match(csv_version) + if m: + tag = f"{m.group(1)}.{m.group(2)}" + print(f"CSV version {csv_version} -> image tag {tag} (major.minor)") + else: + print(f"Unrecognized CSV version format: {csv_version} -- using {latest_img}") + write_result(result_path, latest_img) + return 0 + + candidate = f"{repo}:{tag}" + skopeo = shutil.which("skopeo") + if not skopeo: + print(f"skopeo not found in PATH -- using {latest_img}") + write_result(result_path, latest_img) + return 0 + + probe = run( + [skopeo, "inspect", f"docker://{candidate}", "--no-tags"], + check=False, + capture=True, + ) + if probe.returncode == 0: + print(f"opendatahub-tests tag exists: {candidate}") + write_result(result_path, candidate) + else: + print(f"Tag not found for {candidate} -- falling back to {latest_img}") + write_result(result_path, latest_img) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/run_bvt_pytest.py b/integration-tests/olminstall/helpers/run_bvt_pytest.py new file mode 100644 index 000000000..f342e4d8f --- /dev/null +++ b/integration-tests/olminstall/helpers/run_bvt_pytest.py @@ -0,0 +1,295 @@ +#!/usr/bin/env python3 +"""Run BVT pytest inside the opendatahub-tests image. + +Parameterised via env vars so a single script covers cluster_health, +operator_health, and --collect-only modes. + +Env (required): + ARTIFACT_PREFIX -- filename prefix for JUnit XML + console log +Env (optional): + PYTEST_MARKER -- pytest -m expression (empty = no -m flag) + PYTEST_EXTRA_ARGS -- extra pytest CLI args (e.g. "--collect-only -q" or "-svv") + ARTIFACTS_DIR -- directory for JUnit + logs (default /artifacts; must be under + /artifacts or TEST_ARTIFACTS_DIR when set for local runs) + TEST_ARTIFACTS_DIR -- optional extra allowed root (e.g. ./artifacts) for local debugging + TESTS_SUBDIR -- subdirectory under tests root (default "tests/cluster_health") +""" + +from __future__ import annotations + +import os +import shlex +import shutil +import subprocess +import sys +import threading +from datetime import datetime, timezone +from pathlib import Path + +_KNOWN_ROOTS = [ + "/home/odh/opendatahub-tests", + "/opendatahub-tests", + "/opt/app-root/src", + "/workspace/source", +] + + +def _safe_artifact_prefix(raw: str) -> str | None: + """ARTIFACT_PREFIX must be a single filename segment (no path separators).""" + if not raw or raw in {".", ".."}: + return None + if "/" in raw or "\\" in raw: + return None + p = Path(raw) + if p.is_absolute() or ".." in p.parts or p.name != raw: + return None + return raw + + +def _artifacts_dir_bases() -> tuple[Path, ...]: + """Allowed roots for ARTIFACTS_DIR (pipeline + local dev).""" + bases: list[Path] = [Path("/artifacts")] + extra = os.environ.get("TEST_ARTIFACTS_DIR", "").strip() + if extra: + bases.append(Path(extra)) + return tuple(bases) + + +def _validate_artifacts_dir(raw: str) -> Path: + """Resolve *raw* and ensure it stays under an allowed artifacts root.""" + resolved = Path(raw).resolve() + for base in _artifacts_dir_bases(): + root = base.resolve() + if resolved == root: + return resolved + try: + resolved.relative_to(root) + return resolved + except ValueError: + continue + print( + f"ARTIFACTS_DIR must resolve under {', '.join(str(b) for b in _artifacts_dir_bases())}; got {resolved}", + file=sys.stderr, + ) + sys.exit(1) + + +def _safe_tests_subdir(raw: str) -> str | None: + """TESTS_SUBDIR must be a relative path without .. components.""" + if not raw: + return None + p = Path(raw) + if p.is_absolute() or ".." in p.parts: + return None + return raw + + +def _locate_tests_root(tests_subdir: str) -> str | None: + sub = Path(tests_subdir) + if sub.is_absolute() or ".." in sub.parts: + return None + for d in _KNOWN_ROOTS: + if (Path(d) / sub).is_dir(): + return d + return None + + +def _build_pytest_args( + marker: str, + extra_args: str, + tests_subdir: str, + junit_path: str, +) -> list[str]: + args: list[str] = [] + if marker: + args.extend(["-m", marker]) + if extra_args: + args.extend(shlex.split(extra_args)) + args.extend([tests_subdir, f"--junitxml={junit_path}", "--tb=native"]) + return args + + +def _bvt_timeout_seconds() -> float | None: + raw = os.environ.get("BVT_RUN_TIMEOUT_SECS", "").strip() + if not raw: + return None + try: + secs = float(raw) + except ValueError as exc: + raise ValueError( + f"BVT_RUN_TIMEOUT_SECS must be a positive number, got: {raw!r}" + ) from exc + if secs <= 0: + raise ValueError(f"BVT_RUN_TIMEOUT_SECS must be a positive number, got: {raw!r}") + return secs + + +def _run_with_tee( + cmd: list[str], + log_path: str, + *, + env: dict[str, str] | None = None, + timeout: float | None = None, +) -> int: + """Run *cmd*, tee stdout+stderr to *log_path*, return exit code. + + *timeout* overrides ``BVT_RUN_TIMEOUT_SECS`` when not ``None`` (omit for env-based timeout). + """ + timeout_s = _bvt_timeout_seconds() if timeout is None else (timeout if timeout > 0 else None) + attempt_header = ( + f"\n--- BVT pytest attempt {datetime.now(timezone.utc).isoformat()} " + f"({' '.join(cmd[:3])}...) ---\n" + ) + with open(log_path, "a", encoding="utf-8") as log: + log.write(attempt_header) + proc = subprocess.Popen( + cmd, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + env=env, + ) + assert proc.stdout is not None + + def _tee_stdout() -> None: + for line in proc.stdout: + sys.stdout.write(line) + log.write(line) + + reader = threading.Thread(target=_tee_stdout, daemon=True) + reader.start() + try: + if timeout_s is not None: + proc.wait(timeout=timeout_s) + else: + proc.wait() + except subprocess.TimeoutExpired: + print( + f"ERROR: command timed out after {timeout_s}s (BVT_RUN_TIMEOUT_SECS): {' '.join(cmd)}", + file=sys.stderr, + ) + proc.terminate() + try: + proc.wait(timeout=10) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait(timeout=30) + reader.join(timeout=5) + return 124 + reader.join() + return proc.returncode if proc.returncode is not None else 1 + + +def main() -> int: + artifact_prefix = os.environ.get("ARTIFACT_PREFIX", "").strip() + if not artifact_prefix: + print("ARTIFACT_PREFIX is required", file=sys.stderr) + return 1 + if _safe_artifact_prefix(artifact_prefix) is None: + print("ARTIFACT_PREFIX must be a single filename segment (no / or ..)", file=sys.stderr) + return 1 + + artifacts_path = _validate_artifacts_dir(os.environ.get("ARTIFACTS_DIR", "/artifacts").strip()) + tests_subdir = os.environ.get("TESTS_SUBDIR", "tests/cluster_health").strip() + if _safe_tests_subdir(tests_subdir) is None: + print("TESTS_SUBDIR must be a relative path without ..", file=sys.stderr) + return 1 + pytest_marker = os.environ.get("PYTEST_MARKER", "").strip() + pytest_extra = os.environ.get("PYTEST_EXTRA_ARGS", "").strip() + + root = _locate_tests_root(tests_subdir) + if not root: + print(f"ERROR: could not find {tests_subdir} under known opendatahub-tests paths.", file=sys.stderr) + return 1 + + os.chdir(root) + Path("results").mkdir(exist_ok=True) + + artifacts_path.mkdir(parents=True, exist_ok=True) + junit = str(artifacts_path / f"{artifact_prefix}.xml") + log = str(artifacts_path / f"{artifact_prefix}.console.log") + pytest_args = _build_pytest_args(pytest_marker, pytest_extra, tests_subdir, junit) + + # Official opendatahub-tests image: ENTRYPOINT is `uv run pytest` after + # build-time `uv sync`. Tekton script mode bypasses ENTRYPOINT. Upstream + # Dockerfile puts uv in /.local/bin; ensure that and ~/.local/bin are on PATH. + extra_path = "/.local/bin:/home/odh/.local/bin" + env = dict(os.environ) + env["PATH"] = f"{extra_path}:{env.get('PATH', '')}" + + ec = 1 + ran_uv = False + + uv = shutil.which("uv", path=env["PATH"]) + pyproject = Path("pyproject.toml") + if uv and pyproject.is_file(): + ran_uv = True + ec = _run_with_tee([uv, "run", "pytest", *pytest_args], log, env=env) + if ec == 124: + return ec + # pytest exit 0=pass 1=tests failed; anything ≥2 is an infra/collection error. + if ec >= 2: + print(f"WARN: uv run pytest exited {ec} (infra error); retrying without UV_NO_SYNC...", file=sys.stderr) + retry_env = {k: v for k, v in env.items() if k != "UV_NO_SYNC"} + ec = _run_with_tee([uv, "run", "pytest", *pytest_args], log, env=retry_env) + if ec == 124: + return ec + + if not ran_uv: + # ec was never set by uv; treat as infra so pip+pytest runs instead of exiting 1 without tests. + ec = 2 + + if ec == 124: + return ec + + # Only fall back to bare python when uv is unavailable or produced an infra error. + # ec==1 means tests ran but some failed — keep that result rather than re-running. + if ec >= 2: + print("WARN: uv unavailable or failed; using pip --target + PYTHONPATH (may miss non-pytest deps).", file=sys.stderr) + pylibs = "/tmp/tekton-pytest-libs" + Path(pylibs).mkdir(exist_ok=True) + pip_env = dict(env) + pip_env["PYTHONPATH"] = pylibs + try: + subprocess.run( + ["python3", "-c", "import pytest, shortuuid, kubernetes"], + env=pip_env, check=True, capture_output=True, + ) + except subprocess.CalledProcessError: + pip_proc = subprocess.run( + [ + "python3", + "-m", + "pip", + "install", + "--no-cache-dir", + "-q", + "-t", + pylibs, + "pytest", + "shortuuid", + "kubernetes", + ], + env=env, + check=False, + capture_output=True, + text=True, + ) + if pip_proc.returncode != 0: + tail = ((pip_proc.stdout or "") + (pip_proc.stderr or "")).strip()[:4000] + print( + f"ERROR: pip install pytest shortuuid kubernetes failed (exit {pip_proc.returncode}): " + f"{tail or '(no output)'}", + file=sys.stderr, + ) + return pip_proc.returncode + existing = env.get("PYTHONPATH", "") + pip_env["PYTHONPATH"] = f"{pylibs}:{existing}" if existing else pylibs + ec = _run_with_tee(["python3", "-m", "pytest", *pytest_args], log, env=pip_env) + + print(f"JUnit ({artifact_prefix}): {junit}") + return ec + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/runner.py b/integration-tests/olminstall/helpers/runner.py index b0296c9e7..a31891c51 100644 --- a/integration-tests/olminstall/helpers/runner.py +++ b/integration-tests/olminstall/helpers/runner.py @@ -3,6 +3,7 @@ from __future__ import annotations import argparse +import contextlib import json import os import re @@ -10,20 +11,32 @@ import subprocess import sys import tempfile +import threading import time from dataclasses import dataclass from pathlib import Path -from typing import Any +from collections.abc import Iterator +from typing import Any, TextIO from urllib.parse import quote from .constants import ( + DEFAULT_APP, + DEFAULT_ARTIFACT_BROWSER_REPO_PATH, + DEFAULT_ARTIFACT_BROWSER_URL, + DEFAULT_LIST_COUNT, + DEFAULT_NAMESPACE, + ITS_TESTS_PARAM_DEFAULT, LIST_SUPPORTED_OCP_MAX_PRS, OLMINSTALL_CTX_PRINT_KEYS, + RHOAI_FBCF_IMAGE_REF_PATTERN, + OLMINSTALL_TESTOPS_ITS_NAME, OLMINSTALL_WRITE_ANNOTATION_KEYS, PENDING_REASONS, + STALE_TESTOPS_PLAYPEN_ITS_NAMES, + olminstall_smoke_only_pipelinerun, ) from .errors import AppError -from .kubearchive import KubeArchiveClient +from .kubearchive import KubeArchiveAuthError, KubeArchiveClient from .oc_util import ( derive_konflux_ui_base, derive_kubearchive_host, @@ -35,6 +48,19 @@ ) +_DNS1123_SUBDOMAIN_RE = re.compile(r"^[a-z0-9]([-a-z0-9]*[a-z0-9])?$", re.ASCII) + + +def _snapshot_param_is_resource_name(snap: str) -> bool: + """True when SNAPSHOT param looks like a Kubernetes object name (not inline JSON).""" + s = (snap or "").strip() + if not s or s[0] in "{[": + return False + if len(s) > 253: + return False + return bool(_DNS1123_SUBDOMAIN_RE.fullmatch(s)) + + _CTX_ANNOTATION_LABELS: dict[str, str] = { "olminstall.run-owner": "Run owner", "olminstall.product": "Product", @@ -43,9 +69,144 @@ "olminstall.ocp-version": "OCP version (ephemeral)", "olminstall.scripts-repo-url": "Scripts repo", "olminstall.scripts-repo-revision": "Scripts branch/revision", + "olminstall.tests": "Test phases (TESTS)", + "olminstall.slack-channel-id": "Slack channel ID", + "olminstall.fbcf-image": "FBCF image", + "olminstall.operator-version": "Operator version", + "olminstall.ephemeral-cluster": "Ephemeral CTI", + "olminstall.test-results-url": "Test Results", + "olminstall.artifacts-status": "Artifacts status", + "olminstall.pipeline-test-output": "Pipeline test output", } +# Normalise container log lines: structlog pads levels as "[info ]"; pytest sometimes +# glues "PASSED" to the next ISO timestamp. Used for KubeArchive replay and live tkn streams. +_REPLAY_STATUS_TIMESTAMP_GLUE = re.compile( + r"(PASSED|FAILED|SKIPPED|ERROR|XFAIL|XPASS)(?=20\d{2}-\d{2}-\d{2}T)" +) +_LOG_LEVEL_BRACKET_PAD = re.compile( + r"\[((?:info|warning|error|debug|critical|exception|trace))\s+\]", + re.IGNORECASE, +) +# Other "[Token ]" padding (e.g. rare logger names) — token must start with a letter. +_REPLAY_LOG_BRACKET_PAD = re.compile(r"\[([A-Za-z][A-Za-z0-9_]*)\s{2,}\]") + +# ``tkn pipelinerun logs -f`` exit codes when the user interrupts log streaming (Ctrl-C). +_TKN_LOG_STREAM_DETACH_RC = frozenset({-2, 130, 143}) + + +def _normalize_log_line_for_display(line: str) -> str: + """Collapse padded bracket tags and fix status/timestamp glue for one log line.""" + if not line: + return line + line = _REPLAY_STATUS_TIMESTAMP_GLUE.sub(r"\1\n", line) + line = _LOG_LEVEL_BRACKET_PAD.sub(lambda m: f"[{m.group(1).lower()}]", line) + line = _REPLAY_LOG_BRACKET_PAD.sub(r"[\1]", line) + return line + + +def _format_live_tkn_log_line(raw_line: str) -> str | None: + """Prefix one ``tkn`` log line with a capture time; return ``None`` to skip blank noise lines.""" + body = _normalize_log_line_for_display(raw_line.rstrip("\r\n")) + if not body.strip(): + return None + return f"[{ts_now()}] {body}" + + +def _normalize_replayed_pod_log(text: str) -> str: + """Tidy common formatting glitches in archived container logs when printing.""" + if not text: + return text + return "".join(_normalize_log_line_for_display(line) for line in text.splitlines(keepends=True)) + + +def _interactive_progress_stream() -> TextIO: + """Stream for in-place progress (spinner). Prefer stdout; if not a TTY, use stderr (common in IDE runners).""" + if sys.stdout.isatty(): + return sys.stdout + if sys.stderr.isatty(): + return sys.stderr + return sys.stdout + + +@contextlib.contextmanager +def spin_while(description: str) -> Iterator[None]: + """Show a spinner (or ``…`` on non-TTY) while a slow block runs; TTY ends with ``description ok``.""" + wait_stream = _interactive_progress_stream() + term = (os.environ.get("TERM") or "").strip() + use_spinner = wait_stream.isatty() and term != "dumb" + frames = "|/-\\" + stop = threading.Event() + desc = description.rstrip() + th: threading.Thread | None = None + if use_spinner: + + def _spin() -> None: + i = 0 + while not stop.is_set(): + ch = frames[i % len(frames)] + try: + wait_stream.write(f"\r\033[K{desc} {ch}") + wait_stream.flush() + except BrokenPipeError: + return + i += 1 + if stop.wait(0.12): + return + + th = threading.Thread(target=_spin, name="olminstall-spin", daemon=True) + th.start() + else: + try: + wait_stream.write(f"{desc}...\n") + wait_stream.flush() + except BrokenPipeError: + pass + failed = False + try: + yield + except BaseException: + failed = True + raise + finally: + if th is not None: + stop.set() + th.join(timeout=5.0) + if use_spinner: + try: + wait_stream.write("\r\033[K") + wait_stream.flush() + if failed: + wait_stream.write(f"{desc} (stopped)\n") + else: + wait_stream.write(f"{desc} ok\n") + wait_stream.flush() + except BrokenPipeError: + pass + elif failed: + try: + wait_stream.write(f"{desc} (stopped)\n") + wait_stream.flush() + except BrokenPipeError: + pass + + +def format_olm_pipeline_watch_cli( + *, + olminstall_dir: Path, + namespace: str, + app: str, + pipelinerun: str | None, +) -> str: + """Copy-pastable command: stream logs or KubeArchive replay (see ``--watch``).""" + script = olminstall_dir / "olm_pipeline.py" + base = f"{sys.executable} {script} --watch" + if (pipelinerun or "").strip(): + base = f"{base} {pipelinerun.strip()}" + return f"{base} -n {namespace} --app {app}" + + def first_snapshot_component_name(snapshot_yaml: str) -> str: """Template components[].name from integration-tests/olminstall/test-snapshot.yaml.""" m = re.search(r"(?m)^\s+-\s+name:\s+(\S+)\s*$", snapshot_yaml) @@ -82,21 +243,42 @@ def __init__(self, args: argparse.Namespace) -> None: except ValueError: print(f"WARN Invalid PR_APPEAR_TIMEOUT_SECONDS={raw_to!r}; using 600", file=sys.stderr) self.pr_appear_timeout = 600 + raw_sw = os.environ.get("OLMINSTALL_PIPELINE_START_WAIT_SECONDS", "1200") + try: + self.pipeline_start_wait_seconds = max(60, int(raw_sw)) + except ValueError: + print( + f"WARN Invalid OLMINSTALL_PIPELINE_START_WAIT_SECONDS={raw_sw!r}; using 1200", + file=sys.stderr, + ) + self.pipeline_start_wait_seconds = 1200 self.cleanup_snapshot_on_exit = True + self._user_detached_from_logs = False self.snapshot_name = "" self.its_apply_tmp = "" self.log_file = "" self.pr = "" self.watch_completed = False self.watch_from_archive = False + self._kubearchive_log_replay = False self.ka_succeeded = "Unknown" self.pipeline_exit = 0 self.run_owner = "" self.token = "" self.ka: KubeArchiveClient | None = None + self.konflux_host_api = "" self.resolved_app = "" self.image = args.image or "" self.update_channel_override = args.channel or "" + # Filled after ``create_snapshot`` — used to match PipelineRun ``SNAPSHOT`` JSON params to this trigger. + self._trigger_snapshot_spec: dict[str, Any] | None = None + self._trigger_snapshot_created_ts = "" + + def _tests_its_override(self) -> bool: + """True when CLI should inject TESTS into the ITS (matches annotation logic).""" + return getattr(self.args, "tests_explicit", False) or self.args.tests != getattr( + self.args, "tests_catalog_default_csv", ITS_TESTS_PARAM_DEFAULT + ) def build_olminstall_context_annotations(self) -> dict[str, str]: """Safe, non-secret trigger context for Snapshot / PipelineRun metadata.""" @@ -112,18 +294,39 @@ def build_olminstall_context_annotations(self) -> dict[str, str]: out["olminstall.scripts-repo-url"] = self.args.konflux_repo if self.args.konflux_branch: out["olminstall.scripts-repo-revision"] = self.args.konflux_branch + if self._tests_its_override(): + out["olminstall.tests"] = self.args.tests + if (self.args.slack_channel_id or "").strip(): + out["olminstall.slack-channel-id"] = self.args.slack_channel_id.strip() return out + def early_summary_annotate_argv(self) -> list[str]: + """Predicted BVT artifact URL on the PipelineRun so Konflux UI shows it while the run is in progress.""" + from .bvt_artifacts import tests_include_bvt + from .pipelinerun_summary import predicted_artifacts_browser_url + + if not (self.pr or "").strip(): + return [] + tests_csv = (self.args.tests or "").strip() + if not tests_include_bvt(tests_csv): + return [] + prj = self.get_pipelinerun_json_for_display() + url = predicted_artifacts_browser_url(prj, self.pr) + return [f"olminstall.test-results-url={url}"] + def olminstall_context_annotate_argv(self) -> list[str]: ctx = self.build_olminstall_context_annotations() return [f"{k}={ctx[k]}" for k in OLMINSTALL_WRITE_ANNOTATION_KEYS if k in ctx] def get_pipelinerun_json_for_display(self) -> dict[str, Any]: - if self.watch_from_archive: + if self.watch_from_archive or self._kubearchive_log_replay: assert self.ka is not None - prj = self.ka.get_json( - f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.pr)}" - ) + path = f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.pr)}" + try: + prj = self.ka.get_json(path) + except KubeArchiveAuthError as exc: + print(f"WARN KubeArchive auth failed for display JSON: {exc}", file=sys.stderr) + return {} return prj if isinstance(prj, dict) else {} proc = run_cmd( ["oc", "get", "pipelinerun", self.pr, "-n", self.args.namespace, "-o", "json"], @@ -137,8 +340,7 @@ def get_pipelinerun_json_for_display(self) -> dict[str, Any]: except json.JSONDecodeError: return {} - def print_pipelinerun_context_annotations(self) -> None: - prj = self.get_pipelinerun_json_for_display() + def _trigger_context_lines(self, prj: dict[str, Any]) -> list[str]: ann = prj.get("metadata", {}).get("annotations") or {} lines: list[str] = [] for key in OLMINSTALL_CTX_PRINT_KEYS: @@ -146,14 +348,258 @@ def print_pipelinerun_context_annotations(self) -> None: if val: label = _CTX_ANNOTATION_LABELS.get(key, key) lines.append(f" {label}: {val}") - if not lines: - print("Trigger context : (no olminstall.* annotations on this PipelineRun)") - return - print("Trigger context (PipelineRun annotations):") - for ln in lines: - print(ln) + return lines + + def _pipelinerun_param_value(self, prj: dict[str, Any], name: str, default: str = "") -> str: + for p in prj.get("spec", {}).get("params", []) or []: + if p.get("name") != name: + continue + val = p.get("value") + if isinstance(val, str) and val.strip(): + return val.strip() + return default + + def bvt_artifacts_browser_url(self, prj: dict[str, Any] | None = None) -> str: + """Per-run BVT folder in the OCI artifact browser (URL pattern from pipeline params / defaults).""" + from .pipelinerun_summary import predicted_artifacts_browser_url + + data = prj if prj is not None else self.get_pipelinerun_json_for_display() + return predicted_artifacts_browser_url(data, (self.pr or "").strip()) + + def read_pipeline_install_results(self, prj: dict[str, Any]) -> list[tuple[str, str]]: + """Tekton ``status.pipelineResults`` (install/catalog summary), when the API exposes them.""" + status = prj.get("status") or {} + raw = status.get("pipelineResults") + if not isinstance(raw, list): + return [] + out: list[tuple[str, str]] = [] + for item in raw: + if not isinstance(item, dict): + continue + name = item.get("name") + val = item.get("value") + if name is None: + continue + s = (val if isinstance(val, str) else str(val)).strip() + if s: + out.append((str(name), s)) + return out + + def read_taskruns_for_pr(self) -> list[dict[str, Any]]: + """TaskRun objects for the current PipelineRun (live cluster); empty if unavailable.""" + if self.watch_from_archive or not (self.pr or "").strip(): + return [] + proc = run_cmd( + [ + "oc", + "get", + "taskrun", + "-n", + self.args.namespace, + "-l", + f"tekton.dev/pipelineRun={self.pr}", + "-o", + "json", + ], + capture=True, + check=False, + timeout=90, + ) + if proc.returncode != 0 or not (proc.stdout or "").strip(): + return [] + try: + data = json.loads(proc.stdout) + except json.JSONDecodeError: + return [] + items = data.get("items") + if not isinstance(items, list): + return [] + return [x for x in items if isinstance(x, dict)] + + def read_bvt_artifacts_url_from_taskruns(self) -> str: + """Published ``ARTIFACTS_URL`` from a BVT TaskRun; empty if not uploaded.""" + from .bvt_artifacts import published_artifacts_url_from_taskruns + + return published_artifacts_url_from_taskruns(self.read_taskruns_for_pr()) + + def test_results_url(self, prj: dict[str, Any] | None = None) -> str: + """BVT artifact browser URL when published; otherwise a short reason or omission hint.""" + from .bvt_artifacts import resolve_artifacts_notification_line + + data = prj if prj is not None else self.get_pipelinerun_json_for_display() + ann = (data.get("metadata") or {}).get("annotations") or {} + url_ann = (ann.get("olminstall.test-results-url") or "").strip() + if url_ann: + return url_ann + tests = self._pipelinerun_param_value(data, "TESTS", "") + line = resolve_artifacts_notification_line( + tests_csv=tests, + pipeline_run=(self.pr or "").strip(), + taskruns=self.read_taskruns_for_pr(), + ) + if line is None: + return "(BVT not in TESTS)" + if line.startswith("Artifacts: "): + return line[len("Artifacts: ") :] + return line + + def read_provision_cluster_cti_name(self) -> str: + """Best-effort CTI / HyperShift object name from the provision-cluster TaskRun (live cluster only).""" + prj = self.get_pipelinerun_json_for_display() + ann = (prj.get("metadata") or {}).get("annotations") or {} + cti_ann = (ann.get("olminstall.ephemeral-cluster") or "").strip() + if cti_ann: + return cti_ann + if self.watch_from_archive or not (self.pr or "").strip(): + return "" + proc = run_cmd( + [ + "oc", + "get", + "taskrun", + "-n", + self.args.namespace, + "-l", + f"tekton.dev/pipelineRun={self.pr}", + "-o", + "json", + ], + capture=True, + check=False, + timeout=90, + ) + if proc.returncode != 0 or not (proc.stdout or "").strip(): + return "" + try: + data = json.loads(proc.stdout) + except json.JSONDecodeError: + return "" + for item in data.get("items", []): + labels = (item.get("metadata") or {}).get("labels") or {} + if labels.get("tekton.dev/pipelineTask") != "provision-cluster": + continue + for r in (item.get("status") or {}).get("results", []) or []: + if r.get("name") == "clusterName" and isinstance(r.get("value"), str): + return r["value"].strip() + return "" + + @staticmethod + def _status_label_from_succeeded_condition(cstat: str, reason: str) -> str: + if cstat == "True": + return "Succeeded" + if cstat == "False": + return reason or "Failed" + return reason or "Unknown" + + def _status_label_for_summary_preview(self) -> str: + if self.watch_from_archive: + return self.ka_succeeded or "Unknown" + cstat, reason, _ = self.succeeded_condition_detail(self.pr) + if cstat in ("True", "False"): + return self._status_label_from_succeeded_condition(cstat, reason) + if self.watch_completed: + return reason or "Completed" + return reason or "Running" + + def _terminal_status_label(self) -> str: + if self.watch_from_archive: + return self.ka_succeeded or "Unknown" + cstat, reason, _ = self.succeeded_condition_detail(self.pr) + return self._status_label_from_succeeded_condition(cstat, reason) + + @staticmethod + def _ka_succeeded_from_prj(prj: dict[str, Any]) -> str: + cond = next((c for c in prj.get("status", {}).get("conditions", []) if c.get("type") == "Succeeded"), {}) + if cond.get("status") == "True": + return "Succeeded" + if cond.get("status") == "False": + return (cond.get("reason") or "").strip() or "Failed" + return "Unknown" + + def print_run_summary(self, final_status: str, *, phase: str = "final") -> None: + """Run identity, reattach command, links, trigger context, install results. + + ``phase='preview'`` is printed before log replay/stream; ``phase='final'`` after. + """ + prj = self.get_pipelinerun_json_for_display() + ann = (prj.get("metadata") or {}).get("annotations") or {} + op_ver = (ann.get("olminstall.operator-version") or "").strip() + if phase == "final" and not op_ver and self.log_file and Path(self.log_file).exists(): + txt = Path(self.log_file).read_text(encoding="utf-8", errors="ignore") + m = re.findall(r"Operator version\s*:\s*([^\s]+)", txt) + op_ver = m[-1] if m else "" + watch_cmd = format_olm_pipeline_watch_cli( + olminstall_dir=self.script_dir, + namespace=self.args.namespace, + app=self.args.app, + pipelinerun=self.pr, + ) + title = " Olminstall run summary" + if phase == "preview": + title += " (before logs)" + elif phase == "final": + title += " (after logs)" + print("\n===========================================================") + print(title) + print("===========================================================") + print(f" PipelineRun : {self.pr} [{final_status or 'unknown'}]") + if op_ver: + print(f" Operator : {op_ver}") + artifacts_status = (ann.get("olminstall.artifacts-status") or "").strip() + if artifacts_status: + print(f" Artifacts : {artifacts_status}") + print(f" Watch logs : {watch_cmd}") + if self.watch_from_archive: + print(" Source : KubeArchive (pruned from live cluster)") + if phase == "preview": + if self.watch_from_archive: + print(" Next : Replaying logs from KubeArchive below") + elif self.watch_completed: + print(" Next : Showing pipeline logs below (KubeArchive if live logs are pruned)") + else: + print(" Next : Streaming pipeline logs below") + print("") + print("Related links:") + ui = self._konflux_pipelinerun_url(self.args.app, self.pr) + print(f" Konflux UI : {ui or '(unknown)'}") + print(f" Test Results : {self.test_results_url(prj)}") + if (self.konflux_host_api or "").strip(): + print(f" Konflux API : {self.konflux_host_api}") + cti = self.read_provision_cluster_cti_name() + if cti: + print(f" Ephemeral CTI: {cti}") + ctx = self._trigger_context_lines(prj) + print("") + if ctx: + print("Trigger context (PipelineRun annotations):") + for ln in ctx: + print(ln) + else: + print("Trigger context: (no olminstall.* annotations on this PipelineRun)") + pairs = self.read_pipeline_install_results(prj) + if pairs: + print("") + print("Install results:") + for k, v in pairs: + print(f" - {k}: {v}") + print("===========================================================") + + def mark_detached_from_logs(self) -> None: + """User stopped local log streaming (Ctrl-C); do not delete the trigger Snapshot on exit.""" + self._user_detached_from_logs = True + self.cleanup_snapshot_on_exit = False + + def _print_log_stream_detach_hint(self, watch_hint: str) -> None: + print("\nDetached from logs — PipelineRun still running on the cluster.") + print(f" Reattach with:\n {watch_hint}\n") def cleanup(self) -> None: + if self.its_apply_tmp and Path(self.its_apply_tmp).exists(): + Path(self.its_apply_tmp).unlink(missing_ok=True) + if self.log_file and Path(self.log_file).exists(): + Path(self.log_file).unlink(missing_ok=True) + if self._user_detached_from_logs: + return if self.cleanup_snapshot_on_exit and self.snapshot_name: print("\n-- Cleaning up --") proc = run_cmd( @@ -166,10 +612,6 @@ def cleanup(self) -> None: elif self.snapshot_name: print("\n-- Cleaning up --") print(f" Keeping Snapshot {self.snapshot_name} for delayed trigger/debug") - if self.its_apply_tmp and Path(self.its_apply_tmp).exists(): - Path(self.its_apply_tmp).unlink(missing_ok=True) - if self.log_file and Path(self.log_file).exists(): - Path(self.log_file).unlink(missing_ok=True) def check_login(self) -> None: who = run_cmd(["oc", "whoami"], capture=True, check=False) @@ -181,8 +623,9 @@ def check_login(self) -> None: f"User: {self.run_owner} Product: {self.args.product} " f"Namespace: {self.args.namespace} App: {self.args.app}" ) + self.konflux_host_api = get_jsonpath(["oc", "whoami", "--show-server"]) or "" if not self.ka_host or not self.konflux_ui: - api_server = get_jsonpath(["oc", "whoami", "--show-server"]) + api_server = self.konflux_host_api or get_jsonpath(["oc", "whoami", "--show-server"]) if not self.ka_host: inferred_ka = derive_kubearchive_host(api_server) if inferred_ka: @@ -222,12 +665,143 @@ def get_pipelineruns(self, namespace: str, selector: str | None = None) -> list[ data = parse_json_output(cmd) return data.get("items", []) if data else [] - def succeeded_condition(self, pr_name: str) -> tuple[str, str]: + def succeeded_condition_detail(self, pr_name: str) -> tuple[str, str, str]: + """``Succeeded`` condition: status, reason, message (empty strings if missing).""" data = parse_json_output(["oc", "get", "pipelinerun", pr_name, "-n", self.args.namespace, "-o", "json"]) for cond in data.get("status", {}).get("conditions", []): if cond.get("type") == "Succeeded": - return cond.get("status", "Unknown"), cond.get("reason", "") - return "Unknown", "" + return ( + cond.get("status", "Unknown"), + (cond.get("reason") or "").strip(), + (cond.get("message") or "").strip(), + ) + return "Unknown", "", "" + + def succeeded_condition(self, pr_name: str) -> tuple[str, str]: + c, r, _ = self.succeeded_condition_detail(pr_name) + return c, r + + @staticmethod + def _is_resolver_couldnt_get_pipeline(reason: str, message: str) -> bool: + r = (reason or "").strip() + m = (message or "").lower() + if r == "CouldntGetPipeline": + return True + return "couldntgetpipeline" in r.lower() or "resolver failed" in m or "file does not exist" in m + + @staticmethod + def _coerce_snapshot_payload_to_spec(obj: Any) -> dict[str, Any] | None: + """Normalize Konflux ``SNAPSHOT`` param JSON to a ``Snapshot.spec``-shaped dict.""" + if not isinstance(obj, dict): + return None + if isinstance(obj.get("application"), str) and isinstance(obj.get("components"), list): + return obj + spec = obj.get("spec") + if isinstance(spec, dict) and isinstance(spec.get("application"), str) and isinstance(spec.get("components"), list): + return spec + return None + + @classmethod + def _parse_snapshot_param_as_spec(cls, snap_value: str) -> dict[str, Any] | None: + try: + obj = json.loads(snap_value) + except json.JSONDecodeError: + return None + return cls._coerce_snapshot_payload_to_spec(obj) + + def _raise_resolver_terminal(self, pr_name: str, reason: str, message: str) -> None: + """Fail fast: Tekton never started tasks (CouldntGetPipeline / resolver).""" + self._warn_couldnt_get_pipeline_git_source() + excerpt = (message or reason or "").strip() + if len(excerpt) > 500: + excerpt = excerpt[:500] + "…" + raise AppError( + "PipelineRun failed before tasks started (pipeline definition could not be loaded). " + f"``{pr_name}``: {excerpt}\n" + f"Konflux: {self.konflux_ui}/ns/{self.args.namespace}/applications/{self.args.app}/pipelineruns/{pr_name}" + ) + + def _fail_fast_resolver_terminal(self, pr_name: str) -> None: + """Raise if the run already failed on pipeline resolution (no tasks).""" + cstat, reason, message = self.succeeded_condition_detail(pr_name) + if cstat != "False" or not self._is_resolver_couldnt_get_pipeline(reason, message): + return + self._raise_resolver_terminal(pr_name, reason, message) + + def _warn_couldnt_get_pipeline_git_source(self) -> None: + """Contextual hint after CouldntGetPipeline / missing pipeline file in Git resolver.""" + repo = (getattr(self.args, "konflux_repo", None) or "").strip() + branch = (getattr(self.args, "konflux_branch", None) or "").strip() + head = ( + "WARN Pipeline did not start: Git resolver could not load " + "``integration-tests/olminstall/olminstall-pipeline.yaml`` (CouldntGetPipeline).\n" + ) + if repo and branch: + ns = self.args.namespace + its_name = OLMINSTALL_TESTOPS_ITS_NAME + print( + f"{head}" + f" This run applied the ITS with ``--konflux-repo`` / ``--konflux-branch``: **{repo}** @ **{branch}**.\n" + " Tekton still could not open that path at that ref: confirm the branch is pushed, the file exists " + "on GitHub at that ref, and ``oc get integrationtestscenario -n " + f"{ns} {its_name} -o yaml`` shows the same ``resolverRef`` url/revision after " + "``olm_pipeline.py`` applied it.\n" + " If the branch is correct but the file is missing, add ``olminstall-pipeline.yaml`` (or fix " + "``pathInRepo`` in the ITS) on that branch.", + file=sys.stderr, + ) + return + if repo and not branch: + print( + f"{head}" + f" ``--konflux-repo`` is set ({repo}) but ``--konflux-branch`` is not; the ITS pipeline revision " + "may still be the template default (often **main**), so the resolver may not see your fork branch.\n" + " Pass ``--konflux-branch `` and trigger again so ``resolverRef`` matches the ref that contains this path.", + file=sys.stderr, + ) + return + if branch and not repo: + print( + f"{head}" + f" ``--konflux-branch`` is set ({branch!r}) but ``--konflux-repo`` is not; the ITS URL may still be the " + "template default. Pass ``--konflux-repo`` as well so ``resolverRef`` points at your fork.", + file=sys.stderr, + ) + return + print( + f"{head}" + " With no ``--konflux-repo`` / ``--konflux-branch`` on the CLI, the ITS keeps the committed default: " + "**opendatahub-io/odh-konflux-central** @ **main** (see its-olminstall-*.yaml). That ref may not have this path.\n" + " Re-apply the ITS with a fork + branch, then trigger again, e.g.\n" + " python3 olm_pipeline.py --tests bvt \\\n" + " --konflux-repo https://github.com//odh-konflux-central.git \\\n" + " --konflux-branch ", + file=sys.stderr, + ) + + def _ka_get_json_warn_empty(self, path: str, *, ctx: str) -> dict[str, Any]: + assert self.ka is not None + try: + raw = self.ka.get_json(path) + except KubeArchiveAuthError as exc: + print(f"WARN KubeArchive auth failed ({ctx}): {exc}", file=sys.stderr) + return {} + return raw if isinstance(raw, dict) else {} + + def _ka_get_text_warn_empty(self, path: str, *, ctx: str) -> str: + assert self.ka is not None + try: + return self.ka.get_text(path) + except KubeArchiveAuthError as exc: + print(f"WARN KubeArchive auth failed ({ctx}): {exc}", file=sys.stderr) + if not getattr(self, "_ka_archive_text_auth_tip_shown", False): + self._ka_archive_text_auth_tip_shown = True + print( + "TIP: Re-login (``oc login``) so the KubeArchive client gets a fresh token; " + "see README ``KA_HOST`` / ``--ka-host``.", + file=sys.stderr, + ) + return "" def _merged_pipelinerun_rows(self, limit: int, *, name_substr: str | None) -> list[PipelineRow]: rows: list[PipelineRow] = [] @@ -238,6 +812,9 @@ def _merged_pipelinerun_rows(self, limit: int, *, name_substr: str | None) -> li continue if name_substr is not None and name_substr not in name: continue + pipe = item.get("metadata", {}).get("labels", {}).get("tekton.dev/pipeline", "") + if olminstall_smoke_only_pipelinerun(name, pipe): + continue rows.append( PipelineRow( name=name, @@ -258,11 +835,14 @@ def _merged_pipelinerun_rows(self, limit: int, *, name_substr: str | None) -> li f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns" f"?labelSelector={quote(f'appstudio.openshift.io/application={self.args.app}')}&limit={ka_limit}" ) - data = self.ka.get_json(path) + data = self._ka_get_json_warn_empty(path, ctx="list archived PipelineRuns") for item in data.get("items", []): name = item.get("metadata", {}).get("name", "") if name_substr is not None and name_substr not in name: continue + pipe = item.get("metadata", {}).get("labels", {}).get("tekton.dev/pipeline", "") + if olminstall_smoke_only_pipelinerun(name, pipe): + continue cond = next((c for c in item.get("status", {}).get("conditions", []) if c.get("type") == "Succeeded"), {}) status = cond.get("status") if status == "True": @@ -357,16 +937,27 @@ def _fetch_step_log_archived(self, pr_name: str, pipeline_task: str, container: if not self.ka_available(): return "" assert self.ka is not None - prj = self.ka.get_json(f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(pr_name)}") + prj = self._ka_get_json_warn_empty( + f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(pr_name)}", + ctx=f"archived step log PipelineRun {pr_name}", + ) + if not prj.get("metadata", {}).get("name"): + return "" tr_name = "" for ref in prj.get("status", {}).get("childReferences", []): if ref.get("pipelineTaskName") == pipeline_task: tr_name = ref.get("name", "") or "" break + if not tr_name: + for ttr, task_label in self._archived_pipelinerun_task_refs(prj, pr_name): + if task_label == pipeline_task: + tr_name = ttr + break if not tr_name: return "" - pods = self.ka.get_json( - f"/api/v1/namespaces/{quote(self.args.namespace)}/pods?labelSelector={quote(f'tekton.dev/taskRun={tr_name}')}" + pods = self._ka_get_json_warn_empty( + f"/api/v1/namespaces/{quote(self.args.namespace)}/pods?labelSelector={quote(f'tekton.dev/taskRun={tr_name}')}", + ctx=f"archived step log pods {pipeline_task}", ) items = pods.get("items", []) if not items: @@ -374,7 +965,11 @@ def _fetch_step_log_archived(self, pr_name: str, pipeline_task: str, container: pod = items[0].get("metadata", {}).get("name", "") if not pod: return "" - return self.ka.get_text(f"/api/v1/namespaces/{quote(self.args.namespace)}/pods/{quote(pod)}/log?container={quote(container)}") + log_path = ( + f"/api/v1/namespaces/{quote(self.args.namespace)}/pods/{quote(pod)}" + f"/log?container={quote(container)}" + ) + return self._ka_get_text_warn_empty(log_path, ctx=f"archived step log {pipeline_task}:{container}") def _fetch_provision_cluster_supported_log(self, pr_name: str, source: str) -> str: out = "" @@ -440,6 +1035,12 @@ def list_supported_ocp(self) -> None: 1, ) + def _konflux_pipelinerun_url(self, application: str, pipelinerun_name: str) -> str: + base = (self.konflux_ui or "").rstrip("/") + if not base or not application or application == "-" or not pipelinerun_name: + return "" + return f"{base}/ns/{self.args.namespace}/applications/{application}/pipelineruns/{pipelinerun_name}" + def list_pipelines(self) -> None: merged = self._merged_pipelinerun_rows(self.args.list_pipelines, name_substr=None) @@ -462,9 +1063,10 @@ def list_pipelines(self) -> None: f"Use -n/--namespace {current_ns} or switch: oc project {self.args.namespace}" ) return - print("NAME\tAPP\tSTATE\tCREATED\tSOURCE") + print("NAME\tAPP\tSTATE\tCREATED\tSOURCE\tLINK") for r in merged: - print(f"{r.name}\t{r.app}\t{r.state}\t{r.created}\t{r.source}") + link = self._konflux_pipelinerun_url(r.app, r.name) or "-" + print(f"{r.name}\t{r.app}\t{r.state}\t{r.created}\t{r.source}\t{link}") def get_snapshot_owner(self, snap: str) -> str: if not snap: @@ -482,9 +1084,26 @@ def get_snapshot_owner(self, snap: str) -> str: ] ) + def _pick_newest_owned_pipelinerun(self, rows: list[tuple[str, str, str, str, str]]) -> str: + """Pick newest PipelineRun name from rows (creation_ts, name, snapshot, run-owner, pipeline label).""" + filtered = [r for r in rows if not olminstall_smoke_only_pipelinerun(r[1], r[4])] + owned = [ + row + for row in filtered + if row[3] == self.run_owner + or ( + _snapshot_param_is_resource_name(row[2]) + and self.get_snapshot_owner(row[2]) == self.run_owner + ) + ] + if not owned: + return "" + owned.sort(key=lambda x: x[0], reverse=True) + return owned[0][1] + def find_owned_live_watch_pr(self) -> str: items = self.get_pipelineruns(self.args.namespace) - cands = [] + cands: list[tuple[str, str, str, str, str]] = [] for item in items: name = item.get("metadata", {}).get("name", "") app = item.get("metadata", {}).get("labels", {}).get("appstudio.openshift.io/application", "") @@ -496,37 +1115,178 @@ def find_owned_live_watch_pr(self) -> str: snap = str(p.get("value", "")) break owner = item.get("metadata", {}).get("annotations", {}).get("olminstall.run-owner", "") - cands.append((item.get("metadata", {}).get("creationTimestamp", ""), name, snap, owner)) - for _, name, snap, owner in sorted(cands, reverse=True): - if owner == self.run_owner or self.get_snapshot_owner(snap) == self.run_owner: - return name - return "" + pipe = item.get("metadata", {}).get("labels", {}).get("tekton.dev/pipeline", "") + cands.append((item.get("metadata", {}).get("creationTimestamp", ""), name, snap, owner, pipe)) + return self._pick_newest_owned_pipelinerun(cands) def find_owned_archived_watch_pr(self) -> str: if not self.ka_available(): return "" assert self.ka is not None + # Match find_owned_live_watch_pr + _pick_newest_owned_pipelinerun: ownership may live only on + # the Snapshot (olminstall.run-owner), while the archived PipelineRun annotation is unset or stale. path = f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns?labelSelector={quote(f'appstudio.openshift.io/application={self.args.app}')}" - items = self.ka.get_json(path).get("items", []) - owned = [ - item - for item in items - if "olminstall" in item.get("metadata", {}).get("name", "") - and item.get("metadata", {}).get("annotations", {}).get("olminstall.run-owner", "") == self.run_owner - ] - if not owned: - return "" - owned.sort(key=lambda i: i.get("metadata", {}).get("creationTimestamp", "")) - return owned[-1].get("metadata", {}).get("name", "") + items = self._ka_get_json_warn_empty(path, ctx="archived PipelineRuns for watch").get("items", []) + rows: list[tuple[str, str, str, str, str]] = [] + for item in items: + name = item.get("metadata", {}).get("name", "") + if "olminstall" not in name: + continue + snap = "" + for p in item.get("spec", {}).get("params", []): + if p.get("name") == "SNAPSHOT": + snap = str(p.get("value", "")) + break + owner = item.get("metadata", {}).get("annotations", {}).get("olminstall.run-owner", "") + pipe = item.get("metadata", {}).get("labels", {}).get("tekton.dev/pipeline", "") + rows.append((item.get("metadata", {}).get("creationTimestamp", ""), name, snap, owner, pipe)) + return self._pick_newest_owned_pipelinerun(rows) + + def _pipelinerun_resolver_unusable_for_logs(self, name: str, source: str) -> bool: + """True when Tekton never resolved the Pipeline (e.g. CouldntGetPipeline) — nothing useful to replay.""" + prj: dict[str, Any] = {} + if source == "live": + proc = run_cmd( + ["oc", "get", "pipelinerun", name, "-n", self.args.namespace, "-o", "json"], + capture=True, + check=False, + ) + if proc.returncode != 0: + return False + try: + prj = json.loads(proc.stdout or "{}") + except json.JSONDecodeError: + return False + elif self.ka_available(): + prj = self._ka_get_json_warn_empty( + f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(name)}", + ctx="pipelinerun resolver check (archived)", + ) + if not isinstance(prj, dict) or not prj.get("metadata", {}).get("name"): + return False + for cond in prj.get("status", {}).get("conditions", []) or []: + if cond.get("type") != "Succeeded": + continue + if cond.get("status") != "False": + continue + reason = (cond.get("reason") or "").strip() + if reason == "CouldntGetPipeline": + return True + msg = (cond.get("message") or "").lower() + if "couldntgetpipeline" in msg or "resolver failed" in msg: + return True + return False + + def find_newest_olminstall_any_owner_for_watch(self) -> str: + """Newest non-smoke olminstall PipelineRun for ``--app`` (any owner), same merge order as ``--list``.""" + scan = max(DEFAULT_LIST_COUNT, LIST_SUPPORTED_OCP_MAX_PRS) + merged = self._merged_pipelinerun_rows(scan, name_substr="olminstall") + fallback = "" + for row in merged: + if "olminstall" not in row.name: + continue + if olminstall_smoke_only_pipelinerun(row.name, ""): + continue + if not fallback: + fallback = row.name + if self._pipelinerun_resolver_unusable_for_logs(row.name, row.source): + continue + return row.name + return fallback def get_applications(self, namespace: str) -> list[str]: data = parse_json_output(["oc", "get", "applications", "-n", namespace, "-o", "json"]) return [item.get("metadata", {}).get("name", "") for item in data.get("items", []) if item.get("metadata", {}).get("name")] def latest_matching_image(self, namespace: str, app_name: str, pattern: str) -> tuple[str, str]: - data = parse_json_output( - ["oc", "get", "snapshots", "-n", namespace, "-l", f"appstudio.openshift.io/application={app_name}", "-o", "json"] + """Return (creationTimestamp, containerImage) for the newest Snapshot whose components match ``pattern``. + + Avoids ``oc get … -o json`` for the full Snapshot list (can be tens of MB for busy apps), which can + make ``json.loads`` slow or brittle; use a compact sorted list then small per-Snapshot reads. + """ + proc = run_cmd( + [ + "oc", + "get", + "snapshots", + "-n", + namespace, + "-l", + f"appstudio.openshift.io/application={app_name}", + "--sort-by=.metadata.creationTimestamp", + "-o", + "custom-columns=NAME:.metadata.name,TS:.metadata.creationTimestamp", + ], + capture=True, + check=False, + timeout=120, ) + if proc.returncode != 0 or not (proc.stdout or "").strip(): + return "", "" + rows: list[tuple[str, str]] = [] + for line in (proc.stdout or "").splitlines(): + ln = line.strip() + if not ln or ln.upper().startswith("NAME "): + continue + parts = ln.split() + if len(parts) < 2: + continue + ts = parts[-1] + name = parts[0] + if name and ts and re.match(r"^\d{4}-\d{2}-\d{2}T", ts): + rows.append((ts, name)) + if (proc.stdout or "").strip() and not rows: + print( + f"WARN Snapshot list column parse skipped non-ISO timestamps for app={app_name!r}; " + "using slow path or empty match.", + file=sys.stderr, + ) + if not rows: + return "", "" + # Only walk the newest N snapshots per app; each oc get is small but hundreds add up. + max_walk = 120 + scan = rows[-max_walk:] if len(rows) > max_walk else rows + for ts, snap_name in reversed(scan): + proc2 = run_cmd( + [ + "oc", + "get", + "snapshot", + snap_name, + "-n", + namespace, + "-o", + "jsonpath={range .spec.components[*]}{.containerImage}{'\\n'}{end}", + ], + capture=True, + check=False, + timeout=60, + ) + if proc2.returncode != 0: + continue + for raw in (proc2.stdout or "").splitlines(): + img = raw.strip() + if img and re.search(pattern, img): + return ts, img + if len(rows) <= max_walk: + return "", "" + print( + f"WARN No component matched {pattern!r} in the newest {max_walk} snapshots for {app_name}; " + "falling back to full Snapshot list (may be slow).", + file=sys.stderr, + ) + proc_big = run_cmd( + ["oc", "get", "snapshots", "-n", namespace, "-l", f"appstudio.openshift.io/application={app_name}", "-o", "json"], + capture=True, + check=False, + timeout=300, + ) + data: dict[str, Any] = {} + if proc_big.returncode == 0 and (proc_big.stdout or "").strip(): + try: + data = json.loads(proc_big.stdout) + except json.JSONDecodeError: + data = {} best_ts = "" best_img = "" for item in data.get("items", []): @@ -544,32 +1304,46 @@ def resolve_image(self, odh_overrides: bool) -> None: print(f"Using provided image: {self.image}") return + if self.args.product == "none": + print( + "INFO product=none: skipping rhoai/odh FBC/catalog auto-resolution; " + "Snapshot uses containerImage from test-snapshot.yaml unless --image is set." + ) + return + if self.args.product == "rhoai" and self.args.version: prefix = f"rhoai-v{self.args.version.replace('.', '-')}" - print(f"Resolving latest FBCF image for RHOAI {self.args.version} (apps matching {prefix}*)...") - apps = [a for a in self.get_applications("rhoai-tenant") if re.match(rf"^{re.escape(prefix)}(-|$)", a)] - if not apps: - raise AppError(f"No Konflux application found matching {prefix}* in rhoai-tenant") - best_ts = "" - for app in apps: - ts, img = self.latest_matching_image("rhoai-tenant", app, r"rhoai-fbc-fragment@") - if img and ts > best_ts: - best_ts = ts - self.image = img - self.resolved_app = app - if not self.image: - raise AppError(f"No FBCF snapshot found for RHOAI {self.args.version} (searched {prefix}*)") + with spin_while( + f"Resolving latest FBCF image for RHOAI {self.args.version} (apps matching {prefix}*)" + ): + apps = [a for a in self.get_applications("rhoai-tenant") if re.match(rf"^{re.escape(prefix)}(-|$)", a)] + if not apps: + raise AppError(f"No Konflux application found matching {prefix}* in rhoai-tenant") + best_ts = "" + for app in apps: + ts, img = self.latest_matching_image("rhoai-tenant", app, RHOAI_FBCF_IMAGE_REF_PATTERN) + if img and ts > best_ts: + best_ts = ts + self.image = img + self.resolved_app = app + if not self.image: + apps_s = ", ".join(sorted(apps)) + raise AppError( + f"No FBCF snapshot found for RHOAI {self.args.version} " + f"(apps tried: {apps_s}; image must match {RHOAI_FBCF_IMAGE_REF_PATTERN!r}). " + "If Snapshots only list another image name, use --image or check Konflux application naming." + ) print(f"RHOAI {self.args.version} FBCF image: {self.image} (from {self.resolved_app})") elif self.args.product == "rhoai": - print("Fetching latest FBCF image across all RHOAI apps (highest version)...") - apps = [a for a in self.get_applications("rhoai-tenant") if a.startswith("rhoai-v")] - best_ts = "" - for app in apps: - ts, img = self.latest_matching_image("rhoai-tenant", app, r"rhoai-fbc-fragment@") - if img and ts > best_ts: - best_ts = ts - self.image = img - self.resolved_app = app + with spin_while("Fetching latest FBCF image across all RHOAI apps (highest version)"): + apps = [a for a in self.get_applications("rhoai-tenant") if a.startswith("rhoai-v")] + best_ts = "" + for app in apps: + ts, img = self.latest_matching_image("rhoai-tenant", app, RHOAI_FBCF_IMAGE_REF_PATTERN) + if img and ts > best_ts: + best_ts = ts + self.image = img + self.resolved_app = app if self.image: print(f"Latest FBCF image: {self.image} (from {self.resolved_app})") else: @@ -577,18 +1351,18 @@ def resolve_image(self, odh_overrides: bool) -> None: elif self.args.product == "odh": repo = "quay.io/opendatahub/opendatahub-operator-catalog" tag = "odh-stable" - print("Fetching latest ODH catalog snapshot from open-data-hub-tenant...") - data = parse_json_output(["oc", "get", "snapshots", "-n", "open-data-hub-tenant", "-o", "json"]) - best_ts = "" - for item in data.get("items", []): - if item.get("metadata", {}).get("labels", {}).get("appstudio.openshift.io/application") != "opendatahub-builds": - continue - ts = item.get("metadata", {}).get("creationTimestamp", "") - for comp in item.get("spec", {}).get("components", []): - img = comp.get("containerImage", "") - if re.search(r"opendatahub-operator-catalog@|odh-operator-catalog@", img) and ts > best_ts: - best_ts = ts - self.image = img + with spin_while("Fetching latest ODH catalog snapshot from open-data-hub-tenant"): + data = parse_json_output(["oc", "get", "snapshots", "-n", "open-data-hub-tenant", "-o", "json"]) + best_ts = "" + for item in data.get("items", []): + if item.get("metadata", {}).get("labels", {}).get("appstudio.openshift.io/application") != "opendatahub-builds": + continue + ts = item.get("metadata", {}).get("creationTimestamp", "") + for comp in item.get("spec", {}).get("components", []): + img = comp.get("containerImage", "") + if re.search(r"opendatahub-operator-catalog@|odh-operator-catalog@", img) and ts > best_ts: + best_ts = ts + self.image = img if not self.image: print(" No snapshots found (likely no access to open-data-hub-tenant)") print(f" Resolving from {repo}:{tag} via skopeo...") @@ -605,40 +1379,50 @@ def resolve_image(self, odh_overrides: bool) -> None: if not self.update_channel_override and self.args.product == "odh": self.update_channel_override = "odh-stable" print(f"Auto-selected channel: {self.update_channel_override} (product={self.args.product})") - elif not self.update_channel_override and self.resolved_app.startswith("rhoai-v3-"): + elif ( + not self.update_channel_override + and self.resolved_app + and self.resolved_app.startswith("rhoai-v3-") + ): self.update_channel_override = "stable-3.x" print(f"Auto-selected channel: {self.update_channel_override} (from {self.resolved_app})") - def ensure_its_applied(self, odh_overrides: bool) -> None: - need_yq = any( - [ - self.args.konflux_repo, - self.args.konflux_branch, - self.update_channel_override, - self.args.ocp_version, - odh_overrides, - ] + def prune_stale_integration_test_scenarios(self) -> None: + """Remove legacy ITS objects so one Snapshot does not fan out to smoke / rhoai-test pipelines.""" + if not getattr(self.args, "prune_stale_its", True): + return + if self.args.namespace != DEFAULT_NAMESPACE or self.args.app != DEFAULT_APP: + return + stale = sorted(STALE_TESTOPS_PLAYPEN_ITS_NAMES) + if not stale: + return + print( + "Pruning legacy IntegrationTestScenario objects so the next Snapshot only starts " + f"{OLMINSTALL_TESTOPS_ITS_NAME!r} for application {self.args.app!r}: " + + ", ".join(stale) ) - print("Ensuring IntegrationTestScenario is applied...") - if not need_yq: - proc = run_cmd(["oc", "apply", "-n", self.args.namespace, "-f", str(self.its_file)], capture=True, check=False) - filtered = filter_warning_lines(f"{proc.stdout}\n{proc.stderr}") - if filtered.strip(): - print(filtered, file=sys.stderr) + for name in stale: + proc = run_cmd( + ["oc", "delete", "integrationtestscenario", name, "-n", self.args.namespace, "--ignore-not-found"], + capture=True, + check=False, + ) if proc.returncode != 0: - raise AppError("ITS apply failed") - print("ITS ready") - return + msg = filter_warning_lines(f"{proc.stdout}\n{proc.stderr}").strip() + print(f" WARN oc delete integrationtestscenario/{name}: {msg or proc.returncode}", file=sys.stderr) + def ensure_its_applied(self, odh_overrides: bool) -> None: + print("Ensuring IntegrationTestScenario is applied...") + tests_baseline = getattr(self.args, "tests_catalog_default_csv", ITS_TESTS_PARAM_DEFAULT) if not shutil.which("yq"): raise AppError( - "yq is required for --konflux-repo / --konflux-branch / --channel / --ocp-version / --product odh." + "yq is required to patch the ITS (PRODUCT, --konflux-repo, --channel, etc.)." ) tmp = tempfile.NamedTemporaryFile(delete=False) tmp.close() self.its_apply_tmp = tmp.name - del_names: list[str] = [] + del_names: list[str] = ["PRODUCT"] if self.args.konflux_repo: del_names.append("SCRIPTS_REPO_URL") if self.args.konflux_branch: @@ -649,14 +1433,21 @@ def ensure_its_applied(self, odh_overrides: bool) -> None: del_names.append("OCP_VERSION_PREFIX") if odh_overrides: del_names.extend(["OPERATOR_NAME", "OPERATOR_NAMESPACE", "FBCF_COMPONENT_NAME"]) + if self._tests_its_override(): + del_names.append("TESTS") + if (self.args.slack_channel_id or "").strip(): + del_names.append("SLACK_CHANNEL_ID") - if del_names: - expr = " or ".join(f'.name == "{n}"' for n in del_names) - proc = run_cmd(["yq", "e", f"del(.spec.params[] | select({expr}))", str(self.its_file)], capture=True, check=True) - Path(self.its_apply_tmp).write_text(proc.stdout, encoding="utf-8") - else: - shutil.copyfile(self.its_file, self.its_apply_tmp) + expr = " or ".join(f'.name == "{n}"' for n in del_names) + proc = run_cmd(["yq", "e", f"del(.spec.params[] | select({expr}))", str(self.its_file)], capture=True, check=True) + Path(self.its_apply_tmp).write_text(proc.stdout, encoding="utf-8") + run_cmd( + ["yq", "e", '.spec.params += [{"name":"PRODUCT","value":strenv(YQ_PRODUCT)}]', "-i", self.its_apply_tmp], + capture=True, + check=True, + env={**os.environ, "YQ_PRODUCT": self.args.product}, + ) if self.args.konflux_repo: run_cmd( ["yq", "e", '.spec.params += [{"name":"SCRIPTS_REPO_URL","value":strenv(YQ_SCRIPTS_URL)}]', "-i", self.its_apply_tmp], @@ -698,9 +1489,25 @@ def ensure_its_applied(self, odh_overrides: bool) -> None: env={**os.environ, "YQ_OCP_PREFIX": self.args.ocp_version}, ) if odh_overrides: - run_cmd(["yq", "e", '.spec.params += [{"name":"OPERATOR_NAME","value":"opendatahub-operator"}]', "-i", self.its_apply_tmp], capture=True, check=True) - run_cmd(["yq", "e", '.spec.params += [{"name":"OPERATOR_NAMESPACE","value":"opendatahub-operators"}]', "-i", self.its_apply_tmp], capture=True, check=True) + # Jenkins odhTestConfigOperator: odh-stable Konflux catalog uses rhods-operator + RHOAI namespaces. + run_cmd(["yq", "e", '.spec.params += [{"name":"OPERATOR_NAME","value":"rhods-operator"}]', "-i", self.its_apply_tmp], capture=True, check=True) + run_cmd(["yq", "e", '.spec.params += [{"name":"OPERATOR_NAMESPACE","value":"redhat-ods-operator"}]', "-i", self.its_apply_tmp], capture=True, check=True) run_cmd(["yq", "e", '.spec.params += [{"name":"FBCF_COMPONENT_NAME","value":"odh-operator-catalog"}]', "-i", self.its_apply_tmp], capture=True, check=True) + if self._tests_its_override(): + run_cmd( + ["yq", "e", '.spec.params += [{"name":"TESTS","value":strenv(YQ_TESTS)}]', "-i", self.its_apply_tmp], + capture=True, + check=True, + env={**os.environ, "YQ_TESTS": self.args.tests}, + ) + slack_channel = (self.args.slack_channel_id or "").strip() + if slack_channel: + run_cmd( + ["yq", "e", '.spec.params += [{"name":"SLACK_CHANNEL_ID","value":strenv(YQ_SLACK_CHANNEL)}]', "-i", self.its_apply_tmp], + capture=True, + check=True, + env={**os.environ, "YQ_SLACK_CHANNEL": slack_channel}, + ) print( " ITS overrides:" @@ -708,6 +1515,8 @@ def ensure_its_applied(self, odh_overrides: bool) -> None: f" SCRIPTS_REPO={self.args.konflux_repo or ''}@{self.args.konflux_branch or ''}" f" UPDATE_CHANNEL={self.update_channel_override or ''}" f" OCP_VERSION_PREFIX={self.args.ocp_version or ''}" + f" TESTS={self.args.tests if self.args.tests != tests_baseline else ''}" + f" SLACK_CHANNEL_ID={slack_channel or ''}" f" PRODUCT={self.args.product}" ) proc = run_cmd(["oc", "apply", "-n", self.args.namespace, "-f", self.its_apply_tmp], capture=True, check=False) @@ -736,75 +1545,114 @@ def create_snapshot(self, odh_overrides: bool) -> None: if odh_overrides: tpl_comp = first_snapshot_component_name(snap_yaml) snap_yaml = snap_yaml.replace(f"name: {tpl_comp}", "name: odh-operator-catalog", 1) - print(f"Creating Snapshot to trigger pipeline (app: {self.args.app})...") - proc = run_cmd( - ["oc", "create", "-n", self.args.namespace, "-f", "-", "-o", "jsonpath={.metadata.name}"], - capture=True, - check=True, - input_text=snap_yaml, - ) - self.snapshot_name = proc.stdout.strip() - snap_ann = [ - "oc", - "annotate", - "snapshot", - self.snapshot_name, - "-n", - self.args.namespace, - f"olminstall.run-owner={self.run_owner}", - *self.olminstall_context_annotate_argv(), - "--overwrite", - ] - run_cmd(snap_ann, capture=True, check=False) + with spin_while(f"Creating Snapshot to trigger pipeline (app: {self.args.app})"): + proc = run_cmd( + ["oc", "create", "-n", self.args.namespace, "-f", "-", "-o", "jsonpath={.metadata.name}"], + capture=True, + check=True, + input_text=snap_yaml, + ) + self.snapshot_name = proc.stdout.strip() + snap_ann = [ + "oc", + "annotate", + "snapshot", + self.snapshot_name, + "-n", + self.args.namespace, + f"olminstall.run-owner={self.run_owner}", + *self.olminstall_context_annotate_argv(), + "--overwrite", + ] + self._oc_annotate_required(snap_ann, f"snapshot/{self.snapshot_name}") + snap_rec = parse_json_output( + ["oc", "get", "snapshot", self.snapshot_name, "-n", self.args.namespace, "-o", "json"] + ) + self._trigger_snapshot_spec = snap_rec.get("spec") if isinstance(snap_rec, dict) else None + self._trigger_snapshot_created_ts = ( + (snap_rec.get("metadata") or {}).get("creationTimestamp", "") if isinstance(snap_rec, dict) else "" + ) print(f"Snapshot: {self.snapshot_name}") print(f" Snapshot owner marker: {self.run_owner}") + if not self._trigger_snapshot_spec: + print( + "WARN Could not read Snapshot ``spec`` after create; matching PipelineRuns whose ``SNAPSHOT`` " + "param is JSON (not the snapshot name string) may not find this run until the Snapshot is visible.", + file=sys.stderr, + ) - def wait_for_pipelinerun(self) -> None: - def snapshot_param_matches(snap_value: str) -> bool: - if snap_value == self.snapshot_name: - return True - try: - snap_obj = json.loads(snap_value) - except json.JSONDecodeError: - return False - if snap_obj.get("application") != self.args.app: - return False - components = snap_obj.get("components") or [] - if self.image: - return any((c or {}).get("containerImage") == self.image for c in components) - return False + def snapshot_matches_trigger(self, snap_value: str) -> bool: + """True if this PipelineRun ``SNAPSHOT`` param is the Snapshot we created for this trigger. - attempts = max(1, (self.pr_appear_timeout + 4) // 5) - print(f"Waiting for PipelineRun to start (snapshot: {self.snapshot_name})...") - for i in range(1, attempts + 1): - items = self.get_pipelineruns(self.args.namespace) - cands = [] - for item in items: - name = item.get("metadata", {}).get("name", "") - app = item.get("metadata", {}).get("labels", {}).get("appstudio.openshift.io/application", "") - if "olminstall" not in name or app != self.args.app: - continue - snap = "" - for p in item.get("spec", {}).get("params", []): - if p.get("name") == "SNAPSHOT": - snap = str(p.get("value", "")) - break - if snapshot_param_matches(snap): - cands.append((item.get("metadata", {}).get("creationTimestamp", ""), name)) - if cands: - cands.sort() - self.pr = cands[-1][1] - break + When ``_trigger_snapshot_spec`` is set (normal after ``create_snapshot``), comparison is strict + full-spec equality. The image/substring heuristic runs only if that spec was not captured. + """ + if not self.snapshot_name: + return False + if snap_value == self.snapshot_name: + return True + par = self._parse_snapshot_param_as_spec(snap_value) + if par is None: + return False + if par.get("application") != self.args.app: + return False + ref = self._trigger_snapshot_spec + if ref: + return par.get("application") == ref.get("application") and par.get("components") == ref.get("components") + if self.image: + components = par.get("components") or [] + return any((c or {}).get("containerImage") == self.image for c in components) + return False - print(f" waiting... ({i}/{attempts})") - time.sleep(5) + def _poll_pipelinerun_for_snapshot(self, snap_created: str) -> bool: + """Return True when a matching olminstall PipelineRun name is stored in ``self.pr``.""" + items = self.get_pipelineruns(self.args.namespace) + cands: list[tuple[str, str]] = [] + for item in items: + name = item.get("metadata", {}).get("name", "") + app = item.get("metadata", {}).get("labels", {}).get("appstudio.openshift.io/application", "") + if "olminstall" not in name or app != self.args.app: + continue + snap = "" + for p in item.get("spec", {}).get("params", []): + if p.get("name") == "SNAPSHOT": + snap = str(p.get("value", "")) + break + if not self.snapshot_matches_trigger(snap): + continue + pr_created = (item.get("metadata", {}).get("creationTimestamp", "") or "").strip() + if snap_created and pr_created and pr_created < snap_created: + continue + self._fail_fast_resolver_terminal(name) + cands.append((item.get("metadata", {}).get("creationTimestamp", ""), name)) + if not cands: + return False + cands.sort() + self.pr = cands[-1][1] + return True + def wait_for_pipelinerun(self) -> None: + attempts = max(1, (self.pr_appear_timeout + 4) // 5) + msg_prefix = f"Waiting for PipelineRun to start (snapshot: {self.snapshot_name})" + snap_created = (self._trigger_snapshot_created_ts or "").strip() + with spin_while(msg_prefix): + for attempt in range(1, attempts + 1): + if self._poll_pipelinerun_for_snapshot(snap_created): + return + if attempt < attempts: + time.sleep(5) if not self.pr: self.cleanup_snapshot_on_exit = False + watch_hint = format_olm_pipeline_watch_cli( + olminstall_dir=self.script_dir, + namespace=self.args.namespace, + app=self.args.app, + pipelinerun="", + ) raise AppError( f"PipelineRun did not appear after {self.pr_appear_timeout}s. Check Konflux:\n" f"{self.konflux_ui}/ns/{self.args.namespace}/applications/{self.args.app}/activity/pipelineruns\n" - "Tip: rerun the script in a minute; it will try to reattach first." + f"When the run appears, follow logs with:\n {watch_hint}" ) pr_ann = [ @@ -816,9 +1664,20 @@ def snapshot_param_matches(snap_value: str) -> bool: self.args.namespace, f"olminstall.run-owner={self.run_owner}", *self.olminstall_context_annotate_argv(), + *self.early_summary_annotate_argv(), "--overwrite", ] - run_cmd(pr_ann, capture=True, check=False) + self._oc_annotate_required(pr_ann, f"pipelinerun/{self.pr}") + + def _oc_annotate_required(self, cmd: list[str], resource: str) -> None: + proc = run_cmd(cmd, capture=True, check=False) + if proc.returncode == 0: + return + detail = filter_warning_lines(f"{proc.stdout or ''}\n{proc.stderr or ''}").strip() + raise AppError( + f"Failed to annotate {resource} in namespace {self.args.namespace}: " + f"{detail or f'oc exited {proc.returncode}'}" + ) def ensure_konflux_cluster(self) -> None: res = run_cmd(["oc", "api-resources", "--api-group=appstudio.redhat.com"], capture=True, check=False) @@ -850,7 +1709,10 @@ def run_watch_mode(self) -> None: self.pr = self.args.watch elif self.ka_available(): assert self.ka is not None - prj = self.ka.get_json(f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.args.watch)}") + prj = self._ka_get_json_warn_empty( + f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.args.watch)}", + ctx="watch explicit PipelineRun (archived)", + ) if prj.get("metadata", {}).get("name"): self.pr = prj["metadata"]["name"] self.watch_from_archive = True @@ -861,25 +1723,54 @@ def run_watch_mode(self) -> None: else: raise AppError(f"PipelineRun not found in namespace '{self.args.namespace}': {self.args.watch}") else: - print(f"Watch mode: looking for your latest owned olminstall PipelineRun (app: {self.args.app}, owner: {self.run_owner})...") - self.pr = self.find_owned_live_watch_pr() + print(f"Watch mode: newest non-smoke olminstall PipelineRun for app '{self.args.app}' (same merge order as --list)…") + self.pr = self.find_newest_olminstall_any_owner_for_watch() + if self.pr: + print(f" Selected: {self.pr}") + if run_cmd(["oc", "get", "pipelinerun", self.pr, "-n", self.args.namespace], capture=True, check=False).returncode != 0: + if self.ka_available(): + assert self.ka is not None + prj = self._ka_get_json_warn_empty( + f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.pr)}", + ctx="watch selected PipelineRun (archived)", + ) + if prj.get("metadata", {}).get("name"): + self.watch_from_archive = True + self.watch_completed = True + print(" Live copy pruned — replaying from KubeArchive.") + else: + raise AppError( + f"PipelineRun {self.pr!r} not found on cluster or in KubeArchive " + f"(namespace {self.args.namespace!r})." + ) + else: + raise AppError( + f"PipelineRun {self.pr!r} not found in namespace {self.args.namespace!r} " + f"(KubeArchive unset or unreachable; set KA_HOST / --ka-host for archived runs)." + ) if not self.pr: - self.pr = self.find_owned_archived_watch_pr() + print(f" No olminstall run in --list window; trying run-owner / Snapshot match for {self.run_owner!r}…") + self.pr = self.find_owned_live_watch_pr() if self.pr: - self.watch_from_archive = True - self.watch_completed = True - print(f" Found archived owned PipelineRun: {self.pr}") + print(f" Found latest owned PipelineRun (live): {self.pr}") + if not self.pr: + self.pr = self.find_owned_archived_watch_pr() + if self.pr: + self.watch_from_archive = True + self.watch_completed = True + print(f" Found archived owned PipelineRun: {self.pr}") if not self.pr: raise AppError( - f"No owned olminstall PipelineRun found for app '{self.args.app}' (live or archived).\n" + f"No olminstall PipelineRun found for app '{self.args.app}' (live or archived).\n" "Use '--watch ' to target a specific run, or run with trigger flags (for example --product rhoai)." ) - if not self.watch_from_archive: - print(f"Found latest owned PipelineRun: {self.pr}") if self.watch_from_archive: assert self.ka is not None - prj = self.ka.get_json(f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.pr)}") + prj = self._ka_get_json_warn_empty( + f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.pr)}", + ctx="watch archived PipelineRun metadata", + ) cond = next((c for c in prj.get("status", {}).get("conditions", []) if c.get("type") == "Succeeded"), {}) if cond.get("status") == "True": self.ka_succeeded = "Succeeded" @@ -897,9 +1788,37 @@ def run_watch_mode(self) -> None: self.watch_completed = True print(f"PipelineRun {self.pr} is already completed (completionTime={ctime}). Showing recent logs/status.") + def _infer_konflux_git_pipeline_ref_from_env_only(self) -> tuple[str, str]: + """Optional (repo, revision) from env — both required; otherwise ITS keeps opendatahub-io @ ``main``.""" + env_url = (os.environ.get("OLMINSTALL_PIPELINE_REPO") or os.environ.get("KONFLUX_PIPELINE_REPO") or "").strip() + env_rev = (os.environ.get("OLMINSTALL_PIPELINE_REVISION") or os.environ.get("KONFLUX_PIPELINE_REVISION") or "").strip() + if env_url and env_rev: + return env_url, env_rev + return "", "" + + def _apply_konflux_git_inference_from_clone_or_env(self) -> None: + """Apply ``--konflux-repo`` / ``--konflux-branch`` only from env when both are set there. + + With no CLI flags and no env pair, the committed ITS default applies: opendatahub-io/odh-konflux-central @ main. + """ + if self.args.konflux_repo or self.args.konflux_branch: + return + before_repo, before_branch = self.args.konflux_repo, self.args.konflux_branch + url, rev = self._infer_konflux_git_pipeline_ref_from_env_only() + if url: + self.args.konflux_repo = url + if rev: + self.args.konflux_branch = rev + if self.args.konflux_repo != before_repo or self.args.konflux_branch != before_branch: + print( + "INFO Konflux pipeline Git resolver from OLMINSTALL_PIPELINE_* / KONFLUX_PIPELINE_* " + f"(override with --konflux-repo / --konflux-branch): {self.args.konflux_repo} @ {self.args.konflux_branch}", + file=sys.stderr, + ) + def run_trigger_mode(self) -> None: - print(f"Checking for running olminstall PipelineRun (app: {self.args.app}, owner: {self.run_owner})...") - active = [] + self._apply_konflux_git_inference_from_clone_or_env() + rows: list[tuple[str, str, str, str, str]] = [] for item in self.get_pipelineruns(self.args.namespace): name = item.get("metadata", {}).get("name", "") app = item.get("metadata", {}).get("labels", {}).get("appstudio.openshift.io/application", "") @@ -913,72 +1832,201 @@ def run_trigger_mode(self) -> None: snap = str(p.get("value", "")) break owner = item.get("metadata", {}).get("annotations", {}).get("olminstall.run-owner", "") - active.append((item.get("metadata", {}).get("creationTimestamp", ""), name, snap, owner)) - active.sort(reverse=True) - fallback = active[0][1] if active else "" - self.pr = "" - for _, name, snap, owner in active: - if owner == self.run_owner or self.get_snapshot_owner(snap) == self.run_owner: - self.pr = name - break - if self.pr: - print(f"Found running PipelineRun for app '{self.args.app}' owned by '{self.run_owner}': {self.pr} - attaching...") - return - if fallback: + pipe = item.get("metadata", {}).get("labels", {}).get("tekton.dev/pipeline", "") + rows.append((item.get("metadata", {}).get("creationTimestamp", ""), name, snap, owner, pipe)) + rows.sort(key=lambda x: x[0], reverse=True) + owned_running = self._pick_newest_owned_pipelinerun(rows) + if owned_running: + watch_owned = format_olm_pipeline_watch_cli( + olminstall_dir=self.script_dir, + namespace=self.args.namespace, + app=self.args.app, + pipelinerun=owned_running, + ) + print( + f"INFO Owned olminstall PipelineRun still running: {owned_running}. " + "Trigger mode starts a new run (your flags apply to the new run only). " + f"To stream the existing run instead:\n {watch_owned}" + ) + elif rows: print( f"WARN Found active PipelineRun(s) for app '{self.args.app}' without a matching owner marker; " - "not attaching to an unowned run — triggering a new run." + "triggering a new run." + ) + if self.args.konflux_repo and not self.args.konflux_branch: + print( + "WARN --konflux-repo is set without --konflux-branch; the ITS resolver revision stays the " + "YAML default (``main``). Pass --konflux-branch to use your fork branch.", + file=sys.stderr, ) odh_overrides = self.args.product == "odh" self.resolve_image(odh_overrides) + self.prune_stale_integration_test_scenarios() self.ensure_its_applied(odh_overrides) self.create_snapshot(odh_overrides) self.wait_for_pipelinerun() + def _archived_pipelinerun_task_refs(self, prj: dict[str, Any], pr_name: str) -> list[tuple[str, str]]: + """Resolve TaskRun name + pipeline task label from archived PR JSON. + + KubeArchive sometimes stores a PipelineRun without ``status.childReferences``; + fall back to listing TaskRuns by ``tekton.dev/pipelineRun`` (same as live ``oc`` path). + """ + refs = prj.get("status", {}).get("childReferences", []) + out: list[tuple[str, str]] = [] + for ref in refs: + tr_name = ref.get("name", "") or "" + task_name = ref.get("pipelineTaskName", "") or tr_name + if tr_name: + out.append((tr_name, task_name)) + if out: + return out + sel = f"tekton.dev/pipelineRun={pr_name}" + data = self._ka_get_json_warn_empty( + f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/taskruns?labelSelector={quote(sel)}", + ctx="list archived TaskRuns for PipelineRun", + ) + for item in data.get("items", []): + md = item.get("metadata", {}) or {} + tr_name = md.get("name", "") or "" + labels = md.get("labels", {}) or {} + task_name = labels.get("tekton.dev/pipelineTask", "") or tr_name + if tr_name: + out.append((tr_name, task_name)) + return out + def replay_archived_logs(self) -> None: assert self.ka is not None - prj = self.ka.get_json(f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.pr)}") - refs = prj.get("status", {}).get("childReferences", []) - if not refs: - print("(no child TaskRuns found in archived PipelineRun)") + prj = self._ka_get_json_warn_empty( + f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.pr)}", + ctx="replay archived PipelineRun", + ) + if not prj.get("metadata", {}).get("name"): + print( + "WARN Could not load PipelineRun from KubeArchive (missing, auth, or empty); skipping log replay.", + file=sys.stderr, + ) return - for ref in refs: - tr_name = ref.get("name", "") - task_name = ref.get("pipelineTaskName", tr_name) - pods = self.ka.get_json( - f"/api/v1/namespaces/{quote(self.args.namespace)}/pods?labelSelector={quote(f'tekton.dev/taskRun={tr_name}')}" + task_refs = self._archived_pipelinerun_task_refs(prj, self.pr) + if not task_refs: + cond = next((c for c in prj.get("status", {}).get("conditions", []) if c.get("type") == "Succeeded"), {}) + reason = (cond.get("reason") or "").strip() + message = (cond.get("message") or "").strip() + print("(no TaskRuns found in KubeArchive for this PipelineRun — childReferences missing and no labeled TaskRuns.)") + if reason or message: + print("Tekton Succeeded condition (from archived PipelineRun):") + if reason: + print(f" reason: {reason}") + if message: + for line in message.splitlines(): + print(f" message: {line}") + if self._is_resolver_couldnt_get_pipeline(reason, message): + self._warn_couldnt_get_pipeline_git_source() + else: + print("No condition message on archived object; open the Konflux UI link above for task-level logs.") + return + if not prj.get("status", {}).get("childReferences"): + print( + "INFO Archived PipelineRun has no status.childReferences; " + "replaying logs from TaskRuns listed by label tekton.dev/pipelineRun=…" + ) + for tr_name, task_name in task_refs: + pods = self._ka_get_json_warn_empty( + f"/api/v1/namespaces/{quote(self.args.namespace)}/pods?labelSelector={quote(f'tekton.dev/taskRun={tr_name}')}", + ctx=f"replay archived pods for task {task_name}", ) items = pods.get("items", []) if not items: print(f"[{task_name}] (no pod found)") continue pod = items[0].get("metadata", {}).get("name", "") - pod_obj = self.ka.get_json(f"/api/v1/namespaces/{quote(self.args.namespace)}/pods/{quote(pod)}") + pod_obj = self._ka_get_json_warn_empty( + f"/api/v1/namespaces/{quote(self.args.namespace)}/pods/{quote(pod)}", + ctx=f"replay archived pod {pod}", + ) containers = [c.get("name", "") for c in pod_obj.get("spec", {}).get("initContainers", [])] + [ c.get("name", "") for c in pod_obj.get("spec", {}).get("containers", []) ] for ctr in containers: if ctr in {"prepare", "place-scripts", "place-tools"}: continue - print(f"\n[{task_name} : {ctr}]") - print(self.ka.get_text(f"/api/v1/namespaces/{quote(self.args.namespace)}/pods/{quote(pod)}/log?container={quote(ctr)}"), end="") + print(f"\n[{ts_now()}] [{task_name}:{ctr}]") + t_log = time.monotonic() + log_path = ( + f"/api/v1/namespaces/{quote(self.args.namespace)}/pods/{quote(pod)}" + f"/log?container={quote(ctr)}" + ) + log_text = _normalize_replayed_pod_log( + self._ka_get_text_warn_empty(log_path, ctx=f"replay [{task_name}:{ctr}]") + ) + log_elapsed = time.monotonic() - t_log + print(log_text, end="") + if log_text and not log_text.endswith("\n"): + print(flush=True) + print(f"— KubeArchive: {log_elapsed:.1f}s, {len(log_text)} chars", flush=True) - def stream_live_logs(self) -> None: + def _replay_archived_logs_to_log_file(self) -> None: + tmp_log = tempfile.NamedTemporaryFile(prefix="olminstall-run.", delete=False) + self.log_file = tmp_log.name + tmp_log.close() + print(f"[{ts_now()}] Replaying archived logs from KubeArchive...") + with open(self.log_file, "w", encoding="utf-8") as fh: + old_stdout = sys.stdout + try: + sys.stdout = Tee(sys.stdout, fh) + self.replay_archived_logs() + finally: + sys.stdout = old_stdout + + def _try_kubearchive_log_replay(self) -> bool: + """When the PipelineRun exists on-cluster but tkn has no pod logs, replay from KubeArchive.""" + if not self.ka_available(): + print( + "WARN Live task logs are unavailable and KubeArchive is not configured " + "(set KA_HOST / --ka-host).", + file=sys.stderr, + ) + return False + assert self.ka is not None + prj = self._ka_get_json_warn_empty( + f"/apis/tekton.dev/v1/namespaces/{quote(self.args.namespace)}/pipelineruns/{quote(self.pr)}", + ctx="replay PipelineRun logs from KubeArchive", + ) + if not prj.get("metadata", {}).get("name"): + print( + "WARN PipelineRun not found in KubeArchive; logs unavailable.", + file=sys.stderr, + ) + return False + self._kubearchive_log_replay = True + self.ka_succeeded = self._ka_succeeded_from_prj(prj) + print("Live task logs unavailable — replaying from KubeArchive.") + self._replay_archived_logs_to_log_file() + return True + + def stream_live_logs(self) -> bool: tmp_log = tempfile.NamedTemporaryFile(prefix="olminstall-run.", delete=False) self.log_file = tmp_log.name tmp_log.close() tkn_bin = shutil.which("tkn") pr_name = (self.pr or "").strip() + if not pr_name: + print("WARN No PipelineRun name set; skipping tkn log stream.", file=sys.stderr) + return False + watch_hint = format_olm_pipeline_watch_cli( + olminstall_dir=self.script_dir, + namespace=self.args.namespace, + app=self.args.app, + pipelinerun=pr_name, + ) if tkn_bin: if self.watch_completed: print("Pipeline is already finished - showing last 200 log lines via tkn...") - # tkn often writes log output to stderr; merge into stdout for parsing. - # Put -n before the subcommand (matches tkn global flags); only accept output - # on exit 0 — error text must not satisfy "if lines" or we skip the -a retry. + # tkn expects: pipelinerun logs -n (name after logs, not before -n). lines: list[str] = [] last_rc = 0 for extra in ([], ["-a"]): - cmd = [tkn_bin, "-n", self.args.namespace, "pipelinerun", "logs", pr_name, *extra] + cmd = [tkn_bin, "pipelinerun", "logs", pr_name, "-n", self.args.namespace, *extra] proc = subprocess.run( cmd, stdout=subprocess.PIPE, @@ -996,70 +2044,83 @@ def stream_live_logs(self) -> None: lines = lines[-200:] if not lines or last_rc != 0: print( - f"WARN No log lines from tkn (exit {last_rc}). Try manually:\n" - f" {tkn_bin} -n {self.args.namespace} pipelinerun logs {pr_name}" + f"WARN No log lines from tkn (exit {last_rc}); " + "will try KubeArchive if available.", + file=sys.stderr, ) - else: - with open(self.log_file, "w", encoding="utf-8") as fh: - for line in lines: - out = f"[{ts_now()}] {line}" - print(out) - fh.write(out + "\n") - return + return False + with open(self.log_file, "w", encoding="utf-8") as fh: + for line in lines: + out = _format_live_tkn_log_line(line) + if out is None: + continue + print(out) + fh.write(out + "\n") + return True print("Streaming logs via tkn (Ctrl-C to detach, pipeline keeps running)...") with open(self.log_file, "w", encoding="utf-8") as fh: p = subprocess.Popen( - [tkn_bin, "-n", self.args.namespace, "pipelinerun", "logs", pr_name, "-f"], + [tkn_bin, "pipelinerun", "logs", pr_name, "-n", self.args.namespace, "-f"], text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, ) assert p.stdout is not None + detached = False try: for line in p.stdout: - out = f"[{ts_now()}] {line.rstrip()}" + out = _format_live_tkn_log_line(line) + if out is None: + continue print(out) fh.write(out + "\n") + except KeyboardInterrupt: + self.mark_detached_from_logs() + self._print_log_stream_detach_hint(watch_hint) + raise finally: try: p.stdout.close() except OSError: pass - rc = p.wait(timeout=120) - if rc != 0: + try: + rc = p.wait(timeout=120) + except KeyboardInterrupt: + self.mark_detached_from_logs() + self._print_log_stream_detach_hint(watch_hint) + raise + if rc in _TKN_LOG_STREAM_DETACH_RC: + self.mark_detached_from_logs() + self._print_log_stream_detach_hint(watch_hint) + detached = True + elif rc != 0: print(f"WARN tkn exited with code {rc}", file=sys.stderr) - return - print("tkn not found - polling status (install tkn for live logs)") + print(f" Reattach or replay logs with:\n {watch_hint}", file=sys.stderr) + if detached: + return True + return True + print( + "tkn not found — polling status with oc (install tkn for live streaming in trigger mode). " + f"Or run:\n {watch_hint}" + ) deadline = time.time() + 5400 while time.time() < deadline: cstat, reason = self.succeeded_condition(self.pr) print(f" {ts_now()} succeeded-condition: {cstat} reason: {reason or '?'}") if cstat == "True": print("Pipeline succeeded") - return + return True if cstat == "False": self.pipeline_exit = 1 - print(f"Pipeline failed ({reason or 'Failed'})") - return + _, r, m = self.succeeded_condition_detail(self.pr) + print(f"Pipeline failed ({r or 'Failed'})") + if self._is_resolver_couldnt_get_pipeline(r, m): + self._warn_couldnt_get_pipeline_git_source() + return True time.sleep(15) self.pipeline_exit = 1 raise AppError("Polling timed out before pipeline reached a terminal state") - def print_summary(self, final_status: str) -> None: - op_ver = "" - if self.log_file and Path(self.log_file).exists(): - txt = Path(self.log_file).read_text(encoding="utf-8", errors="ignore") - m = re.findall(r"Operator version\s*:\s*([^\s]+)", txt) - op_ver = m[-1] if m else "" - print("\n===========================================================") - print(" Summary") - print("===========================================================") - print(f" Pipeline : {self.pr} [{final_status or 'unknown'}]") - if op_ver: - print(f" Operator : {op_ver}") - print(f" Konflux UI: {self.konflux_ui}/ns/{self.args.namespace}/applications/{self.args.app}/pipelineruns/{self.pr}") - print("===========================================================") - def run(self) -> int: self.check_login() @@ -1078,40 +2139,26 @@ def run(self) -> int: else: self.run_trigger_mode() - print("") - print(f"PipelineRun : {self.pr}") - if self.watch_from_archive: - print("Source : KubeArchive (pruned from live cluster)") - else: - print(f"Logs : tkn -n {self.args.namespace} pipelinerun logs {self.pr} -f") - print(f"Konflux UI : {self.konflux_ui}/ns/{self.args.namespace}/applications/{self.args.app}/pipelineruns/{self.pr}") - print("") - self.print_pipelinerun_context_annotations() - print("") + self.print_run_summary(self._status_label_for_summary_preview(), phase="preview") if self.watch_from_archive: - tmp_log = tempfile.NamedTemporaryFile(prefix="olminstall-run.", delete=False) - self.log_file = tmp_log.name - tmp_log.close() - print("Replaying archived logs from KubeArchive...") - with open(self.log_file, "w", encoding="utf-8") as fh: - old_stdout = sys.stdout - try: - sys.stdout = Tee(sys.stdout, fh) - self.replay_archived_logs() - finally: - sys.stdout = old_stdout - self.print_summary(self.ka_succeeded) + self._replay_archived_logs_to_log_file() + self.print_run_summary(self.ka_succeeded, phase="final") if self.ka_succeeded == "Failed": self.pipeline_exit = 1 return self.pipeline_exit if not self.watch_completed: - wait_deadline = time.time() + 300 + wait_deadline = time.time() + self.pipeline_start_wait_seconds wait_start = time.time() - print("Waiting for pipeline to start running...") + print( + f"Waiting for pipeline to start running (up to {self.pipeline_start_wait_seconds}s, " + "override with OLMINSTALL_PIPELINE_START_WAIT_SECONDS)..." + ) while time.time() < wait_deadline: - _, reason = self.succeeded_condition(self.pr) + cstat, reason, message = self.succeeded_condition_detail(self.pr) + if cstat == "False" and self._is_resolver_couldnt_get_pipeline(reason, message): + self._raise_resolver_terminal(self.pr, reason, message) if reason in PENDING_REASONS: elapsed = int(time.time() - wait_start) print(f" {ts_now()} {reason or 'pending'} ({elapsed}s)") @@ -1121,11 +2168,27 @@ def run(self) -> int: break else: self.pipeline_exit = 1 - raise AppError(f"Pipeline still pending after 5m. Check Konflux:\n{self.konflux_ui}/ns/{self.args.namespace}/applications/{self.args.app}/pipelineruns/{self.pr}") + wmin = max(1, self.pipeline_start_wait_seconds // 60) + raise AppError( + f"Pipeline still pending after {wmin}m ({self.pipeline_start_wait_seconds}s). Check Konflux:\n" + f"{self.konflux_ui}/ns/{self.args.namespace}/applications/{self.args.app}/pipelineruns/{self.pr}" + ) - self.stream_live_logs() - final_cstat, final_reason = self.succeeded_condition(self.pr) - self.print_summary(final_reason) + try: + logs_shown = self.stream_live_logs() + except KeyboardInterrupt: + self.mark_detached_from_logs() + return 130 + if self._user_detached_from_logs: + return 130 + + if not logs_shown and self.watch_completed: + self._try_kubearchive_log_replay() + + final_cstat, final_reason, final_msg = self.succeeded_condition_detail(self.pr) + if final_cstat == "False" and self._is_resolver_couldnt_get_pipeline(final_reason, final_msg): + self._warn_couldnt_get_pipeline_git_source() + self.print_run_summary(self._terminal_status_label(), phase="final") if final_cstat != "True": self.pipeline_exit = 1 return self.pipeline_exit diff --git a/integration-tests/olminstall/helpers/secure_push_oci_artifacts.sh b/integration-tests/olminstall/helpers/secure_push_oci_artifacts.sh new file mode 100755 index 000000000..e1723bb47 --- /dev/null +++ b/integration-tests/olminstall/helpers/secure_push_oci_artifacts.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# Synced from red-hat-data-services/rhoai-konflux-tasks stepactions/secure-push-oci/0.1/secure-push-oci.yaml +# (StepAction script). Inlined here so bvt-health-checks does not add a second git StepAction +# resolver (Tekton prepare/place-scripts was observed stuck Init:0/2 for long periods). +set -euo pipefail + +main() { + IMAGE_REF_REGEX='^quay\.io/[a-zA-Z0-9_-]+(/[a-zA-Z0-9_-]+)+:[a-zA-Z0-9._-]+$' + + if [[ ! ${OCI_ARTIFACT_REFERENCE:-} =~ $IMAGE_REF_REGEX ]]; then + echo -e "[ERROR]: provided OCI artifact reference '${OCI_ARTIFACT_REFERENCE:-}' is not in correct format 'quay.io/org/repo:tag'" + return 1 + fi + + TAG="${OCI_ARTIFACT_REFERENCE##*:}" + OCI_TAG_EXPIRATION="${OCI_TAG_EXPIRATION:-30d}" + + TEMP_ANNOTATION_FILE="$(mktemp)" + ORAS_ERR="$(mktemp)" + trap 'rm -f "${TEMP_ANNOTATION_FILE}" "${ORAS_ERR}"' EXIT + + # Fetch the manifest annotations for the container + set +e + MANIFESTS_RAW="$(oras manifest fetch "$OCI_ARTIFACT_REFERENCE" 2>"$ORAS_ERR")" + mf_rc=$? + set -e + ORAS_ERR_TEXT="$(cat "$ORAS_ERR" 2>/dev/null || true)" + + if [[ $mf_rc -ne 0 ]]; then + if echo "$ORAS_ERR_TEXT" | grep -qiE 'not found|404|manifest unknown|NAME_UNKNOWN'; then + MANIFESTS_ANNOTATIONS="" + else + echo -e "[ERROR]: oras manifest fetch failed for '$OCI_ARTIFACT_REFERENCE': ${ORAS_ERR_TEXT:-no stderr}" >&2 + return 1 + fi + else + MANIFESTS_ANNOTATIONS="$(printf '%s' "$MANIFESTS_RAW" | jq .annotations)" + fi + + if [ "$MANIFESTS_ANNOTATIONS" == "" ] || [ "$MANIFESTS_ANNOTATIONS" == "null" ]; then + # Create the annotations file, because it does not exist in the OCI artifact + echo -e "[INFO]: provided OCI artifact tag '$OCI_ARTIFACT_REFERENCE' does not exist - will create it" + jq -n --arg exp "$OCI_TAG_EXPIRATION" --arg title "Artifact storage for pipelinerun: $TAG" \ + '{"$manifest": {"quay.expires-after": $exp, "org.opencontainers.image.title": $title}}' > "${TEMP_ANNOTATION_FILE}" + else + echo -e "[INFO]: going to update existing content in '$OCI_ARTIFACT_REFERENCE'" + # Keep the existing annotations file for further use + jq -n --argjson manifest "$MANIFESTS_ANNOTATIONS" '{ "$manifest": $manifest }' > "${TEMP_ANNOTATION_FILE}" + if ! oras pull "$OCI_ARTIFACT_REFERENCE" 2>"$ORAS_ERR"; then + echo -e "[ERROR]: oras pull failed for '$OCI_ARTIFACT_REFERENCE': $(cat "$ORAS_ERR" 2>/dev/null || true)" >&2 + return 1 + fi + fi + + # Scan the working directory using leaktk-scanner and remove problematic files + log_filename="leaktk-scan-$(date +%s).log" + leaktk-scanner scan --kind Files --resource . 2>> "$log_filename" | leaktk-remove-files . &>> "$log_filename" + + # Push the content to remote artifact storage + attempt=1 + while ! oras push "$OCI_ARTIFACT_REFERENCE" --annotation-file "${TEMP_ANNOTATION_FILE}" ./:application/vnd.acme.rocket.docs.layer.v1+tar; do + if [[ $attempt -ge 5 ]]; then + echo -e "[ERROR]: oras push failed after $attempt attempts." + return 1 + fi + echo -e "[WARNING]: oras push failed (attempt $attempt). Retrying in 5 seconds..." + sleep 5 + ((attempt++)) + done +} + +if [[ -e "/workspace/status" && $(cat /workspace/status) == "SKIP" ]]; then + echo "Skipping this step as decided in previous steps.." + exit 0 +fi +if [ "${ALWAYS_PASS:-false}" == "true" ]; then + main || true +else + main +fi diff --git a/integration-tests/olminstall/helpers/send_notification.py b/integration-tests/olminstall/helpers/send_notification.py new file mode 100644 index 000000000..d330b8131 --- /dev/null +++ b/integration-tests/olminstall/helpers/send_notification.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +"""Send a Slack notification summarising the pipeline run. + +Env (required): + TEST_STATUS -- aggregate Tekton tasks status (Succeeded/Failed/Completed) + OPERATOR_NAME -- e.g. rhods-operator, opendatahub-operator + PIPELINE_RUN_NAME +Env (optional): + OPERATOR_VERSION, FBCF_IMAGE, ARTIFACTS_URL, TESTS, + SLACK_CHANNEL_ID, SLACK_WEBHOOK_URL +""" + +from __future__ import annotations + +import json +import os +import sys +from pathlib import Path +from urllib.request import Request, urlopen +from urllib.error import URLError +from urllib.parse import urlparse + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.bvt_artifacts import resolve_artifacts_notification_line, tests_include_bvt +from helpers.tekton_util import require_env + +_PRODUCT_LABELS = { + "opendatahub-operator": "ODH", + "rhods-operator": "RHOAI", +} + + +def _slack_incoming_webhook_ok(url: str) -> bool: + p = urlparse(url.strip()) + if p.scheme != "https" or not p.netloc: + return False + host = p.netloc.lower().split("@")[-1] + return host == "hooks.slack.com" and p.path.startswith("/services/") + + +def main() -> int: + test_status = require_env("TEST_STATUS") + operator_name = require_env("OPERATOR_NAME") + pipeline_run = require_env("PIPELINE_RUN_NAME") + operator_version = os.environ.get("OPERATOR_VERSION", "(unknown)").strip() or "(unknown)" + fbcf_image = os.environ.get("FBCF_IMAGE", "(unknown)").strip() or "(unknown)" + tests_csv = os.environ.get("TESTS", "").strip() + channel_id = os.environ.get("SLACK_CHANNEL_ID", "").strip() + webhook_url = os.environ.get("SLACK_WEBHOOK_URL", "").strip() + + if test_status in ("Succeeded", "Completed"): + status_text = "PASSED" + emoji = "\u2705" + else: + status_text = f"FAILED (pipeline tasks status: {test_status})" + emoji = "\u274c" + + product = _PRODUCT_LABELS.get(operator_name, "ODH/RHOAI") + lines = [ + f"{emoji} {product} olminstall integration test {status_text}", + f"Operator : {operator_version}", + f"FBCF : {fbcf_image}", + f"Run : {pipeline_run}", + ] + artifacts_line = resolve_artifacts_notification_line( + tests_csv=tests_csv, + pipeline_run=pipeline_run, + ) + if not artifacts_line and tests_include_bvt(tests_csv): + predicted = os.environ.get("ARTIFACTS_URL", "").strip() + if predicted: + artifacts_line = f"Artifacts: {predicted}" + if artifacts_line: + lines.append(artifacts_line) + msg = "\n".join(lines) + + if not channel_id: + print("SLACK_CHANNEL_ID not set -- Slack notification disabled") + print("---") + print(msg) + print("---") + return 0 + + if not webhook_url: + print("WARN: SLACK_CHANNEL_ID is set but slack-webhook secret is missing -- skipping Slack") + return 0 + + if not _slack_incoming_webhook_ok(webhook_url): + print( + "WARN: SLACK_WEBHOOK_URL must be https://hooks.slack.com/services/... — skipping Slack", + file=sys.stderr, + ) + return 0 + + payload = json.dumps({"text": msg, "channel": channel_id}).encode() + req = Request( + webhook_url, + data=payload, + headers={"Content-Type": "application/json"}, + method="POST", + ) + try: + with urlopen(req, timeout=30) as resp: + final_url = resp.geturl() + if not _slack_incoming_webhook_ok(final_url): + raise URLError( + f"Slack webhook redirect left allowlisted host (final URL: {final_url})" + ) + resp.read() + print(f"Slack notification sent to channel {channel_id}") + except URLError as exc: + print(f"WARN: Slack POST failed: {exc}", file=sys.stderr) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/helpers/tekton_incluster.py b/integration-tests/olminstall/helpers/tekton_incluster.py new file mode 100644 index 000000000..0dd59fb9c --- /dev/null +++ b/integration-tests/olminstall/helpers/tekton_incluster.py @@ -0,0 +1,153 @@ +"""Shared in-cluster Tekton/Kubernetes API helpers for olminstall pipeline steps.""" + +from __future__ import annotations + +import ipaddress +import json +import os +import socket +import ssl +import sys +import urllib.error +import urllib.parse +import urllib.request +from pathlib import Path +from typing import Any +from urllib.parse import urlparse + + +def pipeline_run_name_from_env(*, required: bool = False) -> str: + for key in ("PIPELINE_RUN_NAME", "PIPELINERUN", "PIPELINE_RUN"): + v = os.environ.get(key, "").strip() + if v: + return v + p = Path("/etc/tekton/pipelineRunName") + if p.is_file(): + file_value = p.read_text(encoding="utf-8").strip() + if file_value: + return file_value + if required: + raise SystemExit("PIPELINE_RUN_NAME missing (and no /etc/tekton/pipelineRunName)") + return "" + + +def namespace_from_env(*, required: bool = False) -> str: + p = Path("/var/run/secrets/kubernetes.io/serviceaccount/namespace") + if p.is_file(): + file_value = p.read_text(encoding="utf-8").strip() + if file_value: + return file_value + v = os.environ.get("NAMESPACE", "").strip() + if v: + return v + if required: + raise SystemExit("cannot determine namespace (no serviceAccount namespace file)") + return "" + + +def _is_allowed_kubernetes_api_host(host: str) -> bool: + """True when *host* looks like the in-cluster API (private IP or cluster DNS name).""" + h = host.strip().lower() + if not h: + return False + if h in ("kubernetes.default.svc", "kubernetes.default.svc.cluster.local"): + return True + if h.endswith(".svc") or h.endswith(".svc.cluster.local"): + try: + ip = ipaddress.ip_address(socket.gethostbyname(host)) + except OSError: + return False + return ip.is_private or ip.is_loopback + try: + ip = ipaddress.ip_address(socket.gethostbyname(host)) + except OSError: + return False + return ip.is_private or ip.is_loopback + + +def kubernetes_api_base_url() -> str | None: + """``https://host:port`` for the in-cluster API, or ``None`` when env host is untrusted.""" + host = os.environ.get("KUBERNETES_SERVICE_HOST", "").strip() + port = os.environ.get("KUBERNETES_SERVICE_PORT", "443").strip() or "443" + if not host or not _is_allowed_kubernetes_api_host(host): + return None + return f"https://{host}:{port}" + + +def validate_kubernetes_api_url(url: str) -> None: + parsed = urlparse(url) + hostname = (parsed.hostname or "").strip() + if not hostname or not _is_allowed_kubernetes_api_host(hostname): + raise ValueError(f"refusing Kubernetes API request to untrusted host: {hostname!r}") + + +def in_cluster_get(url: str, token: str, ca_path: Path) -> dict[str, Any]: + validate_kubernetes_api_url(url) + ctx = ssl.create_default_context(cafile=str(ca_path)) + req = urllib.request.Request(url, headers={"Authorization": f"Bearer {token}"}) + with urllib.request.urlopen(req, context=ctx, timeout=60) as resp: + data = json.loads(resp.read().decode("utf-8")) + if not isinstance(data, dict): + raise ValueError("expected JSON object from API") + return data + + +def task_name(tr: dict[str, Any]) -> str: + labels = (tr.get("metadata") or {}).get("labels") or {} + if not isinstance(labels, dict): + return "" + return str(labels.get("tekton.dev/pipelineTask", "") or "") + + +def task_reason(tr: dict[str, Any]) -> str: + for cond in (tr.get("status") or {}).get("conditions") or []: + if not isinstance(cond, dict): + continue + if cond.get("type") == "Succeeded": + return str(cond.get("reason") or "").strip() + return "" + + +def result_map(tr: dict[str, Any]) -> dict[str, str]: + out: dict[str, str] = {} + for r in (tr.get("status") or {}).get("results") or []: + if not isinstance(r, dict): + continue + name, val = r.get("name"), r.get("value") + if isinstance(name, str) and isinstance(val, str): + out[name] = val + return out + + +def list_taskruns_in_cluster( + pipeline_run: str, + namespace: str, + *, + error_out: list[str] | None = None, +) -> list[dict[str, Any]]: + """List TaskRuns for a PipelineRun. On API failure, return [] and append to *error_out* when set.""" + token_path = Path("/var/run/secrets/kubernetes.io/serviceaccount/token") + ca_path = Path("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt") + base = kubernetes_api_base_url() + if not (pipeline_run and namespace and token_path.is_file() and ca_path.is_file() and base): + if error_out is not None and base is None and os.environ.get("KUBERNETES_SERVICE_HOST", "").strip(): + error_out.append("ERROR: KUBERNETES_SERVICE_HOST is missing or not an allowed in-cluster API host") + return [] + token = token_path.read_text(encoding="utf-8") + sel = urllib.parse.quote(f"tekton.dev/pipelineRun={pipeline_run}") + url = ( + f"{base}/apis/tekton.dev/v1/namespaces/{urllib.parse.quote(namespace)}" + f"/taskruns?labelSelector={sel}" + ) + try: + doc = in_cluster_get(url, token, ca_path) + except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, OSError, ValueError) as exc: + if error_out is not None: + error_out.append(f"ERROR: list TaskRuns: {exc}") + return [] + items = doc.get("items") + if not isinstance(items, list): + if error_out is not None: + error_out.append("ERROR: could not list TaskRuns for this PipelineRun") + return [] + return [x for x in items if isinstance(x, dict)] diff --git a/integration-tests/olminstall/helpers/tekton_util.py b/integration-tests/olminstall/helpers/tekton_util.py new file mode 100644 index 000000000..94ba595c2 --- /dev/null +++ b/integration-tests/olminstall/helpers/tekton_util.py @@ -0,0 +1,299 @@ +"""Shared utilities for Tekton pipeline step scripts. + +Provides reusable functions that replace per-script boilerplate: +env-var reading, Tekton result writing, git cloning (with optional +Red Hat internal TLS workaround), subprocess execution, and JUnit +XML summary parsing. +""" + +from __future__ import annotations + +import os +import re +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Any +from urllib.parse import urlparse +from xml.etree import ElementTree + +_parse_junit_xml: Any = None + + +class DefusedXmlException(Exception): # noqa: N818 — fallback when defusedxml absent + """Raised for unsafe XML; replaced when defusedxml imports successfully.""" + + +def _ensure_defusedxml() -> None: + """Load defusedxml only for JUnit parsing (other helpers import tekton_util without it).""" + global _parse_junit_xml, DefusedXmlException + if _parse_junit_xml is not None: + return + + def _import_defused() -> None: + global _parse_junit_xml, DefusedXmlException + from defusedxml.ElementTree import parse as defused_parse # type: ignore[import-not-found] + from defusedxml.common import DefusedXmlException as _DefusedXmlException # type: ignore[import-not-found] + + _parse_junit_xml = defused_parse + DefusedXmlException = _DefusedXmlException + + try: + _import_defused() + except ImportError: + print("defusedxml not found; installing for JUnit parsing...", file=sys.stderr) + pip = subprocess.run( + [sys.executable, "-m", "pip", "install", "--no-cache-dir", "-q", "defusedxml>=0.7.1"], + capture_output=True, + text=True, + check=False, + ) + if pip.returncode != 0: + tail = ((pip.stdout or "") + (pip.stderr or "")).strip()[:2000] + print( + f"defusedxml is required for JUnit parsing; pip install failed: {tail or pip.returncode}", + file=sys.stderr, + ) + sys.exit(1) + try: + _import_defused() + except ImportError as exc: + print(f"defusedxml still unavailable after pip install: {exc}", file=sys.stderr) + sys.exit(1) + + +def require_env(name: str, default: str | None = None) -> str: + """Return env var *name* (stripped). Exits non-zero when missing and no *default*.""" + v = os.environ.get(name, "").strip() + if v: + return v + if default is not None: + return default + print(f"Required environment variable is missing: {name}", file=sys.stderr) + sys.exit(1) + + +_TEKTON_RESULTS_ROOT = Path("/tekton/results") + + +def _allowed_tekton_result_roots() -> list[Path]: + roots = [_TEKTON_RESULTS_ROOT.resolve()] + extra = os.environ.get("TEKTON_RESULTS_DIR", "").strip() + if extra: + roots.append(Path(extra).resolve()) + return roots + + +def _is_allowed_tekton_result_path(target: Path) -> bool: + """True if *target* is a Tekton step result file path (not arbitrary host paths).""" + for root in _allowed_tekton_result_roots(): + try: + target.relative_to(root) + return True + except ValueError: + continue + parts = target.parts + try: + tekton_idx = parts.index("tekton") + except ValueError: + return False + rest = parts[tekton_idx + 1 :] + # Tekton v1 step results: /tekton/run//status/results/ + return len(rest) >= 4 and rest[0] == "run" and rest[2] == "status" and rest[3] == "results" + + +def write_result(path: str | Path, value: str) -> None: + """Write *value* to a Tekton result file (no trailing newline). + + Refuses paths outside Tekton result directories (``/tekton/results``, ``TEKTON_RESULTS_DIR``, + or ``/tekton/run/*/status/results/*``). + """ + target = Path(path).resolve() + if not _is_allowed_tekton_result_path(target): + print( + f"Refusing to write Tekton result outside allowed directories: {target}", + file=sys.stderr, + ) + sys.exit(1) + target.write_text(value, encoding="utf-8") + + +def run( + cmd: list[str], + *, + check: bool = True, + capture: bool = False, + input_text: str | None = None, + timeout: float | None = 300, + env: dict[str, str] | None = None, + cwd: str | Path | None = None, +) -> subprocess.CompletedProcess[str]: + """Thin wrapper around :func:`subprocess.run` with sensible defaults (always text mode).""" + kw: dict[str, Any] = { + "check": check, + "text": True, + "capture_output": capture, + "timeout": timeout, + } + if input_text is not None: + kw["input"] = input_text + if env is not None: + kw["env"] = env + if cwd is not None: + kw["cwd"] = str(cwd) + return subprocess.run(cmd, **kw) + + +_RH_INTERNAL_HOSTS_RE = re.compile(r"^(?:gitlab\.cee\.redhat\.com|git\.corp\.redhat\.com)$") + +_ALLOW_GIT_SSLVERIFY_FALSE_ENV = "OLMINSTALL_ALLOW_GIT_SSLVERIFY_FALSE" + + +def _safe_junit_int(raw: object, default: int = 0) -> int: + try: + return int(raw) if raw is not None else default + except (TypeError, ValueError): + return default + + +def _env_truthy(name: str) -> bool: + return os.environ.get(name, "").strip().lower() in ("1", "true", "yes") + + +def _git_clone_dest_allowed(dest: Path) -> bool: + """True if *dest* is a non-root path strictly under an allowed clone base (Tekton ``/workspace``, ``/tmp``, …).""" + rp = dest.resolve() + if rp == Path("/"): + return False + bases: list[Path] = [Path("/workspace").resolve(), Path("/tmp").resolve()] + extra = os.environ.get("TEST_WORKSPACE", "").strip() + if extra: + bases.append(Path(extra).resolve()) + for base in bases: + if rp == base: + return False + try: + rp.relative_to(base) + return True + except ValueError: + continue + return False + + +def git_clone( + url: str, + rev: str, + dest: str | Path, + *, + tls_workaround: bool = False, +) -> None: + """Shallow-clone *url* at *rev* into *dest*. + + *dest* must resolve to a directory strictly under ``/workspace``, ``/tmp``, + or ``TEST_WORKSPACE`` (when set); otherwise the clone is refused (whether or + not *dest* already exists) to avoid writing outside allowed bases. + + When *tls_workaround* is ``True`` the Red Hat internal CA trust bundle + is updated. Host-scoped ``sslVerify=false`` is applied only for known + internal hosts and only when ``OLMINSTALL_ALLOW_GIT_SSLVERIFY_FALSE`` is + set to a truthy value (audit log on stderr before mutating git config). + """ + dest = Path(dest).resolve() + if not _git_clone_dest_allowed(dest): + print( + f"Refusing clone destination outside allowed clone bases: {dest}", + file=sys.stderr, + ) + sys.exit(1) + if dest.exists(): + print(f"Removing existing clone directory {dest}", file=sys.stderr) + shutil.rmtree(dest) + dest.mkdir(parents=True, exist_ok=True) + + git_prefix: list[str] = [] + if tls_workaround: + git_prefix = _rh_git_invocation_prefix(url) + + def _git(args: list[str]) -> None: + run(["git", *git_prefix, *args], cwd=dest) + + print(f"Cloning {url}@{rev} -> {dest} ...") + _git(["init", "-q"]) + _git(["remote", "add", "origin", url]) + _git(["fetch", "--depth=1", "origin", rev]) + _git(["checkout", "-q", "FETCH_HEAD"]) + print(f"Cloned {url}@{rev}") + + +def _rh_git_invocation_prefix(url: str) -> list[str]: + """Return per-invocation ``git -c …`` args for RH internal TLS; never mutates global git config.""" + if Path("/etc/pki/ca-trust/source/anchors").is_dir(): + run(["update-ca-trust"], check=False, capture=True) + + probe = run( + ["git", "ls-remote", "--exit-code", url, "HEAD"], + check=False, + capture=True, + ) + if probe.returncode == 0: + return [] + + parsed = urlparse(url) + host = (parsed.hostname or "").strip() + if not host and parsed.netloc: + host = parsed.netloc.split("@")[-1].split(":")[0] + if not _RH_INTERNAL_HOSTS_RE.fullmatch(host): + print(f"TLS verification failed for {url}.") + print("Mount the internal CA bundle into the container so git ls-remote succeeds.") + sys.exit(1) + + if not _env_truthy(_ALLOW_GIT_SSLVERIFY_FALSE_ENV): + print( + f"TLS verification failed for internal host {url}; refusing host-scoped sslVerify=false. " + f"Set {_ALLOW_GIT_SSLVERIFY_FALSE_ENV}=1 after security review, or bake the RH IT CA into the image.", + file=sys.stderr, + ) + sys.exit(1) + + host_origin = f"{parsed.scheme}://{host}" + # TODO(PRIORITY): bake the RH IT root CA into quay.io/rhoai/rhoai-task-toolset:its + # or mount it as a ConfigMap to remove this exception entirely. + print( + f"AUDIT: applying git http.{host_origin}.sslVerify=false url={url!r} host={host!r}", + file=sys.stderr, + ) + print( + " Escalation: replace sslVerify=false by shipping trusted CAs in the task image (see TODO above).", + file=sys.stderr, + ) + return ["-c", f"http.{host_origin}.sslVerify=false"] + + +def parse_junit_summary(artifacts_dir: str | Path) -> dict[str, int]: + """Parse JUnit XML files and return aggregate test counts. + + Returns dict with keys: ``total``, ``passed``, ``failures``, ``errors``, ``skipped``. + """ + _ensure_defusedxml() + total = failures = errors = skipped = 0 + for xml_path in sorted(Path(artifacts_dir).glob("*.xml")): + try: + tree = _parse_junit_xml(xml_path) + except (ElementTree.ParseError, DefusedXmlException): + continue + root = tree.getroot() + suites = [root] if root.tag == "testsuite" else root.findall("testsuite") + for ts in suites: + total += _safe_junit_int(ts.get("tests")) + failures += _safe_junit_int(ts.get("failures")) + errors += _safe_junit_int(ts.get("errors")) + skipped += _safe_junit_int(ts.get("skipped")) + passed = total - failures - errors - skipped + return { + "total": total, + "passed": max(passed, 0), + "failures": failures, + "errors": errors, + "skipped": skipped, + } diff --git a/integration-tests/olminstall/helpers/tests_config.py b/integration-tests/olminstall/helpers/tests_config.py new file mode 100644 index 000000000..ac55d9551 --- /dev/null +++ b/integration-tests/olminstall/helpers/tests_config.py @@ -0,0 +1,166 @@ +"""Load olminstall-tests-config.yaml (PyYAML if available, else Mike Farah ``yq -o=json``).""" + +from __future__ import annotations + +import json +import shutil +import subprocess +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +from .errors import AppError + + +@dataclass(frozen=True) +class TestsCatalog: + """Phases allowed for the pipeline TESTS param and how they map to Tekton result flags.""" + + schema_version: int + phase_ids: tuple[str, ...] + required_ids: frozenset[str] + default_csv: str + # phase_id -> result key -> bool (only True entries stored) + sets_results: dict[str, frozenset[str]] + known_result_keys: frozenset[str] + + +def _load_yaml_document(path: Path) -> dict[str, Any]: + if not path.is_file(): + raise AppError(f"Tests config not found: {path}", 2) + try: + text = path.read_text(encoding="utf-8") + except OSError as exc: + raise AppError(f"Cannot read tests config {path}: {exc}", 2) from exc + + try: + import yaml as pyyaml # type: ignore[import-untyped, import-not-found] + except ImportError: + pyyaml = None + if pyyaml is not None: + try: + loaded = pyyaml.safe_load(text) + except pyyaml.YAMLError as exc: + raise AppError(f"Invalid YAML in {path}: {exc}", 2) from exc + if isinstance(loaded, dict): + return loaded + raise AppError(f"Tests config root must be a mapping: {path}", 2) + + yq_bin = shutil.which("yq") + if yq_bin: + try: + proc = subprocess.run( + [yq_bin, "e", "-o=json", ".", str(path)], + capture_output=True, + text=True, + check=False, + timeout=60, + ) + except subprocess.TimeoutExpired as exc: + raise AppError(f"yq timed out reading {path} (>{exc.timeout}s)", 2) from exc + if proc.returncode == 0 and proc.stdout.strip(): + try: + doc = json.loads(proc.stdout) + except json.JSONDecodeError as exc: + raise AppError(f"Invalid JSON from yq for {path}: {exc}", 2) from exc + if isinstance(doc, dict): + return doc + err = (proc.stderr or proc.stdout or "").strip() + raise AppError(f"yq failed for {path} (exit {proc.returncode}): {err or 'no output'}", 2) + + raise AppError( + f"Cannot read {path}: install PyYAML (`pip install pyyaml`) or install " + "Mike Farah's yq and ensure it is on PATH (`yq e -o=json . file.yaml`).", + 2, + ) + + +def load_tests_catalog(path: Path) -> TestsCatalog: + doc = _load_yaml_document(path) + ver = doc.get("schemaVersion") + if ver != 1: + raise AppError(f"Unsupported tests config schemaVersion {ver!r} in {path} (expected 1).", 2) + raw_phases = doc.get("phases") + if not isinstance(raw_phases, list) or not raw_phases: + raise AppError(f"Tests config must define non-empty phases: {path}", 2) + + phase_ids: list[str] = [] + required: set[str] = set() + default_parts: list[str] = [] + sets_results: dict[str, frozenset[str]] = {} + known_keys: set[str] = set() + + for i, item in enumerate(raw_phases): + if not isinstance(item, dict): + raise AppError(f"phases[{i}] must be a mapping in {path}", 2) + pid = item.get("id") + if not isinstance(pid, str) or not pid.strip(): + raise AppError(f"phases[{i}].id must be a non-empty string in {path}", 2) + pid = pid.strip() + if pid in phase_ids: + raise AppError(f"Duplicate phase id {pid!r} in {path}", 2) + phase_ids.append(pid) + + if item.get("requiredInSelection") is True: + required.add(pid) + if item.get("default") is True: + default_parts.append(pid) + + spr = item.get("setsPipelineResults") + if spr is None: + continue + if not isinstance(spr, dict): + raise AppError(f"phases[{i}].setsPipelineResults must be a mapping in {path}", 2) + keys: set[str] = set() + for rk, rv in spr.items(): + if rv is True: + if not isinstance(rk, str) or not rk.strip(): + raise AppError( + f"phases[{i}].setsPipelineResults[{rk!r}] has an invalid result key; " + f"keys must be non-empty strings in {path}", + 2, + ) + k = rk.strip() + keys.add(k) + known_keys.add(k) + elif rv in (False, None): + continue + else: + raise AppError( + f"phases[{i}].setsPipelineResults[{rk!r}] must be true or false/omitted in {path}", + 2, + ) + if keys: + sets_results[pid] = frozenset(keys) + + if not known_keys: + raise AppError( + f"At least one phase must set pipelineResults (setsPipelineResults) in {path}", 2 + ) + + default_csv = _canonical_csv(default_parts, phase_ids) + return TestsCatalog( + schema_version=1, + phase_ids=tuple(phase_ids), + required_ids=frozenset(required), + default_csv=default_csv, + sets_results=sets_results, + known_result_keys=frozenset(known_keys), + ) + + +def _canonical_csv(selected: list[str], phase_order: list[str]) -> str: + """Order: same as phase_order in config; only include selected ids.""" + sel = [p for p in phase_order if p in selected] + return ",".join(sel) + + +def compute_pipeline_result_flags(selected_ids: frozenset[str], catalog: TestsCatalog) -> dict[str, bool]: + """Map selected phase ids to Tekton parse-pipeline-tests result booleans (string true/false).""" + out = {k: False for k in catalog.known_result_keys} + for pid, keys in catalog.sets_results.items(): + if pid not in selected_ids: + continue + for k in keys: + out[k] = True + return out diff --git a/integration-tests/olminstall/helpers/tests_plan.py b/integration-tests/olminstall/helpers/tests_plan.py new file mode 100644 index 000000000..f959d31b8 --- /dev/null +++ b/integration-tests/olminstall/helpers/tests_plan.py @@ -0,0 +1,54 @@ +"""Validate and normalize the olminstall pipeline ``TESTS`` comma list (CLI + Tekton).""" + +from __future__ import annotations + +from .constants import default_tests_config_path +from .errors import AppError +from .tests_config import TestsCatalog, load_tests_catalog + + +def parse_tests_selection(raw: str, catalog: TestsCatalog) -> frozenset[str]: + """Split and normalize TESTS string; validate tokens and required phases (no canonical order yet).""" + s = (raw or "").strip() + if not s: + raise AppError("TESTS selection is empty.", 2) + seen: set[str] = set() + for part in s.split(","): + tok = part.strip().lower() + if not tok: + continue + if tok not in catalog.phase_ids: + allowed = ", ".join(catalog.phase_ids) + raise AppError(f"Invalid TESTS token {tok!r}. Allowed: {allowed}.", 2) + seen.add(tok) + if not seen: + raise AppError("TESTS selection is empty or normalizes to zero phases.", 2) + missing = catalog.required_ids - seen + if missing: + need = ", ".join(sorted(missing)) + raise AppError( + f"TESTS must include required phase(s): {need}. " + f"Example default from config: {catalog.default_csv}.", + 2, + ) + return frozenset(seen) + + +def canonical_tests_csv(selected: frozenset[str], catalog: TestsCatalog) -> str: + """Stable ordering: follow phase order from olminstall-tests-config.yaml.""" + parts = [p for p in catalog.phase_ids if p in selected] + return ",".join(parts) + + +def validate_and_normalize_tests_csv(raw: str | None, catalog: TestsCatalog | None = None) -> str: + """ + Return a canonical comma-separated TESTS string for ITS / pipeline param. + + If ``raw`` is None or empty, use ``catalog.default_csv``. + """ + cat = catalog if catalog is not None else load_tests_catalog(default_tests_config_path()) + s = (raw or "").strip() + if not s: + return cat.default_csv + selected = parse_tests_selection(s, cat) + return canonical_tests_csv(selected, cat) diff --git a/integration-tests/olminstall/helpers/write_pipeline_test_flags.py b/integration-tests/olminstall/helpers/write_pipeline_test_flags.py new file mode 100644 index 000000000..7d4340914 --- /dev/null +++ b/integration-tests/olminstall/helpers/write_pipeline_test_flags.py @@ -0,0 +1,98 @@ +#!/usr/bin/env python3 +""" +Tekton step: read TESTS param + olminstall-tests-config.yaml, write RUN_SMOKE / RUN_BVT / RUN_TIER1 result files. + +Invoked from parse-pipeline-tests after SCRIPTS_REPO is cloned to REPO_ROOT (e.g. /tmp/repo). +""" + +from __future__ import annotations + +import os +import sys +from pathlib import Path + +_OLMINSTALL = Path(__file__).resolve().parent.parent +if str(_OLMINSTALL) not in sys.path: + sys.path.insert(0, str(_OLMINSTALL)) + +from helpers.errors import AppError +from helpers.tests_config import compute_pipeline_result_flags, load_tests_catalog +from helpers.tests_plan import parse_tests_selection, validate_and_normalize_tests_csv + +try: + import yaml +except ImportError: + yaml = None # type: ignore[assignment] + + +def main() -> int: + tests_raw = os.environ.get("TESTS", "").strip() + repo_root = os.environ.get("REPO_ROOT", "").strip() + if not repo_root: + print("REPO_ROOT is required (clone destination of SCRIPTS_REPO).", file=sys.stderr) + return 1 + root = Path(repo_root) + cfg = root / "integration-tests" / "olminstall" / "olminstall-tests-config.yaml" + + try: + catalog = load_tests_catalog(cfg) + csv = validate_and_normalize_tests_csv(tests_raw if tests_raw else None, catalog) + selected = parse_tests_selection(csv, catalog) + flags = compute_pipeline_result_flags(selected, catalog) + except AppError as exc: + print( + f"ERROR: tests config or selection failed (fix YAML/CSV or paths): {exc}", + file=sys.stderr, + ) + return 1 + except FileNotFoundError as exc: + print( + f"ERROR: file not found — verify REPO_ROOT={repo_root!r} and that the repo contains " + f"integration-tests/olminstall/olminstall-tests-config.yaml: {exc}", + file=sys.stderr, + ) + return 1 + except PermissionError as exc: + print( + f"ERROR: permission denied reading tests config under REPO_ROOT={repo_root!r}: {exc}", + file=sys.stderr, + ) + return 1 + except Exception as exc: + if yaml is not None and isinstance(exc, yaml.YAMLError): + print( + f"ERROR: invalid YAML in tests config ({cfg}): {exc}. Fix indentation/quoting in the file.", + file=sys.stderr, + ) + return 1 + raise + + print(f"TESTS={csv!r} selection={sorted(selected)} -> {flags}") + results_base = Path(os.environ.get("RESULTS_DIR", "/tekton/results")).resolve() + for key, val in flags.items(): + path_var = f"{key}_PATH" + p = os.environ.get(path_var, "").strip() + if not p: + print(f"Missing env {path_var} for result {key}", file=sys.stderr) + return 1 + result_path = Path(p).resolve() + if not result_path.is_relative_to(results_base): + print( + f"ERROR: {path_var}={p!r} resolves outside allowed results directory {results_base}", + file=sys.stderr, + ) + return 1 + try: + result_path.parent.mkdir(parents=True, exist_ok=True) + result_path.write_text("true" if val else "false", encoding="utf-8") + except (FileNotFoundError, PermissionError, OSError) as exc: + print( + f"ERROR: could not write result file {path_var}={p!r}: {exc}", + file=sys.stderr, + ) + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/integration-tests/olminstall/its-olminstall-open-data-hub-tenant.yaml b/integration-tests/olminstall/its-olminstall-open-data-hub-tenant.yaml index fd1d5e0e4..96469f85a 100644 --- a/integration-tests/olminstall/its-olminstall-open-data-hub-tenant.yaml +++ b/integration-tests/olminstall/its-olminstall-open-data-hub-tenant.yaml @@ -3,28 +3,32 @@ kind: IntegrationTestScenario metadata: labels: test.appstudio.openshift.io/optional: "true" - name: odh-olminstall-smoke + name: odh-olminstall namespace: open-data-hub-tenant annotations: openshift.io/description: > - FBCF olminstall smoke for opendatahub-builds. For rhoai-tenant or other apps, duplicate this ITS with + FBCF olminstall integration test for opendatahub-builds. For rhoai-tenant or other apps, duplicate this ITS with updated namespace/application/contexts after stakeholder alignment (see README). spec: application: opendatahub-builds contexts: - - description: Execute olminstall smoke test when FBCF (odh-operator-catalog) is built + - description: Execute olminstall integration test when FBCF (odh-operator-catalog) is built name: component_odh-operator-catalog params: + - name: PRODUCT + value: "odh" - name: FBCF_COMPONENT_NAME value: "odh-operator-catalog" - name: UPDATE_CHANNEL value: "odh-stable" - name: OPERATOR_NAME - value: "opendatahub-operator" + value: "rhods-operator" - name: OPERATOR_NAMESPACE - value: "opendatahub-operators" + value: "redhat-ods-operator" - name: QUAY_PULL_SECRET_NAME value: "odh-quay-secret" + - name: TESTS + value: "bvt,smoke" resolverRef: params: - name: url @@ -32,6 +36,6 @@ spec: - name: revision value: main - name: pathInRepo - value: integration-tests/olminstall/olminstall-smoke-pipeline.yaml + value: integration-tests/olminstall/olminstall-pipeline.yaml resolver: git resourceKind: pipeline diff --git a/integration-tests/olminstall/its-olminstall-rhoai-tenant.yaml b/integration-tests/olminstall/its-olminstall-rhoai-tenant.yaml index 86ebe4bc7..14f420976 100644 --- a/integration-tests/olminstall/its-olminstall-rhoai-tenant.yaml +++ b/integration-tests/olminstall/its-olminstall-rhoai-tenant.yaml @@ -13,22 +13,26 @@ kind: IntegrationTestScenario metadata: labels: test.appstudio.openshift.io/optional: "true" - name: odh-olminstall-smoke-testops + name: odh-olminstall-testops namespace: rhoai-tenant spec: application: testops-playpen contexts: - - description: Execute olminstall smoke test (sandbox / manual trigger) + - description: Execute olminstall integration test (sandbox / manual trigger) name: application # Optional SCRIPTS_REPO_URL / SCRIPTS_REPO_REVISION here to override the pipeline # default (opendatahub-io odh-konflux-central @ main). Or: python3 …/olm_pipeline.py --konflux-repo … params: + - name: PRODUCT + value: "rhoai" - name: FBCF_COMPONENT_NAME value: "rhoai-fbc-fragment-ocp-421" - name: OPERATOR_NAME value: "rhods-operator" - name: QUAY_PULL_SECRET_NAME value: "rhoai-quay-secret" + - name: TESTS + value: "bvt,smoke" resolverRef: params: - name: url @@ -36,6 +40,6 @@ spec: - name: revision value: main - name: pathInRepo - value: integration-tests/olminstall/olminstall-smoke-pipeline.yaml + value: integration-tests/olminstall/olminstall-pipeline.yaml resolver: git resourceKind: pipeline diff --git a/integration-tests/olminstall/olm_pipeline.py b/integration-tests/olminstall/olm_pipeline.py index 0eb536d62..1d6aa2096 100644 --- a/integration-tests/olminstall/olm_pipeline.py +++ b/integration-tests/olminstall/olm_pipeline.py @@ -13,6 +13,7 @@ from helpers.cli import emit_click_style_error, make_parser, parse_cli_args from helpers.errors import AppError +from helpers.kubearchive import KubeArchiveAuthError from helpers.runner import OLMInstallRunner _HELP_DESCRIPTION = ( @@ -21,18 +22,25 @@ ) _HELP_EPILOG = """\ -Tools: oc (required); tkn (live logs); yq (repo/branch/channel/odh); skopeo (odh, optional). -Env: KONFLUX_UI, KA_HOST, KONFLUX_SERVER, PR_APPEAR_TIMEOUT_SECONDS — details in README / contributing doc. +Tools: oc (required); tkn (optional, live logs during trigger mode); yq (repo/branch/channel/odh); skopeo (odh, optional). +Env: KONFLUX_UI, KA_HOST, KONFLUX_SERVER, PR_APPEAR_TIMEOUT_SECONDS — README / contributing doc. -One mode: default (trigger) OR --watch OR --list-pipelines OR --list-supported-ocp. +Modes: default (trigger) OR --watch OR --list-pipelines OR --list-supported-ocp. +Trigger always creates a new PipelineRun; use --watch to stream an existing run. Do not mix trigger flags (--image, --version, …) with --watch/--list* (except --ocp-version with --list-supported-ocp). Examples: - %(prog)s --watch + %(prog)s --watch # newest olminstall for --app (same merge order as --list) + %(prog)s --watch odh-olminstall-testops-xyz %(prog)s --list %(prog)s --list-supported-ocp --ocp-version 4.19 + %(prog)s --tests bvt + %(prog)s --tests smoke %(prog)s --product rhoai --version 3.5 - %(prog)s --konflux-repo https://github.com/you/fork.git --konflux-branch my-feature + %(prog)s --tests bvt,smoke,tier1 + %(prog)s --tests bvt --product rhoai --version 3.5 + %(prog)s --tests bvt --slack-channel-id C01234ABCDE + %(prog)s --konflux-repo https://github.com/you/fork.git --konflux-branch your-branch Exit codes: 0 ok, 1 error, 2 bad args, 130 interrupt.""" @@ -49,10 +57,20 @@ def main(argv: list[str] | None = None) -> int: atexit.register(runner.cleanup) return runner.run() except KeyboardInterrupt: + if "runner" in locals(): + runner.mark_detached_from_logs() return 130 except AppError as exc: emit_click_style_error(parser, str(exc), usage=(exc.code == 2)) return exc.code + except KubeArchiveAuthError as exc: + msg = ( + f"{exc}\n" + "Re-authenticate against the Konflux cluster with the same kubeconfig you use for `oc`, " + "then retry (for example: `KUBECONFIG=… oc login --server= --web`)." + ) + emit_click_style_error(parser, msg, usage=False) + return 1 if __name__ == "__main__": diff --git a/integration-tests/olminstall/olminstall-pipeline.yaml b/integration-tests/olminstall/olminstall-pipeline.yaml new file mode 100644 index 000000000..38e1b1369 --- /dev/null +++ b/integration-tests/olminstall/olminstall-pipeline.yaml @@ -0,0 +1,1355 @@ +--- +apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: odh-olminstall-test + labels: + app.kubernetes.io/version: "1.0" + annotations: + tekton.dev/pipelines.minVersion: "0.12.1" + tekton.dev/tags: odh, rhoai, integration-test +spec: + description: >- + ODH/RHOAI integration test: provisions an ephemeral cluster via Konflux EaaS (HyperShift), + installs the operator with olminstall from the FBCF image in the Konflux snapshot, verifies + CSV Succeeded, then runs post-install BVT (opendatahub-tests cluster_health and operator_health). + + params: + - name: SNAPSHOT + description: | + Spec section of an ApplicationSnapshot as a JSON string. + Provided by Konflux Integration Test Scenario. + type: string + + - name: FBCF_COMPONENT_NAME + description: Name of the FBCF component in the snapshot + type: string + default: "odh-operator-catalog" + + - name: UPDATE_CHANNEL + description: OLM update channel + type: string + default: "stable" + + - name: OPERATOR_NAMESPACE + description: Namespace for operator installation + type: string + default: "redhat-ods-operator" + + - name: OPERATOR_NAME + description: OLM package / deployment name (e.g. rhods-operator for RHOAI) + type: string + default: "rhods-operator" + + - name: OLMINSTALL_CATALOG_NAME + description: CatalogSource name used by olminstall install-operator.sh + type: string + default: "rhoai-catalog-dev" + + - name: HYPERSHIFT_INSTANCE_TYPE + description: AWS instance type for ephemeral HyperShift workers + type: string + default: "m5.2xlarge" + + - name: OCP_VERSION_PREFIX + description: | + OpenShift version prefix for the ephemeral cluster (e.g. 4.20. or 4.19.). Latest patch is resolved via EaaS. + Leave empty to use the newest supported major.minor from the environment. + type: string + default: "" + + - name: SCRIPTS_REPO_URL + description: Git repo containing the pipeline step scripts + type: string + default: "https://github.com/opendatahub-io/odh-konflux-central.git" + + - name: SCRIPTS_REPO_REVISION + description: Branch/tag/SHA of the scripts repo to clone + type: string + default: "main" + + - name: OLMINSTALL_REPO_URL + description: | + olminstall repo URL — provides tested OLM + install manifests (resources/install-rhods-operator.yaml) and utility + scripts (utils/oc_wait.sh, utils/oc_approve.sh). + type: string + default: "https://gitlab.cee.redhat.com/data-hub/olminstall.git" + + - name: OLMINSTALL_REPO_REVISION + description: Branch/tag/SHA of data-hub/olminstall to clone + type: string + default: "main" + + - name: QUAY_PULL_SECRET_NAME + description: Name of the namespace secret containing quay.io/rhoai dockerconfigjson + type: string + default: "rhoai-quay-secret" + + - name: OCI_ARTIFACTS_PUSH_SECRET_NAME + description: | + Secret (type kubernetes.io/dockerconfigjson, key .dockerconfigjson) mounted for BVT + `oras push` to OCI_ARTIFACTS_REPO (e.g. quay.io/opendatahub/odh-ci-artifacts). Separate from + QUAY_PULL_SECRET_NAME which is used for rhoai catalog/image pulls in install-operator. + type: string + default: "odh-registry-secret" + + - name: OPENDATAHUB_TESTS_REPO + description: | + Container image repository (no tag) for BVT — quay.io/opendatahub/opendatahub-tests. + Tag is derived from installed CSV version with skopeo fallback to latest. + type: string + default: "quay.io/opendatahub/opendatahub-tests" + + - name: TESTS + description: | + Comma-separated phase ids declared in integration-tests/olminstall/olminstall-tests-config.yaml + inside SCRIPTS_REPO. parse-pipeline-tests clones that repo and evaluates RUN_* flags from the file. + Default must stay in sync with in-repo ITS examples (bvt,smoke). + type: string + default: "bvt,smoke" + + - name: OCI_ARTIFACTS_REPO + description: | + OCI repository for BVT test artifacts (JUnit XML + console logs). + Uses the same shared repo as other integration test pipelines in this project. + type: string + default: "quay.io/opendatahub/odh-ci-artifacts" + + - name: ARTIFACT_BROWSER_URL + description: | + Base URL of the OCI artifact browser used to view published test artifacts. + Per-run artifacts are at //-bvt/. + type: string + default: "https://app-artifact-browser.apps.rosa.konflux-qe.zmr9.p3.openshiftapps.com" + + - name: ARTIFACT_BROWSER_REPO_PATH + description: | + Path segment (repo slug) in the artifact browser for OCI_ARTIFACTS_REPO. + Default "odh-ci-artifacts" matches the shared QE browser layout for that repo name; override if your OCI target maps to a different slug. + type: string + default: "odh-ci-artifacts" + + - name: PRODUCT + description: | + Product selector: "rhoai", "odh", or "none". + When "none", cluster provisioning and operator installation are skipped + (useful for running tests against an existing cluster in the future). + type: string + default: "rhoai" + + - name: SLACK_CHANNEL_ID + description: | + Optional Slack channel ID (e.g. C01234ABCDE) to receive a run notification. + Leave empty (the default) to suppress all Slack messages. When set, the + slack-webhook secret must be present in the namespace; the channel field + in the payload overrides the webhook's default destination. + type: string + default: "" + + results: + - name: TEST_OUTPUT + description: Overall test result (BVT TEST_OUTPUT or install-operator INSTALL_STATUS, aggregated in finally) + value: $(finally.propagate-pipeline-test-output.results.PROPAGATED_TEST_OUTPUT) + + - name: FBCF_IMAGE + description: FBCF image that was tested + value: $(tasks.extract-fbcf-image.results.FBCF_IMAGE) + + - name: ARTIFACTS_URL + description: BVT JUnit and logs in the OCI artifact browser (Konflux task Results panel) + value: $(finally.post-results.results.ARTIFACTS_URL) + + - name: OLMINSTALL_SUMMARY_TEST_OUTPUT + description: Aggregated TEST_OUTPUT / install status (same as pipeline TEST_OUTPUT) + value: $(finally.post-results.results.TEST_OUTPUT) + + - name: OLMINSTALL_OPERATOR_VERSION + description: Installed operator CSV version from install-operator + value: $(finally.post-results.results.OPERATOR_VERSION) + + - name: EPHEMERAL_CLUSTER + description: EaaS ClusterTemplateInstance name from provision-cluster + value: $(finally.post-results.results.EPHEMERAL_CLUSTER) + + tasks: + - name: parse-pipeline-tests + params: + - name: TESTS + value: $(params.TESTS) + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + taskSpec: + params: + - name: TESTS + type: string + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + results: + - name: RUN_SMOKE + description: '"true" or "false"' + - name: RUN_BVT + description: '"true" or "false"' + - name: RUN_TIER1 + description: '"true" or "false"' + steps: + - name: clone-scripts-repo + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/repo + script: | + #!/bin/bash + set -euo pipefail + echo "Cloning ${REPO_URL}@${REPO_REV} for olminstall-tests-config.yaml..." + rm -rf "${DEST_DIR}" + mkdir -p "${DEST_DIR}" + cd "${DEST_DIR}" + git init -q + git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" + git checkout -q FETCH_HEAD + echo "✓ Cloned scripts repo" + + - name: eval-tests-from-config + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: TESTS + value: $(params.TESTS) + - name: REPO_ROOT + value: /workspace/repo + - name: RUN_SMOKE_PATH + value: $(results.RUN_SMOKE.path) + - name: RUN_BVT_PATH + value: $(results.RUN_BVT.path) + - name: RUN_TIER1_PATH + value: $(results.RUN_TIER1.path) + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/write_pipeline_test_flags.py"]) + + - name: smoke-placeholder + runAfter: + - parse-pipeline-tests + when: + - input: "$(tasks.parse-pipeline-tests.results.RUN_SMOKE)" + operator: in + values: ["true"] + taskSpec: + steps: + - name: noop + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + script: | + #!/usr/bin/env python3 + print("smoke: placeholder — no smoke pytest suite is wired in this pipeline yet.") + + - name: extract-fbcf-image + taskSpec: + params: + - name: SNAPSHOT + type: string + - name: COMPONENT_NAME + type: string + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + results: + - name: FBCF_IMAGE + description: Extracted FBCF image URL + steps: + - name: clone-scripts-repo + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + + - name: parse-snapshot + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: SNAPSHOT + value: $(params.SNAPSHOT) + - name: COMPONENT_NAME + value: $(params.COMPONENT_NAME) + - name: RESULT_PATH + value: $(results.FBCF_IMAGE.path) + script: | + #!/usr/bin/env python3 + import os, sys + + p = "/workspace/repo/integration-tests/olminstall/helpers/extract_fbcf_image.py" + os.execvp(sys.executable, [sys.executable, p]) + params: + - name: SNAPSHOT + value: $(params.SNAPSHOT) + - name: COMPONENT_NAME + value: $(params.FBCF_COMPONENT_NAME) + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + + - name: provision-eaas-space + runAfter: + - extract-fbcf-image + when: + - input: "$(params.PRODUCT)" + operator: notin + values: ["none"] + taskRef: + resolver: git + params: + - name: url + value: https://github.com/konflux-ci/build-definitions.git + - name: revision + value: main + - name: pathInRepo + value: task/eaas-provision-space/0.1/eaas-provision-space.yaml + params: + - name: ownerName + value: $(context.pipelineRun.name) + - name: ownerUid + value: $(context.pipelineRun.uid) + + - name: provision-cluster + runAfter: + - provision-eaas-space + when: + - input: "$(params.PRODUCT)" + operator: notin + values: ["none"] + params: + - name: OCP_VERSION_PREFIX + value: $(params.OCP_VERSION_PREFIX) + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + - name: HYPERSHIFT_INSTANCE_TYPE + value: $(params.HYPERSHIFT_INSTANCE_TYPE) + taskSpec: + params: + - name: OCP_VERSION_PREFIX + type: string + default: "" + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + - name: HYPERSHIFT_INSTANCE_TYPE + type: string + default: "m5.2xlarge" + results: + - name: clusterName + value: "$(steps.create-cluster.results.clusterName)" + steps: + - name: clone-scripts-repo + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + + - name: get-supported-versions + ref: + resolver: git + params: + - name: url + value: https://github.com/konflux-ci/build-definitions.git + - name: revision + value: main + - name: pathInRepo + value: stepactions/eaas-get-supported-ephemeral-cluster-versions/0.1/eaas-get-supported-ephemeral-cluster-versions.yaml + params: + - name: eaasSpaceSecretRef + value: $(tasks.provision-eaas-space.results.secretRef) + - name: resolve-ocp-prefix + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + results: + - name: prefix + type: string + env: + - name: OVERRIDE + value: $(params.OCP_VERSION_PREFIX) + - name: DEFAULT_MINOR + value: $(steps.get-supported-versions.results.versions[0]) + - name: PREFIX_RESULT_PATH + value: $(step.results.prefix.path) + script: | + #!/usr/bin/env python3 + import os, sys + + p = "/workspace/repo/integration-tests/olminstall/helpers/resolve_ocp_prefix.py" + os.execvp(sys.executable, [sys.executable, p]) + - name: pick-version + ref: + resolver: git + params: + - name: url + value: https://github.com/konflux-ci/build-definitions.git + - name: revision + value: main + - name: pathInRepo + value: stepactions/eaas-get-latest-openshift-version-by-prefix/0.1/eaas-get-latest-openshift-version-by-prefix.yaml + params: + - name: prefix + value: "$(steps.resolve-ocp-prefix.results.prefix)" + - name: create-cluster + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + results: + - name: clusterName + type: string + env: + - name: INSTANCE_TYPE + value: $(params.HYPERSHIFT_INSTANCE_TYPE) + - name: VERSION + value: "$(steps.pick-version.results.version)" + - name: KUBECONFIG_VALUE + valueFrom: + secretKeyRef: + name: $(tasks.provision-eaas-space.results.secretRef) + key: kubeconfig + - name: TENANT + value: "$(context.taskRun.namespace)" + - name: TIMEOUT + value: "30m" + - name: ICS_VALUE + value: | + - mirrors: + - quay.io/rhoai + source: registry.redhat.io/rhoai + - name: CLUSTER_NAME_RESULT_PATH + value: "$(step.results.clusterName.path)" + script: | + #!/usr/bin/env python3 + import os, sys + + p = "/workspace/repo/integration-tests/olminstall/helpers/create_eaas_cluster.py" + os.execvp(sys.executable, [sys.executable, p]) + + - name: install-operator + runAfter: + - provision-cluster + - parse-pipeline-tests + when: + - input: "$(params.PRODUCT)" + operator: notin + values: ["none"] + params: + - name: FBCF_IMAGE + value: $(tasks.extract-fbcf-image.results.FBCF_IMAGE) + - name: UPDATE_CHANNEL + value: $(params.UPDATE_CHANNEL) + - name: OPERATOR_NAMESPACE + value: $(params.OPERATOR_NAMESPACE) + - name: OPERATOR_NAME + value: $(params.OPERATOR_NAME) + - name: OLMINSTALL_REPO_URL + value: $(params.OLMINSTALL_REPO_URL) + - name: OLMINSTALL_REPO_REVISION + value: $(params.OLMINSTALL_REPO_REVISION) + - name: OLMINSTALL_CATALOG_NAME + value: $(params.OLMINSTALL_CATALOG_NAME) + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + - name: QUAY_PULL_SECRET_NAME + value: $(params.QUAY_PULL_SECRET_NAME) + taskSpec: + params: + - name: FBCF_IMAGE + type: string + - name: UPDATE_CHANNEL + type: string + - name: OPERATOR_NAMESPACE + type: string + - name: OPERATOR_NAME + type: string + - name: OLMINSTALL_REPO_URL + type: string + - name: OLMINSTALL_REPO_REVISION + type: string + - name: OLMINSTALL_CATALOG_NAME + type: string + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + - name: QUAY_PULL_SECRET_NAME + type: string + results: + - name: INSTALL_STATUS + description: Installation status (SUCCESS or FAILED) + - name: OPERATOR_VERSION + description: Installed operator version + volumes: + - name: credentials + emptyDir: {} + - name: quay-pull-secret + secret: + secretName: $(params.QUAY_PULL_SECRET_NAME) + steps: + - name: get-kubeconfig + ref: + resolver: git + params: + - name: url + value: https://github.com/konflux-ci/build-definitions.git + - name: revision + value: main + - name: pathInRepo + value: stepactions/eaas-get-ephemeral-cluster-credentials/0.1/eaas-get-ephemeral-cluster-credentials.yaml + params: + - name: eaasSpaceSecretRef + value: $(tasks.provision-eaas-space.results.secretRef) + - name: clusterName + value: "$(tasks.provision-cluster.results.clusterName)" + - name: credentials + value: credentials + + - name: clone-repo + image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + + - name: clone-olminstall + image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 + env: + - name: REPO_URL + value: $(params.OLMINSTALL_REPO_URL) + - name: REPO_REV + value: $(params.OLMINSTALL_REPO_REVISION) + - name: DEST_DIR + value: /workspace/olminstall + - name: TLS_WORKAROUND + value: "true" + - name: OLMINSTALL_ALLOW_GIT_SSLVERIFY_FALSE + value: "1" + script: | + #!/usr/bin/env python3 + import os, sys + + sys.path.insert(0, "/workspace/repo/integration-tests/olminstall") + from helpers.tekton_util import git_clone + + tls = os.environ.get("TLS_WORKAROUND", "").strip().lower() in ("1", "true", "yes") + git_clone(os.environ["REPO_URL"], os.environ["REPO_REV"], os.environ["DEST_DIR"], tls_workaround=tls) + + - name: patch-cluster-pull-secret + image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 + volumeMounts: + - name: credentials + mountPath: /credentials + - name: quay-pull-secret + mountPath: /var/secret/quay + readOnly: true + + env: + - name: KUBECONFIG + value: /credentials/$(steps.get-kubeconfig.results.kubeconfig) + script: | + #!/usr/bin/env python3 + import os, sys + + p = "/workspace/repo/integration-tests/olminstall/helpers/patch_cluster_pull_secret.py" + os.execvp(sys.executable, [sys.executable, p]) + + - name: install-and-verify + image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 + volumeMounts: + - name: credentials + mountPath: /credentials + env: + - name: KUBECONFIG + value: /credentials/$(steps.get-kubeconfig.results.kubeconfig) + - name: FBCF_IMAGE + value: $(params.FBCF_IMAGE) + - name: UPDATE_CHANNEL + value: $(params.UPDATE_CHANNEL) + - name: OPERATOR_NAMESPACE + value: $(params.OPERATOR_NAMESPACE) + - name: OPERATOR_NAME + value: $(params.OPERATOR_NAME) + - name: OLMINSTALL_DIR + value: /workspace/olminstall + - name: OLMINSTALL_CATALOG_NAME + value: $(params.OLMINSTALL_CATALOG_NAME) + - name: QUAY_PULL_SECRET_NAME + value: $(params.QUAY_PULL_SECRET_NAME) + - name: INSTALL_STATUS_PATH + value: $(results.INSTALL_STATUS.path) + - name: OPERATOR_VERSION_PATH + value: $(results.OPERATOR_VERSION.path) + script: | + #!/usr/bin/env python3 + import os, sys + + p = "/workspace/repo/integration-tests/olminstall/helpers/install_and_verify.py" + os.execvp(sys.executable, [sys.executable, p]) + + # resolve-opendatahub-tests-image-after-install supplies the opendatahub-tests image when + # PRODUCT is rhoai/odh (EaaS-backed BVT). PRODUCT=none uses placeholder JUnit only — see + # bvt-health-checks-no-eaas (Konflux worker has no DataScienceCluster API). + + - name: resolve-opendatahub-tests-image-after-install + runAfter: + - install-operator + when: + - input: "$(tasks.parse-pipeline-tests.results.RUN_BVT)" + operator: in + values: ["true"] + - input: "$(params.PRODUCT)" + operator: notin + values: ["none"] + params: + - name: OPERATOR_VERSION + value: $(tasks.install-operator.results.OPERATOR_VERSION) + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + - name: OPENDATAHUB_TESTS_REPO + value: $(params.OPENDATAHUB_TESTS_REPO) + taskSpec: + params: + - name: OPERATOR_VERSION + type: string + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + - name: OPENDATAHUB_TESTS_REPO + type: string + results: + - name: OPENDATAHUB_TESTS_IMAGE + description: Full image reference for opendatahub-tests BVT (tag or latest) + steps: + - name: clone-scripts-repo + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + + - name: resolve + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: OPERATOR_VERSION + value: $(params.OPERATOR_VERSION) + - name: OPENDATAHUB_TESTS_REPO + value: $(params.OPENDATAHUB_TESTS_REPO) + - name: RESULT_PATH + value: $(results.OPENDATAHUB_TESTS_IMAGE.path) + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/resolve_opendatahub_tests_image.py"]) + + - name: bvt-health-checks-no-eaas + runAfter: + - parse-pipeline-tests + when: + - input: "$(tasks.parse-pipeline-tests.results.RUN_BVT)" + operator: in + values: ["true"] + - input: "$(params.PRODUCT)" + operator: in + values: ["none"] + params: + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + - name: OCI_ARTIFACTS_REPO + value: $(params.OCI_ARTIFACTS_REPO) + - name: ARTIFACT_BROWSER_URL + value: $(params.ARTIFACT_BROWSER_URL) + - name: ARTIFACT_BROWSER_REPO_PATH + value: $(params.ARTIFACT_BROWSER_REPO_PATH) + - name: OCI_ARTIFACTS_PUSH_SECRET_NAME + value: $(params.OCI_ARTIFACTS_PUSH_SECRET_NAME) + taskSpec: + params: + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + - name: OCI_ARTIFACTS_REPO + type: string + - name: ARTIFACT_BROWSER_URL + type: string + - name: ARTIFACT_BROWSER_REPO_PATH + type: string + - name: OCI_ARTIFACTS_PUSH_SECRET_NAME + type: string + results: + - name: TEST_OUTPUT + description: BVT test summary in Konflux standardised format (rendered in UI task panel) + - name: ARTIFACTS_URL + description: Direct artifact browser URL for this run's BVT JUnit results + volumes: + - name: bvt-artifacts + emptyDir: {} + - name: oci-docker-config-writable + emptyDir: {} + - name: odh-registry-secret-volume + secret: + optional: true + secretName: $(params.OCI_ARTIFACTS_PUSH_SECRET_NAME) + items: + - key: .dockerconfigjson + path: config.json + steps: + - name: clone-scripts-repo + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/scripts-repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + + - name: emit-bvt-placeholder-junit + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + volumeMounts: + - name: bvt-artifacts + mountPath: /artifacts + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp( + sys.executable, + [sys.executable, "helpers/bvt_product_none_placeholder_junit.py", "/artifacts"], + ) + + - name: upload-artifacts + image: quay.io/konflux-qe-incubator/konflux-qe-tools:latest + workingDir: /artifacts + volumeMounts: + - name: bvt-artifacts + mountPath: /artifacts + - name: odh-registry-secret-volume + mountPath: /tmp/oci-secret-readonly + readOnly: true + - name: oci-docker-config-writable + mountPath: /tmp/docker-config + env: + - name: OCI_ARTIFACT_REFERENCE + value: "$(params.OCI_ARTIFACTS_REPO):$(context.pipelineRun.name)-bvt" + - name: OCI_TAG_EXPIRATION + value: "30d" + - name: ALWAYS_PASS + value: "false" + - name: DOCKER_CONFIG + value: /tmp/docker-config + script: | + #!/bin/bash + set -euo pipefail + # Secret mount is read-only; copy into emptyDir so oras/credential helpers can write. + if [[ -f /tmp/oci-secret-readonly/config.json ]]; then + cp /tmp/oci-secret-readonly/config.json "${DOCKER_CONFIG}/config.json" + else + echo "[WARNING] OCI_ARTIFACTS_PUSH_SECRET_NAME secret not mounted; push will likely fail (anonymous)" + fi + exec bash /workspace/scripts-repo/integration-tests/olminstall/helpers/secure_push_oci_artifacts.sh + + - name: emit-test-output + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + volumeMounts: + - name: bvt-artifacts + mountPath: /artifacts + env: + - name: OCI_REPO + value: $(params.OCI_ARTIFACTS_REPO) + - name: ARTIFACT_BROWSER_BASE + value: $(params.ARTIFACT_BROWSER_URL) + - name: ARTIFACT_BROWSER_REPO_PATH + value: $(params.ARTIFACT_BROWSER_REPO_PATH) + - name: PR_NAME + value: $(context.pipelineRun.name) + - name: TEST_OUTPUT_PATH + value: $(results.TEST_OUTPUT.path) + - name: ARTIFACTS_URL_PATH + value: $(results.ARTIFACTS_URL.path) + - name: OCI_TAG_SUFFIX + value: bvt + - name: NOTE_PREFIX + value: BVT + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/emit_test_output.py"]) + + - name: bvt-health-checks-with-eaas + runAfter: + - install-operator + - resolve-opendatahub-tests-image-after-install + when: + - input: "$(tasks.parse-pipeline-tests.results.RUN_BVT)" + operator: in + values: ["true"] + - input: "$(params.PRODUCT)" + operator: notin + values: ["none"] + params: + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + - name: OCI_ARTIFACTS_REPO + value: $(params.OCI_ARTIFACTS_REPO) + - name: ARTIFACT_BROWSER_URL + value: $(params.ARTIFACT_BROWSER_URL) + - name: ARTIFACT_BROWSER_REPO_PATH + value: $(params.ARTIFACT_BROWSER_REPO_PATH) + - name: OCI_ARTIFACTS_PUSH_SECRET_NAME + value: $(params.OCI_ARTIFACTS_PUSH_SECRET_NAME) + taskSpec: + params: + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + - name: OCI_ARTIFACTS_REPO + type: string + - name: ARTIFACT_BROWSER_URL + type: string + - name: ARTIFACT_BROWSER_REPO_PATH + type: string + - name: OCI_ARTIFACTS_PUSH_SECRET_NAME + type: string + results: + - name: TEST_OUTPUT + description: BVT test summary in Konflux standardised format (rendered in UI task panel) + - name: ARTIFACTS_URL + description: Direct artifact browser URL for this run's BVT JUnit results + volumes: + - name: credentials + emptyDir: {} + - name: bvt-artifacts + emptyDir: {} + - name: oci-docker-config-writable + emptyDir: {} + - name: odh-registry-secret-volume + secret: + # optional: pod starts even if the secret is missing; upload-artifacts uses ALWAYS_PASS (default false). + optional: true + secretName: $(params.OCI_ARTIFACTS_PUSH_SECRET_NAME) + items: + - key: .dockerconfigjson + path: config.json + steps: + - name: get-kubeconfig + ref: + resolver: git + params: + - name: url + value: https://github.com/konflux-ci/build-definitions.git + - name: revision + value: main + - name: pathInRepo + value: stepactions/eaas-get-ephemeral-cluster-credentials/0.1/eaas-get-ephemeral-cluster-credentials.yaml + params: + - name: eaasSpaceSecretRef + value: $(tasks.provision-eaas-space.results.secretRef) + - name: clusterName + value: "$(tasks.provision-cluster.results.clusterName)" + - name: credentials + value: credentials + + - name: clone-scripts-repo + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/scripts-repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + + - name: pytest-cluster + image: $(tasks.resolve-opendatahub-tests-image-after-install.results.OPENDATAHUB_TESTS_IMAGE) + volumeMounts: + - name: credentials + mountPath: /credentials + - name: bvt-artifacts + mountPath: /artifacts + env: + - name: KUBECONFIG + value: /credentials/$(steps.get-kubeconfig.results.kubeconfig) + - name: PYTEST_MARKER + value: cluster_health + - name: PYTEST_EXTRA_ARGS + value: "-svv" + - name: ARTIFACT_PREFIX + value: cluster-health + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/run_bvt_pytest.py"]) + + - name: pytest-operator + image: $(tasks.resolve-opendatahub-tests-image-after-install.results.OPENDATAHUB_TESTS_IMAGE) + volumeMounts: + - name: credentials + mountPath: /credentials + - name: bvt-artifacts + mountPath: /artifacts + env: + - name: KUBECONFIG + value: /credentials/$(steps.get-kubeconfig.results.kubeconfig) + - name: PYTEST_MARKER + value: operator_health + - name: PYTEST_EXTRA_ARGS + value: "-svv" + - name: ARTIFACT_PREFIX + value: operator-health + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/run_bvt_pytest.py"]) + + # Inline OCI push (same behavior as rhoai-konflux-tasks secure-push-oci StepAction) so Tekton + # prepare does not clone a second git repo alongside eaas-get-kubeconfig (avoids long Init:0/2). + - name: upload-artifacts + image: quay.io/konflux-qe-incubator/konflux-qe-tools:latest + workingDir: /artifacts + volumeMounts: + - name: bvt-artifacts + mountPath: /artifacts + - name: odh-registry-secret-volume + mountPath: /tmp/oci-secret-readonly + readOnly: true + - name: oci-docker-config-writable + mountPath: /tmp/docker-config + env: + - name: OCI_ARTIFACT_REFERENCE + value: "$(params.OCI_ARTIFACTS_REPO):$(context.pipelineRun.name)-bvt" + - name: OCI_TAG_EXPIRATION + value: "30d" + - name: ALWAYS_PASS + value: "false" + - name: DOCKER_CONFIG + value: /tmp/docker-config + script: | + #!/bin/bash + set -euo pipefail + # Secret mount is read-only; copy into emptyDir so oras/credential helpers can write. + if [[ -f /tmp/oci-secret-readonly/config.json ]]; then + cp /tmp/oci-secret-readonly/config.json "${DOCKER_CONFIG}/config.json" + else + echo "[WARNING] OCI_ARTIFACTS_PUSH_SECRET_NAME secret not mounted; push will likely fail (anonymous)" + fi + exec bash /workspace/scripts-repo/integration-tests/olminstall/helpers/secure_push_oci_artifacts.sh + + - name: emit-test-output + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + volumeMounts: + - name: bvt-artifacts + mountPath: /artifacts + env: + - name: OCI_REPO + value: $(params.OCI_ARTIFACTS_REPO) + - name: ARTIFACT_BROWSER_BASE + value: $(params.ARTIFACT_BROWSER_URL) + - name: ARTIFACT_BROWSER_REPO_PATH + value: $(params.ARTIFACT_BROWSER_REPO_PATH) + - name: PR_NAME + value: $(context.pipelineRun.name) + - name: TEST_OUTPUT_PATH + value: $(results.TEST_OUTPUT.path) + - name: ARTIFACTS_URL_PATH + value: $(results.ARTIFACTS_URL.path) + - name: OCI_TAG_SUFFIX + value: bvt + - name: NOTE_PREFIX + value: BVT + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/emit_test_output.py"]) + + - name: tier1-placeholder + runAfter: + - parse-pipeline-tests + when: + - input: "$(tasks.parse-pipeline-tests.results.RUN_TIER1)" + operator: in + values: ["true"] + taskSpec: + steps: + - name: noop + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + script: | + #!/usr/bin/env python3 + print("tier1: placeholder — no tier1 suite is wired in this pipeline yet.") + + finally: + - name: propagate-pipeline-test-output + params: + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + taskSpec: + params: + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + results: + - name: PROPAGATED_TEST_OUTPUT + description: Konflux TEST_OUTPUT (from BVT TaskRuns or install-operator INSTALL_STATUS) + steps: + - name: clone-scripts-repo + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/scripts-repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + - name: propagate + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: PIPELINE_RUN_NAME + value: $(context.pipelineRun.name) + - name: RESULT_PATH + value: $(results.PROPAGATED_TEST_OUTPUT.path) + script: | + #!/usr/bin/env python3 + import os, sys + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/propagate_pipeline_test_output.py"]) + + - name: collect-diagnostics + when: + - input: "$(tasks.status)" + operator: in + values: ["Failed"] + # Only attempt cluster diagnostics when the cluster was actually provisioned; + # this avoids Tekton silently skipping this task due to missing result + # references when provision-cluster itself failed or was skipped. + # post-results always fires on failure for the notification side. + - input: "$(tasks.provision-cluster.status)" + operator: in + values: ["Succeeded"] + params: + - name: OPERATOR_NAMESPACE + value: $(params.OPERATOR_NAMESPACE) + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) + taskSpec: + params: + - name: OPERATOR_NAMESPACE + type: string + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + results: + - name: DIAGNOSTICS_MANIFEST + description: File listing under /diag (full bundle stays on TaskRun pod until pruned) + volumes: + - name: credentials + emptyDir: {} + - name: diag + emptyDir: {} + steps: + - name: get-kubeconfig + ref: + resolver: git + params: + - name: url + value: https://github.com/konflux-ci/build-definitions.git + - name: revision + value: main + - name: pathInRepo + value: stepactions/eaas-get-ephemeral-cluster-credentials/0.1/eaas-get-ephemeral-cluster-credentials.yaml + params: + - name: eaasSpaceSecretRef + value: $(tasks.provision-eaas-space.results.secretRef) + - name: clusterName + value: "$(tasks.provision-cluster.results.clusterName)" + - name: credentials + value: credentials + + - name: clone-scripts-repo + image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/scripts-repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + + - name: oc-adm-inspect + image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 + volumeMounts: + - name: credentials + mountPath: /credentials + - name: diag + mountPath: /diag + env: + - name: KUBECONFIG + value: /credentials/$(steps.get-kubeconfig.results.kubeconfig) + - name: OPERATOR_NAMESPACE + value: $(params.OPERATOR_NAMESPACE) + - name: DIAG_MANIFEST_RESULT + value: $(results.DIAGNOSTICS_MANIFEST.path) + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/collect_diagnostics.py"]) + + - name: post-results + when: + - input: "$(tasks.status)" + operator: in + values: ["Succeeded", "Failed", "Completed"] + taskSpec: + results: + - name: TEST_OUTPUT + description: BVT / install summary (Konflux standard; same names as bvt-health-checks) + - name: ARTIFACTS_URL + description: BVT artifact browser URL for this PipelineRun + - name: OPERATOR_VERSION + description: Installed operator version when install-operator completed + - name: EPHEMERAL_CLUSTER + description: Ephemeral EaaS cluster name (clusterName result) + params: + - name: FBCF_IMAGE + type: string + default: "(unknown — extract-fbcf-image did not complete)" + - name: TEST_STATUS + type: string + default: "FAILED" + - name: OPERATOR_VERSION + type: string + default: "(unknown)" + - name: PIPELINE_RUN_NAME + type: string + - name: OPERATOR_NAME + type: string + default: "rhods-operator" + - name: ARTIFACTS_URL + type: string + description: BVT artifact browser URL (predicted path; Konflux UI Parameters panel) + default: "" + - name: TESTS + type: string + default: "" + - name: SLACK_CHANNEL_ID + type: string + default: "" + - name: SCRIPTS_REPO_URL + type: string + - name: SCRIPTS_REPO_REVISION + type: string + steps: + - name: clone-scripts-repo + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: REPO_REV + value: $(params.SCRIPTS_REPO_REVISION) + - name: DEST_DIR + value: /workspace/scripts-repo + script: | + #!/bin/bash + set -euo pipefail + rm -rf "${DEST_DIR}" && mkdir -p "${DEST_DIR}" && cd "${DEST_DIR}" + git init -q && git remote add origin "${REPO_URL}" + git fetch --depth=1 origin "${REPO_REV}" && git checkout -q FETCH_HEAD + + # Seed Konflux task Results from params (no scripts repo) so ARTIFACTS_URL appears even + # if patch-summary-annotations fails on an older SCRIPTS_REPO_REVISION. + - name: seed-ui-results + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + script: | + #!/usr/bin/env sh + set -eu + printf '%s' "$(params.ARTIFACTS_URL)" > "$(results.ARTIFACTS_URL.path)" + printf '%s' "result=$(params.TEST_STATUS)" > "$(results.TEST_OUTPUT.path)" + + - name: send-notification + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: TEST_STATUS + value: $(params.TEST_STATUS) + - name: OPERATOR_NAME + value: $(params.OPERATOR_NAME) + - name: OPERATOR_VERSION + value: $(params.OPERATOR_VERSION) + - name: FBCF_IMAGE + value: $(params.FBCF_IMAGE) + - name: PIPELINE_RUN_NAME + value: $(params.PIPELINE_RUN_NAME) + - name: ARTIFACTS_URL + value: $(params.ARTIFACTS_URL) + - name: TESTS + value: $(params.TESTS) + - name: SLACK_CHANNEL_ID + value: $(params.SLACK_CHANNEL_ID) + - name: SLACK_WEBHOOK_URL + valueFrom: + secretKeyRef: + name: slack-webhook + key: webhook-url + optional: true + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/send_notification.py"]) + + - name: patch-summary-annotations + image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d + env: + - name: PIPELINE_RUN_NAME + value: $(params.PIPELINE_RUN_NAME) + - name: TESTS + value: $(params.TESTS) + - name: ARTIFACTS_URL + value: $(params.ARTIFACTS_URL) + - name: ARTIFACTS_URL_PATH + value: $(results.ARTIFACTS_URL.path) + - name: TEST_OUTPUT_PATH + value: $(results.TEST_OUTPUT.path) + - name: OPERATOR_VERSION_PATH + value: $(results.OPERATOR_VERSION.path) + - name: EPHEMERAL_CLUSTER_PATH + value: $(results.EPHEMERAL_CLUSTER.path) + script: | + #!/usr/bin/env python3 + import os, sys + + os.chdir("/workspace/scripts-repo/integration-tests/olminstall") + os.execvp(sys.executable, [sys.executable, "helpers/patch_pipelinerun_summary.py"]) + params: + # NOTE: Do NOT reference $(tasks.bvt-health-checks.results.*) or any other + # task results here. In Tekton v1, a finally task that references a result + # from a skipped/failed task is itself skipped — so Slack notifications + # would silently not fire on the most important failures. + # ARTIFACTS_URL uses only params + context (never $(tasks.*.results)) so post-results + # is not skipped when BVT was skipped. patch-summary-annotations overwrites annotations + # with the published URL when upload-artifacts emitted ARTIFACTS_URL on a TaskRun. + - name: FBCF_IMAGE + value: "$(params.FBCF_COMPONENT_NAME)" + # Aggregate status of all non-finally pipeline tasks (install, resolve, BVT). + - name: TEST_STATUS + value: "$(tasks.status)" + # OPERATOR_VERSION is a best-effort field: pass a safe literal so the + # task is never skipped due to a missing result reference. + - name: OPERATOR_VERSION + value: "(see pipeline run logs)" + - name: PIPELINE_RUN_NAME + value: $(context.pipelineRun.name) + - name: OPERATOR_NAME + value: "$(params.OPERATOR_NAME)" + - name: ARTIFACTS_URL + value: "$(params.ARTIFACT_BROWSER_URL)/$(params.ARTIFACT_BROWSER_REPO_PATH)/$(context.pipelineRun.name)-bvt/" + - name: TESTS + value: "$(params.TESTS)" + - name: SLACK_CHANNEL_ID + value: "$(params.SLACK_CHANNEL_ID)" + - name: SCRIPTS_REPO_URL + value: $(params.SCRIPTS_REPO_URL) + - name: SCRIPTS_REPO_REVISION + value: $(params.SCRIPTS_REPO_REVISION) diff --git a/integration-tests/olminstall/olminstall-smoke-pipeline.yaml b/integration-tests/olminstall/olminstall-smoke-pipeline.yaml deleted file mode 100644 index 56841ccb7..000000000 --- a/integration-tests/olminstall/olminstall-smoke-pipeline.yaml +++ /dev/null @@ -1,643 +0,0 @@ ---- -apiVersion: tekton.dev/v1 -kind: Pipeline -metadata: - name: odh-olminstall-smoke-test - labels: - app.kubernetes.io/version: "1.0" - annotations: - tekton.dev/pipelines.minVersion: "0.12.1" - tekton.dev/tags: odh, rhoai, integration-test -spec: - description: >- - ODH/RHOAI integration test: provisions an ephemeral cluster via Konflux EaaS (HyperShift), - installs the operator with olminstall from the FBCF image in the Konflux snapshot, and runs - basic smoke verification. Post-install BVT is out of scope here; see README (Maintenance). - - params: - - name: SNAPSHOT - description: | - Spec section of an ApplicationSnapshot as a JSON string. - Provided by Konflux Integration Test Scenario. - type: string - - - name: FBCF_COMPONENT_NAME - description: Name of the FBCF component in the snapshot - type: string - default: "odh-operator-catalog" - - - name: UPDATE_CHANNEL - description: OLM update channel - type: string - default: "stable" - - - name: OPERATOR_NAMESPACE - description: Namespace for operator installation - type: string - default: "redhat-ods-operator" - - - name: OPERATOR_NAME - description: OLM package / deployment name (e.g. rhods-operator for RHOAI) - type: string - default: "rhods-operator" - - - name: OLMINSTALL_CATALOG_NAME - description: CatalogSource name used by olminstall install-operator.sh - type: string - default: "rhoai-catalog-dev" - - - name: HYPERSHIFT_INSTANCE_TYPE - description: AWS instance type for ephemeral HyperShift workers - type: string - default: "m5.2xlarge" - - - name: OCP_VERSION_PREFIX - description: | - OpenShift version prefix for the ephemeral cluster (e.g. 4.20. or 4.19.). Latest patch is resolved via EaaS. - Leave empty to use the newest supported major.minor from the environment. - type: string - default: "" - - - name: SCRIPTS_REPO_URL - description: Git repo containing the pipeline step scripts - type: string - default: "https://github.com/opendatahub-io/odh-konflux-central.git" - - - name: SCRIPTS_REPO_REVISION - description: Branch/tag/SHA of the scripts repo to clone - type: string - default: "main" - - - name: OLMINSTALL_REPO_URL - description: | - olminstall repo URL — provides tested OLM - install manifests (resources/install-rhods-operator.yaml) and utility - scripts (utils/oc_wait.sh, utils/oc_approve.sh). - type: string - default: "https://gitlab.cee.redhat.com/data-hub/olminstall.git" - - - name: OLMINSTALL_REPO_REVISION - description: Branch/tag/SHA of data-hub/olminstall to clone - type: string - default: "main" - - - name: QUAY_PULL_SECRET_NAME - description: Name of the namespace secret containing quay.io/rhoai dockerconfigjson - type: string - default: "rhoai-quay-secret" - - results: - - name: TEST_OUTPUT - description: Overall test result (from olminstall task verification) - value: $(tasks.install-operator.results.INSTALL_STATUS) - - - name: FBCF_IMAGE - description: FBCF image that was tested - value: $(tasks.extract-fbcf-image.results.FBCF_IMAGE) - - tasks: - - name: extract-fbcf-image - taskSpec: - params: - - name: SNAPSHOT - type: string - - name: COMPONENT_NAME - type: string - results: - - name: FBCF_IMAGE - description: Extracted FBCF image URL - steps: - - name: parse-snapshot - image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d - env: - - name: SNAPSHOT - value: $(params.SNAPSHOT) - - name: COMPONENT_NAME - value: $(params.COMPONENT_NAME) - script: | - #!/bin/bash - set -euo pipefail - - echo "Parsing Konflux snapshot..." - FBCF_IMAGE=$(echo "$SNAPSHOT" | \ - jq -r --arg name "$COMPONENT_NAME" '.components[] | select(.name==$name) | .containerImage') - - if [ -z "$FBCF_IMAGE" ] || [ "$FBCF_IMAGE" == "null" ]; then - echo "❌ Failed to extract FBCF image from snapshot for component: ${COMPONENT_NAME}" - exit 1 - fi - - echo "✓ Extracted FBCF Image: ${FBCF_IMAGE}" - echo -n "${FBCF_IMAGE}" > "$(results.FBCF_IMAGE.path)" - params: - - name: SNAPSHOT - value: $(params.SNAPSHOT) - - name: COMPONENT_NAME - value: $(params.FBCF_COMPONENT_NAME) - - - name: provision-eaas-space - runAfter: - - extract-fbcf-image - taskRef: - resolver: git - params: - - name: url - value: https://github.com/konflux-ci/build-definitions.git - - name: revision - value: main - - name: pathInRepo - value: task/eaas-provision-space/0.1/eaas-provision-space.yaml - params: - - name: ownerName - value: $(context.pipelineRun.name) - - name: ownerUid - value: $(context.pipelineRun.uid) - - - name: provision-cluster - runAfter: - - provision-eaas-space - taskSpec: - results: - - name: clusterName - value: "$(steps.create-cluster.results.clusterName)" - steps: - - name: get-supported-versions - ref: - resolver: git - params: - - name: url - value: https://github.com/konflux-ci/build-definitions.git - - name: revision - value: main - - name: pathInRepo - value: stepactions/eaas-get-supported-ephemeral-cluster-versions/0.1/eaas-get-supported-ephemeral-cluster-versions.yaml - params: - - name: eaasSpaceSecretRef - value: $(tasks.provision-eaas-space.results.secretRef) - - name: resolve-ocp-prefix - image: quay.io/konflux-ci/task-runner:1.5.0@sha256:200019314a50be5b6dd06f362c794c92a700583a522c5eee9a41e3eab7f706c5 - results: - - name: prefix - type: string - env: - - name: OVERRIDE - value: $(params.OCP_VERSION_PREFIX) - - name: DEFAULT_MINOR - value: $(steps.get-supported-versions.results.versions[0]) - script: | - #!/bin/bash - set -euo pipefail - if [[ -z "${OVERRIDE// }" ]]; then - echo -n "${DEFAULT_MINOR}." > "$(step.results.prefix.path)" - else - P="${OVERRIDE// /}" - case "$P" in *.) ;; - *) P="${P}." ;; - esac - echo -n "$P" > "$(step.results.prefix.path)" - fi - - name: pick-version - ref: - resolver: git - params: - - name: url - value: https://github.com/konflux-ci/build-definitions.git - - name: revision - value: main - - name: pathInRepo - value: stepactions/eaas-get-latest-openshift-version-by-prefix/0.1/eaas-get-latest-openshift-version-by-prefix.yaml - params: - - name: prefix - value: "$(steps.resolve-ocp-prefix.results.prefix)" - - name: create-cluster - image: quay.io/konflux-ci/task-runner:1.5.0@sha256:200019314a50be5b6dd06f362c794c92a700583a522c5eee9a41e3eab7f706c5 - results: - - name: clusterName - type: string - env: - - name: INSTANCE_TYPE - value: $(params.HYPERSHIFT_INSTANCE_TYPE) - - name: VERSION - value: "$(steps.pick-version.results.version)" - - name: KUBECONFIG_VALUE - valueFrom: - secretKeyRef: - name: $(tasks.provision-eaas-space.results.secretRef) - key: kubeconfig - - name: TENANT - value: "$(context.taskRun.namespace)" - - name: TIMEOUT - value: "30m" - - name: ICS_VALUE - value: | - - mirrors: - - quay.io/rhoai - source: registry.redhat.io/rhoai - script: | - #!/bin/bash - set -eo pipefail - export KUBECONFIG=/tmp/kubeconfig - echo "$KUBECONFIG_VALUE" > "$KUBECONFIG" - trap 'rm -f "$KUBECONFIG"' EXIT - - export ICS_VALUE - cat > cti.yaml < "$(step.results.clusterName.path)" - - DEADLINE=$(($(date +%s) + 1800)) - echo "Waiting for cluster to be ready (up to 30m)..." - while [ "$(date +%s)" -lt "$DEADLINE" ]; do - PHASE=$("${KUBECTL[@]}" get cti "$CTI_NAME" -o jsonpath='{.status.phase}' 2>/dev/null || echo "Unknown") - MSG=$("${KUBECTL[@]}" get cti "$CTI_NAME" -o jsonpath='{.status.message}' 2>/dev/null || true) - echo " phase=${PHASE}${MSG:+ ($MSG)}" - if [ "$PHASE" = "Ready" ]; then - echo "✓ Successfully provisioned ${CTI_NAME}" - exit 0 - elif [ "$PHASE" = "Failed" ] || [ "$PHASE" = "Error" ]; then - echo "❌ Cluster provisioning failed" - "${KUBECTL[@]}" get cti "$CTI_NAME" -o yaml - exit 1 - fi - sleep 30 - done - echo "❌ Timed out waiting for cluster" - "${KUBECTL[@]}" get cti "$CTI_NAME" -o yaml - exit 1 - - - name: install-operator - runAfter: - - provision-cluster - params: - - name: FBCF_IMAGE - value: $(tasks.extract-fbcf-image.results.FBCF_IMAGE) - - name: UPDATE_CHANNEL - value: $(params.UPDATE_CHANNEL) - - name: OPERATOR_NAMESPACE - value: $(params.OPERATOR_NAMESPACE) - - name: OPERATOR_NAME - value: $(params.OPERATOR_NAME) - - name: OLMINSTALL_REPO_URL - value: $(params.OLMINSTALL_REPO_URL) - - name: OLMINSTALL_REPO_REVISION - value: $(params.OLMINSTALL_REPO_REVISION) - - name: OLMINSTALL_CATALOG_NAME - value: $(params.OLMINSTALL_CATALOG_NAME) - - name: SCRIPTS_REPO_URL - value: $(params.SCRIPTS_REPO_URL) - - name: SCRIPTS_REPO_REVISION - value: $(params.SCRIPTS_REPO_REVISION) - - name: QUAY_PULL_SECRET_NAME - value: $(params.QUAY_PULL_SECRET_NAME) - taskSpec: - params: - - name: FBCF_IMAGE - type: string - - name: UPDATE_CHANNEL - type: string - - name: OPERATOR_NAMESPACE - type: string - - name: OPERATOR_NAME - type: string - - name: OLMINSTALL_REPO_URL - type: string - - name: OLMINSTALL_REPO_REVISION - type: string - - name: OLMINSTALL_CATALOG_NAME - type: string - - name: SCRIPTS_REPO_URL - type: string - - name: SCRIPTS_REPO_REVISION - type: string - - name: QUAY_PULL_SECRET_NAME - type: string - results: - - name: INSTALL_STATUS - description: Installation status (SUCCESS or FAILED) - - name: OPERATOR_VERSION - description: Installed operator version - volumes: - - name: credentials - emptyDir: {} - - name: quay-pull-secret - secret: - secretName: $(params.QUAY_PULL_SECRET_NAME) - steps: - - name: get-kubeconfig - ref: - resolver: git - params: - - name: url - value: https://github.com/konflux-ci/build-definitions.git - - name: revision - value: main - - name: pathInRepo - value: stepactions/eaas-get-ephemeral-cluster-credentials/0.1/eaas-get-ephemeral-cluster-credentials.yaml - params: - - name: eaasSpaceSecretRef - value: $(tasks.provision-eaas-space.results.secretRef) - - name: clusterName - value: "$(tasks.provision-cluster.results.clusterName)" - - name: credentials - value: credentials - - - name: clone-repo - image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 - env: - - name: REPO_URL - value: $(params.SCRIPTS_REPO_URL) - - name: REPO_REV - value: $(params.SCRIPTS_REPO_REVISION) - script: | - #!/bin/bash - set -euo pipefail - echo "Cloning ${REPO_URL}@${REPO_REV}..." - rm -rf /workspace/repo - mkdir -p /workspace/repo - cd /workspace/repo - git init -q - git remote add origin "${REPO_URL}" - git fetch --depth=1 origin "${REPO_REV}" - git checkout -q FETCH_HEAD - echo "✓ Cloned ${REPO_URL}@${REPO_REV}" - - - name: clone-olminstall - image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 - env: - - name: REPO_URL - value: $(params.OLMINSTALL_REPO_URL) - - name: REPO_REV - value: $(params.OLMINSTALL_REPO_REVISION) - script: | - #!/bin/bash - set -euo pipefail - - # Trust the Red Hat internal CA chain. - if [ -d /etc/pki/ca-trust/source/anchors ]; then - update-ca-trust 2>/dev/null || true - fi - # Verify TLS. For known internal Red Hat hosts (gitlab.cee.redhat.com) - # apply a host-scoped sslVerify=false when the CA bundle isn't in the - # container image — narrower than a global GIT_SSL_NO_VERIFY env var. - # TODO: bake the RH IT root CA into quay.io/rhoai/rhoai-task-toolset:its - # or mount it as a ConfigMap to remove this exception entirely. - if ! git ls-remote --exit-code "${REPO_URL}" HEAD &>/dev/null; then - _host=$(echo "${REPO_URL}" | grep -oE "https?://[^/]+") - if echo "${_host}" | grep -qE "gitlab\.cee\.redhat\.com|git\.corp\.redhat\.com"; then - echo "⚠ TLS verification failed for internal host ${REPO_URL}" - echo " Applying host-scoped sslVerify=false (TODO: bake RH IT CA into image)" - git config --global "http.${_host}.sslVerify" false - else - echo "❌ TLS verification failed for ${REPO_URL}." - echo " Mount the internal CA bundle into the container so git ls-remote succeeds." - exit 1 - fi - fi - - echo "Cloning ${REPO_URL}@${REPO_REV}..." - rm -rf /workspace/olminstall - mkdir -p /workspace/olminstall - cd /workspace/olminstall - git init -q - git remote add origin "${REPO_URL}" - git fetch --depth=1 origin "${REPO_REV}" - git checkout -q FETCH_HEAD - echo "✓ Cloned ${REPO_URL}@${REPO_REV}" - - - name: patch-cluster-pull-secret - image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 - volumeMounts: - - name: credentials - mountPath: /credentials - - name: quay-pull-secret - mountPath: /var/secret/quay - readOnly: true - - env: - - name: KUBECONFIG - value: /credentials/$(steps.get-kubeconfig.results.kubeconfig) - script: | - #!/bin/bash - exec python3 /workspace/repo/integration-tests/olminstall/helpers/patch_cluster_pull_secret.py - - - name: install-and-verify - image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 - volumeMounts: - - name: credentials - mountPath: /credentials - env: - - name: KUBECONFIG - value: /credentials/$(steps.get-kubeconfig.results.kubeconfig) - - name: FBCF_IMAGE - value: $(params.FBCF_IMAGE) - - name: UPDATE_CHANNEL - value: $(params.UPDATE_CHANNEL) - - name: OPERATOR_NAMESPACE - value: $(params.OPERATOR_NAMESPACE) - - name: OPERATOR_NAME - value: $(params.OPERATOR_NAME) - - name: OLMINSTALL_DIR - value: /workspace/olminstall - - name: OLMINSTALL_CATALOG_NAME - value: $(params.OLMINSTALL_CATALOG_NAME) - - name: QUAY_PULL_SECRET_NAME - value: $(params.QUAY_PULL_SECRET_NAME) - - name: INSTALL_STATUS_PATH - value: $(results.INSTALL_STATUS.path) - - name: OPERATOR_VERSION_PATH - value: $(results.OPERATOR_VERSION.path) - script: | - #!/bin/bash - exec python3 /workspace/repo/integration-tests/olminstall/helpers/install_and_verify.py - - finally: - - name: collect-diagnostics - when: - - input: "$(tasks.status)" - operator: in - values: ["Failed"] - # Only attempt cluster diagnostics when the cluster was actually provisioned; - # this avoids Tekton silently skipping this task due to missing result - # references when provision-cluster itself failed or was skipped. - # post-results always fires on failure for the notification side. - - input: "$(tasks.provision-cluster.status)" - operator: in - values: ["Succeeded"] - params: - - name: OPERATOR_NAMESPACE - value: $(params.OPERATOR_NAMESPACE) - taskSpec: - params: - - name: OPERATOR_NAMESPACE - type: string - volumes: - - name: credentials - emptyDir: {} - steps: - - name: get-kubeconfig - ref: - resolver: git - params: - - name: url - value: https://github.com/konflux-ci/build-definitions.git - - name: revision - value: main - - name: pathInRepo - value: stepactions/eaas-get-ephemeral-cluster-credentials/0.1/eaas-get-ephemeral-cluster-credentials.yaml - params: - - name: eaasSpaceSecretRef - value: $(tasks.provision-eaas-space.results.secretRef) - - name: clusterName - value: "$(tasks.provision-cluster.results.clusterName)" - - name: credentials - value: credentials - - name: oc-adm-inspect - image: quay.io/rhoai/rhoai-task-toolset:its@sha256:18b1af9b64d766478792849e49eed156464f678ef53b567a641a28444cacbfc9 - volumeMounts: - - name: credentials - mountPath: /credentials - env: - - name: KUBECONFIG - value: /credentials/$(steps.get-kubeconfig.results.kubeconfig) - - name: OPERATOR_NAMESPACE - value: $(params.OPERATOR_NAMESPACE) - script: | - #!/bin/bash - set -eu - oc adm inspect "ns/${OPERATOR_NAMESPACE}" --dest-dir=/tmp/inspect || true - oc get csv -n "${OPERATOR_NAMESPACE}" -o yaml || true - echo "=== Bundle-unpack job diagnostics ===" - echo "--- jobs in openshift-marketplace ---" - oc get jobs -n openshift-marketplace -o wide || true - echo "--- bundle-unpack job spec (image + SA) ---" - oc get jobs -n openshift-marketplace -o yaml \ - | grep -E "^\s*(image|serviceAccountName|activeDeadlineSeconds):" \ - | sort -u || true - echo "--- bundle-unpack job events ---" - for JOB in $(oc get jobs -n openshift-marketplace \ - -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do - echo "Job: $JOB" - oc describe job "$JOB" -n openshift-marketplace 2>/dev/null \ - | grep -E "(Events|Image|Status|Message|Reason):" -A5 || true - echo "Job logs (last attempt):" - oc logs "job/${JOB}" -n openshift-marketplace \ - --tail=20 2>/dev/null || echo " (no logs)" - done - echo "--- SAs and imagePullSecrets in openshift-marketplace ---" - oc get sa -n openshift-marketplace \ - -o custom-columns='NAME:.metadata.name,PULL_SECRETS:.imagePullSecrets' || true - echo "=== Subscription status ===" - oc describe sub -n "${OPERATOR_NAMESPACE}" || true - - - name: post-results - when: - - input: "$(tasks.status)" - operator: in - values: ["Succeeded", "Failed"] - taskSpec: - params: - - name: FBCF_IMAGE - type: string - default: "(unknown — extract-fbcf-image did not complete)" - - name: TEST_STATUS - type: string - default: "FAILED" - - name: OPERATOR_VERSION - type: string - default: "(unknown)" - - name: PIPELINE_RUN_NAME - type: string - - name: OPERATOR_NAME - type: string - default: "rhods-operator" - steps: - - name: send-notification - image: quay.io/konflux-ci/konflux-test:stable@sha256:09328e5f47da168ffe951ec9c4242cd6761e61e0a235df1bba3f5158e757446d - env: - - name: SLACK_WEBHOOK_URL - valueFrom: - secretKeyRef: - name: slack-webhook - key: webhook-url - optional: true - script: | - #!/bin/bash - set -e - - # TEST_STATUS comes from $(tasks.install-operator.status) which - # returns Tekton task-status strings: Succeeded, Failed, None, Skipped. - if [ "$(params.TEST_STATUS)" == "Succeeded" ]; then - EMOJI="✅" - STATUS_TEXT="PASSED" - else - EMOJI="❌" - STATUS_TEXT="FAILED (task status: $(params.TEST_STATUS))" - fi - - case "$(params.OPERATOR_NAME)" in - opendatahub-operator) PRODUCT_LABEL="ODH" ;; - rhods-operator) PRODUCT_LABEL="RHOAI" ;; - *) PRODUCT_LABEL="ODH/RHOAI" ;; - esac - MSG="${EMOJI} ${PRODUCT_LABEL} olminstall smoke test ${STATUS_TEXT}" - MSG="${MSG}"$'\n'"Operator : $(params.OPERATOR_VERSION)" - MSG="${MSG}"$'\n'"FBCF : $(params.FBCF_IMAGE)" - MSG="${MSG}"$'\n'"Run : $(params.PIPELINE_RUN_NAME)" - - if [ -n "${SLACK_WEBHOOK_URL:-}" ]; then - PAYLOAD=$(echo "$MSG" | jq -Rs '{text: .}') - curl -sS -X POST -H 'Content-type: application/json' \ - --data "${PAYLOAD}" \ - "${SLACK_WEBHOOK_URL}" - echo "✓ Slack notification sent" - else - echo "SLACK_WEBHOOK_URL not set — skipping Slack" - echo "---" - echo "$MSG" - echo "---" - fi - params: - # NOTE: Do NOT reference $(tasks.extract-fbcf-image.results.*) here. - # In Tekton v1, a finally task that references a result from a - # skipped/failed task is itself skipped — so diagnostics and Slack - # notifications would silently not fire on the most important failures. - # We pass the component name as a human-readable identifier instead; - # the pipeline-level FBCF_IMAGE result carries the resolved image if - # extract-fbcf-image succeeded, but we cannot safely reference it here. - - name: FBCF_IMAGE - value: "$(params.FBCF_COMPONENT_NAME)" - # Use $(tasks..status) — not a result reference — so Tekton does - # not skip this finally task when install-operator was itself skipped or - # failed. "None" is the Tekton status value when the task never ran. - - name: TEST_STATUS - value: "$(tasks.install-operator.status)" - # OPERATOR_VERSION is a best-effort field: pass a safe literal so the - # task is never skipped due to a missing result reference. - - name: OPERATOR_VERSION - value: "(see pipeline run logs)" - - name: PIPELINE_RUN_NAME - value: $(context.pipelineRun.name) - - name: OPERATOR_NAME - value: "$(params.OPERATOR_NAME)" diff --git a/integration-tests/olminstall/olminstall-tests-config.yaml b/integration-tests/olminstall/olminstall-tests-config.yaml new file mode 100644 index 000000000..f5fd050f9 --- /dev/null +++ b/integration-tests/olminstall/olminstall-tests-config.yaml @@ -0,0 +1,33 @@ +# Olminstall Konflux pipeline — declarative test phases (Jenkins-style component list). +# Used by: +# - olm_pipeline.py (--tests / --tests-config) to validate the ITS param TESTS +# - Tekton task parse-pipeline-tests (after cloning SCRIPTS_REPO) to set RUN_SMOKE / RUN_BVT / RUN_TIER1 +# +# Adding a phase: extend this file, then wire matching Pipeline tasks + when: in +# olminstall-pipeline.yaml (pipeline cannot discover arbitrary phases without code). +# Phases may set requiredInSelection: true to force them into every TESTS value (optional). +# +# schemaVersion bumps when structure changes (tools may reject unknown versions). +schemaVersion: 1 + +phases: + - id: bvt + description: opendatahub-tests pytest markers cluster_health and operator_health. + default: true + setsPipelineResults: + RUN_BVT: true + + - id: smoke + description: > + Reserved smoke test phase (pytest suite not wired yet); toggles RUN_SMOKE for future tasks. + OLM install + CSV verification always run when PRODUCT is not none, independent of this flag. + requiredInSelection: false + default: true + setsPipelineResults: + RUN_SMOKE: true + + - id: tier1 + description: Reserved placeholder task until a tier1 suite is implemented. + default: false + setsPipelineResults: + RUN_TIER1: true diff --git a/integration-tests/olminstall/requirements.txt b/integration-tests/olminstall/requirements.txt index f090ad4b9..b55c22ea8 100644 --- a/integration-tests/olminstall/requirements.txt +++ b/integration-tests/olminstall/requirements.txt @@ -1,7 +1,9 @@ # integration-tests/olminstall — Python dependencies for this directory. +## Python deps for integration-tests/olminstall helpers and Tekton step scripts. +# defusedxml: required for safe JUnit XML parsing (helpers/tekton_util.py). +defusedxml>=0.7.1 # -# olm_pipeline.py and helpers/*.py use only the Python standard library. -# No packages need to be installed via pip to run them. +# Optional: PyYAML for faster olminstall-tests-config.yaml loading; else helpers use `yq`. # -# Runtime binaries (install separately): oc (required), tkn (recommended), yq (for ITS overrides), -# skopeo (optional, ODH catalog fallback). See module docstring / python3 olm_pipeline.py -h. +# Runtime binaries (install separately): oc (required), tkn (optional; trigger-mode log stream), +# yq (for ITS overrides and YAML→JSON when PyYAML is absent), skopeo (optional, ODH catalog fallback). diff --git a/integration-tests/olminstall/test-pipelinerun.yaml b/integration-tests/olminstall/test-pipelinerun.yaml index 189581011..fa11cf5e8 100644 --- a/integration-tests/olminstall/test-pipelinerun.yaml +++ b/integration-tests/olminstall/test-pipelinerun.yaml @@ -1,5 +1,5 @@ --- -# Manual test PipelineRun for olminstall smoke scenario. +# Manual test PipelineRun for olminstall integration scenario. # Usage: # oc create -n rhoai-tenant -f integration-tests/olminstall/test-pipelinerun.yaml # @@ -21,8 +21,10 @@ spec: - name: revision value: main - name: pathInRepo - value: integration-tests/olminstall/olminstall-smoke-pipeline.yaml + value: integration-tests/olminstall/olminstall-pipeline.yaml params: + - name: PRODUCT + value: odh - name: SNAPSHOT value: | { @@ -36,11 +38,13 @@ spec: - name: UPDATE_CHANNEL value: odh-stable - name: OPERATOR_NAMESPACE - value: opendatahub-operators + value: redhat-ods-operator - name: OPERATOR_NAME - value: opendatahub-operator + value: rhods-operator - name: OLMINSTALL_REPO_URL value: https://gitlab.cee.redhat.com/data-hub/olminstall.git # ODH uses a different pull-secret name; the pipeline default is rhoai-quay-secret - name: QUAY_PULL_SECRET_NAME value: odh-quay-secret + - name: TESTS + value: bvt,smoke diff --git a/integration-tests/olminstall/test-snapshot.yaml b/integration-tests/olminstall/test-snapshot.yaml index 19f75b66f..44b391c8c 100644 --- a/integration-tests/olminstall/test-snapshot.yaml +++ b/integration-tests/olminstall/test-snapshot.yaml @@ -1,5 +1,5 @@ --- -# Manual test Snapshot for olminstall smoke scenario. +# Manual test Snapshot for olminstall integration scenario. # Creates a Snapshot to trigger the ITS pipeline without waiting for a real build. # # Prerequisites: @@ -18,7 +18,10 @@ # integration-tests/olminstall/test-snapshot.yaml | oc create -n rhoai-tenant -f - # # After creation, Konflux detects the Snapshot and triggers a PipelineRun for -# every matching ITS. Watch the run: +# every matching IntegrationTestScenario on spec.application (and Enterprise Contract +# may add separate runs). Prefer ``python3 integration-tests/olminstall/olm_pipeline.py`` (prunes legacy ITS by +# default for rhoai-tenant + testops-playpen) or run ``python3 integration-tests/olminstall/helpers/prune_stale_testops_its.py`` first. +# Watch the olminstall run: # oc get pipelineruns -n rhoai-tenant | grep olminstall # or via the Konflux UI: # https://konflux-ui.apps./ns/rhoai-tenant/applications/testops-playpen/activity/pipelineruns diff --git a/integration-tests/olminstall/verify_cli_args.py b/integration-tests/olminstall/verify_cli_args.py index dc8a82ca5..13712c839 100644 --- a/integration-tests/olminstall/verify_cli_args.py +++ b/integration-tests/olminstall/verify_cli_args.py @@ -2,6 +2,12 @@ """ Batch-verify argparse + parse_cli_args for olm_pipeline.py (no oc / cluster). +This process only exercises ``parse_cli_args`` — it does **not** call ``oc`` or start +``OLMInstallRunner.run()``. It is meant to pass in CI or on a laptop with no Konflux +context. A full ``olm_pipeline.py`` trigger/watch (including ``--tests bvt``) **will +fail** without a logged-in Konflux cluster and tenant; that is expected until a future +option exists to target an existing cluster URL / kubeconfig from this entrypoint. + Run: python3 integration-tests/olminstall/verify_cli_args.py """ @@ -100,6 +106,10 @@ def expect_argparse_fail(argv: list[str]) -> None: failures.append(f"{label} -> {type(exc).__name__}: {exc}") # --- Valid parses --- + expect_ok([], check=lambda a: (_assert(a.product == "none"), _assert(a.list_pipelines == 0), _assert(not a.watch_mode), _assert(a.prune_stale_its))) + + expect_ok(["--no-prune-stale-its"], check=lambda a: (_assert(not a.prune_stale_its),)) + expect_ok( ["--product", "odh"], check=lambda a: ( @@ -109,6 +119,15 @@ def expect_argparse_fail(argv: list[str]) -> None: ), ) + expect_ok( + ["--tests", "bvt"], + check=lambda a: ( + _assert(a.product == "none"), + _assert(a.tests == "bvt"), + _assert(not a.watch_mode), + ), + ) + expect_ok( ["--product", "rhoai", "--version", "3.5"], check=lambda a: (_assert(a.version == "3.5"), _assert(a.product == "rhoai")), @@ -150,12 +169,29 @@ def expect_argparse_fail(argv: list[str]) -> None: check=lambda a: _assert(a.image == "quay.io/rhoai/x@sha256:deadbeef"), ) expect_ok( - ["--konflux-repo", "https://github.com/o/r.git", "--konflux-branch", "feat/x"], - check=lambda a: (_assert(a.konflux_repo.endswith(".git")), _assert(a.konflux_branch == "feat/x")), + ["--konflux-repo", "https://github.com/o/r.git", "--konflux-branch", "your-branch"], + check=lambda a: (_assert(a.konflux_repo.endswith(".git")), _assert(a.konflux_branch == "your-branch")), ) expect_ok(["--ocp-version", "4.20"], check=lambda a: _assert(a.ocp_version == "4.20")) expect_ok(["--ocp-version", " 4.19 "], check=lambda a: _assert(a.ocp_version == "4.19")) + expect_ok( + ["--tests", "bvt", "--ocp-version", "4.19"], + check=lambda a: (_assert(a.tests == "bvt"), _assert(a.ocp_version == "4.19")), + ) + expect_ok( + ["--tests", "bvt", "--image", "quay.io/rhoai/x@sha256:deadbeef"], + check=lambda a: _assert(a.tests == "bvt"), + ) + expect_ok( + ["--tests", "bvt", "--channel", "stable"], + check=lambda a: _assert(a.tests == "bvt"), + ) + expect_ok( + ["--tests", "tier1,bvt"], + check=lambda a: _assert(a.tests == "bvt,tier1"), + ) + expect_ok( ["--ka-host"], env={"KA_HOST": "https://kubearchive.apps.cluster.openshiftapps.com"}, @@ -218,6 +254,7 @@ def expect_argparse_fail(argv: list[str]) -> None: # --- Expected AppError --- expect_err(["--product", "odh", "--version", "1"], "--version is supported only") + expect_err(["--version", "3.5"], "--version is supported only") expect_err(["--ka-host"], "KA_HOST", env={"KA_HOST": ""}) prev_ka = _patch_env({"KA_HOST": None}) @@ -244,8 +281,10 @@ def expect_argparse_fail(argv: list[str]) -> None: expect_err(["--watch", "--ocp-version", "4.19"], "Trigger/install options cannot be used") expect_err(["--watch", "--version", "3.5", "--product", "rhoai"], "Trigger/install options cannot be used") expect_err(["--list-supported-ocp", "--konflux-repo", "https://g/r.git"], "Trigger/install options cannot be used") + expect_err(["--list-pipelines", "--tests", "bvt"], "Trigger/install options cannot be used") expect_argparse_fail(["--product", "invalid"]) + expect_argparse_fail(["--bvt-env-only"]) if failures: print(f"FAIL ({len(failures)}):", file=sys.stderr)