Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/skills/validate-skill
1 change: 1 addition & 0 deletions .codex/skills/validate-skill
1 change: 1 addition & 0 deletions .cursor/skills/validate-skill
1 change: 1 addition & 0 deletions .gemini/skills/validate-skill
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,7 @@ data/
# Temporary files
.tmp/
brain.ai
brain.code-workspace
brain.code-workspace

# Fixture validation reports and captured disposable quickstart output.
test/skills/reports/
16 changes: 12 additions & 4 deletions core/skills/rh-qs-test-suite/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ description: |
- Design doc **`designs/design.md`** defines testing strategy (unit / integration / E2E / evals) — this skill runs inside `.rhoai-qs/<slug>/` itself (the scaffolded quickstart repo), so the design doc sits right alongside the code, no `../` needed
- Optional: replace or extend minimal workflows from **`rh-qs-scaffold`**

## Nightly-only fast path

If the request is scoped to nightly testing specifically (e.g. "add nightly testing to `<slug>`", "update the nightly workflow", "refresh the nightly CI setup") rather than the full CI build-out, skip straight to it: resolve the slug/repo as usual, then invoke **`subagents/nightly-tests-prompt.md`** directly (`invocation_mode: standalone`) and report its result. Do not run the PR-checks/E2E/build-and-push phases for this request — they are unrelated to what was asked. This is the same subagent step 8 below delegates to; the only difference is you stop after it instead of continuing the rest of the workflow.

## What it does

1. Splits CI into **focused workflows** (quality, unit tests, Kind E2E, evals, image build) instead of one monolithic job
Expand All @@ -25,7 +29,7 @@ description: |
4. Defines a **Makefile CI contract** so workflows only call `make` targets
5. Documents required **secrets and repository variables**
6. Configures **branch protection** status checks to match workflow job names
7. Optionally adds **nightly** and **dev→main promotion** workflows for mature quickstarts
7. Optionally adds **dev→main promotion** workflows for mature quickstarts, and delegates **nightly** workflow generation to `subagents/nightly-tests-prompt.md` (also invokable on its own — see "Nightly-only fast path" above)

## Workflow

Expand All @@ -37,7 +41,7 @@ description: |
- [ ] 5. Add PR workflows (checks, e2e, branch policy — as needed)
- [ ] 6. Configure repo secrets/vars (see references/secrets-and-variables.md)
- [ ] 7. Add build-and-push on push to main/dev (if publishing images)
- [ ] 8. Add nightly / promotion workflows only when design requires them
- [ ] 8. When design requires nightly testing, invoke `subagents/nightly-tests-prompt.md`; add dev→main promotion workflow separately if needed
- [ ] 9. Register required status checks in branch protection
- [ ] 10. Open test PR and confirm all workflows green
```
Expand Down Expand Up @@ -65,7 +69,8 @@ description: |
| `pr-evaluation-check.yml` | `pull_request_target` | Known-bad conversation / regression eval gate |
| `pr-branch-check.yml` | `pull_request` → `main` | Enforce `dev` or `dev-<sha>` source branch |
| `build-and-push.yaml` | `push` to `main`/`dev` | Build and push images to Quay (org-owned repos only) |
| `nightly-e2e-*.yml` | `schedule`, `workflow_dispatch` | Longer eval suites on Kind or prod cluster |
| `test-<slug>.yml` | `workflow_call`, `workflow_dispatch` | Reusable deploy + test + teardown against a real OpenShift namespace, via actions copied from `quickstart-ci-testing` — generated by `subagents/nightly-tests-prompt.md` |
| `nightly-main.yml` / `nightly-<dev-branch>.yml` | `schedule`, `workflow_dispatch` | Thin scheduled callers of `test-<slug>.yml`, one per long-lived branch — generated by `subagents/nightly-tests-prompt.md` |
| `create-dev-to-main-pr.yaml` | `workflow_dispatch` | Version-checked promotion PR with temp branch |

Full catalog and naming conventions: [references/workflow-catalog.md](./references/workflow-catalog.md)
Expand Down Expand Up @@ -124,9 +129,11 @@ Full list: [references/makefile-ci-contract.md](./references/makefile-ci-contrac

Configure in GitHub **before** enabling `pull_request_target` E2E workflows:

- **Secrets:** `LLM_API_TOKEN_EVAL`, `LLM_API_TOKEN_INFERENCE`, `QUAY_USERNAME`, `QUAY_PASSWORD`, optional `PROD_TOKEN` / `PROD_SERVER` for prod nightlies
- **Secrets:** `LLM_API_TOKEN_EVAL`, `LLM_API_TOKEN_INFERENCE`, `QUAY_USERNAME`, `QUAY_PASSWORD`
- **Variables:** `LLM_URL_EVAL`, `LLM_ID_INFERENCE`, `LLM_INFERENCE`, `LLM_URL_INFERENCE`

Nightly testing has its own, per-quickstart secret/variable requirements — `PROD_TOKEN`/`PROD_SERVER` are always required, everything else is discovered from that quickstart's `deploy`/`test` targets rather than assumed. See [nightly-actions-catalog.md](./references/nightly-actions-catalog.md) and `subagents/nightly-tests-prompt.md`.

Never commit tokens. Use mock tokens only where workflows explicitly allow (e.g. `HF_TOKEN: mock_hf_token_for_ci` for helm template validation).

Catalog: [references/secrets-and-variables.md](./references/secrets-and-variables.md)
Expand Down Expand Up @@ -187,3 +194,4 @@ When CI workflows are green on a test PR → **`rh-qs-verify-deploy`**, then **`
- [Secrets and variables](./references/secrets-and-variables.md)
- [Makefile CI contract](./references/makefile-ci-contract.md)
- [Workflow skeletons](./references/workflow-skeletons.md) — copy-paste YAML starters
- [Nightly actions catalog](./references/nightly-actions-catalog.md) — `quickstart-ci-testing` actions, reusable test workflow + nightly caller skeletons, provisioning script (used by `subagents/nightly-tests-prompt.md`)
2 changes: 2 additions & 0 deletions core/skills/rh-qs-test-suite/references/composite-actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

Pattern from `it-self-service-agent/.github/actions/`. Prefer composite actions over 50-step workflows so E2E and pip-install PR jobs stay identical.

**These are PR-time actions only.** Nightly testing uses a separate action set copied from `quickstart-ci-testing` (`setup-and-deploy`, `clone-quickstart`, `cleanup`, `run-quickstart-tests`, `run-tests`, `prepare-runner`, `setup-openshift`) — see [nightly-actions-catalog.md](./nightly-actions-catalog.md) and `subagents/nightly-tests-prompt.md`. Both sets live under `.github/actions/` at once without conflict; they back different jobs (ephemeral Kind for PRs vs. real-cluster deploy for nightly). Note the naming collision on `prepare-runner` — the two repos each ship their own version of that action; keep them namespaced correctly and don't let the nightly copy step overwrite the PR one or vice versa.

## prepare-runner

**Path:** `.github/actions/prepare-runner/action.yaml`
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ Examples from it-self-service-agent—replace with design-specific targets:
| `version` | Emit semver for image tags (`build-and-push`) |
| `helm-install-prod` | Nightly against real cluster |

## Nightly contract (canonical names)

`subagents/nightly-tests-prompt.md` expects the canonical names `rh-qs-deploy`/`rh-qs-scaffold` already establish: `deploy` (install), `test`, `undeploy` (uninstall), `verify-deploy` (smoke test). Any quickstart that went through `rh-qs-deploy` should already have all three. For older quickstarts missing one, the subagent searches for an equivalent existing target (e.g. `install-full`, `helm-install-prod`, `e2e-ui`, `helm-uninstall`) rather than renaming anything — it uses whatever real name it finds and records the mapping in `pipeline/nightly-tests-manifest.yaml`.

## Environment variables workflows pass

| Variable | Set by |
Expand Down
145 changes: 145 additions & 0 deletions core/skills/rh-qs-test-suite/references/nightly-actions-catalog.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# Nightly testing: quickstart-ci-testing actions catalog

Used by `subagents/nightly-tests-prompt.md`. Source repo: `rh-ai-quickstart/quickstart-ci-testing`. Reference workflows: `rh-ai-quickstart/ai-supply-chain-agent` — `test-ai-supply-chain-agent.yml`, `nightly-main.yml`, `nightly-development.yml`.

## Why copy instead of reference remotely

`quickstart-ci-testing`'s own README states composite actions cannot be referenced cross-repo (`uses: org/repo/.github/actions/x@ref` doesn't work for composite actions the way it does for whole reusable workflows), so the actions must be copied into `.github/actions/` in the consuming repo. There is no versioning/tagging scheme upstream — copying is a straight directory sync, re-checked (diff + prompt) on every run of the nightly subagent in case the source has changed.

This action set is separate from the `prepare-runner`/`kind`/`inspect` actions `rh-qs-test-suite` may already have added for PR-time Kind E2E (see `composite-actions.md`) — both sets can coexist under `.github/actions/` without conflict; they serve different jobs (ephemeral PR-time Kind vs. real-cluster nightly).

## Actions inventory

| Action | Purpose | Key inputs |
|--------|---------|------------|
| `setup-and-deploy` | Prepares the runner, clones the quickstart at a given ref, deploys it to OpenShift (or Kind, if `use_kind: true`) | `repo_url`, `ref`, `namespace` (default `quickstart-ci-project-1` — override per quickstart), `deploy_command` (required), `setup_commands`, `use_kind`, `build_images`, `build_images_command`, `push_images_command` |
| `clone-quickstart` | Clones the target repo at a ref (called internally by `setup-and-deploy`) | `repo_url`, `ref`, `submodules` |
| `setup-openshift` | `oc` login/context setup for the OpenShift path (called internally by `setup-and-deploy`) | — |
| `prepare-runner` | Frees runner disk space before a deploy/build | `swap-storage` |
| `run-quickstart-tests` | Runs the test command in the cloned quickstart directory, uploads results | `test_command` (required), `test_working_dir` (default `./quickstart`), `pre_test_command`, `upload_results`, `upload_retention_days` (default 2) |
| `run-tests` | Lower-level test execution step called internally by `run-quickstart-tests` | — |
| `cleanup` | Tears down the deployment | `cleanup_command` (required), `working_dir` (default `./quickstart`) |

`setup-and-deploy` validates preconditions before running: `build_images` requires `use_kind: true`; when not using Kind, it errors early if `PROD_TOKEN`/`PROD_SERVER` secrets are missing.

## Reusable test workflow skeleton

`.github/workflows/test-<slug>.yml`:

```yaml
name: Test <slug>

on:
workflow_call:
inputs:
branch:
required: true
type: string
workflow_dispatch:
inputs:
branch:
description: Branch to deploy and test
default: main
type: string

concurrency:
group: <slug>-ci-testing
cancel-in-progress: false

jobs:
test:
name: <slug> (${{ inputs.branch }})
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5

# Toolchain setup steps, derived from this repo's own README/Makefile —
# e.g. astral-sh/setup-uv@v6 pinned to the Makefile's UV_VERSION, only
# if this repo actually uses uv. See "Step 2: Detect toolchain/runtime
# setup requirements" in subagents/nightly-tests-prompt.md.

- name: Setup and deploy
uses: ./.github/actions/setup-and-deploy
timeout-minutes: 15
with:
repo_url: https://github.com/${{ github.repository }}
ref: ${{ inputs.branch }}
namespace: <slug>-ci-project-1
deploy_command: "make deploy"
env:
PROD_TOKEN: ${{ secrets.PROD_TOKEN }}
PROD_SERVER: ${{ secrets.PROD_SERVER }}

- name: Run tests
uses: ./.github/actions/run-quickstart-tests
timeout-minutes: 30
with:
test_command: "make test"
upload_results: "test-results"

- name: Cleanup
if: always()
uses: ./.github/actions/cleanup
with:
cleanup_command: "make undeploy"
```

## Nightly caller skeleton

`.github/workflows/nightly-main.yml` (repeat for the dev branch with a later cron and `branch: <dev-branch>`):

```yaml
name: Nightly - <slug> (main)

on:
workflow_dispatch:
schedule:
- cron: "0 5 * * *"

jobs:
test:
uses: ./.github/workflows/test-<slug>.yml
with:
branch: main
secrets: inherit
```

Stagger cron times across `nightly-main.yml`/`nightly-<dev>.yml` by at least an hour — they share a concurrency group and namespace, so overlapping schedules just queue rather than run in parallel, but staggering keeps failures easy to attribute to one branch.

## Namespace/ServiceAccount provisioning script

`scripts/ci/setup-nightly-namespace.sh` — run by a human against the target cluster, never by an agent:

```bash
#!/usr/bin/env bash
set -euo pipefail

NAMESPACE="${NAMESPACE:-<slug>-ci-project-1}"
SA_NAME="${SA_NAME:-<slug>-ci-1}"
TOKEN_DURATION="${TOKEN_DURATION:-8760h}"

oc create namespace "$NAMESPACE"
oc create sa "$SA_NAME" -n "$NAMESPACE"
oc adm policy add-role-to-user admin \
"system:serviceaccount:${NAMESPACE}:${SA_NAME}" \
-n "$NAMESPACE"

echo
echo "PROD_SERVER: $(oc whoami --show-server)"
echo "PROD_TOKEN:"
oc create token "$SA_NAME" -n "$NAMESPACE" --duration="$TOKEN_DURATION"

echo
echo "Set the two values above as repository secrets PROD_SERVER and PROD_TOKEN"
echo "(Settings -> Secrets and variables -> Actions) before enabling the nightly workflows."
```

This grants **namespace-scoped** `admin` only — not cluster-admin. If the quickstart's chart needs cluster-scoped resources (see the cluster-admin check in `nightly-tests-prompt.md`), this script alone is not sufficient and the nightly subagent's report will say so.

## Toolchain setup — no fixed list either

Like secrets/variables, the runner setup steps before `setup-and-deploy` are not templated from `ai-supply-chain-agent` (its `astral-sh/setup-uv@v6` step is specific to that quickstart being `uv`-based). Each quickstart's `nightly-tests-prompt.md` run first identifies the selected job container or runner image, then derives only missing or version-specific setup from that repo's `README.md`, `Makefile`, lockfiles, and existing PR workflows. A Makefile command alone is not grounds to install a tool: GitHub-hosted `ubuntu-latest`, for example, already provides Helm. See "Step 2: Detect toolchain/runtime setup requirements" in `nightly-tests-prompt.md`.

## Required secrets/variables — no fixed list

`PROD_TOKEN` and `PROD_SERVER` are the only two names ever assumed, because they're structural to `setup-and-deploy`'s cluster-access mechanism, independent of any quickstart's own app. Every other secret/variable a given quickstart's `deploy`/`test` targets need must be discovered per-repo (see "Step 2: Detect required secrets and variables" in `nightly-tests-prompt.md`) — never hardcoded here or assumed from another quickstart's example (e.g. `ai-supply-chain-agent`'s own `MODEL_ID`/`MODEL_URL`/`API_KEY` are specific to that quickstart, not a template to reuse elsewhere).
17 changes: 9 additions & 8 deletions core/skills/rh-qs-test-suite/references/workflow-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,16 +41,17 @@ with:

## Scheduled / nightly

Naming pattern: `nightly-e2e-<scope>-<branch-context>.yml`
Nightly workflows are **not** hand-written per quickstart here — they're generated by `subagents/nightly-tests-prompt.md` (invoked as step 8 of this skill's main workflow, or standalone via the "Nightly-only fast path" in `SKILL.md`). Pattern (from `ai-supply-chain-agent`): one reusable `workflow_call` test workflow, `test-<slug>.yml`, plus one thin scheduled caller per long-lived branch:

| Pattern | Example | Environment |
|---------|---------|-------------|
| Kind + long eval | `nightly-e2e-long-dev.yml` | Checkout `dev`, Kind, `test-long-*` makes |
| Kind + scout prompts | `nightly-e2e-long-scout-prompt-dev.yml` | Variant eval datasets |
| Prod cluster | `nightly-e2e-prod-dev.yml` | `oc login` with `PROD_TOKEN`/`PROD_SERVER`; dedicated namespace; uninstall on success |
| Prod main | `nightly-e2e-prod-main.yml` | Same against `main` |
| File | Trigger | Purpose |
|------|---------|---------|
| `test-<slug>.yml` | `workflow_call`, `workflow_dispatch` | Deploy to a real OpenShift namespace via `setup-and-deploy`, run `make test` via `run-quickstart-tests`, tear down via `cleanup` — actions copied from `quickstart-ci-testing` |
| `nightly-main.yml` | `schedule`, `workflow_dispatch` | Calls `test-<slug>.yml` with `branch: main`, `secrets: inherit` |
| `nightly-<dev-branch>.yml` | `schedule`, `workflow_dispatch` | Same, with `branch: <dev-branch>`, staggered cron so it doesn't collide with the main run in the shared namespace |

**Shared concurrency:** both nightly callers and the reusable workflow itself use the same group (e.g. `<slug>-ci-testing`) with `cancel-in-progress: false`, since they deploy to the same namespace.

**Shared concurrency:** prod deploy workflows use a shared group (e.g. `prod-deploy-shared-namespace`) with `cancel-in-progress: false` to avoid namespace collisions.
Full action inventory, skeletons, and the namespace/ServiceAccount provisioning script: [nightly-actions-catalog.md](./nightly-actions-catalog.md).

## Quickstart mapping (typical)

Expand Down
5 changes: 5 additions & 0 deletions core/skills/rh-qs-test-suite/subagents/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Subagents

| Name | Purpose | Input | Output | When used | Why subagent |
|------|---------|-------|--------|-----------|---------------|
| `nightly-tests-prompt.md` | Generate a reusable `workflow_call` test workflow plus scheduled `nightly-main`/`nightly-<dev>` callers, copying shared composite actions from `quickstart-ci-testing`, a namespace/ServiceAccount provisioning script, and reports on required secrets/variables and toolchain setup | `slug`, `repo_root`, `invocation_mode` (`standalone` or `full-pipeline`) | `pipeline/nightly-tests-manifest.yaml` + a report (toolchain steps used, secrets/variables needed, cluster-admin warning if any, next steps) | Step 8 of the main `rh-qs-test-suite` workflow, or directly via the nightly-only fast path in `SKILL.md` when a quickstart already has CI and just needs nightly testing added or refreshed | Nightly generation is self-contained detection + file-generation logic (Makefile contract resolution, toolchain detection, secret discovery, cluster-admin scan, action copying) that the orchestrator shouldn't have to hold in its own context, and that also needs to be independently invokable without running the rest of the CI build-out |
Loading
Loading