diff --git a/.claude/skills/validate-skill b/.claude/skills/validate-skill new file mode 120000 index 0000000..6239b59 --- /dev/null +++ b/.claude/skills/validate-skill @@ -0,0 +1 @@ +../../core/skills/validate-skill \ No newline at end of file diff --git a/.codex/skills/validate-skill b/.codex/skills/validate-skill new file mode 120000 index 0000000..6239b59 --- /dev/null +++ b/.codex/skills/validate-skill @@ -0,0 +1 @@ +../../core/skills/validate-skill \ No newline at end of file diff --git a/.cursor/skills/validate-skill b/.cursor/skills/validate-skill new file mode 120000 index 0000000..6239b59 --- /dev/null +++ b/.cursor/skills/validate-skill @@ -0,0 +1 @@ +../../core/skills/validate-skill \ No newline at end of file diff --git a/.gemini/skills/validate-skill b/.gemini/skills/validate-skill new file mode 120000 index 0000000..6239b59 --- /dev/null +++ b/.gemini/skills/validate-skill @@ -0,0 +1 @@ +../../core/skills/validate-skill \ No newline at end of file diff --git a/.gitignore b/.gitignore index ea42aaa..46aac84 100644 --- a/.gitignore +++ b/.gitignore @@ -48,4 +48,7 @@ data/ # Temporary files .tmp/ brain.ai -brain.code-workspace \ No newline at end of file +brain.code-workspace + +# Fixture validation reports and captured disposable quickstart output. +test/skills/reports/ diff --git a/core/skills/rh-qs-test-suite/SKILL.md b/core/skills/rh-qs-test-suite/SKILL.md index d469bc8..1538865 100644 --- a/core/skills/rh-qs-test-suite/SKILL.md +++ b/core/skills/rh-qs-test-suite/SKILL.md @@ -17,6 +17,10 @@ description: | - Design doc **`designs/design.md`** defines testing strategy (unit / integration / E2E / evals) — this skill runs inside `.rhoai-qs//` itself (the scaffolded quickstart repo), so the design doc sits right alongside the code, no `../` needed - Optional: replace or extend minimal workflows from **`rh-qs-scaffold`** +## Nightly-only fast path + +If the request is scoped to nightly testing specifically (e.g. "add nightly testing to ``", "update the nightly workflow", "refresh the nightly CI setup") rather than the full CI build-out, skip straight to it: resolve the slug/repo as usual, then invoke **`subagents/nightly-tests-prompt.md`** directly (`invocation_mode: standalone`) and report its result. Do not run the PR-checks/E2E/build-and-push phases for this request — they are unrelated to what was asked. This is the same subagent step 8 below delegates to; the only difference is you stop after it instead of continuing the rest of the workflow. + ## What it does 1. Splits CI into **focused workflows** (quality, unit tests, Kind E2E, evals, image build) instead of one monolithic job @@ -25,7 +29,7 @@ description: | 4. Defines a **Makefile CI contract** so workflows only call `make` targets 5. Documents required **secrets and repository variables** 6. Configures **branch protection** status checks to match workflow job names -7. Optionally adds **nightly** and **dev→main promotion** workflows for mature quickstarts +7. Optionally adds **dev→main promotion** workflows for mature quickstarts, and delegates **nightly** workflow generation to `subagents/nightly-tests-prompt.md` (also invokable on its own — see "Nightly-only fast path" above) ## Workflow @@ -37,7 +41,7 @@ description: | - [ ] 5. Add PR workflows (checks, e2e, branch policy — as needed) - [ ] 6. Configure repo secrets/vars (see references/secrets-and-variables.md) - [ ] 7. Add build-and-push on push to main/dev (if publishing images) -- [ ] 8. Add nightly / promotion workflows only when design requires them +- [ ] 8. When design requires nightly testing, invoke `subagents/nightly-tests-prompt.md`; add dev→main promotion workflow separately if needed - [ ] 9. Register required status checks in branch protection - [ ] 10. Open test PR and confirm all workflows green ``` @@ -65,7 +69,8 @@ description: | | `pr-evaluation-check.yml` | `pull_request_target` | Known-bad conversation / regression eval gate | | `pr-branch-check.yml` | `pull_request` → `main` | Enforce `dev` or `dev-` source branch | | `build-and-push.yaml` | `push` to `main`/`dev` | Build and push images to Quay (org-owned repos only) | -| `nightly-e2e-*.yml` | `schedule`, `workflow_dispatch` | Longer eval suites on Kind or prod cluster | +| `test-.yml` | `workflow_call`, `workflow_dispatch` | Reusable deploy + test + teardown against a real OpenShift namespace, via actions copied from `quickstart-ci-testing` — generated by `subagents/nightly-tests-prompt.md` | +| `nightly-main.yml` / `nightly-.yml` | `schedule`, `workflow_dispatch` | Thin scheduled callers of `test-.yml`, one per long-lived branch — generated by `subagents/nightly-tests-prompt.md` | | `create-dev-to-main-pr.yaml` | `workflow_dispatch` | Version-checked promotion PR with temp branch | Full catalog and naming conventions: [references/workflow-catalog.md](./references/workflow-catalog.md) @@ -124,9 +129,11 @@ Full list: [references/makefile-ci-contract.md](./references/makefile-ci-contrac Configure in GitHub **before** enabling `pull_request_target` E2E workflows: -- **Secrets:** `LLM_API_TOKEN_EVAL`, `LLM_API_TOKEN_INFERENCE`, `QUAY_USERNAME`, `QUAY_PASSWORD`, optional `PROD_TOKEN` / `PROD_SERVER` for prod nightlies +- **Secrets:** `LLM_API_TOKEN_EVAL`, `LLM_API_TOKEN_INFERENCE`, `QUAY_USERNAME`, `QUAY_PASSWORD` - **Variables:** `LLM_URL_EVAL`, `LLM_ID_INFERENCE`, `LLM_INFERENCE`, `LLM_URL_INFERENCE` +Nightly testing has its own, per-quickstart secret/variable requirements — `PROD_TOKEN`/`PROD_SERVER` are always required, everything else is discovered from that quickstart's `deploy`/`test` targets rather than assumed. See [nightly-actions-catalog.md](./references/nightly-actions-catalog.md) and `subagents/nightly-tests-prompt.md`. + Never commit tokens. Use mock tokens only where workflows explicitly allow (e.g. `HF_TOKEN: mock_hf_token_for_ci` for helm template validation). Catalog: [references/secrets-and-variables.md](./references/secrets-and-variables.md) @@ -187,3 +194,4 @@ When CI workflows are green on a test PR → **`rh-qs-verify-deploy`**, then **` - [Secrets and variables](./references/secrets-and-variables.md) - [Makefile CI contract](./references/makefile-ci-contract.md) - [Workflow skeletons](./references/workflow-skeletons.md) — copy-paste YAML starters +- [Nightly actions catalog](./references/nightly-actions-catalog.md) — `quickstart-ci-testing` actions, reusable test workflow + nightly caller skeletons, provisioning script (used by `subagents/nightly-tests-prompt.md`) diff --git a/core/skills/rh-qs-test-suite/references/composite-actions.md b/core/skills/rh-qs-test-suite/references/composite-actions.md index d85bdc7..244ad86 100644 --- a/core/skills/rh-qs-test-suite/references/composite-actions.md +++ b/core/skills/rh-qs-test-suite/references/composite-actions.md @@ -2,6 +2,8 @@ Pattern from `it-self-service-agent/.github/actions/`. Prefer composite actions over 50-step workflows so E2E and pip-install PR jobs stay identical. +**These are PR-time actions only.** Nightly testing uses a separate action set copied from `quickstart-ci-testing` (`setup-and-deploy`, `clone-quickstart`, `cleanup`, `run-quickstart-tests`, `run-tests`, `prepare-runner`, `setup-openshift`) — see [nightly-actions-catalog.md](./nightly-actions-catalog.md) and `subagents/nightly-tests-prompt.md`. Both sets live under `.github/actions/` at once without conflict; they back different jobs (ephemeral Kind for PRs vs. real-cluster deploy for nightly). Note the naming collision on `prepare-runner` — the two repos each ship their own version of that action; keep them namespaced correctly and don't let the nightly copy step overwrite the PR one or vice versa. + ## prepare-runner **Path:** `.github/actions/prepare-runner/action.yaml` diff --git a/core/skills/rh-qs-test-suite/references/makefile-ci-contract.md b/core/skills/rh-qs-test-suite/references/makefile-ci-contract.md index cd0c2f6..df8df2c 100644 --- a/core/skills/rh-qs-test-suite/references/makefile-ci-contract.md +++ b/core/skills/rh-qs-test-suite/references/makefile-ci-contract.md @@ -63,6 +63,10 @@ Examples from it-self-service-agent—replace with design-specific targets: | `version` | Emit semver for image tags (`build-and-push`) | | `helm-install-prod` | Nightly against real cluster | +## Nightly contract (canonical names) + +`subagents/nightly-tests-prompt.md` expects the canonical names `rh-qs-deploy`/`rh-qs-scaffold` already establish: `deploy` (install), `test`, `undeploy` (uninstall), `verify-deploy` (smoke test). Any quickstart that went through `rh-qs-deploy` should already have all three. For older quickstarts missing one, the subagent searches for an equivalent existing target (e.g. `install-full`, `helm-install-prod`, `e2e-ui`, `helm-uninstall`) rather than renaming anything — it uses whatever real name it finds and records the mapping in `pipeline/nightly-tests-manifest.yaml`. + ## Environment variables workflows pass | Variable | Set by | diff --git a/core/skills/rh-qs-test-suite/references/nightly-actions-catalog.md b/core/skills/rh-qs-test-suite/references/nightly-actions-catalog.md new file mode 100644 index 0000000..ec00b9e --- /dev/null +++ b/core/skills/rh-qs-test-suite/references/nightly-actions-catalog.md @@ -0,0 +1,145 @@ +# Nightly testing: quickstart-ci-testing actions catalog + +Used by `subagents/nightly-tests-prompt.md`. Source repo: `rh-ai-quickstart/quickstart-ci-testing`. Reference workflows: `rh-ai-quickstart/ai-supply-chain-agent` — `test-ai-supply-chain-agent.yml`, `nightly-main.yml`, `nightly-development.yml`. + +## Why copy instead of reference remotely + +`quickstart-ci-testing`'s own README states composite actions cannot be referenced cross-repo (`uses: org/repo/.github/actions/x@ref` doesn't work for composite actions the way it does for whole reusable workflows), so the actions must be copied into `.github/actions/` in the consuming repo. There is no versioning/tagging scheme upstream — copying is a straight directory sync, re-checked (diff + prompt) on every run of the nightly subagent in case the source has changed. + +This action set is separate from the `prepare-runner`/`kind`/`inspect` actions `rh-qs-test-suite` may already have added for PR-time Kind E2E (see `composite-actions.md`) — both sets can coexist under `.github/actions/` without conflict; they serve different jobs (ephemeral PR-time Kind vs. real-cluster nightly). + +## Actions inventory + +| Action | Purpose | Key inputs | +|--------|---------|------------| +| `setup-and-deploy` | Prepares the runner, clones the quickstart at a given ref, deploys it to OpenShift (or Kind, if `use_kind: true`) | `repo_url`, `ref`, `namespace` (default `quickstart-ci-project-1` — override per quickstart), `deploy_command` (required), `setup_commands`, `use_kind`, `build_images`, `build_images_command`, `push_images_command` | +| `clone-quickstart` | Clones the target repo at a ref (called internally by `setup-and-deploy`) | `repo_url`, `ref`, `submodules` | +| `setup-openshift` | `oc` login/context setup for the OpenShift path (called internally by `setup-and-deploy`) | — | +| `prepare-runner` | Frees runner disk space before a deploy/build | `swap-storage` | +| `run-quickstart-tests` | Runs the test command in the cloned quickstart directory, uploads results | `test_command` (required), `test_working_dir` (default `./quickstart`), `pre_test_command`, `upload_results`, `upload_retention_days` (default 2) | +| `run-tests` | Lower-level test execution step called internally by `run-quickstart-tests` | — | +| `cleanup` | Tears down the deployment | `cleanup_command` (required), `working_dir` (default `./quickstart`) | + +`setup-and-deploy` validates preconditions before running: `build_images` requires `use_kind: true`; when not using Kind, it errors early if `PROD_TOKEN`/`PROD_SERVER` secrets are missing. + +## Reusable test workflow skeleton + +`.github/workflows/test-.yml`: + +```yaml +name: Test + +on: + workflow_call: + inputs: + branch: + required: true + type: string + workflow_dispatch: + inputs: + branch: + description: Branch to deploy and test + default: main + type: string + +concurrency: + group: -ci-testing + cancel-in-progress: false + +jobs: + test: + name: (${{ inputs.branch }}) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + # Toolchain setup steps, derived from this repo's own README/Makefile — + # e.g. astral-sh/setup-uv@v6 pinned to the Makefile's UV_VERSION, only + # if this repo actually uses uv. See "Step 2: Detect toolchain/runtime + # setup requirements" in subagents/nightly-tests-prompt.md. + + - name: Setup and deploy + uses: ./.github/actions/setup-and-deploy + timeout-minutes: 15 + with: + repo_url: https://github.com/${{ github.repository }} + ref: ${{ inputs.branch }} + namespace: -ci-project-1 + deploy_command: "make deploy" + env: + PROD_TOKEN: ${{ secrets.PROD_TOKEN }} + PROD_SERVER: ${{ secrets.PROD_SERVER }} + + - name: Run tests + uses: ./.github/actions/run-quickstart-tests + timeout-minutes: 30 + with: + test_command: "make test" + upload_results: "test-results" + + - name: Cleanup + if: always() + uses: ./.github/actions/cleanup + with: + cleanup_command: "make undeploy" +``` + +## Nightly caller skeleton + +`.github/workflows/nightly-main.yml` (repeat for the dev branch with a later cron and `branch: `): + +```yaml +name: Nightly - (main) + +on: + workflow_dispatch: + schedule: + - cron: "0 5 * * *" + +jobs: + test: + uses: ./.github/workflows/test-.yml + with: + branch: main + secrets: inherit +``` + +Stagger cron times across `nightly-main.yml`/`nightly-.yml` by at least an hour — they share a concurrency group and namespace, so overlapping schedules just queue rather than run in parallel, but staggering keeps failures easy to attribute to one branch. + +## Namespace/ServiceAccount provisioning script + +`scripts/ci/setup-nightly-namespace.sh` — run by a human against the target cluster, never by an agent: + +```bash +#!/usr/bin/env bash +set -euo pipefail + +NAMESPACE="${NAMESPACE:--ci-project-1}" +SA_NAME="${SA_NAME:--ci-1}" +TOKEN_DURATION="${TOKEN_DURATION:-8760h}" + +oc create namespace "$NAMESPACE" +oc create sa "$SA_NAME" -n "$NAMESPACE" +oc adm policy add-role-to-user admin \ + "system:serviceaccount:${NAMESPACE}:${SA_NAME}" \ + -n "$NAMESPACE" + +echo +echo "PROD_SERVER: $(oc whoami --show-server)" +echo "PROD_TOKEN:" +oc create token "$SA_NAME" -n "$NAMESPACE" --duration="$TOKEN_DURATION" + +echo +echo "Set the two values above as repository secrets PROD_SERVER and PROD_TOKEN" +echo "(Settings -> Secrets and variables -> Actions) before enabling the nightly workflows." +``` + +This grants **namespace-scoped** `admin` only — not cluster-admin. If the quickstart's chart needs cluster-scoped resources (see the cluster-admin check in `nightly-tests-prompt.md`), this script alone is not sufficient and the nightly subagent's report will say so. + +## Toolchain setup — no fixed list either + +Like secrets/variables, the runner setup steps before `setup-and-deploy` are not templated from `ai-supply-chain-agent` (its `astral-sh/setup-uv@v6` step is specific to that quickstart being `uv`-based). Each quickstart's `nightly-tests-prompt.md` run first identifies the selected job container or runner image, then derives only missing or version-specific setup from that repo's `README.md`, `Makefile`, lockfiles, and existing PR workflows. A Makefile command alone is not grounds to install a tool: GitHub-hosted `ubuntu-latest`, for example, already provides Helm. See "Step 2: Detect toolchain/runtime setup requirements" in `nightly-tests-prompt.md`. + +## Required secrets/variables — no fixed list + +`PROD_TOKEN` and `PROD_SERVER` are the only two names ever assumed, because they're structural to `setup-and-deploy`'s cluster-access mechanism, independent of any quickstart's own app. Every other secret/variable a given quickstart's `deploy`/`test` targets need must be discovered per-repo (see "Step 2: Detect required secrets and variables" in `nightly-tests-prompt.md`) — never hardcoded here or assumed from another quickstart's example (e.g. `ai-supply-chain-agent`'s own `MODEL_ID`/`MODEL_URL`/`API_KEY` are specific to that quickstart, not a template to reuse elsewhere). diff --git a/core/skills/rh-qs-test-suite/references/workflow-catalog.md b/core/skills/rh-qs-test-suite/references/workflow-catalog.md index 5265abd..7b7eca8 100644 --- a/core/skills/rh-qs-test-suite/references/workflow-catalog.md +++ b/core/skills/rh-qs-test-suite/references/workflow-catalog.md @@ -41,16 +41,17 @@ with: ## Scheduled / nightly -Naming pattern: `nightly-e2e--.yml` +Nightly workflows are **not** hand-written per quickstart here — they're generated by `subagents/nightly-tests-prompt.md` (invoked as step 8 of this skill's main workflow, or standalone via the "Nightly-only fast path" in `SKILL.md`). Pattern (from `ai-supply-chain-agent`): one reusable `workflow_call` test workflow, `test-.yml`, plus one thin scheduled caller per long-lived branch: -| Pattern | Example | Environment | -|---------|---------|-------------| -| Kind + long eval | `nightly-e2e-long-dev.yml` | Checkout `dev`, Kind, `test-long-*` makes | -| Kind + scout prompts | `nightly-e2e-long-scout-prompt-dev.yml` | Variant eval datasets | -| Prod cluster | `nightly-e2e-prod-dev.yml` | `oc login` with `PROD_TOKEN`/`PROD_SERVER`; dedicated namespace; uninstall on success | -| Prod main | `nightly-e2e-prod-main.yml` | Same against `main` | +| File | Trigger | Purpose | +|------|---------|---------| +| `test-.yml` | `workflow_call`, `workflow_dispatch` | Deploy to a real OpenShift namespace via `setup-and-deploy`, run `make test` via `run-quickstart-tests`, tear down via `cleanup` — actions copied from `quickstart-ci-testing` | +| `nightly-main.yml` | `schedule`, `workflow_dispatch` | Calls `test-.yml` with `branch: main`, `secrets: inherit` | +| `nightly-.yml` | `schedule`, `workflow_dispatch` | Same, with `branch: `, staggered cron so it doesn't collide with the main run in the shared namespace | + +**Shared concurrency:** both nightly callers and the reusable workflow itself use the same group (e.g. `-ci-testing`) with `cancel-in-progress: false`, since they deploy to the same namespace. -**Shared concurrency:** prod deploy workflows use a shared group (e.g. `prod-deploy-shared-namespace`) with `cancel-in-progress: false` to avoid namespace collisions. +Full action inventory, skeletons, and the namespace/ServiceAccount provisioning script: [nightly-actions-catalog.md](./nightly-actions-catalog.md). ## Quickstart mapping (typical) diff --git a/core/skills/rh-qs-test-suite/subagents/README.md b/core/skills/rh-qs-test-suite/subagents/README.md new file mode 100644 index 0000000..4f54f7f --- /dev/null +++ b/core/skills/rh-qs-test-suite/subagents/README.md @@ -0,0 +1,5 @@ +# Subagents + +| Name | Purpose | Input | Output | When used | Why subagent | +|------|---------|-------|--------|-----------|---------------| +| `nightly-tests-prompt.md` | Generate a reusable `workflow_call` test workflow plus scheduled `nightly-main`/`nightly-` callers, copying shared composite actions from `quickstart-ci-testing`, a namespace/ServiceAccount provisioning script, and reports on required secrets/variables and toolchain setup | `slug`, `repo_root`, `invocation_mode` (`standalone` or `full-pipeline`) | `pipeline/nightly-tests-manifest.yaml` + a report (toolchain steps used, secrets/variables needed, cluster-admin warning if any, next steps) | Step 8 of the main `rh-qs-test-suite` workflow, or directly via the nightly-only fast path in `SKILL.md` when a quickstart already has CI and just needs nightly testing added or refreshed | Nightly generation is self-contained detection + file-generation logic (Makefile contract resolution, toolchain detection, secret discovery, cluster-admin scan, action copying) that the orchestrator shouldn't have to hold in its own context, and that also needs to be independently invokable without running the rest of the CI build-out | diff --git a/core/skills/rh-qs-test-suite/subagents/nightly-tests-prompt.md b/core/skills/rh-qs-test-suite/subagents/nightly-tests-prompt.md new file mode 100644 index 0000000..5b03130 --- /dev/null +++ b/core/skills/rh-qs-test-suite/subagents/nightly-tests-prompt.md @@ -0,0 +1,177 @@ +--- +description: Generate reusable nightly test workflows (main + dev) for a quickstart, copying shared actions from quickstart-ci-testing +--- + +# Nightly Tests + +## Your Role + +You add scheduled nightly testing to an AI Quickstart: a reusable `workflow_call` test workflow that deploys the quickstart to a real OpenShift namespace, runs its test suite, and tears it down again, plus two thin scheduled workflows (`nightly-main.yml`, `nightly-.yml`) that call it. You are invoked either as part of the larger `rh-qs-test-suite` pipeline (step 8) or standalone, when a quickstart already has CI and someone just wants nightly testing added or refreshed. + +You run with your working directory at the root of the quickstart's own repo (inside `.rhoai-qs//`, which is that repo's checkout). Every path below is relative to that root. + +**Reference implementation:** `rh-ai-quickstart/ai-supply-chain-agent` — `.github/workflows/test-ai-supply-chain-agent.yml`, `nightly-main.yml`, `nightly-development.yml`. +**Shared actions source:** `rh-ai-quickstart/quickstart-ci-testing` — `.github/actions/` (`setup-and-deploy`, `clone-quickstart`, `cleanup`, `run-quickstart-tests`, `run-tests`, `prepare-runner`, `setup-openshift`, and the `setup-kind` variant `setup-and-deploy` references). Its own README is explicit that these actions must be **copied** into the consuming repo, not referenced remotely (composite actions can't be referenced cross-repo). Before generating a workflow or provisioning script, read `references/nightly-actions-catalog.md`; it is the canonical source for the action inventory and static file skeletons. + +You do **not** run `oc`, `kubectl`, or `gh secret set` at any point (per `rh-qs-secure`). You generate files and a script; a human runs the script and sets the secrets it names. + +## Instructions + +**Input Parameters:** +- `slug`: the quickstart's slug (for naming the namespace, ServiceAccount, and reusable workflow file) +- `repo_root`: absolute or relative path to the quickstart repo root (your working directory) +- `invocation_mode`: `"standalone"` or `"full-pipeline"` — only affects your final report framing, not what you do + +### Step 1: Resolve the Makefile contract + +Read the `Makefile`. Look for the canonical targets the pipeline already establishes via `rh-qs-deploy`/`rh-qs-scaffold`: + +- `deploy` (install), `undeploy` (uninstall), `verify-deploy` (smoke test), `test` + +If all of `deploy`/`test`/`undeploy` exist, use those names directly — this is the expected case for any quickstart that went through `rh-qs-deploy`. + +If one or more is missing, search the Makefile for an equivalent target before assuming nothing exists — look for names like `install-full`, `helm-install-prod`, `helm-install`, `e2e-ui`, `helm-uninstall`, `test-all`. If you find a clear equivalent, use its real name (do not rename it) and record the mapping. If no clear equivalent exists for a given step, ask the user to provide the target name to use for that step and wait for their answer before generating workflows. Do not fabricate a target, silently skip the step, or claim completion without the user's answer. + +Record the three resolved target names (or the missing-step error) — you need them for Steps 4, 5, and 8. + +### Step 2: Detect toolchain/runtime setup requirements + +Before adding a setup action, identify the workflow's execution environment and then determine what this specific repo still needs. Do not infer that every command invoked by `deploy` or `test` needs an installer: the declared job container, self-hosted runner image, or GitHub-hosted runner may already provide it. For example, GitHub-hosted `ubuntu-latest` already provides Helm; do not add `azure/setup-helm` merely because a Makefile invokes `helm`. + +Check, in order: + +1. **Execution environment** — inspect the reusable-workflow skeleton plus existing workflows for `container:`, `runs-on`, or a self-hosted runner label. Treat a declared image or runner configuration that documents a tool as authoritative evidence that the tool is already available. +2. **`README.md`** — any "Prerequisites"/"Requirements"/"Getting started" section naming specific tool versions (Python, Node, `uv`, `pnpm`, Helm, Go, etc.). +3. **The Makefile** — pinned version variables the way `it-self-service-agent` pins `UV_VERSION` (grep for `_VERSION` assignments), and any target that itself installs a toolchain (e.g. a `check-uv-version` target, `go install ...`, `npm install -g pnpm`). Check what the resolved `deploy`/`test` targets invoke — `uv run`, `pnpm`, `helm`, `go run`, plain `python3` — as evidence of a dependency, not automatically of a required setup action. +4. **Lockfiles present at the repo root** (`uv.lock`, `pnpm-lock.yaml`, `package-lock.json`, `go.sum`) as a fallback signal if README/Makefile don't pin a version explicitly. +5. **Existing PR workflows** (`.github/workflows/pr-checks.yml` etc.) — they already solved this problem for the same repo; reuse their exact setup steps and pinned versions rather than re-deriving from scratch when they exist. This is the highest-confidence source. + +Add a concrete `uses:`/`with:` step only when the selected environment does not provide the tool, or when the repo requires a specific version or setup not supplied by that environment. For example, add `astral-sh/setup-uv@v6` with a Makefile-pinned `UV_VERSION`, or `actions/setup-node@v4` with the version from `.nvmrc`/`package.json` `engines`. Do not add a setup step for a tool the repo does not use or one already provided by the selected environment. Do not invent a version or add an unpinned setup action as a fallback; instead, use the provided tool and record any version uncertainty in `unconfirmed` with its environment source. + +Record the concrete list of setup steps you'll insert into the reusable workflow — you need it for Step 8. + +The manifest's `unconfirmed` section must contain only unresolved toolchain or secret/variable findings. If a concrete toolchain setup step is selected, do not retain a generic "No toolchain setup step added" entry; that statement is valid only when `toolchain_setup_steps` is empty. + +### Step 3: Detect required secrets and variables + +Beyond `PROD_TOKEN` and `PROD_SERVER` (which are always required — see Step 7), you must **discover** what this specific quickstart's `deploy`/`test` targets need. Do not assume any names beyond those two; every other quickstart has different requirements. Check, in order of confidence: + +1. **Existing workflows** (`.github/workflows/*.yml`) — `secrets.*`/`vars.*` references in any job that already calls the resolved `deploy`/`test`/`undeploy` targets or their underlying make/helm commands. Highest confidence: if `pr-e2e-tests.yml` already requires something to run the same targets, nightly needs it too. +2. **The Makefile bodies** of the resolved `deploy`/`test`/`undeploy` targets — grep for `$(VAR)` / `${VAR}` references. +3. **Helm `values.yaml`** (and any `secrets.example.yaml`) in the chart(s) `deploy` installs — keys with no default that look like they must be supplied at install time. +4. **`.env.example`**, if present. + +For each name found, classify it as a **secret** (matches `*TOKEN*`, `*KEY*`, `*PASSWORD*`, `*SECRET*`) or a **variable** (everything else), and write a one-line explanation of what it's for, grounded in *where* you found it (e.g. "referenced in the `test` Makefile target", "already required by `pr-e2e-tests.yml` for the same target", "no default in `values.yaml`, passed to the chart at install"). If a reference is ambiguous (referenced once, purpose unclear), list it separately as "found but unconfirmed" rather than silently including or dropping it. + +Also record a target-to-environment map: which detected names are required by +`deploy`, `test`, and `undeploy` respectively. Preserve this mapping through +workflow generation; a name used by more than one target must be available to +each corresponding workflow step. + +You will wire the confidently-classified names into the reusable workflow in Step 8, and report the full list (names + explanations, never values) in your final output. + +### Step 4: Cluster-admin requirement check + +The ServiceAccount the provisioning script creates (Step 7) only gets namespace-scoped `admin` in its own CI namespace — not cluster-admin. Check whether `deploy` needs more than that: + +1. Scan every Helm chart `deploy` installs (including `ai-architecture-charts` subcharts it depends on) for cluster-scoped resource kinds in `templates/`: `ClusterRole`, `ClusterRoleBinding`, `CustomResourceDefinition`, `SecurityContextConstraints`, `ClusterPolicy`, `Subscription`, `OperatorGroup`, `Namespace`, `PriorityClass`, `APIService`. +2. Scan the `deploy`/`test`/`undeploy` Makefile target bodies for direct `oc adm`, `oc create clusterrole*`, or other admin-scoped commands beyond a plain namespace-scoped `helm upgrade --install`. + +If nothing is found, record "No cluster-admin requirement detected." If something is found, record exactly which file/resource triggered it — you will surface this as a warning in your final report (Step 9), but **still generate every workflow and the provisioning script regardless of this result.** This is a warning for the human, not a gate. + +### Step 5: Detect the branch model + +Determine the default branch (normally `main`) and whether a second long-lived branch exists (`dev`, `development`, or `develop` — check for all three; use whichever exists). If no second branch exists, you will generate only `nightly-main.yml` in Step 9 and note in your report that the dev nightly was skipped. + +### Step 6: Copy composite actions from quickstart-ci-testing + +Shallow-clone `https://github.com/rh-ai-quickstart/quickstart-ci-testing` to a temp directory and copy its entire `.github/actions/` tree into this repo's `.github/actions/`, one subdirectory at a time: + +- If a given action directory (e.g. `.github/actions/setup-and-deploy/`) does not exist here → copy it in full. +- If it already exists here → diff it against the freshly cloned source. If identical, skip and note "already up to date." If different, show the user the diff and ask whether to overwrite, keep the existing version, or view more context before deciding. Record the outcome per action. + +Clean up the temp clone when done. + +### Step 7: Generate the namespace/ServiceAccount provisioning script + +Before writing anything, compute `NAMESPACE_VALUE="-ci-project-1"` and `SA_VALUE="-ci-1"` directly from the `slug` **input parameter** given to you — never from `helm/Chart.yaml`'s `name:`, a directory name, or any other repo artifact; those commonly differ from the quickstart's actual slug and will provision the wrong namespace/ServiceAccount if used instead. + +Write `scripts/ci/setup-nightly-namespace.sh` from the **Namespace/ServiceAccount provisioning script** skeleton in `references/nightly-actions-catalog.md`; apply the same skip/diff-and-prompt rule as Step 6 if it already exists, substituting `NAMESPACE_VALUE`/`SA_VALUE` for the `-ci-project-1`/`-ci-1` defaults. This script is for the human to run themselves — you never execute it. `NAMESPACE_VALUE` is the single source of truth: reuse this exact literal verbatim for the `namespace` input to `setup-and-deploy` in Step 8 and for `namespace`/`service_account` in the manifest (Step 10) — do not re-substitute `` independently in those later steps, so the three can never drift apart. + +### Step 8: Generate the reusable test workflow + +Write `.github/workflows/test-.yml` from the **Reusable test workflow skeleton** in `references/nightly-actions-catalog.md`; apply the same skip/diff-and-prompt rule as Step 6 if it already exists. Use the actual resolved target names from Step 1 if they differ from `deploy`/`test`/`undeploy`, replace the toolchain placeholder with the concrete steps from Step 2, and wire detected values according to the target-to-environment map from Step 3: + +- Values required by `deploy` go in `setup-and-deploy.env`. +- Values required by `test` go in `run-quickstart-tests.env`. +- Values required by `undeploy` go in `cleanup.env`. + +Do not put application secrets at job scope unless every step genuinely needs +them; preserve step-level scoping. A value used by both deploy and test must +appear in both step-level `env:` blocks. Use `NAMESPACE_VALUE` computed in Step +7 verbatim for the `namespace` input — do not re-substitute +`-ci-project-1` independently here. Preserve the catalog's concurrency +block. + +Before reporting completion, statically verify that every variable referenced +by each resolved Make target is present in the corresponding workflow action's +`env:` block. Treat a missing mapping as a generation error, not as an +unconfirmed finding. + +### Step 9: Generate the nightly scheduling workflows + +Write `.github/workflows/nightly-main.yml` and, only if Step 5 found a dev branch, `.github/workflows/nightly-.yml` from the **Nightly caller skeleton** in `references/nightly-actions-catalog.md`; apply the same skip/diff-and-prompt rule as Step 6 if they already exist. Use the detected default branch, and stagger the dev workflow's cron by at least an hour so it does not collide on the shared namespace/concurrency group. + +### Step 10: Write the manifest + +Write `pipeline/nightly-tests-manifest.yaml`: + +```yaml +slug: +resolved_targets: + deploy: + test: + undeploy: +toolchain_setup_steps: + - action: + with: { ... } + source: +branch_model: + main: main + dev: +namespace: +service_account: +cluster_admin_check: + status: clean | flagged + detail: +required_secrets: + - name: PROD_TOKEN + reason: ServiceAccount token for the nightly CI namespace + - name: PROD_SERVER + reason: OpenShift API server URL for the cluster nightly deploys to + # ...detected entries from Step 3 +required_variables: + # ...detected entries from Step 3 +unconfirmed: + # ...ambiguous entries from Step 2 (toolchain) and Step 3 (secrets), if any +files: + actions_copied: [...] + actions_skipped_identical: [...] + actions_updated: [...] + workflows_created: [...] + workflows_skipped_or_updated: [...] + provisioning_script: scripts/ci/setup-nightly-namespace.sh +``` + +### Step 11: Report to the user + +Summarize what was created/updated/skipped, then always include: + +1. **Cluster-admin check result** — if flagged, lead with the warning (file/resource + consequence), even though the workflows were created anyway. +2. **Toolchain setup steps used**, and their source — flag any that fell back to a tool's default version because nothing in the repo pinned one, so the user can confirm it's correct. +3. **Required secrets/variables**, names and explanations only, split into Secrets and Variables, plus any "found but unconfirmed" entries — never invent or ask for actual values. +4. **Next steps**: run `scripts/ci/setup-nightly-namespace.sh` against the target cluster, set the printed `PROD_TOKEN`/`PROD_SERVER` plus every other listed secret/variable in GitHub, then trigger `test-.yml` via `workflow_dispatch` once to confirm it's green before relying on the schedule. + +## Output + +`pipeline/nightly-tests-manifest.yaml` (schema above) plus the report described in Step 11, returned as your final message to the orchestrator. diff --git a/core/skills/validate-skill/SKILL.md b/core/skills/validate-skill/SKILL.md new file mode 100644 index 0000000..c78eb1f --- /dev/null +++ b/core/skills/validate-skill/SKILL.md @@ -0,0 +1,188 @@ +--- +name: validate-skill +description: Run fixture-based behavioral validations for a Factory skill and rate each output against its expected-output-repo baseline. +allowed-tools: Bash, Read, Write, Edit, Agent +--- + +# validate-skill + +## Goal + +Validate a Factory skill by running fixture cases under `test/skills//`. By default it runs every case; an optional comma-separated case list runs only those named cases. Each case is copied into the disposable quickstart path `.rhoai-qs/qs-test-quickstart/`, executed through the target skill, and rated from 1 to 10 against its approved, complete post-run repository baseline. + +This skill tests skill behavior. It does not replace unit tests for deterministic scripts, and it does not deploy to a cluster unless the target skill's fixture explicitly requires a human-approved operational test. + +## Mandatory execution rule + +When the user invokes `validate-skill`, run the complete validation workflow below. Do not manually simulate the target skill, synthesize generated output, or assign a validation score from inspection alone. A completed result is valid only when: + +1. The selected fixture was prepared with `prepare_fixture.py`. +2. The target skill was actually executed through the case runner. +3. The generated quickstart output was compared with `compare_baseline.py`. +4. The independent output rater evaluated the result. + +If the case runner or independent output rater cannot be invoked, report the case as `blocked` or `failed`; never mark it passed and never create a substitute score or fabricated validation artifacts. Targeted validation narrows what the rater evaluates, but it does not permit skipping target-skill execution or required target-skill behavior outside the targeted assertion. + +When the workflow is delegated through an external runner, preserve the caller's authenticated execution context. Do not replace authentication-related environment variables with a fresh temporary directory; the case runner must use the existing authenticated context. + +## Input + +- Target skill name, for example `rh-qs-test-suite`. +- One or more test directories in `test/skills//` following [the fixture contract](./references/fixture-contract.md). Each has a `test-config.yaml` with an optional target-skill prompt, question-answer context, and `validation-options` for evaluation behavior. +- Optional test filter, for example `--tests standard-pattern,cluster-privileges`. Test names must exactly match immediate fixture-directory names. Omit it to run every case. + +## Supporting Documents + +**Main agent reads directly:** + +| File | When | +|---|---| +| [reasoning-guardrails.md](./reasoning-guardrails.md) | Before copying fixtures, answering questions, or rating output | +| [spec-template.md](./spec-template.md) | Before recording the validation plan | +| [output-templates.md](./output-templates.md) | Before writing the validation report | +| [references/fixture-contract.md](./references/fixture-contract.md) | Before discovering cases | +| [scripts/discover_cases.py](./scripts/discover_cases.py) | Phase 1 case discovery and validation | +| [scripts/preflight_dependencies.py](./scripts/preflight_dependencies.py) | Before a network-enabled case runner | +| [scripts/prepare_fixture.py](./scripts/prepare_fixture.py) | Before each case, to reset and copy its fixture | +| [scripts/run_case.py](./scripts/run_case.py) | Synchronous case-runner gate and completion manifest | +| [scripts/compare_baseline.py](./scripts/compare_baseline.py) | Before independent output rating | +| [scripts/archive_fixture.py](./scripts/archive_fixture.py) | After each completed case, to retain its generated quickstart tree for debugging | +| [scripts/write_conversation.py](./scripts/write_conversation.py) | After each case, to retain caller-to-target-skill messages | +| [scripts/write_report.py](./scripts/write_report.py) | Phase 4 report rendering | +| [scripts/orchestrate_validation.py](./scripts/orchestrate_validation.py) | Preferred deterministic coordinator for the complete workflow | +| [runners/](./runners/) | Foreground Codex, Claude, Cursor, and Gemini adapters with a shared execution contract | + +**Subagents read (pass by path only):** + +| File | Subagent | +|---|---| +| [subagents/validation-skill-prompt.md](./subagents/validation-skill-prompt.md) | Verifies the fixed disposable quickstart context | +| [subagents/case-runner-prompt.md](./subagents/case-runner-prompt.md) | Executes one case through the target skill | +| [subagents/output-rater-prompt.md](./subagents/output-rater-prompt.md) | Independently compares actual and expected output | + +## Workflow + +### Preferred execution path + +Use `scripts/orchestrate_validation.py` to coordinate the phases below. It still +delegates target-skill execution and independent rating to the supplied commands; +the Python coordinator owns ordering, fixture isolation, evidence capture, schema +validation, archives, and reports. + +The runner and rater can be supplied as JSON command arrays, or selected through +the foreground adapters in `runners/`. Adapter execution is synchronous and +inherits the caller's authenticated environment. Adapters do not silently use +background agents or broaden filesystem access. Use `--runner-adapter` and +`--runner-prompt-file` (and the corresponding `--rater-*` options) when the +client adapter should own command construction. Raw command arrays remain +supported for custom runners. + +The coordinator supports these placeholders in raw command arguments: + +- `{runner_output}` — runner result YAML path +- `{rater_output}` — independent-rater result YAML path +- `{case_path}` — absolute fixture case path +- `{quickstart_root}` — absolute disposable quickstart path +- `{comparison_json}` — baseline comparison JSON path +- `{conversation_json}` — runner conversation JSON path + +Adapter prompt files are passed through as written. The coordinator appends +the case-specific inputs to each prompt; placeholders in the prompt file are +not expanded. + +Unknown or unresolved placeholders in raw command arguments are rejected. The +runner command is responsible for providing the execution capabilities required +by the selected fixtures: + +- For any case whose fixture declares `execution.network: required`, invoke the + coordinator with `--network-profile network` and run the case runner in a + network-capable context. The profile is a validation declaration; it cannot + grant network access by itself. +- The runner must have read/write access to the repository root and the fixed + disposable path `.rhoai-qs/qs-test-quickstart/`. Use the supplied + `{quickstart_root}` placeholder when the runner needs the absolute path. +- Do not use an isolated runner for a network-required case, and do not replace + the disposable path with a temporary or user-selected quickstart path. + +Example shape (with the model-specific commands supplied by the caller): + +```bash +python3 core/skills/validate-skill/scripts/orchestrate_validation.py \ + --tests \ + --network-profile network \ + --runner-command-json '' \ + --rater-command-json '' +``` + +Adapter form: + +```bash +python3 core/skills/validate-skill/scripts/orchestrate_validation.py \ + --tests \ + --runner-adapter codex \ + --runner-prompt-file core/skills/validate-skill/subagents/case-runner-prompt.md \ + --rater-adapter codex \ + --rater-prompt-file core/skills/validate-skill/subagents/output-rater-prompt.md +``` + +For a network-required case, the adapter receives the declared dependency +hostnames. The adapter's execution environment must enforce any domain +allowlist; an environment variable alone is not a network security boundary. + +The coordinator writes per-case evidence under +`test/skills/reports///` and an aggregate result/report +under `test/skills/reports//`. + +### Phase 0: Verify disposable context + +Spawn the validation subagent with the fixed target slug `qs-test-quickstart`, the requested target skill, and the case root. It confirms the target is the test-only path `.rhoai-qs/qs-test-quickstart/`, rather than a user quickstart. Stop if the target name or fixture path differs. + +### Phase 1: Discover and validate cases + +1. Run `python3 core/skills/validate-skill/scripts/discover_cases.py [--tests ,]`. It resolves the target skill, selects all cases or only the named tests, finds them in lexical order, and validates their required inputs. Stop if it reports malformed or unknown selected tests. Do not validate or run unselected cases. +2. Read the emitted JSON instead of manually enumerating or re-checking case paths. +3. For a case with `execution.network: required`, run `python3 core/skills/validate-skill/scripts/preflight_dependencies.py `. Preserve its JSON result as execution evidence. A failed preflight is an infrastructure warning, not permission to skip target execution; run the case runner in a network-enabled execution context and classify the case as `blocked` or `failed` if the runner cannot access the dependency. +4. Read `spec-template.md` and write `test/skills/reports///validation-spec.yaml` for each discovered case, recording the target skill, the case, the fixed quickstart slug `qs-test-quickstart`, its baseline path, and execution requirements. + +### Phase 2: Run each case in isolation + +For each case, in deterministic lexical order: + +1. Run `python3 core/skills/validate-skill/scripts/prepare_fixture.py `. It removes only `.rhoai-qs/qs-test-quickstart/`, recreates it, copies the contents of `input-repo/`, and emits an inventory manifest. +2. Read the emitted inventory; do not repeat its file-system checks manually. +3. Invoke `python3 core/skills/validate-skill/scripts/run_case.py` with the case path, fixed quickstart root, a runner-result path, a completion-record path, and the case-runner command. Use `{runner_output}`, `{quickstart_root}`, and `{case_path}` as command placeholders where needed. The script must block until the runner process exits, require its result status to be `completed`, compute the final output manifest and tree hash itself, and return non-zero otherwise. When `execution.network` is `required`, launch the runner with network access appropriate for the declared dependencies and with access to `.rhoai-qs/qs-test-quickstart/`; do not silently retry in an isolated context or change the disposable path. When `run_skill_command` is absent, invoke the target skill without adding a supplemental user request. +4. If the target skill asks a question, use `question_answer_context` only when it provides an unambiguous, fixture-grounded answer. For a user preference, overwrite decision, credential, cluster action, GitHub mutation, or any other external authority, stop that case and record `blocked` with the question. Never invent credentials or consent. +5. Capture the runner's final report, generated files, blocked question, and chronological `conversation` messages. Retain every caller prompt, target-skill response or question, and any answer supplied from `question_answer_context`; do not include tool output or internal reasoning. + +Do not replace the case runner with hand-authored files in `.rhoai-qs/qs-test-quickstart/`. The target skill must produce the output being evaluated, including files that are outside a targeted assertion. + +### Phase 3: Rate the output + +For each case whose `run_case.py` completion record has status `completed`, run `python3 core/skills/validate-skill/scripts/compare_baseline.py `. Pass its JSON output, `validation_mode`, the optional `additional_validation_instructions` from `test-config.yaml`'s `validation-options` mapping, and the runner's recorded `conversation` to the independent output rater with [subagents/output-rater-prompt.md](./subagents/output-rater-prompt.md), along with the expected baseline and actual quickstart root. Pass the conversation as observable caller-to-target evidence, not as the runner's self-assessment. The rater judges the materiality of reported differences; it does not rediscover file inventories or hashes. + +The rater returns a 1–10 score with evidence. In the default `full-baseline` mode, a score of 10 requires the complete actual repository tree to match the approved post-run baseline, except paths explicitly excluded by the case's `comparison-ignore`. In `targeted` mode, score only the checks in `additional_validation_instructions`; do not penalize baseline-tree differences outside that scope. A blocked or failed execution has no quality score; record it as `blocked` or `failed`, not as a misleading low score. + +Never author `validation-results.json` as a substitute for the runner or rater. Its score and evidence must be copied from the completed execution and independent rating. + +After rating a completed case, add its runner `conversation` to `validation-results.json`, then run `python3 core/skills/validate-skill/scripts/write_conversation.py --input --case --output test/skills/reports///skill-conversation.md`. Also run `python3 core/skills/validate-skill/scripts/archive_fixture.py --output test/skills/reports///quickstart-output.tar.gz`. The archive is a debugging artifact of `.rhoai-qs/qs-test-quickstart/`; never use it as a baseline or input to another case. + +### Phase 4: Report + +Read `output-templates.md`, assemble each case result as `test/skills/reports///validation-results.json`, and run `python3 core/skills/validate-skill/scripts/write_report.py --input --output test/skills/reports///validation-report.md`. Each completed-case directory also contains `quickstart-output.tar.gz` from Phase 3. In the final user message, lead with the aggregate result and identify any case scoring below 8. + +## Guidelines + +- Use `test/skills/`, not `test/skils/`. +- Treat `test-config.yaml`'s `run-skill-command` as a prompt to the target skill, never as shell code. +- Do not modify `input-repo/` or `expected-output-repo/` during a run. Generated output belongs only in `.rhoai-qs/qs-test-quickstart/`. +- Do not let one case's files, answers, or state affect another case. +- Persist only caller-to-target-skill messages in `skill-conversation.md`. Do not store internal reasoning, tool output, credentials, or unredacted secrets. +- Preserve the target skill's normal safety rules. This validator cannot authorize cluster access, secret changes, GitHub writes, or destructive overwrites. +- In `full-baseline` mode, apply additional validation instructions only when they are compatible with the fixture contract and safety rules; they cannot waive a required baseline artifact. In `targeted` mode, they define the exclusive evaluation scope. Neither mode authorizes external actions. +- Use the complete post-run baseline as the primary oracle in `full-baseline` mode. In `targeted` mode, use only the explicit additional validation instructions as the oracle. + +## Success Criteria + +- Every well-formed case is either rated with evidence or explicitly reported as blocked/failed. +- Each completed case has an independent 1–10 rating grounded in `expected-output-repo/`. +- The aggregate report is written without modifying the fixture inputs or expected baselines. diff --git a/core/skills/validate-skill/output-templates.md b/core/skills/validate-skill/output-templates.md new file mode 100644 index 0000000..a5cb381 --- /dev/null +++ b/core/skills/validate-skill/output-templates.md @@ -0,0 +1,65 @@ +# Validation Report Template + +For each case, write these debugging artifacts under `test/skills/reports///`: + +- `validation-results.json` — structured result passed to the report renderer. +- `validation-spec.yaml` — run plan for this case. +- `validation-report.md` — rendered Markdown report. +- `skill-conversation.md` — chronological caller-to-target-skill messages, excluding tool output and internal reasoning. +- `quickstart-output.tar.gz` — archive of the final `.rhoai-qs/qs-test-quickstart/` tree. + +The preferred coordinator also retains machine-readable phase evidence in each +case directory (`preflight.json`, `completion.json`, `comparison.json`, runner +and rater results, and process logs) and writes an aggregate +`test/skills/reports//validation-report.md`. + +The report Markdown is: + +```markdown +# Validation report: + +Run: +Quickstart fixture path: `.rhoai-qs/qs-test-quickstart/` + +| Case | Status | Score | Summary | +|---|---:|---:|---| +| | passed | 9/10 | | + +## + +### Evidence + +- Target command: `` +- Required baseline artifacts: +- Observed artifacts: + +### Differences + +- + +### Rating + +**/10** — . + +### Blocked questions or failures + +. +``` + +The conversation Markdown is: + +~~~~markdown +# Skill conversation: / + +## Caller + +```text + +``` + +## Target skill + +```text + +``` +~~~~ diff --git a/core/skills/validate-skill/reasoning-guardrails.md b/core/skills/validate-skill/reasoning-guardrails.md new file mode 100644 index 0000000..b1ff58d --- /dev/null +++ b/core/skills/validate-skill/reasoning-guardrails.md @@ -0,0 +1,33 @@ +# Reasoning Guardrails for validate-skill + +These concern areas help validate a Factory skill without contaminating its fixtures or bypassing its safety boundaries. + +## How to Use + +Reason freely about each case, then check whether fixture isolation, intent, safety, and evaluation evidence have all been considered. Skip concerns that do not apply. + +## Concern Areas + +### Fixture isolation + +Consider whether the exact disposable target was reset and whether any output can leak between cases. Inspect copied paths and generated artifacts, not assumptions about cleanup. + +### Faithful invocation + +Consider whether the target received the exact case prompt and only the fixture state it would have in a real run. Do not reinterpret `run-skill-command` as a shell command or add requirements not stated by the fixture. + +### Question handling and authority + +Consider whether an answer is objectively established by the fixture. A fixture can answer factual setup questions, but never grants permission for destructive changes, external writes, credentials, or cluster operations. + +### Baseline quality + +Consider whether expected artifacts specify observable behavior rather than incidental formatting. Distinguish legitimate dynamic values from unexpected drift, and distinguish a missing baseline from a target-skill failure. + +### Rating integrity + +Consider whether the score reflects evidence and functional importance. Do not use a score to hide a blocked run; report it as blocked with the exact question instead. + +## When to Stop Checking + +Stop once every case has a reproducible starting state, an execution result, and evidence-backed rating or explicit blocker. Do not continue by changing fixtures to make a target skill pass. diff --git a/core/skills/validate-skill/references/fixture-contract.md b/core/skills/validate-skill/references/fixture-contract.md new file mode 100644 index 0000000..956fbda --- /dev/null +++ b/core/skills/validate-skill/references/fixture-contract.md @@ -0,0 +1,70 @@ +# Fixture Contract + +Each immediate child directory of `test/skills//` is a validation case: + +```text +test/skills/// +├── input-repo/ # Required: source quickstart copied to .rhoai-qs/qs-test-quickstart/ +├── expected-output-repo/ # Required: approved baseline for evaluation +├── test-config.yaml # Required: target-skill prompt, question-answer context, and validation options +└── comparison-ignore # Optional: newline-delimited glob paths excluded from comparison +``` + +`input-repo/` contains the complete starting state the target skill should receive. `expected-output-repo/` is the complete repository tree after a known-good run of the target skill: it includes the original files from `input-repo/` plus every generated or modified file. It must not contain `.git/`, credentials, secrets, or runtime state. + +`test-config.yaml` has this shape: + +```yaml +run-skill-command: Add nightly testing to qs-test-quickstart. +question-answer-context: | + The test fixture is a disposable quickstart. Use only details in its files. +validation-options: + mode: full-baseline + additional-instructions: | + Confirm the generated workflow retains its concurrency group. +execution: + network: required + external-dependencies: + - https://github.com/rh-ai-quickstart/quickstart-ci-testing +``` + +To run the target skill without a supplemental prompt, use an empty mapping: + +```yaml +{} +``` + +`run-skill-command` is optional. When present, it must be a non-empty string and is the exact user request sent to the target skill. When absent, the target skill is invoked without an added user request. It is never executable shell. State the target slug explicitly as `qs-test-quickstart` when the target skill needs a slug. + +`question-answer-context` is optional and must be a non-empty string when present. It is additional factual context the runner can use to answer a target skill's question. It cannot approve external actions, disclose a credential, or choose whether to overwrite unanticipated user work. + +`validation-options` is optional and must be a YAML mapping when present. It controls evaluation only; it does not change how the target skill is invoked. Its optional `additional-instructions` entry must be a non-empty string when present. It is passed only to the output rater to add fixture-specific checks or explain what differences matter. In `full-baseline` mode, it cannot waive required baseline artifacts; in either mode, it cannot authorize external actions. + +`validation-options.mode` is optional and defaults to `full-baseline`. + +`execution` is optional and defaults to no network requirement. Set `execution.network: required` when the target skill must reach an external service or repository. List each required Git repository under `external-dependencies`; the validator preflights these with `git ls-remote` and must launch the case runner in a network-enabled execution context. The runner command must also have read/write access to the repository and the fixed disposable path `.rhoai-qs/qs-test-quickstart/`. `--network-profile network` declares the required execution profile but does not grant permissions by itself. A failed preflight is recorded as infrastructure evidence, but does not authorize skipping target-skill execution. + +The validator's case-runner gate is synchronous: `run_case.py` waits for the runner process to exit, accepts only a runner result with `status: completed`, and then creates the validator-owned output manifest and tree hash. Baseline comparison and rating must not start until that completion record exists. Raw runner and rater command arguments may use their documented placeholders; these are expanded without invoking a shell. Adapter prompt files are passed through as written, with case-specific inputs appended by the coordinator; placeholders in prompt files are not expanded. + +The optional runner adapters (`codex`, `claude`, `cursor`, and `gemini`) invoke foreground clients only. They preserve the caller's environment and workspace, and they do not themselves enforce network allowlists. Network enforcement belongs to the host/container/proxy that launches the adapter. + +- `full-baseline` requires a complete approved `expected-output-repo/`; all baseline artifacts remain part of the evaluation. Additional instructions can add checks but cannot remove baseline requirements. +- `targeted` requires `validation-options.additional-instructions`. Those instructions are the exclusive evaluation scope, so baseline-tree differences outside that scope do not affect the score. `expected-output-repo/` must still exist but may be empty or contain only `.gitkeep`. + +For example, a targeted privilege check is: + +```yaml +validation-options: + mode: targeted + additional-instructions: | + Validate only that the skill flags cluster-admin privilege as required. +``` + +The rater compares the actual copied quickstart to `expected-output-repo/` as follows: + +- Every baseline path is required: regular files are compared by content, and directories (including empty directories) are compared by presence and type. +- Additional actual files or directories are reported as differences. +- Use `comparison-ignore` only for genuinely nondeterministic paths, such as timestamped reports. Each line is a path glob relative to the quickstart root; blank lines and `#` comments are ignored. +- A baseline is an approved oracle, not permission to copy it into the actual output during ordinary validation. + +Keep test reports and debugging archives in `test/skills/reports///`; the top-level `reports/` directory is not a test case. The archive is a capture of generated output only and must never be used as a baseline. An empty `expected-output-repo/` (or one containing only `.gitkeep`) is intentionally invalid until an approved baseline is added through the normal fixture-maintenance workflow. diff --git a/core/skills/validate-skill/runners/README.md b/core/skills/validate-skill/runners/README.md new file mode 100644 index 0000000..1d1b1f3 --- /dev/null +++ b/core/skills/validate-skill/runners/README.md @@ -0,0 +1,32 @@ +# Runner adapters + +These adapters translate the validator's backend-neutral execution request +into a foreground agent invocation. They do not grant permissions themselves. + +The coordinator supplies the discovered target skill name. Each adapter invokes +that skill through the runtime's native mechanism: slash invocation for Claude +Code and Cursor, and explicit skill activation language for Codex and Gemini. +The case-runner prompt is supplemental execution policy; it must not load or +replay the target `SKILL.md` itself. + +The Codex adapter uses `$skill-name` activation and starts from the Factory +repository root so repo-scoped skills are discovered the same way they are in +normal Factory use. The fixed disposable quickstart remains the case input and +target-output path. The launcher keeps Git commands inside that fixture from +inheriting the Factory repository's branch metadata. + +The coordinator or the process wrapper must provide the actual execution +boundary: + +- the workspace mount/write policy; +- network access or egress allowlisting; +- inherited authentication environment; and +- synchronous process completion. + +`VALIDATE_ALLOWED_DOMAINS` is an interface for that outer boundary. It is not +a security control when passed only as an environment variable. + +The Cursor adapter intentionally targets the foreground CLI. It never creates +a Cursor Background Agent implicitly. The Gemini adapter uses headless +`--prompt` mode with explicit approval handling; it never creates a separate +hosted/background session. diff --git a/core/skills/validate-skill/runners/__init__.py b/core/skills/validate-skill/runners/__init__.py new file mode 100644 index 0000000..93581c6 --- /dev/null +++ b/core/skills/validate-skill/runners/__init__.py @@ -0,0 +1,37 @@ +"""Execution adapters for validate-skill case runners. + +Adapters only construct an invocation. The validation coordinator remains +responsible for process lifetime, fixture isolation, completion records, and +reporting. +""" + +from .claude import ClaudeAdapter +from .contract import AdapterError, RunnerRequest, RunnerSpec, domains_from_dependencies +from .codex import CodexAdapter +from .cursor import CursorAdapter +from .gemini import GeminiAdapter + +__all__ = [ + "AdapterError", + "ClaudeAdapter", + "CodexAdapter", + "CursorAdapter", + "GeminiAdapter", + "RunnerRequest", + "RunnerSpec", + "domains_from_dependencies", +] + + +def adapter_for(name: str): + """Return a named foreground adapter.""" + adapters = { + "codex": CodexAdapter, + "claude": ClaudeAdapter, + "cursor": CursorAdapter, + "gemini": GeminiAdapter, + } + try: + return adapters[name]() + except KeyError as error: + raise AdapterError(f"unknown runner adapter: {name}") from error diff --git a/core/skills/validate-skill/runners/claude.py b/core/skills/validate-skill/runners/claude.py new file mode 100644 index 0000000..b04ef5b --- /dev/null +++ b/core/skills/validate-skill/runners/claude.py @@ -0,0 +1,77 @@ +"""Claude Code adapter for validate-skill.""" + +from __future__ import annotations + +from pathlib import Path + +from .contract import RunnerRequest, RunnerSpec + +# core/skills/validate-skill/runners/claude.py -> repo root is 4 parents up. +_REPO_ROOT = Path(__file__).resolve().parents[4] + + +def _strip_frontmatter(prompt: str) -> str: + """Drop a leading YAML frontmatter block, if present. + + A prompt beginning with a bare "---" is parsed by Claude's CLI as an + unknown option rather than positional prompt text, so any frontmatter + left on the prompt (e.g. from an unstripped subagent-prompt markdown + file) must be removed before it reaches the command line. + """ + if not prompt.startswith("---"): + return prompt + lines = prompt.split("\n") + for index, line in enumerate(lines[1:], start=1): + if line.strip() == "---": + return "\n".join(lines[index + 1 :]).lstrip("\n") + return prompt + + +_RAW_YAML_REMINDER = ( + "\n\n**CRITICAL: Output ONLY raw YAML. No markdown code fences. No prose before or after.**\n" + "Return the complete YAML object as plain text without any ```yaml, ```, or other formatting." +) + + +class ClaudeAdapter: + """Build a synchronous Claude Code invocation. + + Claude's permission flags do not enforce network egress. The adapter + exports the requested domains so a container/proxy wrapper can enforce + them, while keeping the agent in its normal permission mode by default. + + The runner subprocess runs with the disposable quickstart as its cwd, so + without extra directory access it cannot read the target skill's + SKILL.md or any repo-relative docs it references. The adapter grants the + repository root so those reads succeed. When the case declares network + is required, it also bypasses permission prompts so fetches/tool calls + against the declared dependencies are not left blocked on a prompt that + nothing can answer in this non-interactive run. + """ + + name = "claude" + + def build(self, request: RunnerRequest) -> RunnerSpec: + prompt = _strip_frontmatter(request.prompt) + if request.skill_name: + prompt = f"/{request.skill_name}\n{prompt}" + prompt += _RAW_YAML_REMINDER + command = [ + "claude", + "--print", + prompt, + "--output-format", + "text", + "--add-dir", + str(request.workspace), + "--add-dir", + str(_REPO_ROOT), + ] + if request.network_profile == "network": + command += ["--permission-mode", "bypassPermissions"] + environment = { + "VALIDATE_NETWORK_PROFILE": request.network_profile, + "VALIDATE_ALLOWED_DOMAINS": ",".join(request.allowed_domains), + "VALIDATE_OUTPUT_PATH": str(request.output_path), + } + return RunnerSpec(tuple(command), environment) diff --git a/core/skills/validate-skill/runners/codex.py b/core/skills/validate-skill/runners/codex.py new file mode 100644 index 0000000..37c1183 --- /dev/null +++ b/core/skills/validate-skill/runners/codex.py @@ -0,0 +1,52 @@ +"""Codex CLI adapter for validate-skill.""" + +from __future__ import annotations + +from .contract import RunnerRequest, RunnerSpec + + +_RAW_YAML_REMINDER = ( + "\n\nCRITICAL OUTPUT CONTRACT: return ONLY the complete YAML object requested by the prompt. " + "Do not use Markdown fences and do not add prose before or after the YAML. " + "Quote scalar values that contain a colon followed by a space, a hash, braces, or newlines; " + "use a YAML block scalar for long or multi-line text.\n" +) + + +class CodexAdapter: + """Build a Codex CLI invocation without broadening filesystem access. + + Network mode uses Codex's workspace-write network setting instead of + falling back to danger-full-access. Domain enforcement remains an + environment concern; the allowlist is exported for a wrapper/container + that can enforce it. + """ + + name = "codex" + + def build(self, request: RunnerRequest) -> RunnerSpec: + prompt = request.prompt + if request.skill_name: + prompt = f"Use the ${request.skill_name} skill for this task.\n\n{prompt}" + prompt += _RAW_YAML_REMINDER + working_root = request.repository_root or request.workspace + command = [ + "codex", + "exec", + "--ephemeral", + "--skip-git-repo-check", + "--config", + f'sandbox_workspace_write.network_access={str(request.network_profile == "network").lower()}', + "--sandbox", + "workspace-write", + "--cd", + str(working_root), + "--output-last-message", + str(request.output_path), + "--", + ] + environment: dict[str, str] = {} + if request.network_profile == "network": + environment["VALIDATE_ALLOWED_DOMAINS"] = ",".join(request.allowed_domains) + command.append(prompt) + return RunnerSpec(tuple(command), environment) diff --git a/core/skills/validate-skill/runners/contract.py b/core/skills/validate-skill/runners/contract.py new file mode 100644 index 0000000..e157895 --- /dev/null +++ b/core/skills/validate-skill/runners/contract.py @@ -0,0 +1,59 @@ +"""Common contract for validate-skill execution adapters.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path +from typing import Protocol + + +class AdapterError(ValueError): + """Raised when an adapter cannot safely construct an invocation.""" + + +@dataclass(frozen=True) +class RunnerRequest: + """Inputs shared by every supported agent client.""" + + prompt: str + workspace: Path + output_path: Path + skill_name: str | None = None + repository_root: Path | None = None + network_profile: str = "isolated" + allowed_domains: tuple[str, ...] = field(default_factory=tuple) + + def __post_init__(self) -> None: + if self.network_profile not in {"isolated", "network"}: + raise AdapterError(f"unsupported network profile: {self.network_profile}") + if self.network_profile == "network" and not self.allowed_domains: + raise AdapterError("network profile requires at least one allowed domain") + + +@dataclass(frozen=True) +class RunnerSpec: + """A process invocation ready for the synchronous case-runner gate.""" + + command: tuple[str, ...] + environment: dict[str, str] = field(default_factory=dict) + + +def domains_from_dependencies(dependencies: tuple[str, ...]) -> tuple[str, ...]: + """Return hostnames from declared dependency URLs.""" + from urllib.parse import urlparse + + domains = [] + for dependency in dependencies: + hostname = urlparse(dependency).hostname + if hostname and hostname not in domains: + domains.append(hostname) + return tuple(domains) + + +class RunnerAdapter(Protocol): + """Build a backend-specific process invocation.""" + + name: str + + def build(self, request: RunnerRequest) -> RunnerSpec: + ... diff --git a/core/skills/validate-skill/runners/cursor.py b/core/skills/validate-skill/runners/cursor.py new file mode 100644 index 0000000..965c9b0 --- /dev/null +++ b/core/skills/validate-skill/runners/cursor.py @@ -0,0 +1,40 @@ +"""Cursor CLI adapter for validate-skill.""" + +from __future__ import annotations + +from .contract import RunnerRequest, RunnerSpec + + +_RAW_YAML_REMINDER = ( + "\n\n**CRITICAL: Output ONLY raw YAML. No markdown code fences. No prose before or after.**\n" + "Return the complete YAML object as plain text without any ```yaml, ```, or other formatting." +) + + +class CursorAdapter: + """Build a foreground Cursor Agent invocation. + + This intentionally targets the local synchronous CLI. It does not create + a Cursor Background Agent or move execution into a separate VM. + """ + + name = "cursor" + + def build(self, request: RunnerRequest) -> RunnerSpec: + prompt = request.prompt + if request.skill_name: + prompt = f"/{request.skill_name}\n{prompt}" + prompt += _RAW_YAML_REMINDER + command = [ + "cursor-agent", + "-p", + prompt, + "--output-format", + "text", + ] + environment = { + "VALIDATE_NETWORK_PROFILE": request.network_profile, + "VALIDATE_ALLOWED_DOMAINS": ",".join(request.allowed_domains), + "VALIDATE_OUTPUT_PATH": str(request.output_path), + } + return RunnerSpec(tuple(command), environment) diff --git a/core/skills/validate-skill/runners/gemini.py b/core/skills/validate-skill/runners/gemini.py new file mode 100644 index 0000000..d8c037b --- /dev/null +++ b/core/skills/validate-skill/runners/gemini.py @@ -0,0 +1,40 @@ +"""Gemini CLI adapter for validate-skill.""" + +from __future__ import annotations + +from .contract import RunnerRequest, RunnerSpec + + +_RAW_YAML_REMINDER = ( + "\n\nCRITICAL OUTPUT CONTRACT: return ONLY the complete YAML object requested by the prompt. " + "Do not use Markdown fences and do not add prose before or after the YAML. " + "Quote scalar values that contain a colon followed by a space, a hash, braces, or newlines; " + "use a YAML block scalar for long or multi-line text.\n" +) + + +class GeminiAdapter: + """Build a synchronous, headless Gemini CLI invocation.""" + + name = "gemini" + + def build(self, request: RunnerRequest) -> RunnerSpec: + prompt = request.prompt + if request.skill_name: + prompt = f"Use the `{request.skill_name}` skill for this task.\n\n{prompt}" + prompt += _RAW_YAML_REMINDER + command = [ + "gemini", + "--prompt", + prompt, + "--output-format", + "text", + "--approval-mode=yolo", + "--skip-trust", + ] + environment = { + "VALIDATE_NETWORK_PROFILE": request.network_profile, + "VALIDATE_ALLOWED_DOMAINS": ",".join(request.allowed_domains), + "VALIDATE_OUTPUT_PATH": str(request.output_path), + } + return RunnerSpec(tuple(command), environment) diff --git a/core/skills/validate-skill/runners/launch.py b/core/skills/validate-skill/runners/launch.py new file mode 100644 index 0000000..7ead363 --- /dev/null +++ b/core/skills/validate-skill/runners/launch.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +"""Launch one foreground agent adapter and capture its final response.""" + +from __future__ import annotations + +import argparse +import os +import subprocess +import sys +from pathlib import Path + +# Allow direct execution as ``python runners/launch.py``. +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from runners import RunnerRequest, adapter_for + + +def normalize_final_response(value: str) -> str: + """Remove one Markdown fence around an otherwise machine-readable result.""" + normalized = value.strip() + lines = normalized.splitlines() + if len(lines) >= 2 and lines[0].strip().startswith("```") and lines[-1].strip() == "```": + normalized = "\n".join(lines[1:-1]).strip() + return normalized + "\n" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--adapter", choices=("codex", "claude", "cursor", "gemini"), required=True) + parser.add_argument("--skill-name") + parser.add_argument("--prompt", required=True) + parser.add_argument("--workspace", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--network-profile", choices=("isolated", "network"), default="isolated") + parser.add_argument("--allowed-domain", action="append", default=[]) + args = parser.parse_args() + + repository_root = None + if args.adapter == "codex" and args.skill_name: + if args.skill_name in {".", ".."} or Path(args.skill_name).name != args.skill_name: + raise SystemExit(f"invalid repo-scoped skill name: {args.skill_name!r}") + # Codex discovers repo-scoped skills from the Factory root. The target + # quickstart remains a separate input used for Git isolation and the + # target skill's generated files. + repository_root = Path(__file__).resolve().parents[4] + skill_entry = repository_root / ".codex" / "skills" / args.skill_name / "SKILL.md" + if not skill_entry.is_file(): + raise SystemExit(f"repo-scoped skill entry not found: {skill_entry}") + + request = RunnerRequest( + prompt=args.prompt, + workspace=args.workspace.resolve(), + output_path=args.output.resolve(), + skill_name=args.skill_name, + repository_root=repository_root, + network_profile=args.network_profile, + allowed_domains=tuple(args.allowed_domain), + ) + spec = adapter_for(args.adapter).build(request) + environment = os.environ.copy() + environment.update(spec.environment) + # Keep Git discovery inside the disposable quickstart boundary. Without + # this, a fixture nested under the Factory checkout inherits the Factory's + # .git metadata and branch list when the target skill runs git commands. + # Set the ceiling at the parent so a real .git in the quickstart remains + # usable, while parent repositories are never discovered. + environment["GIT_CEILING_DIRECTORIES"] = str(request.workspace.parent.resolve()) + args.output.parent.mkdir(parents=True, exist_ok=True) + if args.output.exists(): + args.output.unlink() + + completed = subprocess.run( + list(spec.command), + cwd=repository_root or request.workspace, + env=environment, + capture_output=True, + text=True, + check=False, + ) + if completed.returncode != 0: + sys.stderr.write(completed.stderr) + return completed.returncode + + # Codex writes its final response directly. Claude and Cursor return it on + # stdout, so capture that response for run_case.py's YAML gate. + if args.output.is_file(): + args.output.write_text(normalize_final_response(args.output.read_text())) + else: + args.output.write_text(normalize_final_response(completed.stdout)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/archive_fixture.py b/core/skills/validate-skill/scripts/archive_fixture.py new file mode 100644 index 0000000..1a9a992 --- /dev/null +++ b/core/skills/validate-skill/scripts/archive_fixture.py @@ -0,0 +1,38 @@ +#!/usr/bin/env python3 +"""Archive a completed disposable quickstart for fixture debugging.""" +import argparse +import tarfile +from pathlib import Path + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("skill_name") + parser.add_argument("case_name") + parser.add_argument("--output", required=True, help="archive path relative to the repository root") + args = parser.parse_args() + + root = Path.cwd().resolve() + expected_case = root / "test" / "skills" / args.skill_name / args.case_name + if not expected_case.is_dir(): + parser.error(f"case not found: {expected_case}") + source = root / ".rhoai-qs" / "qs-test-quickstart" + if not source.is_dir() or source.is_symlink(): + parser.error(f"refusing to archive invalid disposable target: {source}") + + output = (root / args.output).resolve() + try: + output.relative_to(root / "test" / "skills" / "reports" / args.skill_name / args.case_name) + except ValueError: + parser.error("output must be under the case report directory") + if output.suffixes[-2:] != [".tar", ".gz"]: + parser.error("output must end in .tar.gz") + output.parent.mkdir(parents=True, exist_ok=True) + with tarfile.open(output, "w:gz") as archive: + archive.add(source, arcname="qs-test-quickstart", recursive=True) + print(output.relative_to(root)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/compare_baseline.py b/core/skills/validate-skill/scripts/compare_baseline.py new file mode 100644 index 0000000..46c96e2 --- /dev/null +++ b/core/skills/validate-skill/scripts/compare_baseline.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +"""Produce deterministic baseline-comparison evidence for one fixture case.""" +import argparse +import fnmatch +import hashlib +import json +from pathlib import Path + + +METADATA_FILES = {".gitkeep"} + + +def digest(path: Path) -> str: + hasher = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + hasher.update(chunk) + return hasher.hexdigest() + + +def entries(root: Path) -> dict[str, dict[str, str | None]]: + """Return every non-root tree entry, including empty directories.""" + result: dict[str, dict[str, str | None]] = {} + for path in root.rglob("*"): + relative = str(path.relative_to(root)) + if path.is_dir(): + result[relative] = {"kind": "directory", "sha256": None} + elif path.is_file(): + result[relative] = {"kind": "file", "sha256": digest(path)} + return result + + +def matches_ignore(path: str, patterns: set[str]) -> bool: + return any(fnmatch.fnmatch(path, pattern) for pattern in patterns) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("case_path", help="case path relative to the repository root") + args = parser.parse_args() + root = Path.cwd().resolve() + case = (root / args.case_path).resolve() + expected = case / "expected-output-repo" + actual = root / ".rhoai-qs" / "qs-test-quickstart" + if not expected.is_dir() or not actual.is_dir(): + parser.error("expected-output-repo and .rhoai-qs/qs-test-quickstart must exist") + ignore_file = case / "comparison-ignore" + ignored = {line.strip() for line in ignore_file.read_text().splitlines() if line.strip() and not line.lstrip().startswith("#")} if ignore_file.is_file() else set() + expected_entries = {path: value for path, value in entries(expected).items() if path not in METADATA_FILES} + actual_entries = entries(actual) + required_status = [] + for path in sorted(expected_entries): + if matches_ignore(path, ignored): + continue + expected_kind = expected_entries[path]["kind"] + actual = actual_entries.get(path) + status = "missing" if actual is None else "present" if actual["kind"] == expected_kind else "differs" + required_status.append({"path": path, "status": status}) + + differences = [] + for path, expected_entry in sorted(expected_entries.items()): + if matches_ignore(path, ignored): + continue + actual_entry = actual_entries.get(path) + if actual_entry is None: + difference = {"path": path, "kind": "missing", "expected_type": expected_entry["kind"]} + if expected_entry["kind"] == "file": + difference["expected_sha256"] = expected_entry["sha256"] + differences.append(difference) + elif actual_entry["kind"] != expected_entry["kind"]: + differences.append({"path": path, "kind": "type_mismatch", "expected_type": expected_entry["kind"], "actual_type": actual_entry["kind"]}) + elif expected_entry["kind"] == "file" and actual_entry["sha256"] != expected_entry["sha256"]: + differences.append({"path": path, "kind": "content_differs", "expected_sha256": expected_entry["sha256"], "actual_sha256": actual_entry["sha256"]}) + + extras = sorted(path for path, entry in actual_entries.items() if entry["kind"] == "file" and path not in expected_entries and not matches_ignore(path, ignored)) + extra_paths = sorted(path for path in actual_entries if path not in expected_entries and not matches_ignore(path, ignored)) + actual_files = sorted(path for path, entry in actual_entries.items() if entry["kind"] == "file") + print(json.dumps({"case": case.name, "required_paths": required_status, "differences": differences, "extra_files": extras, "extra_paths": extra_paths, "actual_files": actual_files, "actual_paths": sorted(actual_entries)}, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/discover_cases.py b/core/skills/validate-skill/scripts/discover_cases.py new file mode 100644 index 0000000..62b2ad0 --- /dev/null +++ b/core/skills/validate-skill/scripts/discover_cases.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""Validate and enumerate fixture cases for a Factory skill.""" +import json +import sys +from pathlib import Path + +import yaml + + +def parse_test_filter(value: str | None) -> tuple[list[str] | None, list[str]]: + """Parse a comma-separated case list and return validation errors.""" + if value is None: + return None, [] + names = [name.strip() for name in value.split(",")] + errors = [] + if not all(names): + errors.append("--tests must be a comma-separated list of non-empty test names") + if len(names) != len(set(names)): + errors.append("--tests must not name a test more than once") + return names, errors + + +def load_test_config(path: Path) -> tuple[dict[str, object] | None, list[str]]: + """Load the test control file and validate the fields the runner consumes.""" + try: + config = yaml.safe_load(path.read_text()) + except yaml.YAMLError as error: + return None, [f"invalid test-config.yaml: {error}"] + if not isinstance(config, dict): + return None, ["test-config.yaml must contain a YAML mapping"] + + errors = [] + command = config.get("run-skill-command") + if command is not None and (not isinstance(command, str) or not command.strip()): + errors.append("run-skill-command must be a non-empty string when provided") + context = config.get("question-answer-context") + if context is not None and (not isinstance(context, str) or not context.strip()): + errors.append("question-answer-context must be a non-empty string when provided") + validation_options = config.get("validation-options", {}) + if not isinstance(validation_options, dict): + return None, ["validation-options must be a YAML mapping when provided"] + validation_instructions = validation_options.get("additional-instructions") + if validation_instructions is not None and (not isinstance(validation_instructions, str) or not validation_instructions.strip()): + errors.append("validation-options.additional-instructions must be a non-empty string when provided") + validation_mode = validation_options.get("mode", "full-baseline") + if validation_mode not in {"full-baseline", "targeted"}: + errors.append("validation-options.mode must be full-baseline or targeted") + if validation_mode == "targeted" and validation_instructions is None: + errors.append("targeted validation-options.mode requires additional-instructions") + execution = config.get("execution", {}) + if not isinstance(execution, dict): + errors.append("execution must be a YAML mapping when provided") + execution = {} + network = execution.get("network", "none") + if network not in {"none", "required"}: + errors.append("execution.network must be none or required") + external_dependencies = execution.get("external-dependencies", []) + if not isinstance(external_dependencies, list) or any( + not isinstance(dependency, str) or not dependency.strip() + for dependency in external_dependencies + ): + errors.append("execution.external-dependencies must be a list of non-empty strings") + external_dependencies = [] + if external_dependencies and network != "required": + errors.append("execution.external-dependencies requires execution.network: required") + if errors: + return None, errors + return { + "run_skill_command": command.strip() if command is not None else None, + "question_answer_context": context.strip() if context is not None else None, + "additional_validation_instructions": validation_instructions.strip() if validation_instructions is not None else None, + "validation_mode": validation_mode, + "execution_network": network, + "external_dependencies": [dependency.strip() for dependency in external_dependencies], + }, [] + + +def main() -> int: + arguments = sys.argv[1:] + if not arguments or len(arguments) > 3: + print("usage: discover_cases.py [--tests test-one,test-two]", file=sys.stderr) + return 2 + skill_name = arguments[0] + test_filter = None + if len(arguments) == 3 and arguments[1] == "--tests": + test_filter = arguments[2] + elif len(arguments) != 1: + print("usage: discover_cases.py [--tests test-one,test-two]", file=sys.stderr) + return 2 + + root = Path.cwd().resolve() + target_skill = root / "core" / "skills" / skill_name + cases_root = root / "test" / "skills" / skill_name + requested_cases, errors = parse_test_filter(test_filter) + if not (target_skill / "SKILL.md").is_file(): + errors.append(f"target skill not found: {target_skill / 'SKILL.md'}") + if not cases_root.is_dir(): + errors.append(f"case directory not found: {cases_root}") + cases = [] + if cases_root.is_dir(): + available_cases = { + case.name: case + for case in cases_root.iterdir() + if case.is_dir() and not case.name.startswith(".") and case.name != "reports" + } + if requested_cases is None: + selected_cases = sorted(available_cases) + else: + selected_cases = sorted(set(requested_cases)) + unknown = sorted(set(selected_cases) - set(available_cases)) + if unknown: + errors.append(f"requested case(s) not found: {', '.join(unknown)}") + + for case_name in selected_cases: + case = available_cases.get(case_name) + if case is None: + continue + config_path = case / "test-config.yaml" + missing = [name for name in ("input-repo", "expected-output-repo", "test-config.yaml") if not (case / name).exists()] + if not missing and not any((case / "input-repo").iterdir()): + missing.append("input-repo contents") + config = None + if not missing: + config, config_errors = load_test_config(config_path) + missing.extend(config_errors) + if not missing and config["validation_mode"] == "full-baseline" and not any(path.name != ".gitkeep" for path in (case / "expected-output-repo").iterdir()): + missing.append("approved expected-output-repo contents") + if missing: + errors.append(f"{case.name}: missing {', '.join(missing)}") + else: + cases.append({ + "name": case.name, + "path": str(case.relative_to(root)), + "test_config": str(config_path.relative_to(root)), + **config, + }) + print(json.dumps({"target_skill": str(target_skill), "cases": cases, "errors": errors}, indent=2)) + return 1 if errors else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/orchestrate_validation.py b/core/skills/validate-skill/scripts/orchestrate_validation.py new file mode 100644 index 0000000..e54146f --- /dev/null +++ b/core/skills/validate-skill/scripts/orchestrate_validation.py @@ -0,0 +1,630 @@ +#!/usr/bin/env python3 +"""Run the deterministic phases of a validate-skill session. + +Model execution remains delegated through the supplied runner and rater command +arrays. This script owns fixture isolation, evidence capture, schema gates, and +report assembly so those handoffs cannot be completed manually or out of order. +""" +from __future__ import annotations + +import argparse +import json +import os +import re +import signal +import shutil +import subprocess +import sys +from datetime import datetime, timezone +from pathlib import Path +from urllib.parse import urlparse +from typing import Any + +import yaml + + +PLACEHOLDER_RE = re.compile(r"\{([a-z_]+)\}") +KNOWN_PLACEHOLDERS = { + "case_path", + "conversation_json", + "comparison_json", + "quickstart_root", + "rater_output", + "runner_output", +} + + +def write_text(path: Path, value: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(value) + + +def write_json(path: Path, value: Any) -> None: + write_text(path, json.dumps(value, indent=2) + "\n") + + +def reset_report_dir(path: Path) -> None: + """Start a case with no evidence left over from an earlier run.""" + if path.is_symlink() or path.is_file(): + path.unlink() + elif path.is_dir(): + shutil.rmtree(path) + path.mkdir(parents=True, exist_ok=True) + + +def run(command: list[str], root: Path, timeout: int) -> tuple[int, str, str, bool]: + process: subprocess.Popen[str] | None = None + try: + process = subprocess.Popen( + command, + cwd=root, + env=os.environ.copy(), + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + start_new_session=True, + ) + stdout, stderr = process.communicate(timeout=timeout) + return process.returncode, stdout, stderr, False + except subprocess.TimeoutExpired as error: + stdout = error.stdout or "" + stderr = error.stderr or "" + if process is not None: + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + remaining_stdout, remaining_stderr = process.communicate() + stdout = stdout or remaining_stdout + stderr = stderr or remaining_stderr + if isinstance(stdout, bytes): + stdout = stdout.decode(errors="replace") + if isinstance(stderr, bytes): + stderr = stderr.decode(errors="replace") + return 124, stdout, stderr, True + + +def save_process_log(path: Path, command: list[str], stdout: str, stderr: str, timed_out: bool) -> None: + lines = ["command:", json.dumps(command), f"timed_out: {str(timed_out).lower()}", "", "stdout:", stdout, "", "stderr:", stderr] + write_text(path, "\n".join(lines)) + + +def load_mapping(path: Path, label: str) -> dict[str, Any]: + try: + value = yaml.safe_load(path.read_text()) + except (OSError, yaml.YAMLError) as error: + raise ValueError(f"{label} is not valid YAML: {error}") from error + if not isinstance(value, dict): + raise ValueError(f"{label} must contain a YAML mapping") + return value + + +def load_json(path: Path, label: str) -> dict[str, Any]: + try: + value = json.loads(path.read_text()) + except (OSError, json.JSONDecodeError) as error: + raise ValueError(f"{label} is not valid JSON: {error}") from error + if not isinstance(value, dict): + raise ValueError(f"{label} must contain a JSON object") + return value + + +def expand(command: list[str], replacements: dict[str, str]) -> list[str]: + expanded = [] + for argument in command: + matches = PLACEHOLDER_RE.findall(argument) + unknown = [name for name in matches if name not in KNOWN_PLACEHOLDERS] + if unknown: + raise ValueError(f"unknown command placeholder(s): {', '.join(sorted(set(unknown)))}") + value = argument + for name, replacement in replacements.items(): + value = value.replace("{" + name + "}", replacement) + if PLACEHOLDER_RE.search(value): + raise ValueError(f"unresolved command placeholder in argument: {argument}") + expanded.append(value) + return expanded + + +def validate_context(root: Path, skill: str, case_path: Path) -> dict[str, Any]: + expected_root = (root / ".rhoai-qs" / "qs-test-quickstart").resolve() + fixture_root = (root / "test" / "skills" / skill).resolve() + valid = ( + expected_root == (root / ".rhoai-qs" / "qs-test-quickstart").resolve() + and case_path.resolve().is_relative_to(fixture_root) + ) + result = { + "resolution": "resolved" if valid else "error", + "slug": "qs-test-quickstart" if valid else None, + "confidence": "high", + "error_message": None if valid else "fixture path is outside the target skill's test directory", + } + return result + + +def write_spec(root: Path, skill: str, case: dict[str, Any], run_at: str, report_dir: Path) -> None: + spec = { + "spec_version": 1, + "skill": "validate-skill", + "target_skill": skill, + "quickstart_slug": "qs-test-quickstart", + "created_at": run_at, + "test_filter": case["name"], + "cases": [ + { + "name": case["name"], + "input_repo": f"test/skills/{skill}/{case['name']}/input-repo", + "expected_output_repo": f"test/skills/{skill}/{case['name']}/expected-output-repo", + "test_config": case["test_config"], + "run_skill_command": case.get("run_skill_command"), + "question_answer_context": case.get("question_answer_context"), + "validation_options": { + "mode": case["validation_mode"], + "additional_instructions": case.get("additional_validation_instructions"), + }, + "execution": { + "network": case["execution_network"], + "external_dependencies": case["external_dependencies"], + }, + "completion": { + "runner_gate": "core/skills/validate-skill/scripts/run_case.py", + "status": "completed", + }, + } + ], + "acceptance_criteria": [ + { + "id": "isolated-fixtures", + "description": "Each case starts with a fresh qs-test-quickstart copy.", + "validation": "prepare_fixture.py runs before every case.", + "requires_user_approval": False, + }, + { + "id": "baseline-rating", + "description": "Every completed case has an evidence-based 1–10 rating.", + "validation": "The independent rater result is schema-validated.", + "requires_user_approval": False, + }, + ], + } + write_text(report_dir / "validation-spec.yaml", yaml.safe_dump(spec, sort_keys=False)) + + +def validate_runner_result(result: dict[str, Any]) -> None: + if result.get("status") != "completed": + return + conversation = result.get("conversation") + if not isinstance(conversation, list) or not conversation: + raise ValueError("completed runner result must contain a non-empty conversation") + for message in conversation: + if not isinstance(message, dict) or message.get("role") not in {"caller", "target-skill"} or not isinstance(message.get("content"), str): + raise ValueError("runner conversation contains an invalid message") + if not isinstance(result.get("artifacts", []), list): + raise ValueError("runner artifacts must be a list") + + +def validate_rater_result(result: dict[str, Any], case_name: str) -> None: + if result.get("case") != case_name: + raise ValueError(f"rater case mismatch: expected {case_name!r}") + score = result.get("score") + if isinstance(score, bool) or not isinstance(score, int) or not 1 <= score <= 10: + raise ValueError("rater score must be an integer from 1 to 10") + if not isinstance(result.get("summary"), str) or not result["summary"].strip(): + raise ValueError("rater summary must be a non-empty string") + if not isinstance(result.get("required_artifacts"), list) or not isinstance(result.get("differences", []), list): + raise ValueError("rater required_artifacts and differences must be lists") + + +def conversation_from_runner(runner_result: dict[str, Any]) -> list[dict[str, str]]: + conversation = runner_result.get("conversation") + return conversation if isinstance(conversation, list) else [] + + +def dependency_domains(dependencies: list[str]) -> list[str]: + domains = [] + for dependency in dependencies: + hostname = urlparse(dependency).hostname + if hostname and hostname not in domains: + domains.append(hostname) + return domains + + +def adapter_command( + root: Path, + adapter: str, + skill_name: str | None, + prompt: str, + output_path: Path, + quickstart_root: Path, + network_profile: str, + domains: list[str], +) -> list[str]: + command = [ + sys.executable, + str(root / "core/skills/validate-skill/runners/launch.py"), + "--adapter", + adapter, + ] + if skill_name: + command.extend(["--skill-name", skill_name]) + command.extend( + [ + "--prompt", + prompt, + "--workspace", + str(quickstart_root), + "--output", + str(output_path), + "--network-profile", + network_profile, + ] + ) + for domain in domains: + command.extend(["--allowed-domain", domain]) + return command + + +def render_case_report(root: Path, report_dir: Path) -> None: + command = [ + sys.executable, + str(root / "core/skills/validate-skill/scripts/write_report.py"), + "--input", + str(report_dir / "validation-results.json"), + "--output", + str(report_dir / "validation-report.md"), + ] + code, stdout, stderr, timed_out = run(command, root, 60) + if code != 0 or timed_out: + raise RuntimeError(f"write_report.py failed: {stderr or stdout}") + + +def render_conversation(root: Path, report_dir: Path, case_name: str) -> None: + command = [ + sys.executable, + str(root / "core/skills/validate-skill/scripts/write_conversation.py"), + "--input", + str(report_dir / "validation-results.json"), + "--case", + case_name, + "--output", + str(report_dir / "skill-conversation.md"), + ] + code, stdout, stderr, timed_out = run(command, root, 60) + if code != 0 or timed_out: + raise RuntimeError(f"write_conversation.py failed: {stderr or stdout}") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("skill_name") + parser.add_argument("--tests", default=None) + parser.add_argument("--runner-command-json", help="JSON array command for the case runner") + parser.add_argument("--rater-command-json", help="JSON array command for the independent rater") + parser.add_argument("--runner-adapter", choices=("codex", "claude", "cursor", "gemini")) + parser.add_argument("--runner-prompt-file", type=Path) + parser.add_argument("--rater-adapter", choices=("codex", "claude", "cursor", "gemini")) + parser.add_argument("--rater-prompt-file", type=Path) + parser.add_argument("--network-profile", choices={"isolated", "network"}, default="isolated") + parser.add_argument("--timeout-seconds", type=int, default=1800) + parser.add_argument("--rater-timeout-seconds", type=int, default=600) + args = parser.parse_args() + + root = Path.cwd().resolve() + if bool(args.runner_adapter) == bool(args.runner_command_json): + parser.error("provide exactly one of --runner-command-json or --runner-adapter") + if args.runner_adapter and not args.runner_prompt_file: + parser.error("--runner-adapter requires --runner-prompt-file") + if bool(args.rater_adapter) == bool(args.rater_command_json): + parser.error("provide exactly one of --rater-command-json or --rater-adapter") + if args.rater_adapter and not args.rater_prompt_file: + parser.error("--rater-adapter requires --rater-prompt-file") + + runner_command = None + rater_command = None + try: + if args.runner_command_json: + runner_command = json.loads(args.runner_command_json) + if args.rater_command_json: + rater_command = json.loads(args.rater_command_json) + except json.JSONDecodeError as error: + parser.error(f"runner/rater command must be valid JSON arrays: {error}") + if runner_command is not None and (not isinstance(runner_command, list) or not all(isinstance(item, str) for item in runner_command)): + parser.error("runner-command-json must be a JSON array of strings") + if rater_command is not None and (not isinstance(rater_command, list) or not all(isinstance(item, str) for item in rater_command)): + parser.error("rater-command-json must be a JSON array of strings") + + discover = [sys.executable, str(root / "core/skills/validate-skill/scripts/discover_cases.py"), args.skill_name] + if args.tests is not None: + discover.extend(["--tests", args.tests]) + code, stdout, stderr, timed_out = run(discover, root, 60) + if code != 0: + print(stdout, end="") + print(stderr, file=sys.stderr, end="") + return code + try: + discovered = json.loads(stdout) + except json.JSONDecodeError as error: + parser.error(f"discover_cases.py did not return JSON: {error}") + if discovered.get("errors"): + print(json.dumps(discovered, indent=2)) + return 1 + network_required = any(case.get("execution_network") == "required" for case in discovered.get("cases", [])) + if network_required and args.network_profile != "network": + print( + json.dumps( + { + "errors": [ + "At least one selected case requires network access; " + "run with --network-profile network and provide a network-capable runner command." + ] + }, + indent=2, + ) + ) + return 1 + + run_at = datetime.now(timezone.utc).isoformat() + context = validate_context(root, args.skill_name, root / "test" / "skills" / args.skill_name) + if context["resolution"] != "resolved": + print(json.dumps(context, indent=2)) + return 1 + + aggregate_cases: list[dict[str, Any]] = [] + for case in discovered["cases"]: + case_path = (root / case["path"]).resolve() + report_dir = root / "test" / "skills" / "reports" / args.skill_name / case["name"] + reset_report_dir(report_dir) + write_spec(root, args.skill_name, case, run_at, report_dir) + + prepare = [sys.executable, str(root / "core/skills/validate-skill/scripts/prepare_fixture.py"), case["path"]] + code, prepare_out, prepare_err, timed_out = run(prepare, root, 60) + save_process_log(report_dir / "prepare.log", prepare, prepare_out, prepare_err, timed_out) + if code != 0: + aggregate_cases.append({"name": case["name"], "status": "failed", "score": None, "summary": "Fixture preparation failed.", "evidence": [], "differences": [], "blocked_or_failure": prepare_err or prepare_out}) + continue + + preflight = [sys.executable, str(root / "core/skills/validate-skill/scripts/preflight_dependencies.py"), case["path"]] + code, preflight_out, preflight_err, timed_out = run(preflight, root, 60) + try: + preflight_json = json.loads(preflight_out) + except json.JSONDecodeError: + preflight_json = {"status": "invalid-output", "stdout": preflight_out, "stderr": preflight_err} + write_json(report_dir / "preflight.json", preflight_json) + save_process_log(report_dir / "preflight.log", preflight, preflight_out, preflight_err, timed_out) + + runner_output = report_dir / "runner-result.yaml" + completion_output = report_dir / "completion.json" + if args.runner_adapter: + runner_prompt = args.runner_prompt_file.read_text() + runner_prompt += ( + "\n\nValidation inputs:\n" + f"target_skill_name={args.skill_name}\n" + f"target_skill_path={(root / 'core/skills' / args.skill_name).resolve()}\n" + f"target_prompt={case.get('run_skill_command') or 'none'}\n" + f"fixture_root={case_path}\n" + f"quickstart_root={(root / '.rhoai-qs' / 'qs-test-quickstart').resolve()}\n" + f"execution_network={case.get('execution_network', 'none')}\n" + f"external_dependencies={case.get('external_dependencies', [])}\n" + ) + qa_context = case.get('question_answer_context') + if qa_context: + runner_prompt += ( + "\n\n---\n\n" + "## Question-Answer Context (HOLD IN RESERVE)\n\n" + "**CRITICAL:** The following context is ONLY for answering questions that the target skill explicitly asks during execution. " + "Do NOT provide this information proactively or use it as general background context. " + "Wait for the skill to ask a specific question, then check if this context provides a fixture-grounded answer. " + "If it does, record the question in the conversation, then provide this answer. " + "If the skill asks a question and this context does NOT provide an answer, stop and return blocked.\n\n" + f"```\n{qa_context}\n```\n" + ) + runner_command_for_case = adapter_command( + root, + args.runner_adapter, + args.skill_name, + runner_prompt, + runner_output.resolve(), + (root / ".rhoai-qs" / "qs-test-quickstart").resolve(), + "network" if case["execution_network"] == "required" else "isolated", + dependency_domains(case.get("external_dependencies", [])), + ) + else: + runner_command_for_case = runner_command + case_runner = [ + sys.executable, + str(root / "core/skills/validate-skill/scripts/run_case.py"), + "--case-path", + case["path"], + "--quickstart-root", + ".rhoai-qs/qs-test-quickstart", + "--runner-output", + str(runner_output), + "--completion-output", + str(completion_output), + "--runner-command", + *runner_command_for_case, + ] + code, runner_out, runner_err, timed_out = run(case_runner, root, args.timeout_seconds) + save_process_log(report_dir / "case-runner.log", case_runner, runner_out, runner_err, timed_out) + if not completion_output.is_file(): + completion = {"status": "failed", "failure": "case runner produced no completion record", "process_exit": code} + write_json(completion_output, completion) + else: + completion = load_json(completion_output, "completion record") + + try: + runner_result = load_mapping(runner_output, "runner result") if runner_output.is_file() else {"status": "failed", "failure": completion.get("failure")} + except ValueError as error: + runner_result = {"status": "failed", "failure": str(error)} + write_text(report_dir / "runner-result.yaml", yaml.safe_dump(runner_result, sort_keys=False)) + try: + validate_runner_result(runner_result) + except ValueError as error: + runner_result = {"status": "failed", "failure": str(error), "runner_result": runner_result} + + if completion.get("status") != "completed" or runner_result.get("status") != "completed": + result = { + "name": case["name"], + "status": "failed" if not runner_result.get("blocked_question") else "blocked", + "score": None, + "summary": "Target-skill execution did not complete.", + "evidence": [f"Network profile: {args.network_profile}", f"Preflight status: {preflight_json.get('status', 'unknown')}"], + "differences": [], + "blocked_or_failure": runner_result.get("blocked_question") or runner_result.get("failure") or completion.get("failure", "unknown runner failure"), + "conversation": conversation_from_runner(runner_result), + } + write_json(report_dir / "validation-results.json", {"target_skill": args.skill_name, "run_at": run_at, "cases": [result]}) + render_conversation(root, report_dir, case["name"]) if result["conversation"] else None + render_case_report(root, report_dir) + aggregate_cases.append(result) + continue + + conversation = conversation_from_runner(runner_result) + conversation_path = report_dir / "conversation.json" + write_json(conversation_path, {"messages": conversation}) + comparison_command = [sys.executable, str(root / "core/skills/validate-skill/scripts/compare_baseline.py"), case["path"]] + code, comparison_out, comparison_err, timed_out = run(comparison_command, root, 60) + save_process_log(report_dir / "comparison.log", comparison_command, comparison_out, comparison_err, timed_out) + try: + comparison = json.loads(comparison_out) + except json.JSONDecodeError as error: + comparison = {"case": case["name"], "differences": [{"kind": "invalid-comparison", "detail": str(error)}], "extra_files": [], "extra_paths": []} + write_json(report_dir / "comparison.json", comparison) + + rater_inputs = { + "conversation_json": str(conversation_path), + "comparison_json": str(report_dir / "comparison.json"), + "quickstart_root": str((root / ".rhoai-qs" / "qs-test-quickstart").resolve()), + } + if args.rater_adapter: + rater_prompt = args.rater_prompt_file.read_text() + rater_prompt += ( + "\n\nEvaluation inputs:\n" + f"fixture_contract_path={(root / 'core/skills/validate-skill/references/fixture-contract.md').resolve()}\n" + f"expected_output_root={(case_path / 'expected-output-repo').resolve()}\n" + f"actual_quickstart_root={rater_inputs['quickstart_root']}\n" + f"case_name={case['name']}\n" + f"comparison_json={rater_inputs['comparison_json']}\n" + f"conversation={rater_inputs['conversation_json']}\n" + f"validation_mode={case.get('validation_mode', 'full-baseline')}\n" + f"additional_validation_instructions={case.get('additional_validation_instructions') or 'null'}\n" + ) + rater = adapter_command( + root, + args.rater_adapter, + None, + rater_prompt, + (report_dir / "rater-result.yaml").resolve(), + (root / ".rhoai-qs" / "qs-test-quickstart").resolve(), + "isolated", + [], + ) + rater_error = None + else: + try: + rater = expand( + rater_command, + { + **rater_inputs, + "case_path": str(case_path), + "rater_output": str((report_dir / "rater-result.yaml").resolve()), + "runner_output": str(runner_output.resolve()), + }, + ) + except ValueError as error: + rater = [] + rater_error = str(error) + else: + rater_error = None + + if rater_error: + rater_code, rater_out, rater_err, rater_timed_out = 1, "", rater_error, False + else: + rater_code, rater_out, rater_err, rater_timed_out = run(rater, root, args.rater_timeout_seconds) + save_process_log(report_dir / "rater.log", rater or ["invalid-command"], rater_out, rater_err, rater_timed_out) + rater_path = report_dir / "rater-result.yaml" + try: + rater_result = load_mapping(rater_path, "rater result") if rater_path.is_file() else {} + except ValueError as error: + rater_result = {"case": case["name"], "error": str(error)} + try: + if rater_code != 0 or rater_timed_out: + raise ValueError(f"rater process failed with exit code {rater_code}") + validate_rater_result(rater_result, case["name"]) + except ValueError as error: + result = { + "name": case["name"], + "status": "failed", + "score": None, + "summary": "Independent output rating failed schema validation.", + "evidence": ["Target-skill execution completed.", f"Preflight status: {preflight_json.get('status', 'unknown')}"], + "differences": [], + "blocked_or_failure": str(error), + "conversation": conversation, + } + else: + differences = [] + for difference in rater_result.get("differences", []): + differences.append(f"{difference.get('path', 'unknown')}: {difference.get('detail', difference)}") + result = { + "name": case["name"], + "status": "passed", + "score": rater_result["score"], + "summary": rater_result["summary"], + "evidence": [ + f"Target command: {case.get('run_skill_command') or 'none'}", + f"Network profile: {args.network_profile}", + f"Preflight status: {preflight_json.get('status', 'unknown')}", + f"Runner tree hash: {completion.get('output_tree_hash', 'unknown')}", + "Independent rater result passed schema validation.", + ], + "differences": differences, + "blocked_or_failure": "None.", + "conversation": conversation, + } + + write_json(report_dir / "validation-results.json", {"target_skill": args.skill_name, "run_at": run_at, "cases": [result]}) + render_conversation(root, report_dir, case["name"]) + render_case_report(root, report_dir) + if completion.get("status") == "completed": + archive_command = [ + sys.executable, + str(root / "core/skills/validate-skill/scripts/archive_fixture.py"), + args.skill_name, + case["name"], + "--output", + str(report_dir / "quickstart-output.tar.gz"), + ] + archive_code, archive_out, archive_err, archive_timed_out = run(archive_command, root, 60) + save_process_log(report_dir / "archive.log", archive_command, archive_out, archive_err, archive_timed_out) + if archive_code != 0: + result["status"] = "failed" + result["score"] = None + result["blocked_or_failure"] = f"archive_fixture.py failed: {archive_err or archive_out}" + write_json(report_dir / "validation-results.json", {"target_skill": args.skill_name, "run_at": run_at, "cases": [result]}) + render_case_report(root, report_dir) + aggregate_cases.append(result) + + aggregate = {"target_skill": args.skill_name, "run_at": run_at, "cases": aggregate_cases} + aggregate_dir = root / "test" / "skills" / "reports" / args.skill_name + write_json(aggregate_dir / "validation-results.json", aggregate) + summary_lines = [ + f"# Validation aggregate: {args.skill_name}", + "", + f"Run: {run_at}", + "", + "| Case | Status | Score | Summary |", + "|---|---|---:|---|", + ] + for result in aggregate_cases: + score = f"{result['score']}/10" if result.get("score") is not None else "—" + summary_lines.append(f"| {result['name']} | {result['status']} | {score} | {result.get('summary', '')} |") + below = [result["name"] for result in aggregate_cases if isinstance(result.get("score"), int) and result["score"] < 8] + summary_lines.extend(["", f"Cases below 8: {', '.join(below) if below else 'None.'}", ""]) + write_text(aggregate_dir / "validation-report.md", "\n".join(summary_lines)) + print(json.dumps(aggregate, indent=2)) + return 0 if all(result["status"] == "passed" for result in aggregate_cases) else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/preflight_dependencies.py b/core/skills/validate-skill/scripts/preflight_dependencies.py new file mode 100644 index 0000000..a1ee2b1 --- /dev/null +++ b/core/skills/validate-skill/scripts/preflight_dependencies.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Check declared Git repositories before a validation case needs them.""" +import argparse +import json +import subprocess +from pathlib import Path + +import yaml + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("case_path") + args = parser.parse_args() + + config_path = Path(args.case_path) / "test-config.yaml" + config = yaml.safe_load(config_path.read_text()) or {} + execution = config.get("execution", {}) + network = execution.get("network", "none") + dependencies = execution.get("external-dependencies", []) + if network != "required" or not dependencies: + print(json.dumps({"network": network, "dependencies": [], "status": "not-required"}, indent=2)) + return 0 + + results = [] + for dependency in dependencies: + try: + completed = subprocess.run( + ["git", "ls-remote", dependency, "HEAD"], + capture_output=True, + text=True, + timeout=20, + check=False, + ) + reachable = completed.returncode == 0 + result = {"dependency": dependency, "status": "reachable" if reachable else "unreachable"} + if not reachable: + result["error"] = (completed.stderr or "git ls-remote failed").strip().splitlines()[-1] + except (OSError, subprocess.TimeoutExpired) as error: + result = {"dependency": dependency, "status": "unreachable", "error": str(error)} + results.append(result) + + status = "reachable" if all(result["status"] == "reachable" for result in results) else "unreachable" + print(json.dumps({"network": network, "dependencies": results, "status": status}, indent=2)) + return 0 if status == "reachable" else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/prepare_fixture.py b/core/skills/validate-skill/scripts/prepare_fixture.py new file mode 100644 index 0000000..78e1c3b --- /dev/null +++ b/core/skills/validate-skill/scripts/prepare_fixture.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Copy one case's input repository into the fixed disposable quickstart path.""" +import argparse +import hashlib +import json +import shutil +from pathlib import Path + + +def digest(path: Path) -> str: + hasher = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + hasher.update(chunk) + return hasher.hexdigest() + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("case_path", help="case path relative to the repository root") + args = parser.parse_args() + root = Path.cwd().resolve() + case = (root / args.case_path).resolve() + try: + case.relative_to(root / "test" / "skills") + except ValueError: + parser.error("case_path must be inside test/skills") + source = case / "input-repo" + if not source.is_dir(): + parser.error(f"input-repo not found: {source}") + target = root / ".rhoai-qs" / "qs-test-quickstart" + if target.is_symlink(): + parser.error("refusing to remove a symlinked disposable target") + if target.exists(): + shutil.rmtree(target) + target.mkdir(parents=True) + for item in source.iterdir(): + destination = target / item.name + if item.is_dir(): + shutil.copytree(item, destination, symlinks=True) + else: + shutil.copy2(item, destination, follow_symlinks=False) + files = [{"path": str(path.relative_to(target)), "sha256": digest(path)} for path in sorted(target.rglob("*")) if path.is_file()] + print(json.dumps({"case": case.name, "target": str(target), "files": files}, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/run_case.py b/core/skills/validate-skill/scripts/run_case.py new file mode 100644 index 0000000..2cc5927 --- /dev/null +++ b/core/skills/validate-skill/scripts/run_case.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +"""Run one validation case and publish a post-run output manifest.""" +import argparse +import hashlib +import json +import os +import re +import subprocess +import tempfile +from pathlib import Path + +import yaml + + +def digest(path: Path) -> str: + hasher = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1024 * 1024), b""): + hasher.update(chunk) + return hasher.hexdigest() + + +def output_manifest(root: Path) -> tuple[list[dict[str, str | None]], str]: + entries: list[dict[str, str | None]] = [] + for path in sorted(root.rglob("*")): + relative = str(path.relative_to(root)) + if path.is_dir(): + entries.append({"path": relative, "kind": "directory", "sha256": None}) + elif path.is_file(): + entries.append({"path": relative, "kind": "file", "sha256": digest(path)}) + encoded = json.dumps(entries, sort_keys=True, separators=(",", ":")).encode() + return entries, hashlib.sha256(encoded).hexdigest() + + +def atomic_write(path: Path, value: dict[str, object]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with tempfile.NamedTemporaryFile("w", dir=path.parent, prefix=f".{path.name}.", delete=False) as stream: + json.dump(value, stream, indent=2) + stream.write("\n") + temporary = Path(stream.name) + os.replace(temporary, path) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--case-path", required=True) + parser.add_argument("--quickstart-root", required=True) + parser.add_argument("--runner-output", required=True, help="Runner result file, written by the runner command") + parser.add_argument("--completion-output", required=True, help="Validator-owned completion record") + parser.add_argument("--runner-command", nargs=argparse.REMAINDER, required=True) + args = parser.parse_args() + + repository = Path.cwd().resolve() + quickstart = Path(args.quickstart_root).resolve() + expected_root = repository / ".rhoai-qs" / "qs-test-quickstart" + if quickstart != expected_root: + parser.error("quickstart-root must be .rhoai-qs/qs-test-quickstart") + if not quickstart.is_dir(): + parser.error(f"quickstart root not found: {quickstart}") + if not args.runner_command: + parser.error("runner-command must not be empty") + + replacements = { + "{runner_output}": str(Path(args.runner_output).resolve()), + "{quickstart_root}": str(quickstart), + "{case_path}": str((repository / args.case_path).resolve()), + } + command = [ + argument.replace("{runner_output}", replacements["{runner_output}"]) + .replace("{quickstart_root}", replacements["{quickstart_root}"]) + .replace("{case_path}", replacements["{case_path}"]) + for argument in args.runner_command + ] + unresolved = [argument for argument in command if re.search(r"\{[a-z_]+\}", argument)] + if unresolved: + parser.error(f"unresolved runner-command placeholder(s): {', '.join(unresolved)}") + process = subprocess.run(command, cwd=repository, env=os.environ.copy(), check=False) + result_path = Path(args.runner_output).resolve() + result: dict[str, object] | None = None + if result_path.is_file(): + try: + loaded = yaml.safe_load(result_path.read_text()) + if isinstance(loaded, dict): + result = loaded + except yaml.YAMLError: + result = None + + status = result.get("status") if result else None + if process.returncode != 0 or status != "completed": + failure = { + "status": "failed", + "process_exit": process.returncode, + "runner_status": status, + "runner_result": result, + "failure": "runner did not complete successfully", + } + atomic_write(Path(args.completion_output).resolve(), failure) + print(json.dumps(failure, indent=2)) + return 1 + + entries, tree_hash = output_manifest(quickstart) + completion = { + "status": "completed", + "process_exit": process.returncode, + "runner_status": status, + "output_tree_hash": f"sha256:{tree_hash}", + "output_manifest": entries, + } + atomic_write(Path(args.completion_output).resolve(), completion) + print(json.dumps(completion, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/write_conversation.py b/core/skills/validate-skill/scripts/write_conversation.py new file mode 100644 index 0000000..cbafda3 --- /dev/null +++ b/core/skills/validate-skill/scripts/write_conversation.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +"""Render a caller-to-skill conversation retained in validation results.""" +import argparse +import json +from pathlib import Path + + +def fenced(value: str) -> list[str]: + marker = "```" + while marker in value: + marker += "`" + return [f"{marker}text", value, marker] + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--input", required=True, help="validation-results.json path") + parser.add_argument("--case", required=True, help="case name in the results") + parser.add_argument("--output", required=True, help="Markdown transcript path") + args = parser.parse_args() + + data = json.loads(Path(args.input).read_text()) + result = next((item for item in data.get("cases", []) if item.get("name") == args.case), None) + if result is None: + parser.error(f"case not found in results: {args.case}") + messages = result.get("conversation") + if not isinstance(messages, list) or not messages: + parser.error("case conversation must be a non-empty list") + + lines = [f"# Skill conversation: {data['target_skill']} / {args.case}", ""] + for message in messages: + role = message.get("role") if isinstance(message, dict) else None + content = message.get("content") if isinstance(message, dict) else None + if role not in {"caller", "target-skill"} or not isinstance(content, str): + parser.error("each conversation item needs role caller|target-skill and string content") + heading = "Caller" if role == "caller" else "Target skill" + lines.extend([f"## {heading}", "", *fenced(content), ""]) + + output = Path(args.output) + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text("\n".join(lines)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/scripts/write_report.py b/core/skills/validate-skill/scripts/write_report.py new file mode 100644 index 0000000..68de442 --- /dev/null +++ b/core/skills/validate-skill/scripts/write_report.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""Render a consistent Markdown validation report from agent-produced JSON results.""" +import argparse +import json +from datetime import datetime, timezone +from pathlib import Path + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--input", required=True, help="JSON file with target_skill and case results") + parser.add_argument("--output", required=True, help="Markdown report path") + args = parser.parse_args() + data = json.loads(Path(args.input).read_text()) + rows = [] + sections = [] + for result in data["cases"]: + score = f"{result['score']}/10" if result.get("score") is not None else "—" + rows.append(f"| {result['name']} | {result['status']} | {score} | {result.get('summary', '')} |") + sections.extend([ + f"## {result['name']}", + "", + "### Evidence", + "", + *([f"- {item}" for item in result.get("evidence", [])] or ["- None recorded."]), + "", + "### Differences", + "", + *([f"- {item}" for item in result.get("differences", [])] or ["- None."]), + "", + "### Blocked questions or failures", + "", + result.get("blocked_or_failure", "None."), + "", + ]) + lines = [ + f"# Validation report: {data['target_skill']}", + "", + f"Run: {data.get('run_at', datetime.now(timezone.utc).isoformat())}", + "Quickstart fixture path: `.rhoai-qs/qs-test-quickstart/`", + "", + "| Case | Status | Score | Summary |", + "|---|---|---:|---|", + *rows, + "", + *sections, + ] + output = Path(args.output) + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text("\n".join(lines)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/core/skills/validate-skill/spec-template.md b/core/skills/validate-skill/spec-template.md new file mode 100644 index 0000000..d0da860 --- /dev/null +++ b/core/skills/validate-skill/spec-template.md @@ -0,0 +1,37 @@ +# Validation Spec Template + +For each case, write `test/skills/reports///validation-spec.yaml` before running it: + +```yaml +spec_version: 1 +skill: validate-skill +target_skill: +quickstart_slug: qs-test-quickstart +created_at: +test_filter: +cases: + - name: + input_repo: test/skills///input-repo + expected_output_repo: test/skills///expected-output-repo + test_config: test/skills///test-config.yaml + run_skill_command: + question_answer_context: + validation_options: + mode: full-baseline | targeted + additional_instructions: + execution: + network: none | required + external_dependencies: + completion: + runner_gate: core/skills/validate-skill/scripts/run_case.py + status: completed +acceptance_criteria: + - id: isolated-fixtures + description: Each case starts with a fresh qs-test-quickstart copy. + validation: Confirm the target path is removed and recreated before every case. + requires_user_approval: false + - id: baseline-rating + description: Every completed case has an evidence-based 1–10 rating. + validation: Inspect the validation report. + requires_user_approval: false +``` diff --git a/core/skills/validate-skill/subagents/README.md b/core/skills/validate-skill/subagents/README.md new file mode 100644 index 0000000..e090cd8 --- /dev/null +++ b/core/skills/validate-skill/subagents/README.md @@ -0,0 +1,7 @@ +# validate-skill Subagents + +| Name | Purpose | Input | Output | When used | Why subagent | +|---|---|---|---|---|---| +| `validation-skill-prompt.md` | Verify that this meta-skill uses only its fixed disposable quickstart context | Fixed slug, target skill, fixture root | Context status | Phase 0 | Prevents a fixture run from overwriting a real quickstart | +| `case-runner-prompt.md` | Run one target-skill case from a fresh fixture | Target skill path, prompt, fixture root, copied quickstart root, optional answers | Execution status, artifacts, questions, final target-skill report | Once per case | Keeps target-skill execution isolated from orchestration and evaluation | +| `output-rater-prompt.md` | Independently rate actual output against baseline | Actual quickstart root, expected-output root, fixture contract | Score, evidence, differences | After each completed case | Avoids allowing the executor to grade its own work | diff --git a/core/skills/validate-skill/subagents/case-runner-prompt.md b/core/skills/validate-skill/subagents/case-runner-prompt.md new file mode 100644 index 0000000..8a4006e --- /dev/null +++ b/core/skills/validate-skill/subagents/case-runner-prompt.md @@ -0,0 +1,52 @@ +--- +description: Run one fixture case through a target Factory skill without changing its baseline. +--- + +# Case Runner + +## Your Role + +Execute the target Factory skill exactly as a user would, using the copied fixture at `.rhoai-qs/qs-test-quickstart/`. Report facts about execution; do not rate the output. + +The case runner is already the execution agent. Perform the target skill in the current +runner context; do not launch another model client or nested agent process from a shell. +A nested model invocation introduces separate authentication and network requirements and +is not part of the fixture case. + +Preserve the runner's existing authentication context. Never replace authentication-related +environment variables with a new temporary directory. + +## Inputs + +- `target_skill_path` +- `target_prompt`: optional `run-skill-command` value from `test-config.yaml` +- `fixture_root` +- `quickstart_root` +- `question_answer_context`: optional text from `test-config.yaml` +- `execution_network`: `none` or `required` +- `external_dependencies`: repositories the runner must be able to reach when network is required + +## Instructions + +1. Invoke the already-discovered target skill by its supplied name using the runtime's native skill mechanism. Do not manually read or replay its `SKILL.md`. When present, treat `target_prompt` as the user's request. When absent, invoke the target skill without inventing an additional user request. + If `execution_network` is `required`, verify that the runner has network access to the declared `external_dependencies` before starting the target skill. If access is unavailable, return `failed` with the dependency error; do not substitute local files or fabricate target output. The runner execution context must also have read/write access to `quickstart_root`; a network-capable context that cannot access the disposable quickstart is not valid. +2. Work only in `quickstart_root` and ordinary target-skill output locations. `quickstart_root` is the fixed `.rhoai-qs/qs-test-quickstart/` path for this validation run; do not substitute another temporary directory or user-selected quickstart. Never modify `fixture_root/input-repo` or `fixture_root/expected-output-repo`. +3. Record a chronological `conversation` entry for the initial caller prompt, every target-skill user-facing response or question, and every caller answer. Do not record tool output or internal reasoning. If no `target_prompt` was provided, record that the target skill was invoked without a supplemental prompt. +4. If the target skill asks a question, use `question_answer_context` only for an exact, fixture-grounded answer. Record the question and answer in `conversation`. If it does not establish one, or the question requests user judgment, destructive overwrite approval, credentials, cluster access, GitHub mutation, or any external action, stop and return it as blocked. +5. Preserve all target-skill safety restrictions. In particular, do not run `oc`, `kubectl`, provisioning scripts, or secret-setting commands merely because a fixture exists. +6. If the target skill references a subagent prompt, follow that prompt within the current runner context or use the platform's already-available delegation mechanism; never invoke another model through a CLI or nested process. + +## Output + +Return raw YAML only, with no prose, markdown fences, or other formatting: + +--- +status: completed | blocked | failed +target_skill_report: +artifacts: + - +conversation: + - role: caller | target-skill + content: +blocked_question: +failure: diff --git a/core/skills/validate-skill/subagents/output-rater-prompt.md b/core/skills/validate-skill/subagents/output-rater-prompt.md new file mode 100644 index 0000000..0e3c01e --- /dev/null +++ b/core/skills/validate-skill/subagents/output-rater-prompt.md @@ -0,0 +1,44 @@ +--- +description: Independently compare one target-skill output with its expected fixture baseline and assign an evidence-based score. +--- + +# Output Rater + +## Your Role + +Rate one completed fixture case from 1 to 10 against its approved expected-output baseline. You did not execute the target skill and must base conclusions on observable artifacts. + +## Inputs + +- `fixture_contract_path` +- `expected_output_root` +- `actual_quickstart_root` +- `case_name` +- `comparison_json`: output from `scripts/compare_baseline.py` +- `validation_mode`: `full-baseline` or `targeted`, from `test-config.yaml`'s `validation-options.mode` +- `additional_validation_instructions`: optional text from `test-config.yaml`'s `validation-options.additional-instructions` +- `conversation`: the recorded caller-to-target-skill messages from the completed case runner; use this as evidence for interaction-specific assertions, not as a self-assessment or score + +## Instructions + +1. Read the fixture contract and the comparison JSON. In `full-baseline` mode, treat every file and directory path in the expected-output repository as a required post-run artifact. In `targeted` mode, use the comparison inventory only as evidence and do not treat its differences as defects unless they are named by the additional validation instructions. +2. Use the script's file inventory, required-path results, and hashes as evidence. Inspect content only where needed to judge a reported difference or targeted criterion. + For targeted criteria about questions, answers, or other user interaction, use the supplied runner conversation as the observable evidence. Do not require that conversational behavior to be duplicated in generated repository files. +3. In `full-baseline` mode, allow only documented dynamic differences. Do not penalize unrelated input files that predate the run. +4. In `targeted` mode, follow `additional_validation_instructions` as the exclusive evaluation scope. It must be present and cannot authorize external actions. +5. Score functional conformity: 10 means complete conformity with the applicable oracle; 8–9 means minor nonfunctional differences; 5–7 means partial functionality; 1–4 means major required behavior is missing or wrong. Never score a blocked or failed execution. + +## Output + +```yaml +case: +score: +summary: +required_artifacts: + - path: + status: present | missing | differs +differences: + - path: + impact: critical | material | minor + detail: +``` diff --git a/core/skills/validate-skill/subagents/validation-skill-prompt.md b/core/skills/validate-skill/subagents/validation-skill-prompt.md new file mode 100644 index 0000000..6782d96 --- /dev/null +++ b/core/skills/validate-skill/subagents/validation-skill-prompt.md @@ -0,0 +1,33 @@ +--- +description: Verify that validate-skill is operating only on its fixed disposable quickstart fixture. +--- + +# Validation Context + +## Your Role + +Confirm that this meta-skill will use the dedicated disposable quickstart path and cannot be confused with a real quickstart selection. + +## Inputs + +- `slug` +- `quickstart_root` +- `fixture_root` +- `target_skill` + +## Resolution Logic + +The only valid slug is `qs-test-quickstart`. The only valid quickstart root is `.rhoai-qs/qs-test-quickstart/`, and the fixture root must be under `test/skills//`. + +Return `error` if any input differs. Do not attempt fuzzy matching or ask the user to select another quickstart; this skill intentionally has a fixed disposable context. + +## Output + +```json +{ + "resolution": "resolved | error", + "slug": "qs-test-quickstart | null", + "confidence": "high", + "error_message": "string | null" +} +``` diff --git a/test/skills/rh-qs-test-suite/alternate-make-targets/expected-output-repo/.gitkeep b/test/skills/rh-qs-test-suite/alternate-make-targets/expected-output-repo/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/Makefile b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/Makefile new file mode 100644 index 0000000..d87234d --- /dev/null +++ b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/Makefile @@ -0,0 +1,17 @@ +NAMESPACE ?= qs-test-quickstart-ci-project-1 +RELEASE ?= qs-test-quickstart +CHART ?= helm + +.PHONY: helm-install-prod test-all helm-uninstall verify-deploy + +helm-install-prod: + helm upgrade --install $(RELEASE) $(CHART) --namespace $(NAMESPACE) --create-namespace + +test-all: + @echo "Nightly tests passed" + +helm-uninstall: + helm uninstall $(RELEASE) --namespace $(NAMESPACE) + +verify-deploy: + helm status $(RELEASE) --namespace $(NAMESPACE) diff --git a/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/README.md b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/README.md new file mode 100644 index 0000000..fdcacd9 --- /dev/null +++ b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/README.md @@ -0,0 +1,4 @@ +# Alternate target fixture + +This fixture deliberately uses equivalent Makefile target names for nightly +deployment, testing, and teardown. diff --git a/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/Chart.yaml b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/Chart.yaml new file mode 100644 index 0000000..a631d91 --- /dev/null +++ b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: nightly-fixture +description: Minimal chart for alternate Makefile target testing +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/templates/deployment.yaml b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/templates/deployment.yaml new file mode 100644 index 0000000..4addaea --- /dev/null +++ b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/templates/deployment.yaml @@ -0,0 +1,18 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nightly-fixture +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app: nightly-fixture + template: + metadata: + labels: + app: nightly-fixture + spec: + containers: + - name: fixture + image: registry.access.redhat.com/ubi9/ubi-micro:latest + command: ["/bin/sh", "-c", "sleep 3600"] diff --git a/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/values.yaml b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/values.yaml new file mode 100644 index 0000000..ebf6be8 --- /dev/null +++ b/test/skills/rh-qs-test-suite/alternate-make-targets/input-repo/helm/values.yaml @@ -0,0 +1 @@ +replicaCount: 1 diff --git a/test/skills/rh-qs-test-suite/alternate-make-targets/test-config.yaml b/test/skills/rh-qs-test-suite/alternate-make-targets/test-config.yaml new file mode 100644 index 0000000..cdfcaf3 --- /dev/null +++ b/test/skills/rh-qs-test-suite/alternate-make-targets/test-config.yaml @@ -0,0 +1,12 @@ +run-skill-command: Add nightly testing to qs-test-quickstart. +execution: + network: required + external-dependencies: + - https://github.com/rh-ai-quickstart/quickstart-ci-testing +validation-options: + mode: targeted + additional-instructions: | + Validate that the skill resolves deploy to helm-install-prod, test to test-all, + and undeploy to helm-uninstall, and uses those exact commands in the generated + reusable nightly workflow and manifest. Do not require unrelated output files + to match the baseline. diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/cleanup/action.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/cleanup/action.yml new file mode 100644 index 0000000..977fafc --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/cleanup/action.yml @@ -0,0 +1,27 @@ +name: Cleanup Deployment +description: Run cleanup command to tear down a deployed quickstart + +inputs: + cleanup_command: + description: Command to tear down the deployment + required: true + working_dir: + description: Directory to run cleanup in + required: false + default: ./quickstart + +runs: + using: composite + steps: + - name: Cleanup deployment + shell: bash + working-directory: ${{ inputs.working_dir }} + continue-on-error: true + env: + CLEANUP_COMMAND: ${{ inputs.cleanup_command }} + run: eval "$CLEANUP_COMMAND" + + - name: Delete remaining resources + shell: bash + continue-on-error: true + run: oc delete pods,deployments,replicasets,statefulsets,services,jobs,cronjobs,daemonsets,pvc --all -n "$NAMESPACE" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/clone-quickstart/action.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/clone-quickstart/action.yml new file mode 100644 index 0000000..ec18450 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/clone-quickstart/action.yml @@ -0,0 +1,52 @@ +name: Clone Quickstart +description: Clone a quickstart repository at a specified ref + +inputs: + repo_url: + description: Full GitHub URL of the quickstart repo + required: true + ref: + description: Branch, tag, or SHA to check out + required: false + default: main + checkout_path: + description: Directory to clone into + required: false + default: ./quickstart + submodules: + description: Whether to clone submodules (true/false) + required: false + default: 'false' + +runs: + using: composite + steps: + - name: Clone quickstart repository + shell: bash + env: + REPO_URL: ${{ inputs.repo_url }} + REF: ${{ inputs.ref }} + CHECKOUT_PATH: ${{ inputs.checkout_path }} + SUBMODULES: ${{ inputs.submodules }} + run: | + CLONE_ARGS="--depth 1 --branch $REF" + if [ "$SUBMODULES" = "true" ]; then + CLONE_ARGS="$CLONE_ARGS --recurse-submodules" + fi + + git clone $CLONE_ARGS "$REPO_URL" "$CHECKOUT_PATH" + + - name: Print clone info + shell: bash + working-directory: ${{ inputs.checkout_path }} + env: + REPO_URL: ${{ inputs.repo_url }} + REF: ${{ inputs.ref }} + CHECKOUT_PATH: ${{ inputs.checkout_path }} + run: | + echo "=== Clone Info ===" + echo "Repository: $REPO_URL" + echo "Ref: $REF" + echo "Commit SHA: $(git rev-parse HEAD)" + echo "Branch: $(git branch --show-current || echo 'detached')" + echo "Clone path: $CHECKOUT_PATH" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/prepare-runner/action.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/prepare-runner/action.yml new file mode 100644 index 0000000..62f8266 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/prepare-runner/action.yml @@ -0,0 +1,39 @@ +name: Prepare Runner +description: Free disk space and check memory/swap status before running tests + +inputs: + swap-storage: + description: Enable swap storage (may slow down if heavily used) + required: false + default: 'false' + +runs: + using: composite + steps: + - name: Free Disk Space + uses: jlumbroso/free-disk-space@v1.3.1 + with: + android: true + dotnet: true + haskell: true + large-packages: false + docker-images: false + swap-storage: ${{ inputs.swap-storage }} + tool-cache: false + + - name: Install oc CLI + shell: bash + run: | + curl -sL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz | tar xzf - -C /usr/local/bin oc kubectl + + - name: Check memory and swap status + shell: bash + run: | + echo "=== Memory Status ===" + free -h + echo "" + echo "=== Swap Usage ===" + swapon --show || echo "No swap configured" + echo "" + echo "=== Disk Space ===" + df -h diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/run-quickstart-tests/action.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/run-quickstart-tests/action.yml new file mode 100644 index 0000000..fd54341 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/run-quickstart-tests/action.yml @@ -0,0 +1,71 @@ +name: Run Tests +description: Run pre-test setup, tests, and optionally upload result artifacts + +inputs: + test_command: + description: Command to run tests + required: true + test_working_dir: + description: Directory for tests + required: false + default: ./quickstart + pre_test_command: + description: Setup command to run before tests + required: false + default: '' + upload_results: + description: >- + Space-separated list of directories (relative to test_working_dir) + to upload as artifacts after tests complete. Example: "evaluations/results test/reports" + required: false + default: '' + upload_retention_days: + description: Number of days to retain uploaded artifacts + required: false + default: '2' + +runs: + using: composite + steps: + - name: Pre-test setup + if: inputs.pre_test_command != '' + shell: bash + working-directory: ./quickstart + env: + PRE_TEST_COMMAND: ${{ inputs.pre_test_command }} + run: eval "$PRE_TEST_COMMAND" + + - name: Run tests + uses: ./.github/actions/run-tests + with: + test_command: ${{ inputs.test_command }} + working_dir: ${{ inputs.test_working_dir }} + + - name: Prepare upload paths + if: always() && inputs.upload_results != '' + id: upload-paths + shell: bash + env: + UPLOAD_RESULTS: ${{ inputs.upload_results }} + WORKING_DIR: ${{ inputs.test_working_dir }} + run: | + PATHS="" + for dir in $UPLOAD_RESULTS; do + if [[ "$dir" == /* ]] || [[ "$dir" == *..* ]]; then + echo "::error::upload_results paths must be relative to working dir and cannot contain '..': $dir" + exit 1 + fi + PATHS="${PATHS}${WORKING_DIR}/${dir}"$'\n' + done + echo "paths<> "$GITHUB_OUTPUT" + echo "$PATHS" >> "$GITHUB_OUTPUT" + echo "EOF" >> "$GITHUB_OUTPUT" + + - name: Upload test results + if: always() && inputs.upload_results != '' + uses: actions/upload-artifact@v4 + with: + name: test-results-${{ github.job }}-${{ github.run_id }} + path: ${{ steps.upload-paths.outputs.paths }} + retention-days: ${{ inputs.upload_retention_days }} + if-no-files-found: warn diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/run-tests/action.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/run-tests/action.yml new file mode 100644 index 0000000..61b7699 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/run-tests/action.yml @@ -0,0 +1,22 @@ +name: Run Tests +description: Run tests via a configurable command + +inputs: + test_command: + description: Command to run tests (e.g. "make test-all" or "pytest tests/ -v") + required: true + working_dir: + description: Directory to run tests in + required: false + default: ./quickstart + +runs: + using: composite + steps: + - name: Run tests + shell: bash + working-directory: ${{ inputs.working_dir }} + env: + TEST_COMMAND: ${{ inputs.test_command }} + run: | + eval "$TEST_COMMAND" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-and-deploy/action.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-and-deploy/action.yml new file mode 100644 index 0000000..5309ba8 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-and-deploy/action.yml @@ -0,0 +1,153 @@ +name: Setup and Deploy Quickstart +description: Prepare runner, clone quickstart repo, and deploy to OpenShift or KinD + +inputs: + repo_url: + description: GitHub repo URL to clone + required: true + ref: + description: Branch/tag/SHA to check out + required: false + default: main + submodules: + description: Whether to clone submodules + required: false + default: 'false' + use_kind: + description: Use KinD instead of OpenShift + required: false + default: 'false' + deploy_command: + description: Command to deploy the application + required: true + namespace: + description: Kubernetes namespace + required: false + default: quickstart-ci-project-1 + setup_commands: + description: Pre-deploy setup commands + required: false + default: '' + kind_pre_deploy_command: + description: Pre-deploy setup for KinD + required: false + default: '' + extra_deploy_args: + description: Extra args appended to deploy command + required: false + default: '' + kind_extra_deploy_args: + description: Extra args appended to deploy command only when using KinD + required: false + default: '' + build_images: + description: Build and push images to local KinD registry (only valid with use_kind) + required: false + default: 'false' + build_images_command: + description: Command to build images + required: false + default: 'make build-all-images' + push_images_command: + description: Command to push images to local registry + required: false + default: 'make push-all-images' + +runs: + using: composite + steps: + - name: Prepare Runner + uses: ./.github/actions/prepare-runner + + - name: Export namespace + shell: bash + run: echo "NAMESPACE=${{ inputs.namespace }}" >> "$GITHUB_ENV" + + - name: Validate build_images requires use_kind + if: inputs.build_images == 'true' && inputs.use_kind != 'true' + shell: bash + run: | + echo "::error::build_images can only be used when use_kind is 'true'" + exit 1 + + - name: Validate OpenShift secrets + if: inputs.use_kind != 'true' + shell: bash + run: | + if [ -z "${PROD_TOKEN}" ]; then + echo "::error::PROD_TOKEN secret is missing or empty." + exit 1 + fi + if [ -z "${PROD_SERVER}" ]; then + echo "::error::PROD_SERVER secret is missing or empty." + exit 1 + fi + + - name: Clone quickstart + uses: ./.github/actions/clone-quickstart + with: + repo_url: ${{ inputs.repo_url }} + ref: ${{ inputs.ref }} + submodules: ${{ inputs.submodules }} + + - name: Setup OpenShift + if: inputs.use_kind != 'true' + uses: ./.github/actions/setup-openshift + with: + namespace: ${{ inputs.namespace }} + setup_commands: ${{ inputs.setup_commands }} + + - name: Print environment versions + if: inputs.use_kind != 'true' + shell: bash + env: + NAMESPACE: ${{ inputs.namespace }} + run: | + echo "=== OpenShift Version ===" + oc version || true + + echo "" + echo "=== OpenShift AI Operator Version ===" + oc get csv -n "$NAMESPACE" \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.version}{"\n"}{end}' 2>/dev/null \ + | grep -iE 'rhods|rhoai|opendatahub' \ + || echo "(OpenShift AI operator not found)" + + - name: Setup KinD + if: inputs.use_kind == 'true' + uses: ./.github/actions/setup-kind + with: + namespace: ${{ inputs.namespace }} + pre_deploy_command: ${{ inputs.kind_pre_deploy_command || inputs.setup_commands }} + build_images: ${{ inputs.build_images }} + build_images_command: ${{ inputs.build_images_command }} + push_images_command: ${{ inputs.push_images_command }} + + - name: Deploy application + if: inputs.build_images != 'true' + shell: bash + working-directory: ./quickstart + env: + DEPLOY_COMMAND: ${{ inputs.deploy_command }} + EXTRA_DEPLOY_ARGS: ${{ inputs.extra_deploy_args }} ${{ inputs.use_kind == 'true' && inputs.kind_extra_deploy_args || '' }} + NAMESPACE: ${{ inputs.namespace }} + run: eval "$DEPLOY_COMMAND" $EXTRA_DEPLOY_ARGS + + - name: Deploy application (local images) + if: inputs.build_images == 'true' + shell: bash + working-directory: ./quickstart + env: + DEPLOY_COMMAND: ${{ inputs.deploy_command }} + EXTRA_DEPLOY_ARGS: ${{ inputs.extra_deploy_args }} ${{ inputs.use_kind == 'true' && inputs.kind_extra_deploy_args || '' }} + NAMESPACE: ${{ inputs.namespace }} + REGISTRY: localhost:5001 + run: eval "$DEPLOY_COMMAND" $EXTRA_DEPLOY_ARGS + + - name: Verify deployment + shell: bash + env: + NAMESPACE: ${{ inputs.namespace }} + run: | + echo "=== Pod Status ===" + oc get pods -n "$NAMESPACE" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-kind/action.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-kind/action.yaml new file mode 100644 index 0000000..bef0838 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-kind/action.yaml @@ -0,0 +1,78 @@ +name: Setup KinD cluster +description: Create a KinD cluster with local registry and prepare for deployment + +inputs: + namespace: + description: Kubernetes namespace to deploy into + required: true + working_dir: + description: Directory containing the quickstart code + required: false + default: ./quickstart + kind_version: + description: KinD version to install + required: false + default: v0.30.0 + pre_deploy_command: + description: Command to run before deploy (e.g. helm dependency build) + required: false + default: '' + build_images: + description: Build and push images to the local KinD registry + required: false + default: 'false' + build_images_command: + description: Command to build images + required: false + default: 'make build-all-images' + push_images_command: + description: Command to push images to local registry + required: false + default: 'make push-all-images' + +runs: + using: composite + steps: + - name: Install Kind + shell: bash + env: + KIND_VERSION: ${{ inputs.kind_version }} + run: go install "sigs.k8s.io/kind@${KIND_VERSION}" + + - name: Create Kind Cluster with local registry + run: bash ${GITHUB_ACTION_PATH}/../../../scripts/ci/kind-with-registry.sh + shell: bash + + - name: Pre-deploy setup + if: inputs.pre_deploy_command != '' + working-directory: ${{ inputs.working_dir }} + shell: bash + env: + PRE_DEPLOY_COMMAND: ${{ inputs.pre_deploy_command }} + NAMESPACE: ${{ inputs.namespace }} + run: eval "$PRE_DEPLOY_COMMAND" + + - name: Build images + if: inputs.build_images == 'true' + working-directory: ${{ inputs.working_dir }} + shell: bash + env: + BUILD_COMMAND: ${{ inputs.build_images_command }} + REGISTRY: localhost:5001 + run: eval "$BUILD_COMMAND" + + - name: Push images to local registry + if: inputs.build_images == 'true' + working-directory: ${{ inputs.working_dir }} + shell: bash + env: + PUSH_COMMAND: ${{ inputs.push_images_command }} + REGISTRY: localhost:5001 + PUSH_EXTRA_AGRS: "--tls-verify=false" + run: eval "$PUSH_COMMAND" + + - name: Create namespace + shell: bash + env: + NAMESPACE: ${{ inputs.namespace }} + run: kubectl create namespace "$NAMESPACE" || true diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-openshift/action.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-openshift/action.yml new file mode 100644 index 0000000..4beb476 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/actions/setup-openshift/action.yml @@ -0,0 +1,33 @@ +name: Setup OpenShift +description: Log into OpenShift and run any pre-deploy setup commands + +inputs: + working_dir: + description: Directory to run commands in + required: false + default: ./quickstart + namespace: + description: OpenShift namespace/project to deploy into + required: true + setup_commands: + description: Pre-deploy setup commands (newline-separated) + required: false + default: '' + +runs: + using: composite + steps: + - name: Log into OpenShift cluster + shell: bash + run: | + oc login --token="$PROD_TOKEN" --server="$PROD_SERVER" + + - name: Run setup commands + if: inputs.setup_commands != '' + shell: bash + working-directory: ${{ inputs.working_dir }} + env: + SETUP_COMMANDS: ${{ inputs.setup_commands }} + NAMESPACE: ${{ inputs.namespace }} + run: | + eval "$SETUP_COMMANDS" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/workflows/nightly-main.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/workflows/nightly-main.yml new file mode 100644 index 0000000..673f50b --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/workflows/nightly-main.yml @@ -0,0 +1,13 @@ +name: Nightly - qs-test-quickstart (main) + +on: + workflow_dispatch: + schedule: + - cron: "0 5 * * *" + +jobs: + test: + uses: ./.github/workflows/test-qs-test-quickstart.yml + with: + branch: main + secrets: inherit diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/workflows/test-qs-test-quickstart.yml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/workflows/test-qs-test-quickstart.yml new file mode 100644 index 0000000..37b53c0 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/.github/workflows/test-qs-test-quickstart.yml @@ -0,0 +1,57 @@ +name: Test qs-test-quickstart + +on: + workflow_call: + inputs: + branch: + required: true + type: string + workflow_dispatch: + inputs: + branch: + description: Branch to deploy and test + default: main + type: string + +concurrency: + group: qs-test-quickstart-ci-testing + cancel-in-progress: false + +jobs: + test: + name: qs-test-quickstart (${{ inputs.branch }}) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - name: Setup and deploy + uses: ./.github/actions/setup-and-deploy + timeout-minutes: 15 + with: + repo_url: https://github.com/${{ github.repository }} + ref: ${{ inputs.branch }} + namespace: qs-test-quickstart-ci-project-1 + deploy_command: "make deploy" + env: + PROD_TOKEN: ${{ secrets.PROD_TOKEN }} + PROD_SERVER: ${{ secrets.PROD_SERVER }} + MODEL_ID: ${{ vars.MODEL_ID }} + MODEL_URL: ${{ vars.MODEL_URL }} + MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }} + + - name: Run tests + uses: ./.github/actions/run-quickstart-tests + timeout-minutes: 30 + with: + test_command: "make test" + upload_results: "test-results" + env: + MODEL_ID: ${{ vars.MODEL_ID }} + MODEL_URL: ${{ vars.MODEL_URL }} + MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }} + + - name: Cleanup + if: always() + uses: ./.github/actions/cleanup + with: + cleanup_command: "make undeploy" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/Makefile b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/Makefile new file mode 100644 index 0000000..e65708f --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/Makefile @@ -0,0 +1,28 @@ +NAMESPACE ?= nightly-ci-fixture-ci-project-1 +RELEASE ?= nightly-ci-fixture +CHART ?= helm +MODEL_ID ?= +MODEL_URL ?= +MODEL_API_KEY ?= + +.PHONY: check-model-config deploy test undeploy verify-deploy + +check-model-config: + @test -n "$(MODEL_ID)" || (echo "MODEL_ID must be set"; exit 1) + @test -n "$(MODEL_URL)" || (echo "MODEL_URL must be set"; exit 1) + @test -n "$(MODEL_API_KEY)" || (echo "MODEL_API_KEY must be set"; exit 1) + +deploy: check-model-config + helm upgrade --install $(RELEASE) $(CHART) --namespace $(NAMESPACE) --create-namespace \ + --set-string model.id="$(MODEL_ID)" \ + --set-string model.url="$(MODEL_URL)" \ + --set-string model.apiKey="$(MODEL_API_KEY)" + +test: check-model-config + @echo "Nightly CI fixture model tests passed" + +undeploy: + helm uninstall $(RELEASE) --namespace $(NAMESPACE) + +verify-deploy: + helm status $(RELEASE) --namespace $(NAMESPACE) diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/README.md b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/README.md new file mode 100644 index 0000000..2e966f3 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/README.md @@ -0,0 +1,43 @@ +# Nightly CI Demo + +A minimal Helm-deployed service for demonstrating a Quickstart deployment workflow. + +## Prerequisites + +- Access to an OpenShift cluster +- A model deployment ID, API URL, and API key + +## Deploy + +Set the target namespace: + +```bash +export NAMESPACE=your-namespace +export MODEL_ID=granite-3.1-8b-instruct +export MODEL_URL=https://model.example.com/v1 +export MODEL_API_KEY=your-model-api-key +``` + +Deploy the chart: + +```bash +make deploy NAMESPACE=$NAMESPACE MODEL_ID=$MODEL_ID MODEL_URL=$MODEL_URL MODEL_API_KEY=$MODEL_API_KEY +``` + +Check the release: + +```bash +make verify-deploy NAMESPACE=$NAMESPACE +``` + +## Test + +```bash +make test MODEL_ID=$MODEL_ID MODEL_URL=$MODEL_URL MODEL_API_KEY=$MODEL_API_KEY +``` + +## Remove + +```bash +make undeploy NAMESPACE=$NAMESPACE +``` diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/Chart.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/Chart.yaml new file mode 100644 index 0000000..e5ef944 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: nightly-ci-fixture +description: Minimal namespaced chart for rh-qs-test-suite tests +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/_helpers.tpl b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/_helpers.tpl new file mode 100644 index 0000000..285088e --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/_helpers.tpl @@ -0,0 +1,13 @@ +{{- define "nightly-ci-fixture.fullname" -}} +{{- .Release.Name -}} +{{- end -}} + +{{- define "nightly-ci-fixture.labels" -}} +app.kubernetes.io/name: {{ .Chart.Name }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "nightly-ci-fixture.selectorLabels" -}} +app.kubernetes.io/name: {{ .Chart.Name }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/cluster-role.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/cluster-role.yaml new file mode 100644 index 0000000..6e6942e --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/cluster-role.yaml @@ -0,0 +1,10 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "nightly-ci-fixture.fullname" . }}-model-reader + labels: + {{- include "nightly-ci-fixture.labels" . | nindent 4 }} +rules: + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list"] diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/deployment.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/deployment.yaml new file mode 100644 index 0000000..d36182e --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/templates/deployment.yaml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "nightly-ci-fixture.fullname" . }} + labels: + {{- include "nightly-ci-fixture.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + {{- include "nightly-ci-fixture.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "nightly-ci-fixture.selectorLabels" . | nindent 8 }} + spec: + containers: + - name: fixture + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + command: ["/bin/sh", "-c", "sleep 3600"] + env: + - name: MODEL_ID + value: {{ .Values.model.id | quote }} + - name: MODEL_URL + value: {{ .Values.model.url | quote }} + - name: MODEL_API_KEY + value: {{ .Values.model.apiKey | quote }} diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/values.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/values.yaml new file mode 100644 index 0000000..ce2c3a5 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/helm/values.yaml @@ -0,0 +1,10 @@ +replicaCount: 1 + +image: + repository: registry.access.redhat.com/ubi9/ubi-micro + tag: latest + +model: + id: "" + url: "" + apiKey: "" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/pipeline/nightly-tests-manifest.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/pipeline/nightly-tests-manifest.yaml new file mode 100644 index 0000000..ebf0df6 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/pipeline/nightly-tests-manifest.yaml @@ -0,0 +1,45 @@ +slug: qs-test-quickstart +resolved_targets: + deploy: deploy + test: test + undeploy: undeploy +toolchain_setup_steps: [] +branch_model: + main: main + dev: null +namespace: qs-test-quickstart-ci-project-1 +service_account: qs-test-quickstart-ci-1 +cluster_admin_check: + status: flagged + detail: helm/templates/cluster-role.yaml defines a ClusterRole, which requires cluster-scoped permissions. +required_secrets: + - name: PROD_TOKEN + reason: ServiceAccount token for the nightly CI namespace + - name: PROD_SERVER + reason: OpenShift API server URL for the cluster nightly deploys to + - name: MODEL_API_KEY + reason: API key required by the deploy and test Makefile targets +required_variables: + - name: MODEL_ID + reason: Model deployment ID required by the deploy and test Makefile targets + - name: MODEL_URL + reason: Model API URL required by the deploy and test Makefile targets +unconfirmed: + - No toolchain setup step added; Helm is invoked by the Makefile, but no pinned version or existing workflow establishes a setup action. +files: + actions_copied: + - cleanup + - clone-quickstart + - prepare-runner + - run-quickstart-tests + - run-tests + - setup-and-deploy + - setup-kind + - setup-openshift + actions_skipped_identical: [] + actions_updated: [] + workflows_created: + - .github/workflows/test-qs-test-quickstart.yml + - .github/workflows/nightly-main.yml + workflows_skipped_or_updated: [] + provisioning_script: scripts/ci/setup-nightly-namespace.sh diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/scripts/ci/setup-nightly-namespace.sh b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/scripts/ci/setup-nightly-namespace.sh new file mode 100755 index 0000000..dcbd646 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/expected-output-repo/scripts/ci/setup-nightly-namespace.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +set -euo pipefail + +NAMESPACE="${NAMESPACE:-qs-test-quickstart-ci-project-1}" +SA_NAME="${SA_NAME:-qs-test-quickstart-ci-1}" +TOKEN_DURATION="${TOKEN_DURATION:-8760h}" + +oc create namespace "$NAMESPACE" +oc create sa "$SA_NAME" -n "$NAMESPACE" +oc adm policy add-role-to-user admin \ + "system:serviceaccount:${NAMESPACE}:${SA_NAME}" \ + -n "$NAMESPACE" + +echo +echo "PROD_SERVER: $(oc whoami --show-server)" +echo "PROD_TOKEN:" +oc create token "$SA_NAME" -n "$NAMESPACE" --duration="$TOKEN_DURATION" + +echo +echo "Set the two values above as repository secrets PROD_SERVER and PROD_TOKEN" +echo "(Settings -> Secrets and variables -> Actions) before enabling the nightly workflows." diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/Makefile b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/Makefile new file mode 100644 index 0000000..e65708f --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/Makefile @@ -0,0 +1,28 @@ +NAMESPACE ?= nightly-ci-fixture-ci-project-1 +RELEASE ?= nightly-ci-fixture +CHART ?= helm +MODEL_ID ?= +MODEL_URL ?= +MODEL_API_KEY ?= + +.PHONY: check-model-config deploy test undeploy verify-deploy + +check-model-config: + @test -n "$(MODEL_ID)" || (echo "MODEL_ID must be set"; exit 1) + @test -n "$(MODEL_URL)" || (echo "MODEL_URL must be set"; exit 1) + @test -n "$(MODEL_API_KEY)" || (echo "MODEL_API_KEY must be set"; exit 1) + +deploy: check-model-config + helm upgrade --install $(RELEASE) $(CHART) --namespace $(NAMESPACE) --create-namespace \ + --set-string model.id="$(MODEL_ID)" \ + --set-string model.url="$(MODEL_URL)" \ + --set-string model.apiKey="$(MODEL_API_KEY)" + +test: check-model-config + @echo "Nightly CI fixture model tests passed" + +undeploy: + helm uninstall $(RELEASE) --namespace $(NAMESPACE) + +verify-deploy: + helm status $(RELEASE) --namespace $(NAMESPACE) diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/README.md b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/README.md new file mode 100644 index 0000000..2e966f3 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/README.md @@ -0,0 +1,43 @@ +# Nightly CI Demo + +A minimal Helm-deployed service for demonstrating a Quickstart deployment workflow. + +## Prerequisites + +- Access to an OpenShift cluster +- A model deployment ID, API URL, and API key + +## Deploy + +Set the target namespace: + +```bash +export NAMESPACE=your-namespace +export MODEL_ID=granite-3.1-8b-instruct +export MODEL_URL=https://model.example.com/v1 +export MODEL_API_KEY=your-model-api-key +``` + +Deploy the chart: + +```bash +make deploy NAMESPACE=$NAMESPACE MODEL_ID=$MODEL_ID MODEL_URL=$MODEL_URL MODEL_API_KEY=$MODEL_API_KEY +``` + +Check the release: + +```bash +make verify-deploy NAMESPACE=$NAMESPACE +``` + +## Test + +```bash +make test MODEL_ID=$MODEL_ID MODEL_URL=$MODEL_URL MODEL_API_KEY=$MODEL_API_KEY +``` + +## Remove + +```bash +make undeploy NAMESPACE=$NAMESPACE +``` diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/Chart.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/Chart.yaml new file mode 100644 index 0000000..e5ef944 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: nightly-ci-fixture +description: Minimal namespaced chart for rh-qs-test-suite tests +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/_helpers.tpl b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/_helpers.tpl new file mode 100644 index 0000000..285088e --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/_helpers.tpl @@ -0,0 +1,13 @@ +{{- define "nightly-ci-fixture.fullname" -}} +{{- .Release.Name -}} +{{- end -}} + +{{- define "nightly-ci-fixture.labels" -}} +app.kubernetes.io/name: {{ .Chart.Name }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "nightly-ci-fixture.selectorLabels" -}} +app.kubernetes.io/name: {{ .Chart.Name }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/cluster-role.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/cluster-role.yaml new file mode 100644 index 0000000..6e6942e --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/cluster-role.yaml @@ -0,0 +1,10 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "nightly-ci-fixture.fullname" . }}-model-reader + labels: + {{- include "nightly-ci-fixture.labels" . | nindent 4 }} +rules: + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list"] diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/deployment.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/deployment.yaml new file mode 100644 index 0000000..d36182e --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/templates/deployment.yaml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "nightly-ci-fixture.fullname" . }} + labels: + {{- include "nightly-ci-fixture.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + {{- include "nightly-ci-fixture.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "nightly-ci-fixture.selectorLabels" . | nindent 8 }} + spec: + containers: + - name: fixture + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + command: ["/bin/sh", "-c", "sleep 3600"] + env: + - name: MODEL_ID + value: {{ .Values.model.id | quote }} + - name: MODEL_URL + value: {{ .Values.model.url | quote }} + - name: MODEL_API_KEY + value: {{ .Values.model.apiKey | quote }} diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/values.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/values.yaml new file mode 100644 index 0000000..ce2c3a5 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/input-repo/helm/values.yaml @@ -0,0 +1,10 @@ +replicaCount: 1 + +image: + repository: registry.access.redhat.com/ubi9/ubi-micro + tag: latest + +model: + id: "" + url: "" + apiKey: "" diff --git a/test/skills/rh-qs-test-suite/cluster-privileges/test-config.yaml b/test/skills/rh-qs-test-suite/cluster-privileges/test-config.yaml new file mode 100644 index 0000000..0d37701 --- /dev/null +++ b/test/skills/rh-qs-test-suite/cluster-privileges/test-config.yaml @@ -0,0 +1,8 @@ +run-skill-command: Add nightly testing to qs-test-quickstart. +execution: + network: required + external-dependencies: + - https://github.com/rh-ai-quickstart/quickstart-ci-testing +validation-options: + mode: targeted + additional-instructions: only validate that the skill flags that cluster admin privilege is needed, nothing else diff --git a/test/skills/rh-qs-test-suite/missing-make-target/expected-output-repo/.gitkeep b/test/skills/rh-qs-test-suite/missing-make-target/expected-output-repo/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/test/skills/rh-qs-test-suite/missing-make-target/input-repo/Makefile b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/Makefile new file mode 100644 index 0000000..1a60b9f --- /dev/null +++ b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/Makefile @@ -0,0 +1,17 @@ +NAMESPACE ?= qs-test-quickstart-ci-project-1 +RELEASE ?= qs-test-quickstart +CHART ?= helm + +.PHONY: deploy test helm-do-something verify-deploy + +deploy: + helm upgrade --install $(RELEASE) $(CHART) --namespace $(NAMESPACE) --create-namespace + +helm-do-something: + helm uninstall $(RELEASE) --namespace $(NAMESPACE) + +test: + @echo "Nightly tests passed" + +verify-deploy: + helm status $(RELEASE) --namespace $(NAMESPACE) diff --git a/test/skills/rh-qs-test-suite/missing-make-target/input-repo/README.md b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/README.md new file mode 100644 index 0000000..790de20 --- /dev/null +++ b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/README.md @@ -0,0 +1,5 @@ +# Missing target fixture + +This fixture has canonical deploy and test targets but intentionally has no +clear undeploy equivalent. The ambiguous helm-do-something target requires +user confirmation before it can be used. diff --git a/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/Chart.yaml b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/Chart.yaml new file mode 100644 index 0000000..dbfc612 --- /dev/null +++ b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: nightly-fixture +description: Minimal chart for missing Makefile target testing +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/templates/deployment.yaml b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/templates/deployment.yaml new file mode 100644 index 0000000..4addaea --- /dev/null +++ b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/templates/deployment.yaml @@ -0,0 +1,18 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nightly-fixture +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app: nightly-fixture + template: + metadata: + labels: + app: nightly-fixture + spec: + containers: + - name: fixture + image: registry.access.redhat.com/ubi9/ubi-micro:latest + command: ["/bin/sh", "-c", "sleep 3600"] diff --git a/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/values.yaml b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/values.yaml new file mode 100644 index 0000000..ebf6be8 --- /dev/null +++ b/test/skills/rh-qs-test-suite/missing-make-target/input-repo/helm/values.yaml @@ -0,0 +1 @@ +replicaCount: 1 diff --git a/test/skills/rh-qs-test-suite/missing-make-target/test-config.yaml b/test/skills/rh-qs-test-suite/missing-make-target/test-config.yaml new file mode 100644 index 0000000..a041c47 --- /dev/null +++ b/test/skills/rh-qs-test-suite/missing-make-target/test-config.yaml @@ -0,0 +1,14 @@ +run-skill-command: Add nightly testing to qs-test-quickstart. +execution: + network: required + external-dependencies: + - https://github.com/rh-ai-quickstart/quickstart-ci-testing +question-answer-context: | + Use helm-do-something as the undeploy target for this fixture. +validation-options: + mode: targeted + additional-instructions: | + Validate that the skill asks the user for the missing undeploy target, + accepts helm-do-something as the answer, and uses that exact command in + the generated reusable nightly workflow and manifest. Do not require + unrelated output files to match the baseline. diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/cleanup/action.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/cleanup/action.yml new file mode 100644 index 0000000..977fafc --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/cleanup/action.yml @@ -0,0 +1,27 @@ +name: Cleanup Deployment +description: Run cleanup command to tear down a deployed quickstart + +inputs: + cleanup_command: + description: Command to tear down the deployment + required: true + working_dir: + description: Directory to run cleanup in + required: false + default: ./quickstart + +runs: + using: composite + steps: + - name: Cleanup deployment + shell: bash + working-directory: ${{ inputs.working_dir }} + continue-on-error: true + env: + CLEANUP_COMMAND: ${{ inputs.cleanup_command }} + run: eval "$CLEANUP_COMMAND" + + - name: Delete remaining resources + shell: bash + continue-on-error: true + run: oc delete pods,deployments,replicasets,statefulsets,services,jobs,cronjobs,daemonsets,pvc --all -n "$NAMESPACE" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/clone-quickstart/action.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/clone-quickstart/action.yml new file mode 100644 index 0000000..ec18450 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/clone-quickstart/action.yml @@ -0,0 +1,52 @@ +name: Clone Quickstart +description: Clone a quickstart repository at a specified ref + +inputs: + repo_url: + description: Full GitHub URL of the quickstart repo + required: true + ref: + description: Branch, tag, or SHA to check out + required: false + default: main + checkout_path: + description: Directory to clone into + required: false + default: ./quickstart + submodules: + description: Whether to clone submodules (true/false) + required: false + default: 'false' + +runs: + using: composite + steps: + - name: Clone quickstart repository + shell: bash + env: + REPO_URL: ${{ inputs.repo_url }} + REF: ${{ inputs.ref }} + CHECKOUT_PATH: ${{ inputs.checkout_path }} + SUBMODULES: ${{ inputs.submodules }} + run: | + CLONE_ARGS="--depth 1 --branch $REF" + if [ "$SUBMODULES" = "true" ]; then + CLONE_ARGS="$CLONE_ARGS --recurse-submodules" + fi + + git clone $CLONE_ARGS "$REPO_URL" "$CHECKOUT_PATH" + + - name: Print clone info + shell: bash + working-directory: ${{ inputs.checkout_path }} + env: + REPO_URL: ${{ inputs.repo_url }} + REF: ${{ inputs.ref }} + CHECKOUT_PATH: ${{ inputs.checkout_path }} + run: | + echo "=== Clone Info ===" + echo "Repository: $REPO_URL" + echo "Ref: $REF" + echo "Commit SHA: $(git rev-parse HEAD)" + echo "Branch: $(git branch --show-current || echo 'detached')" + echo "Clone path: $CHECKOUT_PATH" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/prepare-runner/action.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/prepare-runner/action.yml new file mode 100644 index 0000000..62f8266 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/prepare-runner/action.yml @@ -0,0 +1,39 @@ +name: Prepare Runner +description: Free disk space and check memory/swap status before running tests + +inputs: + swap-storage: + description: Enable swap storage (may slow down if heavily used) + required: false + default: 'false' + +runs: + using: composite + steps: + - name: Free Disk Space + uses: jlumbroso/free-disk-space@v1.3.1 + with: + android: true + dotnet: true + haskell: true + large-packages: false + docker-images: false + swap-storage: ${{ inputs.swap-storage }} + tool-cache: false + + - name: Install oc CLI + shell: bash + run: | + curl -sL https://mirror.openshift.com/pub/openshift-v4/clients/ocp/stable/openshift-client-linux.tar.gz | tar xzf - -C /usr/local/bin oc kubectl + + - name: Check memory and swap status + shell: bash + run: | + echo "=== Memory Status ===" + free -h + echo "" + echo "=== Swap Usage ===" + swapon --show || echo "No swap configured" + echo "" + echo "=== Disk Space ===" + df -h diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/run-quickstart-tests/action.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/run-quickstart-tests/action.yml new file mode 100644 index 0000000..fd54341 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/run-quickstart-tests/action.yml @@ -0,0 +1,71 @@ +name: Run Tests +description: Run pre-test setup, tests, and optionally upload result artifacts + +inputs: + test_command: + description: Command to run tests + required: true + test_working_dir: + description: Directory for tests + required: false + default: ./quickstart + pre_test_command: + description: Setup command to run before tests + required: false + default: '' + upload_results: + description: >- + Space-separated list of directories (relative to test_working_dir) + to upload as artifacts after tests complete. Example: "evaluations/results test/reports" + required: false + default: '' + upload_retention_days: + description: Number of days to retain uploaded artifacts + required: false + default: '2' + +runs: + using: composite + steps: + - name: Pre-test setup + if: inputs.pre_test_command != '' + shell: bash + working-directory: ./quickstart + env: + PRE_TEST_COMMAND: ${{ inputs.pre_test_command }} + run: eval "$PRE_TEST_COMMAND" + + - name: Run tests + uses: ./.github/actions/run-tests + with: + test_command: ${{ inputs.test_command }} + working_dir: ${{ inputs.test_working_dir }} + + - name: Prepare upload paths + if: always() && inputs.upload_results != '' + id: upload-paths + shell: bash + env: + UPLOAD_RESULTS: ${{ inputs.upload_results }} + WORKING_DIR: ${{ inputs.test_working_dir }} + run: | + PATHS="" + for dir in $UPLOAD_RESULTS; do + if [[ "$dir" == /* ]] || [[ "$dir" == *..* ]]; then + echo "::error::upload_results paths must be relative to working dir and cannot contain '..': $dir" + exit 1 + fi + PATHS="${PATHS}${WORKING_DIR}/${dir}"$'\n' + done + echo "paths<> "$GITHUB_OUTPUT" + echo "$PATHS" >> "$GITHUB_OUTPUT" + echo "EOF" >> "$GITHUB_OUTPUT" + + - name: Upload test results + if: always() && inputs.upload_results != '' + uses: actions/upload-artifact@v4 + with: + name: test-results-${{ github.job }}-${{ github.run_id }} + path: ${{ steps.upload-paths.outputs.paths }} + retention-days: ${{ inputs.upload_retention_days }} + if-no-files-found: warn diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/run-tests/action.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/run-tests/action.yml new file mode 100644 index 0000000..61b7699 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/run-tests/action.yml @@ -0,0 +1,22 @@ +name: Run Tests +description: Run tests via a configurable command + +inputs: + test_command: + description: Command to run tests (e.g. "make test-all" or "pytest tests/ -v") + required: true + working_dir: + description: Directory to run tests in + required: false + default: ./quickstart + +runs: + using: composite + steps: + - name: Run tests + shell: bash + working-directory: ${{ inputs.working_dir }} + env: + TEST_COMMAND: ${{ inputs.test_command }} + run: | + eval "$TEST_COMMAND" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-and-deploy/action.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-and-deploy/action.yml new file mode 100644 index 0000000..5309ba8 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-and-deploy/action.yml @@ -0,0 +1,153 @@ +name: Setup and Deploy Quickstart +description: Prepare runner, clone quickstart repo, and deploy to OpenShift or KinD + +inputs: + repo_url: + description: GitHub repo URL to clone + required: true + ref: + description: Branch/tag/SHA to check out + required: false + default: main + submodules: + description: Whether to clone submodules + required: false + default: 'false' + use_kind: + description: Use KinD instead of OpenShift + required: false + default: 'false' + deploy_command: + description: Command to deploy the application + required: true + namespace: + description: Kubernetes namespace + required: false + default: quickstart-ci-project-1 + setup_commands: + description: Pre-deploy setup commands + required: false + default: '' + kind_pre_deploy_command: + description: Pre-deploy setup for KinD + required: false + default: '' + extra_deploy_args: + description: Extra args appended to deploy command + required: false + default: '' + kind_extra_deploy_args: + description: Extra args appended to deploy command only when using KinD + required: false + default: '' + build_images: + description: Build and push images to local KinD registry (only valid with use_kind) + required: false + default: 'false' + build_images_command: + description: Command to build images + required: false + default: 'make build-all-images' + push_images_command: + description: Command to push images to local registry + required: false + default: 'make push-all-images' + +runs: + using: composite + steps: + - name: Prepare Runner + uses: ./.github/actions/prepare-runner + + - name: Export namespace + shell: bash + run: echo "NAMESPACE=${{ inputs.namespace }}" >> "$GITHUB_ENV" + + - name: Validate build_images requires use_kind + if: inputs.build_images == 'true' && inputs.use_kind != 'true' + shell: bash + run: | + echo "::error::build_images can only be used when use_kind is 'true'" + exit 1 + + - name: Validate OpenShift secrets + if: inputs.use_kind != 'true' + shell: bash + run: | + if [ -z "${PROD_TOKEN}" ]; then + echo "::error::PROD_TOKEN secret is missing or empty." + exit 1 + fi + if [ -z "${PROD_SERVER}" ]; then + echo "::error::PROD_SERVER secret is missing or empty." + exit 1 + fi + + - name: Clone quickstart + uses: ./.github/actions/clone-quickstart + with: + repo_url: ${{ inputs.repo_url }} + ref: ${{ inputs.ref }} + submodules: ${{ inputs.submodules }} + + - name: Setup OpenShift + if: inputs.use_kind != 'true' + uses: ./.github/actions/setup-openshift + with: + namespace: ${{ inputs.namespace }} + setup_commands: ${{ inputs.setup_commands }} + + - name: Print environment versions + if: inputs.use_kind != 'true' + shell: bash + env: + NAMESPACE: ${{ inputs.namespace }} + run: | + echo "=== OpenShift Version ===" + oc version || true + + echo "" + echo "=== OpenShift AI Operator Version ===" + oc get csv -n "$NAMESPACE" \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.version}{"\n"}{end}' 2>/dev/null \ + | grep -iE 'rhods|rhoai|opendatahub' \ + || echo "(OpenShift AI operator not found)" + + - name: Setup KinD + if: inputs.use_kind == 'true' + uses: ./.github/actions/setup-kind + with: + namespace: ${{ inputs.namespace }} + pre_deploy_command: ${{ inputs.kind_pre_deploy_command || inputs.setup_commands }} + build_images: ${{ inputs.build_images }} + build_images_command: ${{ inputs.build_images_command }} + push_images_command: ${{ inputs.push_images_command }} + + - name: Deploy application + if: inputs.build_images != 'true' + shell: bash + working-directory: ./quickstart + env: + DEPLOY_COMMAND: ${{ inputs.deploy_command }} + EXTRA_DEPLOY_ARGS: ${{ inputs.extra_deploy_args }} ${{ inputs.use_kind == 'true' && inputs.kind_extra_deploy_args || '' }} + NAMESPACE: ${{ inputs.namespace }} + run: eval "$DEPLOY_COMMAND" $EXTRA_DEPLOY_ARGS + + - name: Deploy application (local images) + if: inputs.build_images == 'true' + shell: bash + working-directory: ./quickstart + env: + DEPLOY_COMMAND: ${{ inputs.deploy_command }} + EXTRA_DEPLOY_ARGS: ${{ inputs.extra_deploy_args }} ${{ inputs.use_kind == 'true' && inputs.kind_extra_deploy_args || '' }} + NAMESPACE: ${{ inputs.namespace }} + REGISTRY: localhost:5001 + run: eval "$DEPLOY_COMMAND" $EXTRA_DEPLOY_ARGS + + - name: Verify deployment + shell: bash + env: + NAMESPACE: ${{ inputs.namespace }} + run: | + echo "=== Pod Status ===" + oc get pods -n "$NAMESPACE" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-kind/action.yaml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-kind/action.yaml new file mode 100644 index 0000000..bef0838 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-kind/action.yaml @@ -0,0 +1,78 @@ +name: Setup KinD cluster +description: Create a KinD cluster with local registry and prepare for deployment + +inputs: + namespace: + description: Kubernetes namespace to deploy into + required: true + working_dir: + description: Directory containing the quickstart code + required: false + default: ./quickstart + kind_version: + description: KinD version to install + required: false + default: v0.30.0 + pre_deploy_command: + description: Command to run before deploy (e.g. helm dependency build) + required: false + default: '' + build_images: + description: Build and push images to the local KinD registry + required: false + default: 'false' + build_images_command: + description: Command to build images + required: false + default: 'make build-all-images' + push_images_command: + description: Command to push images to local registry + required: false + default: 'make push-all-images' + +runs: + using: composite + steps: + - name: Install Kind + shell: bash + env: + KIND_VERSION: ${{ inputs.kind_version }} + run: go install "sigs.k8s.io/kind@${KIND_VERSION}" + + - name: Create Kind Cluster with local registry + run: bash ${GITHUB_ACTION_PATH}/../../../scripts/ci/kind-with-registry.sh + shell: bash + + - name: Pre-deploy setup + if: inputs.pre_deploy_command != '' + working-directory: ${{ inputs.working_dir }} + shell: bash + env: + PRE_DEPLOY_COMMAND: ${{ inputs.pre_deploy_command }} + NAMESPACE: ${{ inputs.namespace }} + run: eval "$PRE_DEPLOY_COMMAND" + + - name: Build images + if: inputs.build_images == 'true' + working-directory: ${{ inputs.working_dir }} + shell: bash + env: + BUILD_COMMAND: ${{ inputs.build_images_command }} + REGISTRY: localhost:5001 + run: eval "$BUILD_COMMAND" + + - name: Push images to local registry + if: inputs.build_images == 'true' + working-directory: ${{ inputs.working_dir }} + shell: bash + env: + PUSH_COMMAND: ${{ inputs.push_images_command }} + REGISTRY: localhost:5001 + PUSH_EXTRA_AGRS: "--tls-verify=false" + run: eval "$PUSH_COMMAND" + + - name: Create namespace + shell: bash + env: + NAMESPACE: ${{ inputs.namespace }} + run: kubectl create namespace "$NAMESPACE" || true diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-openshift/action.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-openshift/action.yml new file mode 100644 index 0000000..4beb476 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/actions/setup-openshift/action.yml @@ -0,0 +1,33 @@ +name: Setup OpenShift +description: Log into OpenShift and run any pre-deploy setup commands + +inputs: + working_dir: + description: Directory to run commands in + required: false + default: ./quickstart + namespace: + description: OpenShift namespace/project to deploy into + required: true + setup_commands: + description: Pre-deploy setup commands (newline-separated) + required: false + default: '' + +runs: + using: composite + steps: + - name: Log into OpenShift cluster + shell: bash + run: | + oc login --token="$PROD_TOKEN" --server="$PROD_SERVER" + + - name: Run setup commands + if: inputs.setup_commands != '' + shell: bash + working-directory: ${{ inputs.working_dir }} + env: + SETUP_COMMANDS: ${{ inputs.setup_commands }} + NAMESPACE: ${{ inputs.namespace }} + run: | + eval "$SETUP_COMMANDS" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/workflows/nightly-main.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/workflows/nightly-main.yml new file mode 100644 index 0000000..673f50b --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/workflows/nightly-main.yml @@ -0,0 +1,13 @@ +name: Nightly - qs-test-quickstart (main) + +on: + workflow_dispatch: + schedule: + - cron: "0 5 * * *" + +jobs: + test: + uses: ./.github/workflows/test-qs-test-quickstart.yml + with: + branch: main + secrets: inherit diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/workflows/test-qs-test-quickstart.yml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/workflows/test-qs-test-quickstart.yml new file mode 100644 index 0000000..4e0dd42 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/.github/workflows/test-qs-test-quickstart.yml @@ -0,0 +1,62 @@ +name: Test qs-test-quickstart + +on: + workflow_call: + inputs: + branch: + required: true + type: string + workflow_dispatch: + inputs: + branch: + description: Branch to deploy and test + default: main + type: string + +concurrency: + group: qs-test-quickstart-ci-testing + cancel-in-progress: false + +jobs: + test: + name: qs-test-quickstart (${{ inputs.branch }}) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v5 + + - name: Set up uv + uses: astral-sh/setup-uv@v6 + with: + version: "0.5.24" + + - name: Setup and deploy + uses: ./.github/actions/setup-and-deploy + timeout-minutes: 15 + with: + repo_url: https://github.com/${{ github.repository }} + ref: ${{ inputs.branch }} + namespace: qs-test-quickstart-ci-project-1 + deploy_command: "make deploy" + env: + PROD_TOKEN: ${{ secrets.PROD_TOKEN }} + PROD_SERVER: ${{ secrets.PROD_SERVER }} + MODEL_ID: ${{ vars.MODEL_ID }} + MODEL_URL: ${{ vars.MODEL_URL }} + MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }} + + - name: Run tests + uses: ./.github/actions/run-quickstart-tests + timeout-minutes: 30 + with: + test_command: "make test" + upload_results: "test-results" + env: + MODEL_ID: ${{ vars.MODEL_ID }} + MODEL_URL: ${{ vars.MODEL_URL }} + MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }} + + - name: Cleanup + if: always() + uses: ./.github/actions/cleanup + with: + cleanup_command: "make undeploy" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/Makefile b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/Makefile new file mode 100644 index 0000000..d284b81 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/Makefile @@ -0,0 +1,32 @@ +NAMESPACE ?= nightly-ci-fixture-ci-project-1 +RELEASE ?= nightly-ci-fixture +CHART ?= helm +MODEL_ID ?= +MODEL_URL ?= +MODEL_API_KEY ?= +UV_VERSION := 0.5.24 + +.PHONY: check-model-config check-uv-version deploy test undeploy verify-deploy + +check-model-config: + @test -n "$(MODEL_ID)" || (echo "MODEL_ID must be set"; exit 1) + @test -n "$(MODEL_URL)" || (echo "MODEL_URL must be set"; exit 1) + @test -n "$(MODEL_API_KEY)" || (echo "MODEL_API_KEY must be set"; exit 1) + +check-uv-version: + @uv --version | grep -q "$(UV_VERSION)" || (echo "uv $(UV_VERSION) must be installed"; exit 1) + +deploy: check-model-config + helm upgrade --install $(RELEASE) $(CHART) --namespace $(NAMESPACE) --create-namespace \ + --set-string model.id="$(MODEL_ID)" \ + --set-string model.url="$(MODEL_URL)" \ + --set-string model.apiKey="$(MODEL_API_KEY)" + +test: check-model-config check-uv-version + @uv run --no-project python -c "print('Nightly CI fixture model tests passed')" + +undeploy: + helm uninstall $(RELEASE) --namespace $(NAMESPACE) + +verify-deploy: + helm status $(RELEASE) --namespace $(NAMESPACE) diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/README.md b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/README.md new file mode 100644 index 0000000..0561c3c --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/README.md @@ -0,0 +1,44 @@ +# Nightly CI Demo + +A minimal Helm-deployed service for demonstrating a Quickstart deployment workflow. + +## Prerequisites + +- Access to an OpenShift cluster +- A model deployment ID, API URL, and API key +- `uv` version `0.5.24` to run the Python test harness used by `make test` + +## Deploy + +Set the target namespace: + +```bash +export NAMESPACE=your-namespace +export MODEL_ID=granite-3.1-8b-instruct +export MODEL_URL=https://model.example.com/v1 +export MODEL_API_KEY=your-model-api-key +``` + +Deploy the chart: + +```bash +make deploy NAMESPACE=$NAMESPACE MODEL_ID=$MODEL_ID MODEL_URL=$MODEL_URL MODEL_API_KEY=$MODEL_API_KEY +``` + +Check the release: + +```bash +make verify-deploy NAMESPACE=$NAMESPACE +``` + +## Test + +```bash +make test MODEL_ID=$MODEL_ID MODEL_URL=$MODEL_URL MODEL_API_KEY=$MODEL_API_KEY +``` + +## Remove + +```bash +make undeploy NAMESPACE=$NAMESPACE +``` diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/Chart.yaml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/Chart.yaml new file mode 100644 index 0000000..e5ef944 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: nightly-ci-fixture +description: Minimal namespaced chart for rh-qs-test-suite tests +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/templates/_helpers.tpl b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/templates/_helpers.tpl new file mode 100644 index 0000000..285088e --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/templates/_helpers.tpl @@ -0,0 +1,13 @@ +{{- define "nightly-ci-fixture.fullname" -}} +{{- .Release.Name -}} +{{- end -}} + +{{- define "nightly-ci-fixture.labels" -}} +app.kubernetes.io/name: {{ .Chart.Name }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "nightly-ci-fixture.selectorLabels" -}} +app.kubernetes.io/name: {{ .Chart.Name }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/templates/deployment.yaml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/templates/deployment.yaml new file mode 100644 index 0000000..d36182e --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/templates/deployment.yaml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "nightly-ci-fixture.fullname" . }} + labels: + {{- include "nightly-ci-fixture.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + {{- include "nightly-ci-fixture.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "nightly-ci-fixture.selectorLabels" . | nindent 8 }} + spec: + containers: + - name: fixture + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + command: ["/bin/sh", "-c", "sleep 3600"] + env: + - name: MODEL_ID + value: {{ .Values.model.id | quote }} + - name: MODEL_URL + value: {{ .Values.model.url | quote }} + - name: MODEL_API_KEY + value: {{ .Values.model.apiKey | quote }} diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/values.yaml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/values.yaml new file mode 100644 index 0000000..ce2c3a5 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/helm/values.yaml @@ -0,0 +1,10 @@ +replicaCount: 1 + +image: + repository: registry.access.redhat.com/ubi9/ubi-micro + tag: latest + +model: + id: "" + url: "" + apiKey: "" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/pipeline/nightly-tests-manifest.yaml b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/pipeline/nightly-tests-manifest.yaml new file mode 100644 index 0000000..b7996e8 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/pipeline/nightly-tests-manifest.yaml @@ -0,0 +1,48 @@ +slug: qs-test-quickstart +resolved_targets: + deploy: deploy + test: test + undeploy: undeploy +toolchain_setup_steps: + - action: astral-sh/setup-uv@v6 + with: + version: "0.5.24" + source: README.md and Makefile UV_VERSION +branch_model: + main: main + dev: null +namespace: qs-test-quickstart-ci-project-1 +service_account: qs-test-quickstart-ci-1 +cluster_admin_check: + status: clean + detail: No cluster-admin requirement detected. +required_secrets: + - name: PROD_TOKEN + reason: ServiceAccount token for the nightly CI namespace + - name: PROD_SERVER + reason: OpenShift API server URL for the cluster nightly deploys to + - name: MODEL_API_KEY + reason: API key required by the deploy and test Makefile targets +required_variables: + - name: MODEL_ID + reason: Model deployment ID required by the deploy and test Makefile targets + - name: MODEL_URL + reason: Model API URL required by the deploy and test Makefile targets +unconfirmed: [] +files: + actions_copied: + - cleanup + - clone-quickstart + - prepare-runner + - run-quickstart-tests + - run-tests + - setup-and-deploy + - setup-kind + - setup-openshift + actions_skipped_identical: [] + actions_updated: [] + workflows_created: + - .github/workflows/test-qs-test-quickstart.yml + - .github/workflows/nightly-main.yml + workflows_skipped_or_updated: [] + provisioning_script: scripts/ci/setup-nightly-namespace.sh diff --git a/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/scripts/ci/setup-nightly-namespace.sh b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/scripts/ci/setup-nightly-namespace.sh new file mode 100755 index 0000000..dcbd646 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/expected-output-repo/scripts/ci/setup-nightly-namespace.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +set -euo pipefail + +NAMESPACE="${NAMESPACE:-qs-test-quickstart-ci-project-1}" +SA_NAME="${SA_NAME:-qs-test-quickstart-ci-1}" +TOKEN_DURATION="${TOKEN_DURATION:-8760h}" + +oc create namespace "$NAMESPACE" +oc create sa "$SA_NAME" -n "$NAMESPACE" +oc adm policy add-role-to-user admin \ + "system:serviceaccount:${NAMESPACE}:${SA_NAME}" \ + -n "$NAMESPACE" + +echo +echo "PROD_SERVER: $(oc whoami --show-server)" +echo "PROD_TOKEN:" +oc create token "$SA_NAME" -n "$NAMESPACE" --duration="$TOKEN_DURATION" + +echo +echo "Set the two values above as repository secrets PROD_SERVER and PROD_TOKEN" +echo "(Settings -> Secrets and variables -> Actions) before enabling the nightly workflows." diff --git a/test/skills/rh-qs-test-suite/standard-pattern/input-repo/Makefile b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/Makefile new file mode 100644 index 0000000..d284b81 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/Makefile @@ -0,0 +1,32 @@ +NAMESPACE ?= nightly-ci-fixture-ci-project-1 +RELEASE ?= nightly-ci-fixture +CHART ?= helm +MODEL_ID ?= +MODEL_URL ?= +MODEL_API_KEY ?= +UV_VERSION := 0.5.24 + +.PHONY: check-model-config check-uv-version deploy test undeploy verify-deploy + +check-model-config: + @test -n "$(MODEL_ID)" || (echo "MODEL_ID must be set"; exit 1) + @test -n "$(MODEL_URL)" || (echo "MODEL_URL must be set"; exit 1) + @test -n "$(MODEL_API_KEY)" || (echo "MODEL_API_KEY must be set"; exit 1) + +check-uv-version: + @uv --version | grep -q "$(UV_VERSION)" || (echo "uv $(UV_VERSION) must be installed"; exit 1) + +deploy: check-model-config + helm upgrade --install $(RELEASE) $(CHART) --namespace $(NAMESPACE) --create-namespace \ + --set-string model.id="$(MODEL_ID)" \ + --set-string model.url="$(MODEL_URL)" \ + --set-string model.apiKey="$(MODEL_API_KEY)" + +test: check-model-config check-uv-version + @uv run --no-project python -c "print('Nightly CI fixture model tests passed')" + +undeploy: + helm uninstall $(RELEASE) --namespace $(NAMESPACE) + +verify-deploy: + helm status $(RELEASE) --namespace $(NAMESPACE) diff --git a/test/skills/rh-qs-test-suite/standard-pattern/input-repo/README.md b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/README.md new file mode 100644 index 0000000..0561c3c --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/README.md @@ -0,0 +1,44 @@ +# Nightly CI Demo + +A minimal Helm-deployed service for demonstrating a Quickstart deployment workflow. + +## Prerequisites + +- Access to an OpenShift cluster +- A model deployment ID, API URL, and API key +- `uv` version `0.5.24` to run the Python test harness used by `make test` + +## Deploy + +Set the target namespace: + +```bash +export NAMESPACE=your-namespace +export MODEL_ID=granite-3.1-8b-instruct +export MODEL_URL=https://model.example.com/v1 +export MODEL_API_KEY=your-model-api-key +``` + +Deploy the chart: + +```bash +make deploy NAMESPACE=$NAMESPACE MODEL_ID=$MODEL_ID MODEL_URL=$MODEL_URL MODEL_API_KEY=$MODEL_API_KEY +``` + +Check the release: + +```bash +make verify-deploy NAMESPACE=$NAMESPACE +``` + +## Test + +```bash +make test MODEL_ID=$MODEL_ID MODEL_URL=$MODEL_URL MODEL_API_KEY=$MODEL_API_KEY +``` + +## Remove + +```bash +make undeploy NAMESPACE=$NAMESPACE +``` diff --git a/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/Chart.yaml b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/Chart.yaml new file mode 100644 index 0000000..e5ef944 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/Chart.yaml @@ -0,0 +1,6 @@ +apiVersion: v2 +name: nightly-ci-fixture +description: Minimal namespaced chart for rh-qs-test-suite tests +type: application +version: 0.1.0 +appVersion: "0.1.0" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/templates/_helpers.tpl b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/templates/_helpers.tpl new file mode 100644 index 0000000..285088e --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/templates/_helpers.tpl @@ -0,0 +1,13 @@ +{{- define "nightly-ci-fixture.fullname" -}} +{{- .Release.Name -}} +{{- end -}} + +{{- define "nightly-ci-fixture.labels" -}} +app.kubernetes.io/name: {{ .Chart.Name }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} + +{{- define "nightly-ci-fixture.selectorLabels" -}} +app.kubernetes.io/name: {{ .Chart.Name }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- end -}} diff --git a/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/templates/deployment.yaml b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/templates/deployment.yaml new file mode 100644 index 0000000..d36182e --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/templates/deployment.yaml @@ -0,0 +1,27 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "nightly-ci-fixture.fullname" . }} + labels: + {{- include "nightly-ci-fixture.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + {{- include "nightly-ci-fixture.selectorLabels" . | nindent 6 }} + template: + metadata: + labels: + {{- include "nightly-ci-fixture.selectorLabels" . | nindent 8 }} + spec: + containers: + - name: fixture + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + command: ["/bin/sh", "-c", "sleep 3600"] + env: + - name: MODEL_ID + value: {{ .Values.model.id | quote }} + - name: MODEL_URL + value: {{ .Values.model.url | quote }} + - name: MODEL_API_KEY + value: {{ .Values.model.apiKey | quote }} diff --git a/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/values.yaml b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/values.yaml new file mode 100644 index 0000000..ce2c3a5 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/input-repo/helm/values.yaml @@ -0,0 +1,10 @@ +replicaCount: 1 + +image: + repository: registry.access.redhat.com/ubi9/ubi-micro + tag: latest + +model: + id: "" + url: "" + apiKey: "" diff --git a/test/skills/rh-qs-test-suite/standard-pattern/test-config.yaml b/test/skills/rh-qs-test-suite/standard-pattern/test-config.yaml new file mode 100644 index 0000000..066a996 --- /dev/null +++ b/test/skills/rh-qs-test-suite/standard-pattern/test-config.yaml @@ -0,0 +1,5 @@ +run-skill-command: Add nightly testing to qs-test-quickstart. +execution: + network: required + external-dependencies: + - https://github.com/rh-ai-quickstart/quickstart-ci-testing