diff --git a/.github/extension/README.md b/.github/extension/README.md new file mode 100644 index 00000000000..60fe5fb6930 --- /dev/null +++ b/.github/extension/README.md @@ -0,0 +1,98 @@ +# Repo Radius workflow assets + +This folder holds the **Repo Radius** workflow templates that are written into a user's repository so that Radius can run on a GitHub Actions runner. + +These files are templates: a copy is committed into the target repository under `.github/workflows/` and dispatched there. They are not run from this repository directly. + +They live here so the workflow contract has a canonical, reviewed home that any frontend (the Copilot app, the CLI, etc.) can drive. See [radius-project/radius#12118](https://github.com/radius-project/radius/issues/12118) for background. + +## Credential verification (upstream) + +Before this deploy workflow runs, a separate **`Radius - Verify Credentials`** workflow confirms the GitHub Environment's cloud credentials are wired up correctly. It is generated by the environment-setup flow (see the `radius-environment` skill), not by this folder, so its template is documented there rather than here. The deploy dispatcher (below) chains off it: a successful verify run auto-triggers a deploy via its `workflow_run` trigger. + +The OIDC trust that the verify and deploy workflows both rely on must already exist before either can authenticate: + +- **Azure:** a federated credential on the AAD app whose subject is exactly `repo:/:environment:`, audience `api://AzureADTokenExchange`. +- **AWS:** an IAM role trust policy that allows `sts:AssumeRoleWithWebIdentity` from `token.actions.githubusercontent.com` with audience `sts.amazonaws.com` and subject `repo:/:environment:`. + +## `run-rad-commands.yml` (dispatcher, provider workflows, and shared actions) + +The run-rad-commands workflow Radius uses to run one or more `rad` CLI commands on demand against a user's target cluster — deploying by default, but able to run any allowed command (`deploy`, `app graph`, `app delete`, and so on). It stands up an ephemeral [k3d](https://k3d.io) control plane on the runner, restores persisted state, runs the requested commands, then persists state again and tears the control plane down. + +To keep the two provider paths from duplicating the ~80% of steps they share, it ships as a unified dispatcher, two thin provider workflows, and shared composite actions: + +- **`run-rad-commands.yml`** — the unified **dispatcher** and the only file that is dispatched. It owns the dispatch contract (`workflow_dispatch` inputs and the `Radius - Verify Credentials` auto-trigger). A `detect` job binds the GitHub Environment, reads which provider variable is set (`AZURE_CLIENT_ID` / `AWS_ROLE_ARN`), and calls the matching provider workflow via `workflow_call` with `secrets: inherit`. +- **`run-rad-commands-azure.yml`** — a reusable (`workflow_call`) workflow with only the Azure-specific steps: Azure OIDC login, AKS connection (`az aks get-credentials`), workload-identity credential registration, and the `azure-avm` recipe pack (Azure Verified Modules) downloaded from [resource-types-contrib](https://github.com/radius-project/resource-types-contrib). +- **`run-rad-commands-aws.yml`** — a reusable (`workflow_call`) workflow with only the AWS-specific steps: AWS OIDC login, EKS connection (access entry + static token kubeconfig), IRSA credential registration, and the `aws-terraform` recipe pack. +- **`actions/*`** — composite actions holding the provider-agnostic phases both provider workflows share: [`setup-control-plane`](actions/setup-control-plane/action.yml), [`restore-state`](actions/restore-state/action.yml), [`run-rad-commands`](actions/run-rad-commands/action.yml), and [`teardown`](actions/teardown/action.yml). The provider workflows reference them from `radius-project/radius` at a pinned ref (the `{{RADIUS_REF}}` placeholder the generator fills in), so the shared logic has a single reviewed home and is not copied into user repos. + +The deploy flow (see the `radius-deploy` skill) generates the dispatcher and both provider workflows, commits them to the target repo under `.github/workflows/`, and dispatches `run-rad-commands.yml`. + +### What it does + +The dispatcher routes to the matching provider workflow, which runs on `ubuntu-latest`. It stands up an ephemeral [k3d](https://k3d.io) cluster to host the Radius control plane on the runner, points that control plane at the user's existing EKS/AKS cluster, and deploys the application there. The control-plane setup, state restore, and run/teardown phases below run from the shared composite actions; the OIDC login, cluster connection, token projection, credential registration, and recipe-pack creation are the provider-specific steps. When a provider's identifying variable is empty, its steps are skipped and resources deploy to the ephemeral control-plane cluster instead of an external target. + +1. **Authenticate via OIDC.** Runs `azure/login` (Azure) or `aws-actions/configure-aws-credentials` (AWS) so the runner exchanges its GitHub OIDC token for short-lived cloud credentials. No long-lived cloud secrets are stored. +2. **Build the target-cluster kubeconfig.** Exports `RADIUS_TARGET_KUBECONFIG` to a path under `$HOME/.kube`, then connects to the workload cluster: Azure runs `az aks get-credentials --file`; AWS ensures an EKS access entry and cluster-admin access policy for the IAM role and writes a static, token-based kubeconfig. +3. **Create the ephemeral control plane.** Installs k3d and creates the `radius-cp` cluster, then installs `oras`, the `rad` CLI (edge), and Terraform. +4. **Create the target-kubeconfig secret.** Stores the target kubeconfig as the `target-kubeconfig` secret in `radius-system` (skipped when no target kubeconfig is present). +5. **Install Radius on the control plane.** Runs `rad install kubernetes` with `database.enabled=true` (control-plane PostgreSQL for durable state), `rp.publicEndpointOverride=localhost`, `dynamicrp.buildkit.enabled=true`, and — when a target kubeconfig is present — `global.targetCluster.enabled=true`. The chart mounts the secret into `applications-rp`, `dynamic-rp`, and `bicep-de` and sets `RADIUS_TARGET_KUBECONFIG`, so recipe execution and directly-rendered resources land on the external cluster. The Terraform state backend deliberately stays on the control-plane cluster. +6. **Project cloud OIDC tokens.** Mints a GitHub OIDC token for the provider and patches it into the RP/DE pods at the fixed path each reads for the federated token exchange (AWS IRSA `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`; Azure workload identity `/var/run/secrets/azure/tokens/azure-identity-token`). +7. **Refresh external target credentials.** AWS re-mints the short-lived EKS token; both providers rewrite the `target-kubeconfig` secret and restart `applications-rp`, `dynamic-rp`, and `bicep-de` so they re-read it. +8. **Configure the workspace.** Runs `rad workspace create kubernetes default` and `rad group create` / `rad group switch default`. +9. **Restore persisted state (`rad startup`).** Restores the control-plane databases and the Terraform recipe-state Secrets saved by the previous run, so `rad deploy` plans against prior state rather than an empty backend. A no-op on the first run. +10. **Register cloud credentials.** Registers the cloud identity with `rad credential register azure wi` / `aws irsa` so Radius holds the identity selector and reads the projected token at runtime. +11. **Create the Radius environment and recipe pack.** `rad deploy`s a `radius-env.bicep` that defines a `Radius.Core/recipePacks` resource and the `Radius.Core/environments` resource that references it. Azure downloads the `azure-avm` pack (Azure Verified Modules) from [resource-types-contrib](https://github.com/radius-project/resource-types-contrib); AWS generates an inline `aws-terraform` pack. `radius-env.bicep` is written to the app file's directory (e.g. `.radius/`) and deployed from there, so `rad deploy` resolves the repo's own `bicepconfig.json` (which declares the `radius` extension) — bicep resolves the config nearest the `.bicep` file. The `Radius.Compute/containerImages` type ships with the Radius extension, so no separate resource-type registration is needed. +12. **Provision registry credentials on the control plane.** Creates the `ghcr-registry-creds` secret from `github.actor` and the built-in `GITHUB_TOKEN` so the containerImages recipe's in-pod BuildKit can push the application image. +13. **Run the requested rad commands.** Validates each command in `rad_commands` against the allowed-command set, then runs them in order (stopping on the first failure) and writes a combined `rad-commands-result` artifact. When `rad_commands` is empty it runs the default `rad deploy --environment `, passing the `image` parameter (the `image` input, defaulting to `github.sha`) and any application parameters from the `RADIUS_DEPLOY_PARAMS` secret. +14. **Persist state (`rad shutdown`).** Backs the control-plane databases and Terraform recipe-state Secrets up to the `radius-state` git orphan branch. This runs even when the deploy fails (`if: always()`), so a partially-applied Terraform run is not lost. +15. **Tear down.** Runs `rad app list`, and always deletes the ephemeral `radius-cp` cluster. On failure, Radius and application logs are collected and uploaded as the `radius-logs` artifact (three-day retention). + +### Triggers and permissions + +Triggers and permissions live on the **dispatcher** (`run-rad-commands.yml`); the provider workflows are `workflow_call`-only and inherit permissions and secrets from it. + +- **Triggers:** + - `workflow_dispatch` with an `environment` input (the GitHub Environment name) plus optional `image` and `rad_commands` inputs. The `detect` job binds that environment via `environment: ${{ inputs.environment }}` to read the provider variables. + - `workflow_run` after the `Radius - Verify Credentials` workflow completes. The `detect` job runs only when the upstream verify run concluded `success`, so a successful credential check auto-triggers a deploy. +- **Inputs:** + + | Input | Required | Description | + |---|---|---| + | `environment` | Yes | The GitHub Environment name, used as the Radius environment. | + | `image` | No | Container image for the application, passed to the default deploy as the `image` parameter. Defaults to the commit SHA (`github.sha`) when unset. | + | `rad_commands` | No | A single `rad` command string, or a JSON array of command strings run in order (the `rad` prefix omitted, e.g. `["deploy .radius/app.bicep --environment dev", "app graph my-app -o json"]`). Each command is validated against the allowed-command set. Falls back to the `RADIUS_RAD_COMMANDS` variable. When empty, the workflow runs its default `rad deploy` of the app bicep. | + +- **Outputs:** a combined `rad-commands-result` artifact — a JSON document with a top-level `outcome`/`exitCode` and a `commands` array (one entry per command, in input order, with each command's exit code and output). +- **Permissions:** `id-token: write` (required for OIDC), `contents: write` (so `rad shutdown` can push the `radius-state` branch), and `packages: write` (to push the application image built by the containerImages recipe). + +### Required environment variables + +The workflow reads cloud and cluster configuration from GitHub Actions **variables** (`vars`). Configure the relevant provider's set on the target GitHub Environment: + +| Provider | Variables | +|---|---| +| Common | `KUBERNETES_NAMESPACE` (default `default`), `RADIUS_BUILD_REGISTRY` (default `ghcr.io//`), `RADIUS_RAD_COMMANDS` (optional fallback for `rad_commands`) | +| Azure (`run-rad-commands-azure.yml`) | `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`, `AZURE_RESOURCE_GROUP`, `AZURE_AKS_CLUSTER_NAME` | +| AWS (`run-rad-commands-aws.yml`) | `AWS_ROLE_ARN`, `AWS_REGION`, `AWS_ACCOUNT_ID`, `AWS_EKS_CLUSTER_NAME`, `RADIUS_VPC_ID`, `RADIUS_SUBNET_IDS` | + +The provider steps run only when the identifying variable (`AZURE_CLIENT_ID` or `AWS_ROLE_ARN`) is non-empty. When it is unset, resources deploy to the ephemeral control-plane cluster instead of an external target. + +This workflow also reads GitHub Actions **secrets** for image push and application configuration: + +| Secret | Purpose | +|---|---| +| `GITHUB_TOKEN` | Built-in. Used with `github.actor` to authenticate the containerImages recipe's image push to GHCR. | +| `RADIUS_DEPLOY_PARAMS` | Optional. A JSON object of application parameters (`{"password":"…","apiKey":"…"}`) expanded into `--parameters name=value` pairs on the default deploy. | + +### State persistence (`rad startup` / `rad shutdown`) + +`rad startup` and `rad shutdown` are kind-agnostic CLI commands that restore and back up all durable Radius state (control-plane PostgreSQL + Terraform recipe-state Secrets) to a `radius-state` git orphan branch. They do not manage cluster lifecycle — the workflow owns creating and destroying the ephemeral control plane around them. `rad startup` runs after the install (so `rad deploy` plans against prior state) and `rad shutdown` runs after the commands with `if: always()` (so state survives a failed deploy). + +### Prerequisites + +- OIDC trust for the environment (federated credential on Azure, IAM role trust policy on AWS). Run the verify workflow first to confirm the environment is wired up correctly — a successful verify run also auto-triggers this workflow. +- The target cluster (`AWS_EKS_CLUSTER_NAME` / `AZURE_AKS_CLUSTER_NAME`) must already exist and be reachable; the assumed identity needs cluster-admin-level access to it. +- The application must define its app bicep file in the target repo. + +For the full deploy flow and troubleshooting, see the `radius-deploy` skill. diff --git a/.github/extension/actions/restore-state/action.yml b/.github/extension/actions/restore-state/action.yml new file mode 100644 index 00000000000..10e1eb2ba02 --- /dev/null +++ b/.github/extension/actions/restore-state/action.yml @@ -0,0 +1,50 @@ +# Provider-agnostic state restore shared by run-rad-commands-aws.yml and +# run-rad-commands-azure.yml. Sets up the Radius workspace/resource group and +# restores durable state from the previous run via `rad startup`. +name: Radius - Restore state +description: Configure the Radius workspace and restore persisted state (rad startup). + +inputs: + namespace: + description: Target Kubernetes namespace (vars.KUBERNETES_NAMESPACE, defaults to default). + required: false + default: default + +runs: + using: composite + steps: + - name: Configure Radius workspace + shell: bash + env: + NAMESPACE: ${{ inputs.namespace }} + run: | + # Ensure namespace exists on target cluster before Radius deploys into it. + TARGET_KUBECONFIG="$RADIUS_TARGET_KUBECONFIG" + if [ -f "$TARGET_KUBECONFIG" ]; then + echo "Ensuring namespace $NAMESPACE exists on target cluster..." + kubectl --kubeconfig "$TARGET_KUBECONFIG" get namespace "$NAMESPACE" 2>/dev/null || \ + kubectl --kubeconfig "$TARGET_KUBECONFIG" create namespace "$NAMESPACE" + fi + + # The Radius.Core/environments resource (with its recipe pack and + # cloud providers) is created later via Bicep. `rad install kubernetes` + # already creates the 'default' resource group server-side, but we create + # it explicitly (idempotent) as a defensive measure against older control + # planes, then set up the local CLI workspace and point it at that group. + rad workspace create kubernetes default + rad group create default + rad group switch default + + - name: Restore Radius state (rad startup) + shell: bash + run: | + # rad startup restores the control-plane PostgreSQL databases and the + # Terraform recipe-state Secrets saved by the previous run's `rad shutdown` + # from the `radius-state` git orphan branch, so commands plan against prior + # state instead of an empty backend. On the first ever run there is nothing + # to restore and it is a no-op. It needs the workspace created above and + # waits for PostgreSQL to be ready itself. Configure a git identity so it can + # manage the state branch. + git config --global user.email "radius-deploy@users.noreply.github.com" + git config --global user.name "radius-deploy" + rad startup diff --git a/.github/extension/actions/run-rad-commands/action.yml b/.github/extension/actions/run-rad-commands/action.yml new file mode 100644 index 00000000000..62f9519615c --- /dev/null +++ b/.github/extension/actions/run-rad-commands/action.yml @@ -0,0 +1,276 @@ +# Provider-agnostic deploy shared by run-rad-commands-aws.yml and +# run-rad-commands-azure.yml. Provisions registry credentials on the control plane +# and runs the requested rad commands (deploying by default), writing the combined +# rad-commands-result artifact. Teardown (rad shutdown, log collection, k3d delete) +# lives in the separate `teardown` action so it can run unconditionally. +name: Radius - Run rad commands +description: Provision registry credentials and run the requested rad commands (deploying by default). + +inputs: + environment: + description: Radius environment name. + required: true + app-file: + description: Application bicep file to deploy by default. + required: true + app-image: + description: Container image passed to the default deploy as the image parameter. + required: false + default: "" + namespace: + description: Target Kubernetes namespace (vars.KUBERNETES_NAMESPACE, defaults to default). + required: false + default: default + rad-commands: + description: rad command string or JSON array (rad prefix omitted). Overrides the default deploy. + required: false + default: "" + deploy-params: + description: Application parameters as a JSON object, expanded into --parameters name=value pairs. + required: false + default: "" + registry-username: + description: Username for the containerImages recipe's image-push registry secret. + required: true + registry-password: + description: Password/token for the containerImages recipe's image-push registry secret. + required: true + +runs: + using: composite + steps: + - name: Verify app namespace on target cluster + shell: bash + run: | + TARGET_KUBECONFIG="$RADIUS_TARGET_KUBECONFIG" + if [ -f "$TARGET_KUBECONFIG" ]; then + BICEP_APP_NAME=$(grep -oP "name:\s*'\K[^']+" ".radius/app.bicep" 2>/dev/null | head -1) + if [ -z "$BICEP_APP_NAME" ]; then + BICEP_APP_NAME="app" + fi + APP_NS="default-$BICEP_APP_NAME" + echo "Ensuring namespace $APP_NS exists on target cluster..." + kubectl --kubeconfig "$TARGET_KUBECONFIG" get namespace "$APP_NS" 2>/dev/null || \ + kubectl --kubeconfig "$TARGET_KUBECONFIG" create namespace "$APP_NS" + fi + + - name: Provision registry credentials on control plane + shell: bash + env: + # The Radius.Compute/containerImages recipe authenticates its BuildKit + # image push by loading the Kubernetes Secret named by registrySecretName + # via the in-cluster provider -- i.e. from the CONTROL PLANE cluster where + # the dynamic-rp/BuildKit pods run, NOT the target cluster. Provision that + # secret here directly from the default GitHub token so no registry + # credentials need to live in the application bicep. + REGISTRY_USERNAME: ${{ inputs.registry-username }} + REGISTRY_PASSWORD: ${{ inputs.registry-password }} + ENV_NS: ${{ inputs.namespace }} + run: | + set -eu + REGISTRY_SECRET_NAME="ghcr-registry-creds" + BICEP_APP_NAME=$(grep -oP "name:\s*'\K[^']+" ".radius/app.bicep" 2>/dev/null | head -1) + [ -z "$BICEP_APP_NAME" ] && BICEP_APP_NAME="app" + APP_NS="default-$BICEP_APP_NAME" + # The recipe reads the secret from context.runtime.kubernetes.namespace on + # the control plane. Materialize it in both the environment namespace and + # the application namespace so the read resolves regardless of scope. These + # kubectl calls run against the default kubeconfig (the control plane), not + # the target cluster. + for NS in "$ENV_NS" "$APP_NS"; do + kubectl get namespace "$NS" >/dev/null 2>&1 || kubectl create namespace "$NS" + kubectl create secret generic "$REGISTRY_SECRET_NAME" \ + --namespace "$NS" \ + --from-literal=username="$REGISTRY_USERNAME" \ + --from-literal=password="$REGISTRY_PASSWORD" \ + --dry-run=client -o yaml | kubectl apply -f - + echo "Provisioned $REGISTRY_SECRET_NAME in control-plane namespace $NS" + done + + - name: Run rad commands + shell: bash + env: + ENVIRONMENT: ${{ inputs.environment }} + APP_FILE: ${{ inputs.app-file }} + APP_IMAGE: ${{ inputs.app-image }} + # Pass caller input through the environment to avoid command injection. + # Caller-supplied rad commands (falls back to the RADIUS_RAD_COMMANDS + # variable upstream) so the command applies on both an explicit dispatch + # and the verify→deploy auto trigger (where inputs are empty). + RAD_COMMANDS: ${{ inputs.rad-commands }} + # Application parameters (user-supplied + auto-generated by the Radius + # extension) as a JSON object: {"password":"…","apiKey":"…"}. Consumed + # by the default deploy below. Read via the environment (not inlined into + # the script) so values with quotes/newlines can't break the shell. + RADIUS_DEPLOY_PARAMS: ${{ inputs.deploy-params }} + run: | + # pipefail so a failed `rad` whose output is piped through `tee` is detected + # via PIPESTATUS rather than masked by tee's exit code. + set -o pipefail + mkdir -p /tmp/radius-output + RESULT_FILE=/tmp/radius-output/rad-commands-result.json + + # Aggregate result accumulators. The result is written by a trap on EXIT so + # the combined `rad-commands-result` artifact is complete even when a command + # fails and the step exits early. + COMMANDS_JSON='[]' + OVERALL_OUTCOME="succeeded" + OVERALL_EXIT=0 + REQUESTED=0 + RAN=0 + + write_result() { + jq -n \ + --arg outcome "$OVERALL_OUTCOME" \ + --argjson exitCode "$OVERALL_EXIT" \ + --arg environment "$ENVIRONMENT" \ + --argjson requested "$REQUESTED" \ + --argjson ran "$RAN" \ + --argjson commands "$COMMANDS_JSON" \ + '{schemaVersion:"1.0", outcome:$outcome, exitCode:$exitCode, environment:$environment, commandsRequested:$requested, commandsRan:$ran, commands:$commands}' \ + > "$RESULT_FILE" + } + trap write_result EXIT + + # Allowed-command set. Each command's leading verb must be in this list, so + # commands that do not fit the ephemeral, per-run model — managing or + # upgrading a control plane, switching workspaces, or changing install state — + # cannot run. Validation happens before any command runs, so a disallowed + # command fails fast (exit 2) without deploying anything. + ALLOWED_VERBS="deploy app resource env recipe group credential resource-type version bicep" + is_allowed() { + local verb="${1%% *}" + case " $ALLOWED_VERBS " in *" $verb "*) return 0 ;; *) return 1 ;; esac + } + + # Record one command's result into the combined artifact. Keeps live console + # output via tee while capturing it for the artifact. + record() { + local index="$1"; shift + local display="$1"; shift + local outfile code + outfile=$(mktemp) + echo "::group::rad $display" + "$@" 2>&1 | tee "$outfile" + code=${PIPESTATUS[0]} + echo "::endgroup::" + RAN=$((RAN + 1)) + COMMANDS_JSON=$(jq \ + --argjson index "$index" \ + --arg cmd "$display" \ + --argjson code "$code" \ + --arg outcome "$([ "$code" -eq 0 ] && echo succeeded || echo failed)" \ + --rawfile out "$outfile" \ + '. + [{index:$index, command:$cmd, outcome:$outcome, exitCode:$code, output:$out}]' \ + <<<"$COMMANDS_JSON") + rm -f "$outfile" + return "$code" + } + + if [ -n "${RAD_COMMANDS//[[:space:]]/}" ]; then + # Caller-supplied commands: a single command string or a JSON array, run in + # order with the `rad` prefix omitted. + if printf '%s' "$RAD_COMMANDS" | jq -e 'type == "array"' >/dev/null 2>&1; then + mapfile -t COMMANDS < <(printf '%s' "$RAD_COMMANDS" | jq -r '.[]') + else + COMMANDS=("$RAD_COMMANDS") + fi + + # Drop blank entries and validate every command before running any of them. + CLEAN=() + for cmd in "${COMMANDS[@]}"; do + [ -z "${cmd//[[:space:]]/}" ] && continue + CLEAN+=("$cmd") + done + REQUESTED=${#CLEAN[@]} + + for cmd in "${CLEAN[@]}"; do + if ! is_allowed "$cmd"; then + OVERALL_OUTCOME="disallowed_command" + OVERALL_EXIT=2 + COMMANDS_JSON=$(jq --arg cmd "$cmd" \ + '. + [{command:$cmd, outcome:"disallowed", exitCode:2, errorMessage:"Command leading verb is not in the allowed-command set."}]' \ + <<<"$COMMANDS_JSON") + echo "Disallowed command: rad $cmd" >&2 + exit 2 + fi + done + + idx=0 + for cmd in "${CLEAN[@]}"; do + verb="${cmd%% *}" + if [ "$verb" = "deploy" ]; then + # For deploy commands, append the image and the secret application + # parameters. Build an argv array so secret values with special + # characters are never word-split, and keep them out of the + # recorded command string (only the non-secret `$cmd` is recorded). + # shellcheck disable=SC2086 + read -ra CMD_ARGV <<< "$cmd" + EXTRA_PARAMS=() + if [ -n "$APP_IMAGE" ]; then + EXTRA_PARAMS+=(--parameters "image=$APP_IMAGE") + fi + if [ -n "${RADIUS_DEPLOY_PARAMS//[[:space:]]/}" ]; then + while IFS= read -r _pname; do + [ -z "$_pname" ] && continue + _pval=$(printf '%s' "$RADIUS_DEPLOY_PARAMS" | jq -r --arg k "$_pname" '.[$k]') + EXTRA_PARAMS+=(--parameters "$_pname=$_pval") + done < <(printf '%s' "$RADIUS_DEPLOY_PARAMS" | jq -r 'keys_unsorted[]') + fi + if ! record "$idx" "$cmd" rad "${CMD_ARGV[@]}" "${EXTRA_PARAMS[@]}"; then + OVERALL_OUTCOME="command_failed" + OVERALL_EXIT=1 + echo "Command failed: rad $cmd" >&2 + exit 1 + fi + else + # Intentional word-split: the command string carries its own args. + # shellcheck disable=SC2086 + if ! record "$idx" "$cmd" rad $cmd; then + OVERALL_OUTCOME="command_failed" + OVERALL_EXIT=1 + echo "Command failed: rad $cmd" >&2 + exit 1 + fi + fi + idx=$((idx + 1)) + done + echo "✅ Radius commands complete." + else + # Default behavior: deploy the app bicep. Build the parameters as an array + # so values with special characters are not mangled by the shell. + REQUESTED=1 + DEPLOY_PARAMS=() + if [ -n "$APP_IMAGE" ]; then + DEPLOY_PARAMS+=(--parameters "image=$APP_IMAGE") + fi + # Expand the application parameters JSON ({"name":"value", …}) into + # --parameters name=value pairs. Each value is read back from the JSON + # by key with jq so embedded '=', spaces, or newlines are preserved and + # never re-split by the shell. + if [ -n "${RADIUS_DEPLOY_PARAMS//[[:space:]]/}" ]; then + while IFS= read -r _pname; do + [ -z "$_pname" ] && continue + _pval=$(printf '%s' "$RADIUS_DEPLOY_PARAMS" | jq -r --arg k "$_pname" '.[$k]') + DEPLOY_PARAMS+=(--parameters "$_pname=$_pval") + done < <(printf '%s' "$RADIUS_DEPLOY_PARAMS" | jq -r 'keys_unsorted[]') + fi + # The recorded command string omits the parameters so secret values are + # not written into the result artifact. + if ! record 0 "deploy $APP_FILE --environment $ENVIRONMENT" \ + rad deploy "$APP_FILE" --environment "$ENVIRONMENT" "${DEPLOY_PARAMS[@]}"; then + OVERALL_OUTCOME="command_failed" + OVERALL_EXIT=1 + echo "Deployment failed" >&2 + exit 1 + fi + echo "✅ Deployment complete." + fi + + - name: Upload command result + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: rad-commands-result + path: /tmp/radius-output/ + retention-days: 1 diff --git a/.github/extension/actions/setup-control-plane/action.yml b/.github/extension/actions/setup-control-plane/action.yml new file mode 100644 index 00000000000..9e7030c5272 --- /dev/null +++ b/.github/extension/actions/setup-control-plane/action.yml @@ -0,0 +1,76 @@ +# Provider-agnostic control-plane setup shared by run-rad-commands-aws.yml and +# run-rad-commands-azure.yml. Installs the runner tooling, stands up the ephemeral +# k3d control-plane cluster, stores the target-cluster kubeconfig as a Secret, and +# installs Radius. Reads RADIUS_TARGET_KUBECONFIG from the job environment (exported +# via $GITHUB_ENV by the caller), so it needs no inputs. +name: Radius - Set up control plane +description: Install tooling, create the ephemeral k3d control plane, and install Radius. + +runs: + using: composite + steps: + - name: Install k3d + shell: bash + run: curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash + + - name: Create ephemeral Radius control plane cluster + shell: bash + run: | + k3d cluster create radius-cp \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + --volume "${{ github.workspace }}/:/app/demo" \ + --k3s-arg "--disable=traefik@server:*" \ + --wait + kubectl wait --for=condition=Ready node --all --timeout=120s + + - name: Install oras + uses: oras-project/setup-oras@22ce207df3b08e061f537244349aac6ae1d214f6 # v1 + + - name: Install Radius CLI + shell: bash + run: | + wget -q "https://raw.githubusercontent.com/radius-project/radius/main/deploy/install.sh" -O - | /bin/bash -s edge + rad version + + - name: Install Terraform + uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # v3 + with: + terraform_wrapper: false + + - name: Create target-kubeconfig secret + # The chart's targetCluster seam mounts this secret into applications-rp, + # dynamic-rp, and bicep-de at install time and sets RADIUS_TARGET_KUBECONFIG, + # so the secret must exist before `rad install`. + shell: bash + run: | + if [ ! -f "$RADIUS_TARGET_KUBECONFIG" ]; then + echo "No target kubeconfig; resources will deploy to the control-plane cluster." + exit 0 + fi + kubectl create namespace radius-system --dry-run=client -o yaml | kubectl apply -f - + kubectl create secret generic target-kubeconfig \ + --namespace radius-system \ + --from-file=kubeconfig="$RADIUS_TARGET_KUBECONFIG" \ + --dry-run=client -o yaml | kubectl apply -f - + + - name: Install Radius on control plane + shell: bash + run: | + # When a target kubeconfig is present, enable the chart's targetCluster + # seam. It mounts the target-kubeconfig secret into applications-rp, + # dynamic-rp, and bicep-de and sets RADIUS_TARGET_KUBECONFIG so recipe + # execution and directly-rendered resources target the external cluster. + # The Terraform state backend deliberately stays on the control plane. + # + # database.enabled=true installs the control-plane PostgreSQL so durable + # Radius state can be backed up and restored by `rad shutdown`/`rad startup`. + TARGET_FLAGS=() + if [ -f "$RADIUS_TARGET_KUBECONFIG" ]; then + TARGET_FLAGS+=(--set global.targetCluster.enabled=true) + fi + rad install kubernetes \ + --set database.enabled=true \ + --set rp.publicEndpointOverride=localhost \ + --set dynamicrp.buildkit.enabled=true \ + "${TARGET_FLAGS[@]}" + kubectl wait --for=condition=Available deployment --all -n radius-system --timeout=300s diff --git a/.github/extension/actions/teardown/action.yml b/.github/extension/actions/teardown/action.yml new file mode 100644 index 00000000000..05012715921 --- /dev/null +++ b/.github/extension/actions/teardown/action.yml @@ -0,0 +1,66 @@ +# Provider-agnostic teardown shared by run-rad-commands-aws.yml and +# run-rad-commands-azure.yml. Persists durable state via `rad shutdown`, collects +# logs on failure, and always tears the k3d control plane down. Invoked with +# `if: always()` so it runs even when an earlier step failed. +name: Radius - Teardown +description: Persist state (rad shutdown), collect logs on failure, and delete the k3d control plane. + +runs: + using: composite + steps: + - name: Persist Radius state (rad shutdown) + if: always() + shell: bash + run: | + # rad shutdown backs up the control-plane PostgreSQL databases and the + # Terraform recipe-state Secrets to the `radius-state` git orphan branch and + # pushes it to origin (contents: write). It runs even when a command fails so + # a partially-applied Terraform run is not lost. It does not manage the + # cluster; the cleanup step tears k3d down afterwards. + git config --global user.email "radius-deploy@users.noreply.github.com" + git config --global user.name "radius-deploy" + rad shutdown + + - name: Show application status + if: always() + shell: bash + run: | + rad app list || true + + - name: Collect Radius logs + if: failure() + shell: bash + run: | + mkdir -p /tmp/radius-logs + for deploy in applications-rp dynamic-rp bicep-de controller ucpd; do + kubectl logs -n radius-system -l app.kubernetes.io/name=$deploy --tail=200 >> /tmp/radius-logs/$deploy.log 2>&1 || true + done + kubectl get pods -n radius-system -o wide >> /tmp/radius-logs/pods.txt 2>&1 || true + kubectl get events -n radius-system --sort-by=.lastTimestamp >> /tmp/radius-logs/events.txt 2>&1 || true + + # Collect app pod logs and events from target cluster + APP_NS="default-todo-list-app" + TKC="$RADIUS_TARGET_KUBECONFIG" + KC=""; [ -f "$TKC" ] && KC="--kubeconfig $TKC" + if kubectl $KC get ns "$APP_NS" &>/dev/null; then + kubectl $KC get pods -n "$APP_NS" -o wide >> /tmp/radius-logs/app-pods.txt 2>&1 || true + kubectl $KC get events -n "$APP_NS" --sort-by=.lastTimestamp >> /tmp/radius-logs/app-events.txt 2>&1 || true + for pod in $(kubectl $KC get pods -n "$APP_NS" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do + kubectl $KC logs -n "$APP_NS" "$pod" --all-containers --tail=100 >> "/tmp/radius-logs/app-pod-$pod.log" 2>&1 || true + kubectl $KC describe pod -n "$APP_NS" "$pod" >> "/tmp/radius-logs/app-describe-$pod.txt" 2>&1 || true + done + kubectl $KC get secret dbsecret -n "$APP_NS" -o jsonpath='{.data}' >> /tmp/radius-logs/app-dbsecret-keys.txt 2>&1 || true + fi + + - name: Upload Radius logs + if: failure() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: radius-logs + path: /tmp/radius-logs/ + retention-days: 3 + + - name: Cleanup control plane cluster + if: always() + shell: bash + run: k3d cluster delete radius-cp || true diff --git a/.github/extension/run-rad-commands-aws.yml b/.github/extension/run-rad-commands-aws.yml new file mode 100644 index 00000000000..06e31fd79a2 --- /dev/null +++ b/.github/extension/run-rad-commands-aws.yml @@ -0,0 +1,327 @@ +# This workflow is auto-generated by Radius to run rad commands against an AWS +# environment. It is a reusable (workflow_call) workflow invoked by the unified +# run-rad-commands.yml dispatcher; it is not dispatched directly. It creates an +# ephemeral k3d cluster for the Radius control plane, connects to the user's EKS +# cluster, restores persisted state, runs the requested rad commands (deploying by +# default), then persists state again and tears the cluster down. The provider- +# agnostic phases are shared composite actions in radius-project/radius; only the +# AWS-specific steps live here. This is the run-rad-commands action in the +# two-action Repo Radius model (the other being verify-cloud-auth). +name: Radius - Run rad Commands (AWS) + +on: + workflow_call: + inputs: + environment: + description: 'GitHub Environment name' + type: string + required: true + image: + description: 'Container image for the application' + type: string + required: false + default: '' + rad_commands: + description: 'rad CLI command string, or JSON array of command strings (rad prefix omitted, run verbatim). Overrides the default deploy.' + type: string + required: false + default: '' + +permissions: + id-token: write + contents: write + packages: write + +env: + ENVIRONMENT: ${{ inputs.environment }} + APP_FILE: '{{APP_FILE}}' + APP_IMAGE: ${{ inputs.image || github.sha || 'latest' }} + RESOURCE_TYPES_CONTRIB_REPO: https://github.com/radius-project/resource-types-contrib.git + RESOURCE_TYPES_CONTRIB_REF: main + +jobs: + deploy: + name: Deploy with Radius + runs-on: ubuntu-latest + environment: ${{ inputs.environment }} + steps: + - name: Checkout + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + + - name: Configure AWS Credentials (OIDC) + if: ${{ vars.AWS_ROLE_ARN != '' }} + uses: aws-actions/configure-aws-credentials@7474bc4690e29a8392af63c5b98e7449536d5c3a # v4 + with: + role-to-assume: ${{ vars.AWS_ROLE_ARN }} + aws-region: ${{ vars.AWS_REGION }} + + - name: Get target cluster kubeconfig + run: | + mkdir -p "$HOME/.kube" + echo "RADIUS_TARGET_KUBECONFIG=$HOME/.kube/target-cluster" >> "$GITHUB_ENV" + + - name: Connect to EKS cluster + if: ${{ vars.AWS_EKS_CLUSTER_NAME != '' }} + run: | + CLUSTER="${{ vars.AWS_EKS_CLUSTER_NAME }}" + REGION="${{ vars.AWS_REGION }}" + ROLE_ARN="${{ vars.AWS_ROLE_ARN }}" + TARGET="$RADIUS_TARGET_KUBECONFIG" + + # Ensure the IAM role has access to the EKS cluster. + # Creates an access entry if one doesn't already exist. + echo "Ensuring EKS access entry for $ROLE_ARN..." + aws eks create-access-entry \ + --cluster-name "$CLUSTER" \ + --principal-arn "$ROLE_ARN" \ + --type STANDARD \ + --region "$REGION" 2>/dev/null || echo "Access entry already exists" + aws eks associate-access-policy \ + --cluster-name "$CLUSTER" \ + --principal-arn "$ROLE_ARN" \ + --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \ + --access-scope type=cluster \ + --region "$REGION" 2>/dev/null || echo "Access policy already associated" + + # Build a static kubeconfig with a bearer token instead of exec-based auth. + # The exec-based config requires aws CLI inside the container, which Radius + # images don't have. + ENDPOINT=$(aws eks describe-cluster --name "$CLUSTER" --region "$REGION" --query 'cluster.endpoint' --output text) + CA_DATA=$(aws eks describe-cluster --name "$CLUSTER" --region "$REGION" --query 'cluster.certificateAuthority.data' --output text) + TOKEN=$(aws eks get-token --cluster-name "$CLUSTER" --region "$REGION" --output json | jq -r '.status.token') + printf 'apiVersion: v1\nclusters:\n- cluster:\n certificate-authority-data: %s\n server: %s\n name: eks\ncontexts:\n- context:\n cluster: eks\n user: eks-user\n name: eks\ncurrent-context: eks\nkind: Config\nusers:\n- name: eks-user\n user:\n token: %s\n' "$CA_DATA" "$ENDPOINT" "$TOKEN" > "$TARGET" + echo "EKS kubeconfig saved with static token" + kubectl --kubeconfig "$TARGET" cluster-info || echo "WARNING: Could not connect to EKS cluster" + + - name: Set up control plane + uses: radius-project/radius/.github/extension/actions/setup-control-plane@{{RADIUS_REF}} + + - name: Project cloud OIDC tokens into Radius pods + run: | + # Mint GitHub OIDC tokens and mount them where Radius expects them. + # GitHub Actions is already a trusted OIDC issuer for both AWS and Azure + # (via the CloudFormation template / AAD federated credential), so no + # custom issuer setup is needed. + + if [ -n "${{ vars.AWS_ROLE_ARN }}" ]; then + echo "Projecting AWS OIDC token..." + AWS_TOKEN=$(curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ + "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=sts.amazonaws.com" | jq -r '.value') + kubectl create secret generic aws-oidc-token -n radius-system \ + --from-literal=token="$AWS_TOKEN" --dry-run=client -o yaml | kubectl apply -f - + fi + + # Mount the projected token at the fixed paths Radius reads from. The + # cloud identity selectors (AWS role ARN, Azure client/tenant id) are NOT + # set as SDK-native env vars here -- they come from the UCP credential + # registered later via `rad credential register`. We only project the raw + # token file that the credential needs for the federated token exchange. + + # AWS IRSA: the UCP AWS proxy (ucp) and the Terraform AWS provider + # (dynamic-rp) both read the token from the hard-coded path + # /var/run/secrets/eks.amazonaws.com/serviceaccount/token. + if [ -n "${{ vars.AWS_ROLE_ARN }}" ]; then + AWS_PATCH='[ + {"op":"add","path":"/spec/template/spec/volumes/-","value":{"name":"aws-oidc-token","secret":{"secretName":"aws-oidc-token"}}}, + {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/-","value":{"name":"aws-oidc-token","mountPath":"/var/run/secrets/eks.amazonaws.com/serviceaccount","readOnly":true}} + ]' + for deploy in applications-rp dynamic-rp ucp; do + kubectl patch deployment $deploy -n radius-system --type=json -p="$AWS_PATCH" 2>/dev/null || true + done + fi + + # Wait for any patched deployments to roll out. + for deploy in applications-rp dynamic-rp bicep-de ucp; do + kubectl rollout status deployment/$deploy -n radius-system --timeout=300s || true + done + echo "✅ Cloud OIDC tokens projected into Radius pods." + + - name: Refresh external deployment target credentials + run: | + TARGET_KUBECONFIG="$RADIUS_TARGET_KUBECONFIG" + + if [ ! -f "$TARGET_KUBECONFIG" ]; then + echo "No target kubeconfig found, resources will deploy to k3d cluster" + exit 0 + fi + + # Refresh the EKS token right before deploy (EKS tokens are short-lived + # and the one minted earlier may have expired during install). + if [ -n "${{ vars.AWS_ROLE_ARN }}" ] && [ -n "${{ vars.AWS_EKS_CLUSTER_NAME }}" ]; then + echo "Generating fresh EKS token..." + CLUSTER="${{ vars.AWS_EKS_CLUSTER_NAME }}" + REGION="${{ vars.AWS_REGION }}" + ENDPOINT=$(aws eks describe-cluster --name "$CLUSTER" --region "$REGION" --query 'cluster.endpoint' --output text) + CA_DATA=$(aws eks describe-cluster --name "$CLUSTER" --region "$REGION" --query 'cluster.certificateAuthority.data' --output text) + TOKEN=$(aws eks get-token --cluster-name "$CLUSTER" --region "$REGION" --output json | jq -r '.status.token') + printf 'apiVersion: v1\nclusters:\n- cluster:\n certificate-authority-data: %s\n server: %s\n name: eks\ncontexts:\n- context:\n cluster: eks\n user: eks-user\n name: eks\ncurrent-context: eks\nkind: Config\nusers:\n- name: eks-user\n user:\n token: %s\n' "$CA_DATA" "$ENDPOINT" "$TOKEN" > "$TARGET_KUBECONFIG" + fi + + # Update the secret the chart mounted at install with the refreshed + # kubeconfig, then restart the recipe-executing pods so they re-read it. + # The chart owns the volume mount and RADIUS_TARGET_KUBECONFIG env var via + # global.targetCluster.enabled, so no manual deployment patching is needed. + # All three pods honor RADIUS_TARGET_KUBECONFIG natively: applications-rp + # (direct-rendered resources) and dynamic-rp (Terraform kubernetes provider) + # via the cluster access resolver, and bicep-de (Bicep 'extension kubernetes' + # provider). The Terraform state backend deliberately stays on the + # control-plane cluster, so no backend override is needed here. + kubectl create secret generic target-kubeconfig --namespace radius-system \ + --from-file=kubeconfig="$TARGET_KUBECONFIG" --dry-run=client -o yaml | kubectl apply -f - + for deploy in applications-rp dynamic-rp bicep-de; do + kubectl rollout restart deployment/$deploy -n radius-system + done + + echo "Waiting for rollouts..." + kubectl rollout status deployment/applications-rp -n radius-system --timeout=300s + kubectl rollout status deployment/dynamic-rp -n radius-system --timeout=300s + kubectl rollout status deployment/bicep-de -n radius-system --timeout=300s + echo "External deployment target configured." + + - name: Restore Radius state + uses: radius-project/radius/.github/extension/actions/restore-state@{{RADIUS_REF}} + with: + namespace: ${{ vars.KUBERNETES_NAMESPACE || 'default' }} + + - name: Register cloud credentials with Radius + run: | + # Register the cloud identity with Radius using its UCP credential model. + # Radius stores only the identity selector (AWS role ARN / Azure client + + # tenant id); the short-lived token is read at runtime from the GitHub + # OIDC token file projected into the pods in the earlier step. This keeps + # the credential visible to `rad credential show`, rotatable, and usable + # by both the Bicep and Terraform code paths. + if [ -n "${{ vars.AWS_ROLE_ARN }}" ]; then + echo "Registering AWS IRSA credential..." + rad credential register aws irsa --iam-role "${{ vars.AWS_ROLE_ARN }}" + fi + echo "✅ Cloud credentials registered with Radius." + + - name: Create Radius environment and recipe pack + run: | + REPO="${{ env.RESOURCE_TYPES_CONTRIB_REPO }}" + REF="${{ env.RESOURCE_TYPES_CONTRIB_REF }}" + NAMESPACE="${{ vars.KUBERNETES_NAMESPACE || 'default' }}" + + # Registry and credentials for the Radius.Compute/containerImages recipe. + # The recipe builds images with the in-pod BuildKit and pushes them to + # $BUILD_REGISTRY, authenticating with the Kubernetes Secret named + # $REGISTRY_SECRET_NAME. Because the recipe loads that Secret via the + # in-cluster provider (the control-plane cluster where dynamic-rp/BuildKit + # run), the secret is provisioned directly onto the control plane by the + # run-and-teardown action. Only the secret NAME is wired here via the + # recipe pack's registrySecretName. $BUILD_REGISTRY is ghcr.io// + # so images land under the repository's package namespace; it must be lowercase. + BUILD_REGISTRY=$(echo "${{ vars.RADIUS_BUILD_REGISTRY || format('ghcr.io/{0}', github.repository) }}" | tr '[:upper:]' '[:lower:]') + REGISTRY_SECRET_NAME="ghcr-registry-creds" + + # Select the provider-specific Terraform recipe pack. Both provider packs + # bundle the shared Kubernetes compute/data recipes and differ only in the + # mySQL database recipe and cloud provider config. + PACK_NAME="aws-terraform" + MYSQL_RECIPE=$(cat < "$ENV_BICEP" <> "$GITHUB_ENV" + + - name: Connect to AKS cluster + if: ${{ vars.AZURE_AKS_CLUSTER_NAME != '' }} + run: | + az aks get-credentials \ + --resource-group "${{ vars.AZURE_RESOURCE_GROUP }}" \ + --name "${{ vars.AZURE_AKS_CLUSTER_NAME }}" \ + --subscription "${{ vars.AZURE_SUBSCRIPTION_ID }}" \ + --file "$RADIUS_TARGET_KUBECONFIG" + + - name: Set up control plane + uses: radius-project/radius/.github/extension/actions/setup-control-plane@{{RADIUS_REF}} + + - name: Project cloud OIDC tokens into Radius pods + run: | + # Mint GitHub OIDC tokens and mount them where Radius expects them. + # GitHub Actions is already a trusted OIDC issuer for both AWS and Azure + # (via the CloudFormation template / AAD federated credential), so no + # custom issuer setup is needed. + + if [ -n "${{ vars.AZURE_CLIENT_ID }}" ]; then + echo "Projecting Azure OIDC token..." + AZ_TOKEN=$(curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ + "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=api://AzureADTokenExchange" | jq -r '.value') + # The secret data key becomes the mounted file name, so it must be + # 'azure-identity-token' -- the file Radius and the Terraform azurerm + # provider read from /var/run/secrets/azure/tokens/. + kubectl create secret generic azure-oidc-token -n radius-system \ + --from-literal=azure-identity-token="$AZ_TOKEN" --dry-run=client -o yaml | kubectl apply -f - + fi + + # Mount the projected token at the fixed paths Radius reads from. The + # cloud identity selectors (AWS role ARN, Azure client/tenant id) are NOT + # set as SDK-native env vars here -- they come from the UCP credential + # registered later via `rad credential register`. We only project the raw + # token file that the credential needs for the federated token exchange. + + # Azure workload identity: the Go UCPCredential (applications-rp), the + # .NET Bicep DE (bicep-de) and the Terraform azurerm provider (dynamic-rp) + # all read /var/run/secrets/azure/tokens/azure-identity-token. + # AZURE_FEDERATED_TOKEN_FILE is still required because armauth builds the + # WorkloadIdentityCredential with an empty TokenFilePath and falls back to + # this env var to locate the token. + if [ -n "${{ vars.AZURE_CLIENT_ID }}" ]; then + AZ_PATCH='[ + {"op":"add","path":"/spec/template/spec/volumes/-","value":{"name":"azure-oidc-token","secret":{"secretName":"azure-oidc-token"}}}, + {"op":"add","path":"/spec/template/spec/containers/0/volumeMounts/-","value":{"name":"azure-oidc-token","mountPath":"/var/run/secrets/azure/tokens","readOnly":true}}, + {"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"AZURE_FEDERATED_TOKEN_FILE","value":"/var/run/secrets/azure/tokens/azure-identity-token"}} + ]' + for deploy in applications-rp dynamic-rp bicep-de; do + kubectl patch deployment $deploy -n radius-system --type=json -p="$AZ_PATCH" 2>/dev/null || true + done + fi + + # Wait for any patched deployments to roll out. + for deploy in applications-rp dynamic-rp bicep-de ucp; do + kubectl rollout status deployment/$deploy -n radius-system --timeout=300s || true + done + echo "✅ Cloud OIDC tokens projected into Radius pods." + + - name: Refresh external deployment target credentials + run: | + TARGET_KUBECONFIG="$RADIUS_TARGET_KUBECONFIG" + + if [ ! -f "$TARGET_KUBECONFIG" ]; then + echo "No target kubeconfig found, resources will deploy to k3d cluster" + exit 0 + fi + + # Update the secret the chart mounted at install with the refreshed + # kubeconfig, then restart the recipe-executing pods so they re-read it. + # The chart owns the volume mount and RADIUS_TARGET_KUBECONFIG env var via + # global.targetCluster.enabled, so no manual deployment patching is needed. + # All three pods honor RADIUS_TARGET_KUBECONFIG natively: applications-rp + # (direct-rendered resources) and dynamic-rp (Terraform kubernetes provider) + # via the cluster access resolver, and bicep-de (Bicep 'extension kubernetes' + # provider). The Terraform state backend deliberately stays on the + # control-plane cluster, so no backend override is needed here. + kubectl create secret generic target-kubeconfig --namespace radius-system \ + --from-file=kubeconfig="$TARGET_KUBECONFIG" --dry-run=client -o yaml | kubectl apply -f - + for deploy in applications-rp dynamic-rp bicep-de; do + kubectl rollout restart deployment/$deploy -n radius-system + done + + echo "Waiting for rollouts..." + kubectl rollout status deployment/applications-rp -n radius-system --timeout=300s + kubectl rollout status deployment/dynamic-rp -n radius-system --timeout=300s + kubectl rollout status deployment/bicep-de -n radius-system --timeout=300s + echo "External deployment target configured." + + - name: Restore Radius state + uses: radius-project/radius/.github/extension/actions/restore-state@{{RADIUS_REF}} + with: + namespace: ${{ vars.KUBERNETES_NAMESPACE || 'default' }} + + - name: Register cloud credentials with Radius + run: | + # Register the cloud identity with Radius using its UCP credential model. + # Radius stores only the identity selector (AWS role ARN / Azure client + + # tenant id); the short-lived token is read at runtime from the GitHub + # OIDC token file projected into the pods in the earlier step. This keeps + # the credential visible to `rad credential show`, rotatable, and usable + # by both the Bicep and Terraform code paths. + if [ -n "${{ vars.AZURE_CLIENT_ID }}" ]; then + echo "Registering Azure workload identity credential..." + rad credential register azure wi \ + --client-id "${{ vars.AZURE_CLIENT_ID }}" \ + --tenant-id "${{ vars.AZURE_TENANT_ID }}" + fi + echo "✅ Cloud credentials registered with Radius." + + - name: Create Radius environment and recipe pack + run: | + NAMESPACE="${{ vars.KUBERNETES_NAMESPACE || 'default' }}" + + # Registry and credentials for the Radius.Compute/containerImages recipe. + # The recipe builds images with the in-pod BuildKit and pushes them to + # $BUILD_REGISTRY, authenticating with the Kubernetes Secret named + # $REGISTRY_SECRET_NAME. Because the recipe loads that Secret via the + # in-cluster provider (the control-plane cluster where dynamic-rp/BuildKit + # run), the secret is provisioned directly onto the control plane by the + # run-and-teardown action. Only the secret NAME is wired here via the + # recipe pack's containerImagesRegistrySecretName. $BUILD_REGISTRY is + # ghcr.io// so images land under the repository's package + # namespace; it must be lowercase. + BUILD_REGISTRY=$(echo "${{ vars.RADIUS_BUILD_REGISTRY || format('ghcr.io/{0}', github.repository) }}" | tr '[:upper:]' '[:lower:]') + REGISTRY_SECRET_NAME="ghcr-registry-creds" + + # Download the default Azure recipe pack from resource-types-contrib, + # pinned to $RECIPE_PACK_REF. The recipe logic lives upstream; here we + # only supply parameters. The pack creates a Radius.Core/recipePacks + # resource and a Radius.Core/environments resource referencing it. + RECIPE_PACK_URL="https://raw.githubusercontent.com/radius-project/resource-types-contrib/${{ env.RECIPE_PACK_REF }}/${{ env.RECIPE_PACK_PATH }}" + echo "Downloading recipe pack from $RECIPE_PACK_URL" + # Place the recipe pack next to the app file so `rad deploy` resolves the + # repo's own bicepconfig.json (which declares the `radius` extension). bicep + # resolves the config nearest the .bicep file, so writing it to $APP_DIR + # (e.g. .radius/) picks up .radius/bicepconfig.json instead of the workspace + # root, where no config exists. + APP_DIR=$(dirname "$APP_FILE") + ENV_BICEP="$APP_DIR/radius-env.bicep" + curl -fsSL "$RECIPE_PACK_URL" -o "$ENV_BICEP" + + # routesGatewayName is a required pack parameter, but it only matters when + # a Radius.Compute/routes resource is actually deployed. Default to empty + # unless the repo configures an existing Kubernetes Gateway. + ROUTES_GATEWAY_NAME="${{ vars.RADIUS_ROUTES_GATEWAY_NAME }}" + ROUTES_GATEWAY_NAMESPACE="${{ vars.RADIUS_ROUTES_GATEWAY_NAMESPACE || 'default' }}" + + echo "Recipe pack file:" + cat "$ENV_BICEP" + echo "" + echo "Deploying Radius.Core/environments and recipe pack..." + rad deploy "$ENV_BICEP" \ + --parameters environmentName="$ENVIRONMENT" \ + --parameters environmentNamespace="$NAMESPACE" \ + --parameters azureSubscriptionId="${{ vars.AZURE_SUBSCRIPTION_ID }}" \ + --parameters azureResourceGroup="${{ vars.AZURE_RESOURCE_GROUP }}" \ + --parameters routesGatewayName="$ROUTES_GATEWAY_NAME" \ + --parameters routesGatewayNamespace="$ROUTES_GATEWAY_NAMESPACE" \ + --parameters containerImagesRegistry="$BUILD_REGISTRY" \ + --parameters containerImagesRegistrySecretName="$REGISTRY_SECRET_NAME" + echo "✅ Environment '$ENVIRONMENT' created with recipe pack." + + - name: Run rad commands + uses: radius-project/radius/.github/extension/actions/run-rad-commands@{{RADIUS_REF}} + with: + environment: ${{ inputs.environment }} + app-file: ${{ env.APP_FILE }} + app-image: ${{ env.APP_IMAGE }} + namespace: ${{ vars.KUBERNETES_NAMESPACE || 'default' }} + rad-commands: ${{ inputs.rad_commands || vars.RADIUS_RAD_COMMANDS }} + deploy-params: ${{ secrets.RADIUS_DEPLOY_PARAMS }} + registry-username: ${{ github.actor }} + registry-password: ${{ secrets.GITHUB_TOKEN }} + + - name: Teardown + if: always() + uses: radius-project/radius/.github/extension/actions/teardown@{{RADIUS_REF}} diff --git a/.github/extension/run-rad-commands.yml b/.github/extension/run-rad-commands.yml new file mode 100644 index 00000000000..19827e27b89 --- /dev/null +++ b/.github/extension/run-rad-commands.yml @@ -0,0 +1,92 @@ +# This workflow is auto-generated by Radius. It is the unified entry point for +# running rad commands against an environment: it detects whether the selected +# GitHub Environment is wired for Azure or AWS and calls the matching reusable +# workflow (run-rad-commands-azure.yml / run-rad-commands-aws.yml). Keeping the +# dispatch contract (inputs, environment binding, verify→deploy auto-trigger) in +# one place lets the provider workflows stay thin and focused on cloud-specific +# steps. This is the run-rad-commands action in the two-action Repo Radius model +# (the other being verify-cloud-auth). +name: Radius - Run rad Commands + +on: + workflow_dispatch: + inputs: + environment: + description: 'GitHub Environment name' + required: true + default: '{{ENV}}' + image: + description: 'Container image for the application' + required: false + default: '' + rad_commands: + # The Repo Radius dispatch contract. A single rad CLI command string, or a + # JSON-encoded array of command strings run in order, with the `rad` prefix + # omitted (e.g. `deploy .radius/app.bicep --environment dev` or + # `["deploy .radius/app.bicep --environment dev", "app graph my-app -o json"]`). + # Commands run verbatim, so the caller must include any flags a command needs. + # Each command is validated against the allowed-command set before anything + # runs. When empty, the workflow runs its default `rad deploy` of the app file. + description: 'rad CLI command string, or JSON array of command strings (rad prefix omitted, run verbatim). Overrides the default deploy.' + required: false + default: '' + # Auto-trigger after the Verify Credentials workflow completes. The extension's + # environment-setup flow dispatches only the verify workflow; this chain starts + # the deploy once verify succeeds (the job-level `if` gates on the conclusion). + workflow_run: + workflows: ["Radius - Verify Credentials"] + types: [completed] + +permissions: + id-token: write + contents: write + packages: write + +jobs: + # Bind to the GitHub Environment so environment-scoped variables are visible, then + # pick the provider from whichever identifying variable is set. A job that calls a + # reusable workflow (`uses:`) can't bind an environment itself, so this routing + # decision has to happen in a regular job first. + detect: + name: Detect provider + runs-on: ubuntu-latest + environment: ${{ inputs.environment || '{{ENV}}' }} + # On workflow_dispatch run unconditionally; on workflow_run only when the + # upstream Verify Credentials run succeeded. + if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }} + outputs: + provider: ${{ steps.detect.outputs.provider }} + steps: + - name: Determine provider from environment variables + id: detect + run: | + if [ -n "${{ vars.AZURE_CLIENT_ID }}" ]; then + echo "provider=azure" >> "$GITHUB_OUTPUT" + elif [ -n "${{ vars.AWS_ROLE_ARN }}" ]; then + echo "provider=aws" >> "$GITHUB_OUTPUT" + else + echo "No AZURE_CLIENT_ID or AWS_ROLE_ARN set on this environment." >&2 + echo "provider=none" >> "$GITHUB_OUTPUT" + fi + + azure: + name: Azure + needs: detect + if: ${{ needs.detect.outputs.provider == 'azure' }} + uses: ./.github/workflows/run-rad-commands-azure.yml + with: + environment: ${{ inputs.environment || '{{ENV}}' }} + image: ${{ inputs.image }} + rad_commands: ${{ inputs.rad_commands }} + secrets: inherit + + aws: + name: AWS + needs: detect + if: ${{ needs.detect.outputs.provider == 'aws' }} + uses: ./.github/workflows/run-rad-commands-aws.yml + with: + environment: ${{ inputs.environment || '{{ENV}}' }} + image: ${{ inputs.image }} + rad_commands: ${{ inputs.rad_commands }} + secrets: inherit diff --git a/.github/skills/radius-deploy/SKILL.md b/.github/skills/radius-deploy/SKILL.md new file mode 100644 index 00000000000..dc3fc994089 --- /dev/null +++ b/.github/skills/radius-deploy/SKILL.md @@ -0,0 +1,72 @@ +--- +name: radius-deploy +description: Deploy a Radius application to a configured environment via the auto-generated GitHub Actions workflow. Use when the user asks to deploy, redeploy, trigger a deployment, or troubleshoot a failed Radius deploy. +--- + +# Radius — Deploy Application + +Trigger the `Radius - Run rad Commands` workflow which spins up an ephemeral k3d Radius control plane, connects to the target AKS/EKS cluster, registers the right recipes for the env's provider, restores persisted state, runs the requested `rad` commands (deploying by default), and persists state again before tearing the control plane down. + +## When to use this skill + +- "Deploy my app" +- "Redeploy to the test environment" +- "Trigger a deploy" +- "Why did my deploy fail?" +- "Deploy app X to env Y" + +## Prerequisites + +Before invoking this skill, all of these must exist: +1. A GitHub Environment configured with cloud credentials → use the `radius-environment` skill if missing. +2. A `.radius/app.bicep` file → use the `radius-app-bicep` skill if missing. +3. Authenticated access to dispatch the workflow (e.g. a logged-in `gh` CLI, or a token with `actions: write` on the repo). The token only triggers the run; it is never passed into the workflow. + +## How to invoke + +Trigger the workflow via the GitHub API (or `gh`). Omit `rad_commands` to run the default `rad deploy` of `.radius/app.bicep`: + +``` +POST /repos/{owner}/{repo}/actions/workflows/run-rad-commands.yml/dispatches +{ "ref": "main", "inputs": { "environment": "", "image": "" } } +``` + +```bash +gh workflow run run-rad-commands.yml -f environment= [-f image=] +``` + +Then follow the run (`gh run watch` or the run URL) until it succeeds, fails, or times out. `run-rad-commands.yml` is a dispatcher: it detects the environment's provider and calls the matching reusable workflow (`run-rad-commands-azure.yml` / `run-rad-commands-aws.yml`), so the actual deploy work runs as a called workflow underneath it. + +## What the workflow does + +1. The dispatcher detects the environment's provider (from `AZURE_CLIENT_ID` / `AWS_ROLE_ARN`) and calls the matching provider workflow, which authenticates to that cloud via OIDC. +2. Fetches a kubeconfig for the target cluster into `RADIUS_TARGET_KUBECONFIG` (EKS via `aws eks describe-cluster` + a static bearer-token kubeconfig; AKS via `az aks get-credentials`). +3. Installs `k3d`, creates the ephemeral `radius-cp` cluster, and installs the `rad` CLI (edge) and Terraform. +4. When a target kubeconfig exists, creates the `target-kubeconfig` secret and installs Radius with `--set global.targetCluster.enabled=true` (plus `--set database.enabled=true` for state backup/restore and `--set dynamicrp.buildkit.enabled=true` for in-pod image builds). The chart mounts the secret into `applications-rp`, `dynamic-rp`, and `bicep-de` and sets `RADIUS_TARGET_KUBECONFIG` so recipes and directly-rendered resources target the external cluster. Without a target kubeconfig, resources deploy to the k3d control plane. The Terraform state backend stays on the control plane. +5. Projects GitHub OIDC tokens into the pods and registers the cloud identity with `rad credential register` (`aws irsa` / `azure wi`). +6. Refreshes the (short-lived EKS) target token, updates the `target-kubeconfig` secret, and restarts the recipe-executing pods so they re-read it. +7. Creates the CLI workspace/group, then runs `rad startup` to restore the control-plane databases and Terraform recipe-state Secrets saved by the previous run (a no-op on the first run). The `Radius.Compute/containerImages` type ships with the published `radius` Bicep extension, so no resource-type registration or local Bicep-extension build is needed at deploy time. +8. Deploys a `Radius.Core/environments` resource and recipe pack from the app file's directory (e.g. `.radius/`) so the repo's own `bicepconfig.json` resolves the `radius` extension. Azure downloads the `azure-avm` pack from `resource-types-contrib`; AWS generates an inline pack bundling the Kubernetes recipes (`containers`, `containerImages`, `persistentVolumes`, `routes`, `postgreSqlDatabases`, `secrets`) plus a provider-gated `mySqlDatabases` recipe (AWS RDS). +9. Creates registry credentials for image builds, then runs `rad deploy` on `.radius/app.bicep` (passing the `image` parameter, and any application parameters from the `RADIUS_DEPLOY_PARAMS` secret when set). Afterwards `rad shutdown` (`if: always()`) backs the control-plane databases and Terraform recipe-state Secrets up to the `radius-state` git orphan branch. On failure, logs are uploaded as the `radius-logs` artifact; the k3d cluster is always deleted. + +## Common failure modes + +- **`RecipeDeploymentFailed` with `the resource with id '/planes/aws/aws/providers/System.AWS/credentials/default' was not found`** + → The `mySqlDatabases` recipe was registered for the wrong provider. The fix lives in the `Create Radius environment and recipe pack` step: an AWS env uses the inline `recipes/aws/terraform` recipe, an Azure env uses the `azure-avm` pack. If you see this error, the committed workflow is stale — re-commit the updated workflow and re-trigger the deploy. + +- **`RecipeDownloadFailed` with `subdir not found`** + → The recipe path doesn't exist on the configured `RESOURCE_TYPES_CONTRIB_REF` branch. Check the actual layout in `radius-project/resource-types-contrib` for that branch. `mySqlDatabases` specifically has **no** `recipes/kubernetes/terraform` directory — only aws, azure, and a kubernetes/bicep variant. + +- **Workflow runs but pod never reaches Ready** + → Look at the `Install Radius on control plane` and `Refresh external deployment target credentials` steps in the run logs. Usually a target cluster kubeconfig issue (expired EKS token, AKS network restriction). + +## After a successful deploy + +- Tell the user the deploy succeeded and include the workflow run URL. + +## Related files + +- `.github/extension/run-rad-commands.yml` (this repo) — the unified dispatcher template; a copy is committed into the user repo at `.github/workflows/run-rad-commands.yml` and is the file that gets dispatched. +- `.github/extension/run-rad-commands-azure.yml` and `.github/extension/run-rad-commands-aws.yml` — the provider-specific reusable (`workflow_call`) workflows the dispatcher calls; committed alongside the dispatcher. +- `.github/extension/actions/*` — the shared composite actions (`setup-control-plane`, `restore-state`, `run-rad-commands`, `teardown`) the provider workflows reference from `radius-project/radius`; not copied into the user repo. +- `.github/extension/README.md` — the workflow contract: trigger/inputs, required `vars`, secrets, and prerequisites. diff --git a/.github/workflows/functional-test-noncloud.yaml b/.github/workflows/functional-test-noncloud.yaml index 422dc4b921a..caff52fa9f4 100644 --- a/.github/workflows/functional-test-noncloud.yaml +++ b/.github/workflows/functional-test-noncloud.yaml @@ -50,6 +50,9 @@ on: permissions: {} env: + # yq version + YQ_VERSION: v4.44.3 + YQ_LINUX_AMD64_SHA256: a2c097180dd884a8d50c956ee16a9cec070f30a7947cf4ebf87d5f36213e9ed7 # Dapr runtime version DAPR_RUNTIME_VER: 1.15.4 # Dapr dashboard version @@ -84,6 +87,9 @@ env: GIT_HTTP_PASSWORD: not-a-secret-password # Kubernetes client QPS and Burst settings for high-concurrency CI environments RADIUS_QPS_AND_BURST: "800" + # Bicep CLI pinned: v0.40+ rejects br:localhost:5000/... (ThrowIfRegistryNotTrusted). + # Bump only after verifying localhost support or adding allowedUntrustedRegistries to bicepconfig.json. + BICEP_VER: v0.42.1 jobs: changes: @@ -167,6 +173,7 @@ jobs: kubernetes-noncloud, msgrp-noncloud, multicluster-noncloud, + statestore-noncloud, samples-noncloud, ucp-noncloud, datastoresrp-noncloud, @@ -214,12 +221,12 @@ jobs: } >> "${GITHUB_ENV}" - name: Checkout - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 with: persist-credentials: false - name: Checkout samples repo - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 if: matrix.name == 'samples-noncloud' with: repository: radius-project/samples @@ -241,7 +248,12 @@ jobs: - name: Install yq # Required by make generate-bicep-types-contrib to parse defaults.yaml. - run: make install-yq + run: | + mkdir -p "${RUNNER_TEMP}/bin" + curl -fsSL "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_amd64" -o "${RUNNER_TEMP}/bin/yq" + echo "${YQ_LINUX_AMD64_SHA256} ${RUNNER_TEMP}/bin/yq" | sha256sum -c - + chmod +x "${RUNNER_TEMP}/bin/yq" + echo "${RUNNER_TEMP}/bin" >> "${GITHUB_PATH}" - name: Generate Bicep extensibility types from OpenAPI specs env: @@ -265,9 +277,12 @@ jobs: registry-server: ${{ env.LOCAL_REGISTRY_SERVER }} registry-port: ${{ env.LOCAL_REGISTRY_PORT }} - - name: Install bicep CLI - # Pinned version + checksum live in build/tools.mk. - run: make install-bicep + - name: Setup and verify bicep CLI + run: | + curl -Lo bicep "https://github.com/Azure/bicep/releases/download/${BICEP_VER}/bicep-linux-x64" + chmod +x ./bicep + sudo mv ./bicep /usr/local/bin/bicep + bicep --version - name: Publish bicep types env: @@ -398,6 +413,10 @@ jobs: } >> "${GITHUB_ENV}" - name: Install Radius + # The statestore leg drives its own install/uninstall/reinstall lifecycle + # inside the test (it must reinstall to simulate an ephemeral control + # plane), so the shared install is skipped for it. + if: matrix.name != 'statestore-noncloud' run: | export PATH=$GITHUB_WORKSPACE/bin:$PATH which rad || { echo "cannot find rad"; exit 1; } @@ -591,7 +610,9 @@ jobs: make "test-functional-${MATRIX_NAME}" env: DOCKER_REGISTRY: ${{ env.LOCAL_REGISTRY_NAME }}:${{ env.LOCAL_REGISTRY_PORT }} - TEST_TIMEOUT: ${{ env.FUNCTIONALTEST_TIMEOUT }} + # The statestore leg installs/uninstalls/reinstalls Radius, so it needs + # far longer than the standard per-leg timeout. + TEST_TIMEOUT: ${{ matrix.name == 'statestore-noncloud' && '40m' || env.FUNCTIONALTEST_TIMEOUT }} RADIUS_CONTAINER_LOG_PATH: ${{ github.workspace }}/${{ env.RADIUS_CONTAINER_LOG_BASE }} RADIUS_SAMPLES_REPO_ROOT: ${{ github.workspace }}/samples BICEP_RECIPE_REGISTRY: ${{ env.LOCAL_REGISTRY_NAME }}:${{ env.LOCAL_REGISTRY_PORT }} @@ -601,6 +622,9 @@ jobs: RADIUS_TEST_FAST_CLEANUP: true GIT_HTTP_PASSWORD: ${{ env.GIT_HTTP_PASSWORD }} MATRIX_NAME: ${{ matrix.name }} + # Used by the statestore leg's in-test `rad install` to trust the secure + # local registry; ignored by other legs. + RADIUS_REGISTRY_CERT_FILE: ${{ steps.create-local-registry.outputs.temp-cert-dir }}/certs/${{ env.LOCAL_REGISTRY_SERVER }}/client.crt - name: Process Functional Test Results uses: ./.github/actions/process-test-results diff --git a/build/test.mk b/build/test.mk index 0750cb586fc..68646e8dbf4 100644 --- a/build/test.mk +++ b/build/test.mk @@ -148,6 +148,13 @@ test-functional-multicluster-noncloud: ## Runs multi-cluster functional tests th # because of that extra setup. CGO_ENABLED=1 $(GOTEST_TOOL) ./test/functional-portable/multicluster/noncloud/... -timeout ${TEST_TIMEOUT} -v -parallel 1 $(GOTEST_OPTS) +.PHONY: test-functional-statestore-noncloud +test-functional-statestore-noncloud: ## Runs the rad startup/shutdown state-storage lifecycle test + # Destructive: the test installs, uninstalls (--purge), and reinstalls Radius + # to simulate an ephemeral control plane, so it must run on a dedicated cluster + # and never alongside other functional legs. Not part of test-functional-all-noncloud. + CGO_ENABLED=1 $(GOTEST_TOOL) ./test/functional-portable/statestore/noncloud/... -timeout ${TEST_TIMEOUT} -v -parallel 1 $(GOTEST_OPTS) + .PHONY: test-functional-upgrade test-functional-upgrade: test-functional-upgrade-noncloud ## Runs all Upgrade functional tests diff --git a/eng/design-notes/environments/2026-06-repo-radius-deploy-workflow.md b/eng/design-notes/environments/2026-06-repo-radius-deploy-workflow.md new file mode 100644 index 00000000000..d60a6d9bcb1 --- /dev/null +++ b/eng/design-notes/environments/2026-06-repo-radius-deploy-workflow.md @@ -0,0 +1,181 @@ +# Repo Radius — Deploy Workflow (Technical Design) + +- **Authors**: Shruthi Kannan (@sk593), Sylvain Niles (@sylvainsf) +- **Status**: Draft +- **Feature spec**: Repo Radius (Zach Casper) — [PR #12078](https://github.com/radius-project/radius/pull/12078) +- **Issue**: [#12118 Add Repo Radius verify/deploy workflows to the repo](https://github.com/radius-project/radius/issues/12118) +- **Depends on**: [#12106 Multi-cluster deployment v1](https://github.com/radius-project/radius/pull/12106) (merged), [#12214 Repo Radius state storage (`rad startup` / `rad shutdown`)](https://github.com/radius-project/radius/pull/12214) (merged) + +## Scope + +This document covers **Investment 3 of the Repo Radius feature spec: the Repo Radius workflow with standardized inputs and outputs** — the `deploy` workflow that runs Radius on demand inside a GitHub Actions runner — together with **Investment 4: cloud credential integration**. + +The workflow ships as a **unified dispatcher plus two thin provider workflows and shared composite actions**: + +- [`run-rad-commands.yml`](../../../.github/extension/run-rad-commands.yml) — the dispatcher and the only file that is dispatched. It owns the dispatch contract and, via a `detect` job bound to the GitHub Environment, routes to the matching provider workflow (`workflow_call`, `secrets: inherit`). +- [`run-rad-commands-azure.yml`](../../../.github/extension/run-rad-commands-azure.yml) and [`run-rad-commands-aws.yml`](../../../.github/extension/run-rad-commands-aws.yml) — reusable workflows carrying only the cloud-specific steps (OIDC login, cluster connection, token projection, credential registration, and recipe pack). +- [`.github/extension/actions/`](../../../.github/extension/actions/) — composite actions for the provider-agnostic phases (`setup-control-plane`, `restore-state`, `run-rad-commands`, `teardown`), referenced from `radius-project/radius` at a pinned ref. + +A frontend (the Copilot app, the CLI, etc.) writes the dispatcher and both provider workflows into a user's repository under `.github/workflows/` and dispatches `run-rad-commands.yml`. The provider workflows are not dispatched directly, and the composite actions are never copied into the user repo — they stay at [`.github/extension/`](../../../.github/extension/) so the shared logic has a canonical, reviewed home. + +The workflow was originally a single file that branched on which provider variables were present. It was first split into two self-contained per-provider files, which duplicated the ~80% of provider-agnostic steps between them; the current shape keeps the two provider surfaces separate while hoisting the shared steps into composite actions behind one dispatcher, so each duplicated phase is defined once. + +Explicitly **out of scope**: + +- **Cloud-side OIDC / permission provisioning** — creating the AWS IAM role + trust policy or the Entra app registration + federated credential. The workflow *consumes* an environment that is already federated; standing that up is tracked separately. +- **The state-storage mechanism** (`rad startup` / `rad shutdown`, the `radius-state` git orphan branch) — owned by the [state-storage design](../2026-06-repo-radius-state-storage.md). +- **The multi-cluster seam internals** (`global.targetCluster`, the cluster access resolver) — owned by the [multi-cluster design](2026-06-multi-cluster.md). This document only describes how the workflow *drives* that seam. +- **Mid-run cloud-token refresh** beyond the single pre-deploy EKS refresh — a long Azure run may outlive the one-time token exchange; refreshing it mid-run is a deferred fast follow. + +## Background + +Repo Radius runs the Radius control plane on an **ephemeral k3d cluster** inside a GitHub Actions runner. The cluster is created at the start of a run and destroyed at the end; application workloads deploy to the developer's **external** AKS/EKS cluster, not the runner cluster. Each provider-specific workflow composes three independently landed building blocks: + +| Piece | Provides | Owner | +|------------------|-------------------------------------------------------------------------------------------|---------------------------------------------------------------| +| Multi-cluster v1 | `RADIUS_TARGET_KUBECONFIG` seam (chart `global.targetCluster.enabled`) | [#12106](https://github.com/radius-project/radius/pull/12106) | +| State storage | `rad startup` / `rad shutdown` + `database.enabled=true` chart wiring | [#12214](https://github.com/radius-project/radius/pull/12214) | +| Workflow (this) | The orchestration that installs Radius, restores state, runs commands, and persists state | this design | + +An earlier proof of concept validated the end-to-end flow but kept the workflow as a generated string outside Radius, where the contract it depends on had no reviewed home. Bringing the workflow in-tree gives that contract a canonical home and removes any reliance on an external project. + +## The dispatch contract (stable; frontends depend on it) + +The frontend drives Repo Radius through the GitHub API. The contract is the `workflow_dispatch` input set plus the GitHub Environment the run binds to. + +### Inputs + +| Input | Required | Description | +|----------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `environment` | Yes | The GitHub Environment name. Used as the Radius environment name and to bind the job (`environment: ${{ inputs.environment }}`) so its variables and OIDC subject apply. | +| `image` | No | Container image for the application, passed to the default deploy as the `image` parameter. Defaults to the commit SHA. | +| `rad_commands` | No | A single `rad` CLI command string, **or** a JSON-encoded array of command strings run in order, with the `rad` prefix omitted (e.g. `deploy .radius/app.bicep --environment dev` or `["deploy .radius/app.bicep --environment dev", "app graph my-app -o json"]`). Falls back to the `RADIUS_RAD_COMMANDS` variable when the input is empty. Each command is validated against the allowed-command set before any command runs. When both are empty, the workflow runs its default `rad deploy` of `.radius/app.bicep`. | + +`rad_commands` is the dispatch contract: it lets a frontend drive `rad` commands through the documented seam rather than being limited to a single deploy. Commands run in order and the run **stops on the first failure**, then still persists state (below). Because commands run verbatim, the caller owns each command's flags — notably `--environment ` for `deploy` (there is no workspace default set) and any `--parameters` (image, password, etc.) the app expects. `image` is retained as a convenience for the common single-deploy case and so the workflow remains usable without constructing a command string. + +#### Allowed-command set + +Each command's leading verb is validated against an allow-list before any command runs, so commands that do not fit the ephemeral, per-run model — managing or upgrading a control plane, switching workspaces, or changing install state — cannot run, and the contract stays narrow and reviewable. A disallowed command fails fast (overall `outcome: disallowed_command`, exit 2) without deploying anything. The allowed leading verbs are `deploy`, `app`, `resource`, `env`, `recipe`, `group`, `credential`, `resource-type`, `version`, and `bicep`; this set is part of the stable contract and may grow in backward-compatible releases. + +### Outputs + +The run produces a single `rad-commands-result` artifact: a JSON document with a top-level `outcome` (`succeeded`, `command_failed`, or `disallowed_command`) and `exitCode`, plus a `commands` array with one entry per command, in input order, each carrying the command string, its `exitCode`, `outcome`, and captured `output`. The artifact name is stable and known in advance, so the frontend downloads one file to get the full result of the run and reads the run conclusion from the matching exit code. The artifact is written even when a command fails (a trap finalizes it on exit), so a failed or disallowed run still yields a complete result; entries for commands that did not run are absent. On failure, additional control-plane and application logs upload as the `radius-logs` artifact. + +A single combined artifact is preferred over per-command artifacts (`rad-command-0`, `rad-command-1`, …): the name is fixed rather than varying with the number of commands, ordering is explicit via the array, and one download yields the whole run. The trade-off is that the frontend cannot poll a single command's output mid-run; that is acceptable because the GitHub artifacts API does not expose an artifact until its upload completes anyway, so per-command artifacts would not stream either. + +### GitHub Environment variables + +The workflow reads Actions **variables** (`vars`) for cloud configuration; the provider steps run only when the identifying variable (`AZURE_CLIENT_ID` or `AWS_ROLE_ARN`) is non-empty. + +| Provider | Variables | +|----------|--------------------------------------------------------------------------------------------------------------------| +| Azure | `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`, `AZURE_RESOURCE_GROUP`, `AZURE_AKS_CLUSTER_NAME` | +| AWS | `AWS_ROLE_ARN`, `AWS_REGION`, `AWS_ACCOUNT_ID`, `AWS_EKS_CLUSTER_NAME`, `RADIUS_VPC_ID`, `RADIUS_SUBNET_IDS` | +| Common | `KUBERNETES_NAMESPACE` (target namespace, defaults to `default`), `RADIUS_BUILD_REGISTRY` (image-build push target), `RADIUS_RAD_COMMANDS` (fallback for the `rad_commands` input) | + +Secrets used: the built-in `GITHUB_TOKEN` (with `github.actor`) for image-build registry auth, and `RADIUS_DEPLOY_PARAMS` — a JSON object of application parameters expanded into `--parameters name=value` pairs on the default deploy. + +### Auto-trigger + +The `workflow_dispatch` and `workflow_run` triggers live on the **dispatcher** (`run-rad-commands.yml`); the provider workflows are `workflow_call`-only. Besides an explicit dispatch, the dispatcher has a `workflow_run` trigger on the `Radius - Verify Credentials` workflow completing. The `detect` job gates on `github.event.workflow_run.conclusion == 'success'`, so a successful credential check auto-triggers a deploy; on an explicit dispatch it runs unconditionally. + +### Composition (dispatcher, provider workflows, composite actions) + +The provider paths share ~80% of their steps, so the workflow is factored to define each shared phase once while keeping the two cloud surfaces separate and reviewable: + +- The **dispatcher** owns the trigger surface and the routing. A reusable-workflow caller job (`uses:`) cannot itself bind an `environment:`, so it cannot read environment-scoped variables to pick a provider. The dispatcher works around this with a `detect` job that *does* bind `environment: ${{ inputs.environment }}`, reads `AZURE_CLIENT_ID` / `AWS_ROLE_ARN`, and emits `provider`; two downstream `uses:` jobs (`azure`, `aws`) gate on that output and pass inputs through with `secrets: inherit`. +- The **provider workflows** are reusable (`workflow_call`) and hold only the cloud-specific steps. They bind the environment themselves, so environment variables and the OIDC subject apply inside them. +- The **composite actions** (`setup-control-plane`, `restore-state`, `run-rad-commands`, `teardown`) hold the provider-agnostic phases and are referenced from `radius-project/radius` at a pinned ref. `run-rad-commands` and `teardown` are deliberately separate: the provider workflow invokes `run-rad-commands` on success but `teardown` with `if: always()`, so `rad shutdown`, log collection, and the k3d delete still run when an earlier step fails without the deploy itself running after a failed prerequisite. Because composite-action steps cannot read the `vars`/`secrets` contexts or the caller's `env:` block, every value they need (namespace, app file, image, rad-commands, deploy params, registry credentials) is passed explicitly as an action input; the target-cluster kubeconfig path is the exception, shared via `RADIUS_TARGET_KUBECONFIG` exported to `$GITHUB_ENV`. + +### Permissions + +`id-token: write` (OIDC), `contents: write` (so `rad shutdown` can push the `radius-state` branch), and `packages: write` (so container-image recipes can push to GHCR). Declared on the dispatcher and, because reusable workflows run with the caller's grants, inherited by the provider workflows. + +## Workflow stages + +```mermaid +flowchart TD + A[OIDC login
azure/login or configure-aws-credentials] --> B[Build target-cluster kubeconfig
az aks get-credentials / aws eks get-token] + B --> C[Create k3d control plane
+ install rad CLI, oras, Terraform] + C --> D[Create target-kubeconfig secret] + D --> E[rad install kubernetes
database.enabled=true
global.targetCluster.enabled=true
dynamicrp.buildkit.enabled=true] + E --> F[Project cloud OIDC tokens into pods] + F --> G[Refresh external target creds
re-mint EKS token, update secret, restart RP/DE] + G --> H[Configure workspace
rad workspace/group] + H --> I[rad startup
restore PostgreSQL + Terraform state] + I --> J[rad credential register
aws irsa / azure wi] + J --> K[Create environment + recipe pack] + K --> L[Provision registry creds on control plane] + L --> M[Run rad_commands or default deploy
upload rad-commands-result artifact] + M --> N[rad shutdown
back up + push radius-state] + N --> O[Delete k3d cluster] +``` + +### Why the order matters + +- **`rad startup` runs after install but before any command**, so the first deploy plans against restored state rather than an empty backend. +- **`rad shutdown` runs after the commands with `if: always()`**, so a partially-applied Terraform run is not lost. +- **Cloud OIDC tokens are projected into the RP/DE pods right after install**, and the external target credentials are refreshed (EKS token re-mint, Secret rewrite, RP/DE restart) before `rad startup`, so the restored control plane comes up already able to reach the cloud and the target cluster. +- **`rad credential register` runs after `rad startup` and before the environment/recipe-pack deploy** that consumes the credential, so the registered credential lands in the restored control-plane state. + +## The integration contract (owned by Radius) + +### Target cluster — `RADIUS_TARGET_KUBECONFIG` + +The workflow builds a kubeconfig for the external workload cluster on the runner and stores it as the `target-kubeconfig` Secret in `radius-system`. Installing the chart with `--set global.targetCluster.enabled=true` mounts that Secret into `applications-rp`, `dynamic-rp`, and `bicep-de` and sets `RADIUS_TARGET_KUBECONFIG`. Radius then directs recipe execution **and** directly-rendered output resources at that cluster; the Terraform kubernetes provider follows the same kubeconfig through the cluster access resolver. The Terraform **state** backend deliberately stays on the control-plane cluster. The Secret's lifecycle (creation, EKS-token refresh) is the workflow's responsibility, not the chart's. + +### Cloud credentials — UCP credential + projected OIDC token + +Credentials use each provider's native OIDC model, and the cloud identity is registered with Radius's UCP credential model so it is visible to `rad credential show`, rotatable, and usable by both the Bicep and Terraform code paths: + +- **AWS (IRSA)** — `rad credential register aws irsa --iam-role ` records the role ARN. The workflow mints the GitHub Actions OIDC JWT (audience `sts.amazonaws.com`) and projects it into the UCP AWS proxy and the Terraform AWS provider pods at the IRSA token path `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`. +- **Azure (Workload Identity)** — `rad credential register azure wi --client-id --tenant-id` records the identity. The workflow mints the GitHub Actions OIDC JWT (audience `api://AzureADTokenExchange`) and projects it at `/var/run/secrets/azure/tokens/azure-identity-token`, the path the Go `WorkloadIdentityCredential`, the .NET Bicep DE, and the Terraform `azurerm` provider read (with `AZURE_FEDERATED_TOKEN_FILE` set for the armauth fallback). + +Radius stores only the identity selector; the short-lived token is read at runtime from the projected file. This intentionally diverges from feature-spec note SN29 ("no need for `rad credential` commands") because the IRSA/WI token-file model genuinely needs the registered credential to perform the federated token exchange. + +#### Credential lifetime + +- **Azure** — the GitHub Actions OIDC JWT is short-lived, but the Azure SDKs exchange it once for an ~1-hour AAD token. The workflow mints it once and does not refresh it; a run whose Azure work outlives that window may fail. Refreshing mid-run is a deferred fast follow. +- **AWS** — the EKS bearer token used to *reach* the target cluster is ~15 minutes and is used directly on every API call, so the workflow re-mints it and rewrites the `target-kubeconfig` Secret in its refresh step (after install, before `rad startup`), then restarts the RP/DE deployments to pick it up. + +### Cluster credential model — injected kubeconfig (v1) versus cloud-derived (v2) + +This design ships on the **v1 injected-kubeconfig** seam described above: the workflow builds a kubeconfig on the runner and mounts it via `global.targetCluster.enabled`. This is the mechanism that is merged and working today, and it keeps the cluster-access logic in the workflow where it can be iterated quickly. + +The target state is the **v2 cloud-derived** model from the [multi-cluster design](2026-06-multi-cluster.md) and the [external-kubernetes feature](2026-05-external-kubernetes.md): the environment names the cluster on its cloud-provider block (`aws.eksClusterName` / `azure.aksClusterName`) and Radius acquires Kubernetes API access **in-process** from the cloud credential it already holds (EKS `DescribeCluster` + STS presign; AKS `ListClusterUserCredentials`). The Repo Radius feature spec's Investment 1 points here, and it is the better long-term backend for this workflow specifically because it **deletes the most fragile part of v1**: minting the 15-minute EKS bearer token, rewriting the `target-kubeconfig` Secret before every deploy, and restarting the RP/DE pods to pick it up (see [Credential lifetime](#credential-lifetime) above). In-process acquisition refreshes the short-lived cluster credential where it is used, with no Secret remount and no pod restart. + +The action contract hides which model is in use: the dispatch inputs and the result artifact are identical either way, so moving from v1 to v2 is a backend change behind the same stable contract and does not require frontends or committed workflows to change. v2 is gated only on the in-process acquisition being implemented for both providers; until then v1 is the sanctioned interim. + +### State persistence — `rad startup` / `rad shutdown` + +`rad startup` and `rad shutdown` are kind-agnostic CLI commands that back up and restore all durable Radius state (control-plane PostgreSQL + Terraform recipe-state Secrets) to a `radius-state` git orphan branch pushed to the repo's `origin`. They do not manage cluster lifecycle — the workflow owns creating and destroying the ephemeral control plane around them. The mechanism is the plan of record; see the [state-storage design](../2026-06-repo-radius-state-storage.md). + +### Recipe pack and environment + +The workflow provisions a `Radius.Core/recipePacks` resource and a `Radius.Core/environments` resource that references the pack and carries the cloud provider scope. **Azure** downloads the `azure-avm` pack from [resource-types-contrib](https://github.com/radius-project/resource-types-contrib) (`recipepack/azure/aks-recipepack.bicep`), which provisions data/messaging/AI types with Azure Verified Modules and keeps compute types on the shared Kubernetes recipes. **AWS** generates an inline Bicep pack bundling the Kubernetes compute/data recipes (`containers`, `containerImages`, `persistentVolumes`, `routes`, `postgreSqlDatabases`, `secrets`) plus a provider-gated `mySqlDatabases` recipe (AWS RDS). The `containerImages` recipe builds the application image with the in-pod BuildKit (`dynamicrp.buildkit.enabled=true`) and pushes it to the configured registry, authenticated by a Kubernetes Secret created in the app's runtime namespace. + +The `radius-env.bicep` that carries the pack is written to the app file's directory (e.g. `.radius/`) and deployed from there. bicep resolves `bicepconfig.json` nearest the `.bicep` file, so deploying from that directory picks up the repo's own `.radius/bicepconfig.json` — which declares the `radius` extension — rather than a (non-existent) config at the workspace root. The `Radius.Compute/containerImages` type ships with the published `radius` Bicep extension, so the workflow no longer registers resource types or wires a local Bicep extension at deploy time. + +### Control plane startup (Investment 5) + +Because the control plane is created and torn down on every operation, startup time is on the critical path for every user-facing action and is the primary determinant of perceived responsiveness. Two backend decisions follow from that, both owned by this technical design rather than the feature spec: + +- **Package the engine as a composite action, not a Docker action.** A Docker action adds an image-pull on the critical path of every run; a composite action of shell steps adds none. The engine is a sequence of CLI invocations (`k3d`, `rad`, `kubectl`), which composites express directly. +- **Pre-bake the control-plane image.** The dominant startup cost is not action packaging but `k3d cluster create` plus `rad install` pulling the Radius images. The mitigation is a pre-built k3d node image with the Radius control-plane images already loaded, so install becomes a local image reference rather than a registry pull. This is the highest-leverage startup optimization and is tracked as the concrete deliverable for Investment 5. + +## Testing + +The `test/functional-portable/statestore` lifecycle test (its own isolated `statestore-noncloud` CI leg) exercises the state path this workflow protects: install → deploy a Terraform-backed resource → `rad shutdown` → teardown → reinstall → `rad startup` → deploy an update. It drives `rad install` / `rad startup` / `rad shutdown` directly with the build under test, hardened against the install/uninstall flakes seen in the upgrade test (poll for control-plane readiness treating 503 as retryable; poll discovery until `api.ucp.dev/v1alpha3` deregisters before reinstalling). + +## Alternatives considered + +- **Keep the workflow outside Radius.** Rejected: the contract Radius owns (`RADIUS_TARGET_KUBECONFIG`, `rad startup`/`rad shutdown`, the dispatch inputs) would live only in a generated string in a separate project, with no review or stability guarantee for the frontends that depend on it. +- **One self-contained file per provider (no shared actions).** Rejected: it duplicates the ~80% provider-agnostic steps (including the ~200-line command runner) across both files, so every fix has to be made twice and the two can drift. The dispatcher + composite-action shape keeps the provider surfaces separate while defining each shared phase once. +- **A single reusable workflow parameterized by a `provider` input (branch internally with `if:`).** Rejected: it re-introduces the provider `if:` branching the split was meant to remove, interleaving both clouds' steps in one file. Separate provider workflows over shared composite actions keep each cloud path readable on its own. +- **Commit the shared composite actions into the user's repo.** Rejected: it multiplies the generated files and forks the shared logic per repo. Referencing them from `radius-project/radius` at a pinned ref keeps one reviewed copy; the trade-off is that committed templates pin a Radius ref. +- **Deploy-only `image` input (no `rad_commands`).** Rejected as the sole contract: it cannot express `app graph` or multi-command flows the spec requires. `image` is retained as a convenience alongside `rad_commands`. +- **Per-command result artifacts (`rad-command-`) instead of one combined artifact.** Rejected: the artifact set would vary with the number and order of commands, forcing the frontend to discover names dynamically, and the GitHub artifacts API does not expose an artifact until its upload completes, so per-command artifacts would not stream mid-run anyway. One `rad-commands-result` artifact with an ordered `commands` array has a stable known name and is downloaded once. +- **Inject AWS/Azure credentials as plain env vars instead of registering them.** Rejected: the merged code paths read the federated token from a fixed file and resolve the identity through the UCP credential, so the registered credential plus projected token file is the supported model. +- **Refresh the Azure federated token mid-run.** Deferred; the one-time exchange covers ~1 hour, sufficient for current deploys. +- **Stay on the injected-kubeconfig model permanently.** Rejected as the target state: the cloud-derived model removes the EKS token-refresh dance entirely. The injected-kubeconfig seam is retained only as the v1 interim behind the same action contract (see [Cluster credential model](#cluster-credential-model--injected-kubeconfig-v1-versus-cloud-derived-v2)). +- **Package the engine as a Docker action.** Rejected: the image-pull latency lands on the per-run critical path that Investment 5 works to minimize; a composite action avoids it (see [Control plane startup](#control-plane-startup-investment-5)). diff --git a/test/functional-portable/statestore/noncloud/statestore_lifecycle_test.go b/test/functional-portable/statestore/noncloud/statestore_lifecycle_test.go index a738b445ee2..fa2b72adbfc 100644 --- a/test/functional-portable/statestore/noncloud/statestore_lifecycle_test.go +++ b/test/functional-portable/statestore/noncloud/statestore_lifecycle_test.go @@ -23,25 +23,23 @@ limitations under the License. // proves that both the control-plane databases and the Terraform state Secrets survived the // teardown. // -// The test is destructive (it uninstalls and reinstalls Radius on the target cluster) and requires -// a real cluster plus the Terraform recipe module server, so it does not run as part of the normal -// functional suite. It is skipped unless RADIUS_STATE_E2E is set to a truthy value. -// -// Test dependency: rad startup/shutdown do not create clusters or install Radius. This test -// currently drives the cluster install/uninstall itself (installRadius/uninstallRadius below). It -// is expected to depend on the separate Repo Radius workflow code (in flight) that creates the -// ephemeral cluster, installs Radius, and runs the deploy. Once that lands, re-point the helpers -// at the shared workflow code instead of duplicating the install/uninstall steps here. +// The test is destructive: it uninstalls and reinstalls Radius (`--purge`) to simulate the +// ephemeral control plane that Repo Radius runs on. Because of that it runs on its own dedicated +// cluster in CI (the `statestore-noncloud` leg), never alongside other functional tests, and +// drives its own install/uninstall instead of relying on the shared "Install Radius" CI step. package statestore import ( "context" + "fmt" "os" - "strconv" + "os/exec" "testing" "time" "github.com/stretchr/testify/require" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "github.com/radius-project/radius/test" @@ -54,51 +52,191 @@ const ( stateNamespace = "radius-system" secretPrefix = "tfstate-default-" + // resourceGroup is the Radius resource group the test deploys into. It must match the group + // segment of resourceID below so the Terraform state secret name resolves correctly. + resourceGroup = "kind-radius" + + // relativeChartPath points at the in-repo Helm chart so the test installs the build under test. + // The path is relative to this test package directory + // (test/functional-portable/statestore/noncloud), which is four levels below the repo root. + relativeChartPath = "../../../../deploy/Chart" + // redisRecipeTemplate is the Terraform recipe fixture shared with the corerp recipe tests. redisRecipeTemplate = "../../corerp/noncloud/resources/testdata/corerp-resources-terraform-redis.bicep" + + // controlPlaneTimeout is how long to wait for the control plane API to become available after an + // install. It is generous because the UCP aggregated APIService may briefly return 503 while the + // pods roll. (Lesson from the flaky upgrade test, PR #12245.) + controlPlaneTimeout = 5 * time.Minute + controlPlanePollInterval = 5 * time.Second + + // apiServiceDeregistrationTimeout bounds the wait for the Radius aggregated APIService to + // deregister after an uninstall. Reinstalling while `api.ucp.dev/v1alpha3` is still registered + // makes the API server return 503, which flakes the next install. (Lesson from PR #12245.) + apiServiceDeregistrationTimeout = 60 * time.Second + apiServiceDeregistrationInterval = 2 * time.Second + radiusAPIGroupVersion = "api.ucp.dev/v1alpha3" + + // podTerminationTimeout bounds the wait for Radius pods to disappear after an uninstall. + podTerminationTimeout = 2 * time.Minute + podTerminationPoll = 5 * time.Second + radiusPodSelector = "app.kubernetes.io/part-of=radius" ) -// shouldRun reports whether the destructive lifecycle test has been opted into. -func shouldRun(t *testing.T) { +// installRadius installs Radius with the PostgreSQL state backend enabled, using the images and +// chart of the build under test. In CI the registry/tag come from DOCKER_REGISTRY/REL_VERSION and +// the secure local registry's CA is supplied via RADIUS_REGISTRY_CERT_FILE; locally it falls back +// to the public images. +func installRadius(ctx context.Context, t *testing.T, cli *radcli.CLI) { t.Helper() - v, _ := strconv.ParseBool(os.Getenv("RADIUS_STATE_E2E")) - if !v { - t.Skip("set RADIUS_STATE_E2E=1 to run the destructive rad startup/shutdown lifecycle test") + registry, tag := testutil.SetDefault() + + args := []string{ + "install", "kubernetes", + "--chart", relativeChartPath, + "--set", fmt.Sprintf("rp.image=%s/applications-rp,rp.tag=%s", registry, tag), + "--set", fmt.Sprintf("dynamicrp.image=%s/dynamic-rp,dynamicrp.tag=%s", registry, tag), + "--set", fmt.Sprintf("controller.image=%s/controller,controller.tag=%s", registry, tag), + "--set", fmt.Sprintf("ucp.image=%s/ucpd,ucp.tag=%s", registry, tag), + "--set", fmt.Sprintf("bicep.image=%s/bicep,bicep.tag=%s", registry, tag), + "--set", fmt.Sprintf("preupgrade.image=%s/pre-upgrade,preupgrade.tag=%s", registry, tag), + "--set", "database.enabled=true", + } + if deImage := os.Getenv("DE_IMAGE"); deImage != "" { + args = append(args, "--set", fmt.Sprintf("de.image=%s,de.tag=%s", deImage, os.Getenv("DE_TAG"))) + } + if cert := os.Getenv("RADIUS_REGISTRY_CERT_FILE"); cert != "" { + args = append(args, "--set-file", "global.rootCA.cert="+cert) } -} -// installRadius installs Radius with the PostgreSQL state backend enabled. -func installRadius(ctx context.Context, t *testing.T, cli *radcli.CLI) { - t.Helper() - out, err := cli.RunCommand(ctx, []string{"install", "kubernetes", "--set", "database.enabled=true"}) + out, err := cli.RunCommand(ctx, args) require.NoErrorf(t, err, "rad install failed: %s", out) + waitForControlPlane(t, ctx) } // uninstallRadius removes Radius and its state so the next install starts from an empty control -// plane, simulating an ephemeral teardown. +// plane, simulating an ephemeral teardown. It then waits for the Radius pods to terminate and the +// aggregated APIService to deregister so a subsequent install does not race the teardown. func uninstallRadius(ctx context.Context, t *testing.T, cli *radcli.CLI) { t.Helper() - out, err := cli.RunCommand(ctx, []string{"uninstall", "kubernetes", "--purge"}) + out, err := cli.RunCommand(ctx, []string{"uninstall", "kubernetes", "--purge", "--yes"}) require.NoErrorf(t, err, "rad uninstall failed: %s", out) + waitForCleanTeardown(t, ctx) +} + +// waitForControlPlane polls until every Radius control-plane deployment in radius-system reports +// Available, treating transient API errors (including 503 from the aggregated APIService while +// pods roll) as retryable. It is deliberately workspace-independent: it talks to Kubernetes +// directly, so it can run immediately after install and before any rad workspace exists. +func waitForControlPlane(t *testing.T, ctx context.Context) { + t.Helper() + k8s := test.NewTestOptions(t).K8sClient + require.Eventually(t, func() bool { + deployments, err := k8s.AppsV1().Deployments(stateNamespace).List(ctx, metav1.ListOptions{LabelSelector: radiusPodSelector}) + if err != nil { + t.Logf("waiting to list control-plane deployments: %v", err) + return false + } + if len(deployments.Items) == 0 { + return false + } + for _, d := range deployments.Items { + available := false + for _, c := range d.Status.Conditions { + if c.Type == appsv1.DeploymentAvailable && c.Status == corev1.ConditionTrue { + available = true + break + } + } + if !available { + t.Logf("waiting for deployment %s to become Available...", d.Name) + return false + } + } + return true + }, controlPlaneTimeout, controlPlanePollInterval, "control plane did not become available within timeout") +} + +// waitForCleanTeardown waits for Radius pods to terminate and the aggregated APIService to +// deregister after an uninstall, so the next install does not race a half-torn-down control plane. +func waitForCleanTeardown(t *testing.T, ctx context.Context) { + t.Helper() + k8s := test.NewTestOptions(t).K8sClient + + require.Eventually(t, func() bool { + pods, err := k8s.CoreV1().Pods(stateNamespace).List(ctx, metav1.ListOptions{LabelSelector: radiusPodSelector}) + if err != nil { + t.Logf("waiting to list pods: %v", err) + return false + } + if len(pods.Items) == 0 { + return true + } + t.Logf("waiting for %d Radius pod(s) to terminate...", len(pods.Items)) + return false + }, podTerminationTimeout, podTerminationPoll, "Radius pods did not terminate within timeout") + + // A 503 from the aggregated APIService means it is still registered but its backend is gone; + // poll discovery until the Radius API group is no longer served. + require.Eventually(t, func() bool { + _, resources, err := k8s.Discovery().ServerGroupsAndResources() + if err != nil { + // Partial results are expected mid-deregistration; inspect what we got. + t.Logf("discovery returned partial results (expected during deregistration): %v", err) + } + for _, rl := range resources { + if rl != nil && rl.GroupVersion == radiusAPIGroupVersion { + t.Log("Radius aggregated APIService still registered, waiting...") + return false + } + } + return true + }, apiServiceDeregistrationTimeout, apiServiceDeregistrationInterval, "aggregated APIService did not deregister within timeout") +} + +// newStateRepo creates a throwaway git repository with no remote for `rad shutdown` / `rad startup` +// to persist state into. gitstate resolves the repo from the rad process's working directory and +// pushes to `origin` when a remote exists; running from this remote-less repo exercises the +// design's supported local/test case (commit-only, no push) instead of trying to push to the +// checkout's GitHub origin, which has no credentials in CI. +func newStateRepo(t *testing.T) string { + t.Helper() + dir := t.TempDir() + for _, args := range [][]string{ + {"init"}, + {"config", "user.email", "statestore-test@radapp.io"}, + {"config", "user.name", "statestore-test"}, + {"commit", "--allow-empty", "-m", "init"}, + } { + cmd := exec.Command("git", args...) + cmd.Dir = dir + out, err := cmd.CombinedOutput() + require.NoErrorf(t, err, "git %v failed: %s", args, out) + } + return dir } // Test_StateStore_ShutdownStartup_TerraformCrossDeploy exercises every state path: // install, deploy a Terraform resource, shut down (backup), tear down, start up (restore), then // deploy an update to the same resource. func Test_StateStore_ShutdownStartup_TerraformCrossDeploy(t *testing.T) { - shouldRun(t) - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute) defer cancel() cli := radcli.NewCLI(t, "") + // shutdown/startup persist state into a remote-less git repo so the backup commits locally + // without trying to push to the checkout's GitHub origin (no credentials in CI). Both commands + // must use the same repo so the state committed by shutdown survives into startup. + stateCLI := radcli.NewCLI(t, "") + stateCLI.WorkingDirectory = newStateRepo(t) + appName := "statestore-tf-redis-app" envName := "statestore-tf-redis-env" resourceName := "statestore-tf-redis" redisCacheName := "statestore-redis" - resourceID := "/planes/radius/local/resourcegroups/kind-radius/providers/Applications.Core/extenders/" + resourceName + resourceID := "/planes/radius/local/resourcegroups/" + resourceGroup + "/providers/Applications.Core/extenders/" + resourceName secretSuffix, err := corerp.GetSecretSuffix(resourceID, envName, appName) require.NoError(t, err) secretName := secretPrefix + secretSuffix @@ -122,17 +260,26 @@ func Test_StateStore_ShutdownStartup_TerraformCrossDeploy(t *testing.T) { return getErr == nil } - // 1. Fresh install with PostgreSQL state backend. + // 1. Fresh install with the PostgreSQL state backend. installRadius(ctx, t, cli) t.Cleanup(func() { uninstallRadius(context.Background(), t, cli) }) + // Create the workspace and resource group the test deploys into (the shared CI "Install Radius" + // step is skipped for this leg, so the test owns this setup). + out, err := cli.RunCommand(ctx, []string{"workspace", "create", "kubernetes", "--force"}) + require.NoErrorf(t, err, "rad workspace create failed: %s", out) + out, err = cli.RunCommand(ctx, []string{"group", "create", resourceGroup}) + require.NoErrorf(t, err, "rad group create failed: %s", out) + out, err = cli.RunCommand(ctx, []string{"group", "switch", resourceGroup}) + require.NoErrorf(t, err, "rad group switch failed: %s", out) + // 2. Deploy the Terraform-backed resource. This creates control-plane state and a Terraform // state Secret. deploy() require.True(t, secretExists(), "Terraform state secret should exist after the first deploy") // 3. Back up all durable state. - out, err := cli.RunCommand(ctx, []string{"shutdown"}) + out, err = stateCLI.RunCommand(ctx, []string{"shutdown"}) require.NoErrorf(t, err, "rad shutdown failed: %s", out) // 4. Tear the control plane down completely (ephemeral teardown). @@ -143,7 +290,7 @@ func Test_StateStore_ShutdownStartup_TerraformCrossDeploy(t *testing.T) { require.False(t, secretExists(), "Terraform state secret must be gone after reinstall (teardown was real)") // 6. Restore the saved state. - out, err = cli.RunCommand(ctx, []string{"startup"}) + out, err = stateCLI.RunCommand(ctx, []string{"startup"}) require.NoErrorf(t, err, "rad startup failed: %s", out) // 7. Both stores must be restored: the Terraform state Secret is back, and the control-plane