From 1bf2c9caf5cf87a5c1e2bb5fac15fe6ca1d98fed Mon Sep 17 00:00:00 2001 From: Saba Date: Mon, 29 Jun 2026 17:14:41 -0700 Subject: [PATCH 1/6] [DE-4343] Add VKS install page under Install on clusters Add a "VMware Kubernetes Service (VKS)" page to the Calico Enterprise install-on-clusters section and wire it into the sidebar after TKG. Covers installing Calico Enterprise on VKS: Part 1 Supervisor Service setup (Carvel package build/publish), Part 2 guest-cluster deployment with optional Windows node support, and Parts 3-4 air-gapped bundle/image relocation. Ported from tigera/vks CE-install.md: blockquote callouts converted to Docusaurus admonitions, an internal vSphere lab URL removed, the private- registry cross-link made site-relative, and $[prodname] used in body prose. Co-Authored-By: Claude Opus 4.8 --- .../install-on-clusters/vks.mdx | 324 ++++++++++++++++++ sidebars-calico-enterprise.js | 1 + 2 files changed, 325 insertions(+) create mode 100644 calico-enterprise/getting-started/install-on-clusters/vks.mdx diff --git a/calico-enterprise/getting-started/install-on-clusters/vks.mdx b/calico-enterprise/getting-started/install-on-clusters/vks.mdx new file mode 100644 index 0000000000..985c8ed498 --- /dev/null +++ b/calico-enterprise/getting-started/install-on-clusters/vks.mdx @@ -0,0 +1,324 @@ +--- +description: Install Calico Enterprise as a Supervisor Service on VMware Kubernetes Service (VKS) and deploy it to guest clusters, with optional Windows node support. +--- + +# VMware Kubernetes Service (VKS) + +This guide walks through setting up $[prodname] as a Supervisor Service on VKS, and deploying it to guest clusters with optional Windows node support. + +## Prerequisites + +- **VM Images**: Ubuntu (and optionally Windows) OVA images registered for a supported TKR version, with the TKR in a healthy state. +- **vSphere Resources**: Sufficient capacity to provision guest clusters with at minimum: + - 1 control plane node + - 3 Linux worker nodes + - 1 Windows worker node (optional, if Windows workloads are required) +- **OCI Registry**: An OCI-compliant registry configured in vSphere for storing package manifests and container images. + +## Networking Overview + +| Configuration | Dataplane | Encapsulation | BGP | +|---|---|---|---| +| Linux only (default) | IPTables | IPIP | Enabled | +| Linux + Windows | IPTables | VXLAN | Disabled | + +:::note + +When Windows nodes are enabled, the encapsulation is automatically set to VXLAN and BGP is disabled. This is handled internally when `calicoWindowsDataplane` is set to `true` in the AddonConfig. + +::: + +--- + +## Part 1: Supervisor Service Setup + +### Prerequisites + +- `helm`, `kbld`, `ytt`, and `imgpkg` binaries available for the host OS architecture in a `./bin` directory. + +### Configuration + +| Variable | Description | +|---|---| +| `CALICO_ENTERPRISE_VERSION` | $[prodname] version to package | +| `CALICO_ENTERPRISE_SVS` | (Optional) Custom name for the Supervisor Service | +| `ARTIFACTORY_WRITE` | OCI registry for pushing manifests and images | +| `ARTIFACTORY_READ` | OCI registry from which manifests and images are pulled | +| `REPO_VERSION` | Addon repository version | +| `SERVICE_VERSION` | Addon service version | + +### Build and Publish + +1. Remove previously generated artifacts: + ```bash + make clean + ``` + +2. Build the Carvel package and push to the OCI registry. This creates the Addon package and uploads the artifacts to the OCI registry, also creates the Package and PackageMetadata resources (`calico-svs.yaml`) to be uploaded to the vSphere Supervisor UI: + ```bash + make all + ``` + +3. In the vSphere Supervisor UI: + - Create a new Supervisor Service (or add a new version to an existing one). + - Upload the generated `calico-svs.yaml` and complete the UI workflow. + - Navigate to the service management page and set `registryName` to match the registry name configured in the UI. + +### What `make all` Does + +- Packages the Carvel manifests (AddonConfigDefinition, Package, AddonInstall, etc.). +- Pulls and bundles the Tigera Operator Helm chart. +- Resolves and bundles all required container images (`tigera-operator`, `tigera-prometheus-operator`, etc.). +- Renders the `crd.projectcalico.org.v1` chart and bundles the $[prodname] CRDs. As of $[prodname] v3.23.0-2.0 the CRDs ship as a separate chart (no longer inside `tigera-operator`); they are applied alongside the operator with the `kapp.k14s.io/disable-original` annotation so the large CRDs (e.g. `installations.operator.tigera.io`) clear kapp's annotation-size limit — the kapp equivalent of the upstream server-side apply step. +- Includes the `tigera-operator` namespace and `tigera-elasticsearch` StorageClass in the bundle. + +--- + +## Part 2: Guest Cluster Deployment + +### Prerequisites + +- **Tigera pull secret** for accessing the Tigera container registry (contact the Tigera team). +- **Cluster ResourceSet for tigera pull secret**: The tigera-pull-secret applied as CAPI Cluster ResourceSet in the Supervisor Cluster for all namespaces where guest clusters with $[prodname] will be provisioned. The `./sample-configs/cluster-resourceset-pull-secret.yaml` can be used for this, by replacing the `.dockerconfigjson` value with the value of the private/tigera registry. +- **Tigera license file** (contact the Tigera team). +- **Dynamic PV provisioning** via a StorageClass, required for Elasticsearch storage. + +### Step 1: Apply the AddonConfig (from Supervisor) + +SSH into the Supervisor machine and apply the AddonConfig resource. A sample configuration is provided in `./sample-configs/addon_config.yaml`. + +:::info + +Update the following fields before applying: +- `metadata.name` and `metadata.namespace` +- `spec.addonConfigDefinitionRef` (name and namespace) +- `spec.clusterName` +- `spec.values.installation.calicoWindowsDataplane.enabled` (set to `true` for Windows support, `false` otherwise) +- `spec.values.installation.registry` to use $[prodname] images from a private registry (for air-gapped installs). This should be the same registry where $[prodname] images are pushed to using [Part 4](#part-4-for-airgapped-env-relocate-calico-enterprise-private-images-to-your-or-customers-private-registry) of this doc. Optionally `spec.values.installation.imagePath` can be set if the $[prodname] images were pushed to a specific folder within the private registry. + +::: + +:::note + +All $[prodname] features (`logCollector`, `logStorage`, `manager`, `monitor`, `policyRecommendation`) are optional. Only `apiServer` is required. Set any unwanted features to `false`. + +::: + +```bash +kubectl apply -f ./sample-configs/addon_config.yaml +``` + +### Step 2: Apply the Cluster Resource (from Supervisor) + +While still on the Supervisor machine, apply the Cluster resource. A sample is provided in `./sample-configs/cluster.yaml`. + +Update the following fields before applying: +- `metadata.name` and `metadata.namespace` +- `spec.topology.variables[].value.cniRef` (name and namespace of the $[prodname] addon) +- Add the label `tigera.io/pull-secret: required` to `metadata.labels` so that tigera-pull-secret will be created for the cluster. +- Node pool configuration (VM class, storage class, replicas) as needed for your environment + +```bash +kubectl apply -f ./sample-configs/cluster.yaml +``` + +### Step 3: Retrieve the Guest Cluster Kubeconfig + +```bash +kubectl -n get secret -o jsonpath='{.data.value}' | base64 --decode > kubeconfig +``` + +:::note + +From this point onwards, Supervisor SSH access is no longer required. All remaining steps can be performed from any machine with access to the guest cluster's API server. + +::: + +### Step 4: Create the Tigera Pull Secret + +```bash +kubectl create secret generic tigera-pull-secret \ + --type=kubernetes.io/dockerconfigjson -n tigera-operator \ + --from-file=.dockerconfigjson= +``` + +### Step 5: [Windows only] Apply IPAM Configuration + +Apply the strict affinity IPAM configuration, after Calico `apiserver` is `Available`. + +:::tip + +Status can be checked using `kubectl get tigerastatus`. + +::: + +```bash +kubectl apply -f ./sample-configs/ipam_configuration.yaml +``` + +### Step 6: [Windows only] Apply the kubernetes-services-endpoint Configmap + +```bash +kubectl apply -f - << EOF +kind: ConfigMap +apiVersion: v1 +metadata: + name: kubernetes-services-endpoint + namespace: tigera-operator +data: + KUBERNETES_SERVICE_HOST: "${APISERVER_ADDR}" + KUBERNETES_SERVICE_PORT: "${APISERVER_PORT}" +EOF +``` + +:::tip + +The `APISERVER_ADDR` and `APISERVER_PORT` can be retrieved via `kubectl get endpoints kubernetes -o wide`. + +::: + +### Step 7: [Windows only] Apply the kube-proxy on Windows nodes + +Apply the Windows kube-proxy: + +```bash +curl -L https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml | sed "s/KUBE_PROXY_VERSION//g" | kubectl apply -f - +``` + +Replace `` with the kube-proxy version, e.g. `v1.35.2`. + +:::tip + +The Kubernetes version can be retrieved with `kubectl version -o json | jq -r '.serverVersion.gitVersion'`. + +::: + +:::note + +If the version returned is of the form `v1.35.2+vmware.1`, strip the `+vmware.1` suffix unless a VMware-provided kube-proxy image exists in the `sigwindowstools/kube-proxy` repository. + +::: + +### Step 8: Wait for Core Components + +Monitor the Tigera status until `calico` and `apiserver` report as `Available`: + +```bash +kubectl get tigerastatus +``` + +At this point, all nodes should be in `Ready` state. + +### Step 9: Apply the Tigera License + +```bash +kubectl create -f tigera-license.yaml +``` + +### Step 10: Verify the Installation + +Wait for all remaining components to become available: + +```bash +kubectl get tigerastatus +kubectl get pods -A | grep tigera\|calico +``` + +Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. + +--- + +## Part 3 [FOR AIRGAPPED ENV]: Relocating Bundles to a Private Registry for Air-gapped install + +If the published bundles need to be moved into a customer's private OCI registry (for example, an air-gapped install or a customer who cannot pull from your registry), follow these steps. `make all` produces three bundles and one customer-facing YAML; relocation moves the bundles and updates one reference in the YAML. + +### Prerequisites + +- `imgpkg` and `yq` available locally. +- Network access to the source registry (or a tarball produced from it) and to the customer registry. +- Credentials to push to the customer registry (`imgpkg` will prompt or use the local docker config). + +### Bundles to relocate + +The three bundles correspond to the three Make targets and use the registry paths configured in the Makefile: + +| Bundle | Source path | Default tag | +|---|---|---| +| Package bundle ($[prodname] chart + container images) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}` | `v${CALICO_ENTERPRISE_VERSION}` | +| Repository bundle (Package + PackageMetadata for the package bundle) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}-repo` | `v${REPO_VERSION}` | +| Service bundle (AddonInstall, AddonRepository, AddonRepositoryInstall) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}-svs` | `v${SERVICE_VERSION}` | + +Cross-bundle references (service → repository → package → container images) are stored as digests in each bundle's `.imgpkg/images.yml` lock file. `imgpkg copy` rewrites those lock files automatically when relocating, so no manifests inside the bundles need manual editing. + +### Step 1: Copy the bundles to a tarball (source side) + +Run on a host that has access to the source registry: + +```bash +imgpkg copy --bundle /calico-enterprise:v3.22.2 --to-tar calico-package.tar +imgpkg copy --bundle /calico-enterprise-repo:v3.6.1 --to-tar calico-repo.tar +imgpkg copy --bundle /calico-enterprise-svs:v3.6.1 --to-tar calico-svs.tar +``` + +Transfer the three tarballs along with `calico-svs.yaml` to the destination environment. + +:::tip + +Direct registry-to-registry relocation is also supported. Replace `--to-tar .tar` with `--to-repo /` and skip Step 2. + +::: + +### Step 2: Push the bundles into the customer registry (destination side) + +```bash +imgpkg copy --tar calico-package.tar --to-repo /calico-enterprise +imgpkg copy --tar calico-repo.tar --to-repo /calico-enterprise-repo +imgpkg copy --tar calico-svs.tar --to-repo /calico-enterprise-svs +``` + +After each `imgpkg copy --to-repo` finishes, the relocated bundle's `images.yml` references the customer registry by digest. Container images and cross-bundle references resolve to the customer registry automatically at deploy time. + +### Step 3: Patch `calico-svs.yaml` + +The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-svs.yaml`. Update it to the customer registry: + +```bash +yq eval ' + (select(.kind == "Package") | .spec.template.spec.fetch[0].imgpkgBundle.image) + = "/calico-enterprise-svs:v3.6.1" +' -i calico-svs.yaml +``` + +:::tip + +**Content-pin to digest (optional):** `imgpkg copy --to-repo` prints the destination digest of the relocated bundle on its last line. Substitute `:v3.6.1` with `@sha256:` in the `yq` command above to make the reference immutable. + +::: + +### Step 4: Upload `calico-svs.yaml` to the Supervisor + +Follow the original [Part 1, Build and Publish](#build-and-publish) workflow with the patched `calico-svs.yaml`. From here, [Part 2](#part-2-guest-cluster-deployment) proceeds unchanged. + +### What does *not* work + +- `docker pull` / `docker push` on individual images. Re-tagging can change manifest digests, and the bundles' `images.yml` lock files are not rewritten — kapp-controller fetches will fail with `manifest unknown`. +- Editing `addon_repository.yaml` or `package.yaml` directly. Those files live inside the relocated bundles; their image references are overridden at fetch time by the lock files that `imgpkg copy` already rewrote. Manual edits are unnecessary and won't take effect anyway. +- Skipping any of the three `imgpkg copy` runs. All three bundles must be present in the customer registry — the service bundle references the repository bundle, which references the package bundle. + +--- + +## Part 4 [FOR AIRGAPPED ENV]: Relocate $[prodname] private images to your (or Customer's) private registry + +Follow the steps in [Push $[prodname] images to your private registry](/calico-enterprise/latest/getting-started/install-on-clusters/private-registry/private-registry-regular#push-calico-enterprise-images-to-your-private-registry) to pull $[prodname] images from Tigera's private image registry to your private registry. + +:::note + +Only steps 1, 2, 3, 4 — i.e. `docker pull`, `docker tag` and `docker push` for Linux only; and the `crane cp` for Windows images — need to be performed. + +::: + +:::caution + +Do not push $[prodname] private images to a public registry. + +::: diff --git a/sidebars-calico-enterprise.js b/sidebars-calico-enterprise.js index dd20b28151..c091945751 100644 --- a/sidebars-calico-enterprise.js +++ b/sidebars-calico-enterprise.js @@ -54,6 +54,7 @@ module.exports = { 'getting-started/install-on-clusters/rke2', 'getting-started/install-on-clusters/rancher-ui', 'getting-started/install-on-clusters/tkg', + 'getting-started/install-on-clusters/vks', 'getting-started/install-on-clusters/charmed-k8s', { type: 'category', From e5c356e05c7b65a97e343fdca400a6ea9f2d0d5d Mon Sep 17 00:00:00 2001 From: Saba Date: Mon, 29 Jun 2026 17:26:15 -0700 Subject: [PATCH 2/6] [DE-4343] Add VKS install page to 3.23-2 versioned docs Copy the VKS install page into calico-enterprise_versioned_docs/version-3.23-2 and add it to that version's sidebar after TKG. Also switch the private-registry cross-link from an absolute /latest path to a relative .mdx link so it resolves to the correct version on both the latest and versioned pages. Co-Authored-By: Claude Opus 4.8 --- .../install-on-clusters/vks.mdx | 2 +- .../install-on-clusters/vks.mdx | 324 ++++++++++++++++++ .../version-3.23-2-sidebars.json | 1 + 3 files changed, 326 insertions(+), 1 deletion(-) create mode 100644 calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx diff --git a/calico-enterprise/getting-started/install-on-clusters/vks.mdx b/calico-enterprise/getting-started/install-on-clusters/vks.mdx index 985c8ed498..463231a1e8 100644 --- a/calico-enterprise/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise/getting-started/install-on-clusters/vks.mdx @@ -309,7 +309,7 @@ Follow the original [Part 1, Build and Publish](#build-and-publish) workflow wit ## Part 4 [FOR AIRGAPPED ENV]: Relocate $[prodname] private images to your (or Customer's) private registry -Follow the steps in [Push $[prodname] images to your private registry](/calico-enterprise/latest/getting-started/install-on-clusters/private-registry/private-registry-regular#push-calico-enterprise-images-to-your-private-registry) to pull $[prodname] images from Tigera's private image registry to your private registry. +Follow the steps in [Push $[prodname] images to your private registry](./private-registry/private-registry-regular.mdx#push-calico-enterprise-images-to-your-private-registry) to pull $[prodname] images from Tigera's private image registry to your private registry. :::note diff --git a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx new file mode 100644 index 0000000000..463231a1e8 --- /dev/null +++ b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx @@ -0,0 +1,324 @@ +--- +description: Install Calico Enterprise as a Supervisor Service on VMware Kubernetes Service (VKS) and deploy it to guest clusters, with optional Windows node support. +--- + +# VMware Kubernetes Service (VKS) + +This guide walks through setting up $[prodname] as a Supervisor Service on VKS, and deploying it to guest clusters with optional Windows node support. + +## Prerequisites + +- **VM Images**: Ubuntu (and optionally Windows) OVA images registered for a supported TKR version, with the TKR in a healthy state. +- **vSphere Resources**: Sufficient capacity to provision guest clusters with at minimum: + - 1 control plane node + - 3 Linux worker nodes + - 1 Windows worker node (optional, if Windows workloads are required) +- **OCI Registry**: An OCI-compliant registry configured in vSphere for storing package manifests and container images. + +## Networking Overview + +| Configuration | Dataplane | Encapsulation | BGP | +|---|---|---|---| +| Linux only (default) | IPTables | IPIP | Enabled | +| Linux + Windows | IPTables | VXLAN | Disabled | + +:::note + +When Windows nodes are enabled, the encapsulation is automatically set to VXLAN and BGP is disabled. This is handled internally when `calicoWindowsDataplane` is set to `true` in the AddonConfig. + +::: + +--- + +## Part 1: Supervisor Service Setup + +### Prerequisites + +- `helm`, `kbld`, `ytt`, and `imgpkg` binaries available for the host OS architecture in a `./bin` directory. + +### Configuration + +| Variable | Description | +|---|---| +| `CALICO_ENTERPRISE_VERSION` | $[prodname] version to package | +| `CALICO_ENTERPRISE_SVS` | (Optional) Custom name for the Supervisor Service | +| `ARTIFACTORY_WRITE` | OCI registry for pushing manifests and images | +| `ARTIFACTORY_READ` | OCI registry from which manifests and images are pulled | +| `REPO_VERSION` | Addon repository version | +| `SERVICE_VERSION` | Addon service version | + +### Build and Publish + +1. Remove previously generated artifacts: + ```bash + make clean + ``` + +2. Build the Carvel package and push to the OCI registry. This creates the Addon package and uploads the artifacts to the OCI registry, also creates the Package and PackageMetadata resources (`calico-svs.yaml`) to be uploaded to the vSphere Supervisor UI: + ```bash + make all + ``` + +3. In the vSphere Supervisor UI: + - Create a new Supervisor Service (or add a new version to an existing one). + - Upload the generated `calico-svs.yaml` and complete the UI workflow. + - Navigate to the service management page and set `registryName` to match the registry name configured in the UI. + +### What `make all` Does + +- Packages the Carvel manifests (AddonConfigDefinition, Package, AddonInstall, etc.). +- Pulls and bundles the Tigera Operator Helm chart. +- Resolves and bundles all required container images (`tigera-operator`, `tigera-prometheus-operator`, etc.). +- Renders the `crd.projectcalico.org.v1` chart and bundles the $[prodname] CRDs. As of $[prodname] v3.23.0-2.0 the CRDs ship as a separate chart (no longer inside `tigera-operator`); they are applied alongside the operator with the `kapp.k14s.io/disable-original` annotation so the large CRDs (e.g. `installations.operator.tigera.io`) clear kapp's annotation-size limit — the kapp equivalent of the upstream server-side apply step. +- Includes the `tigera-operator` namespace and `tigera-elasticsearch` StorageClass in the bundle. + +--- + +## Part 2: Guest Cluster Deployment + +### Prerequisites + +- **Tigera pull secret** for accessing the Tigera container registry (contact the Tigera team). +- **Cluster ResourceSet for tigera pull secret**: The tigera-pull-secret applied as CAPI Cluster ResourceSet in the Supervisor Cluster for all namespaces where guest clusters with $[prodname] will be provisioned. The `./sample-configs/cluster-resourceset-pull-secret.yaml` can be used for this, by replacing the `.dockerconfigjson` value with the value of the private/tigera registry. +- **Tigera license file** (contact the Tigera team). +- **Dynamic PV provisioning** via a StorageClass, required for Elasticsearch storage. + +### Step 1: Apply the AddonConfig (from Supervisor) + +SSH into the Supervisor machine and apply the AddonConfig resource. A sample configuration is provided in `./sample-configs/addon_config.yaml`. + +:::info + +Update the following fields before applying: +- `metadata.name` and `metadata.namespace` +- `spec.addonConfigDefinitionRef` (name and namespace) +- `spec.clusterName` +- `spec.values.installation.calicoWindowsDataplane.enabled` (set to `true` for Windows support, `false` otherwise) +- `spec.values.installation.registry` to use $[prodname] images from a private registry (for air-gapped installs). This should be the same registry where $[prodname] images are pushed to using [Part 4](#part-4-for-airgapped-env-relocate-calico-enterprise-private-images-to-your-or-customers-private-registry) of this doc. Optionally `spec.values.installation.imagePath` can be set if the $[prodname] images were pushed to a specific folder within the private registry. + +::: + +:::note + +All $[prodname] features (`logCollector`, `logStorage`, `manager`, `monitor`, `policyRecommendation`) are optional. Only `apiServer` is required. Set any unwanted features to `false`. + +::: + +```bash +kubectl apply -f ./sample-configs/addon_config.yaml +``` + +### Step 2: Apply the Cluster Resource (from Supervisor) + +While still on the Supervisor machine, apply the Cluster resource. A sample is provided in `./sample-configs/cluster.yaml`. + +Update the following fields before applying: +- `metadata.name` and `metadata.namespace` +- `spec.topology.variables[].value.cniRef` (name and namespace of the $[prodname] addon) +- Add the label `tigera.io/pull-secret: required` to `metadata.labels` so that tigera-pull-secret will be created for the cluster. +- Node pool configuration (VM class, storage class, replicas) as needed for your environment + +```bash +kubectl apply -f ./sample-configs/cluster.yaml +``` + +### Step 3: Retrieve the Guest Cluster Kubeconfig + +```bash +kubectl -n get secret -o jsonpath='{.data.value}' | base64 --decode > kubeconfig +``` + +:::note + +From this point onwards, Supervisor SSH access is no longer required. All remaining steps can be performed from any machine with access to the guest cluster's API server. + +::: + +### Step 4: Create the Tigera Pull Secret + +```bash +kubectl create secret generic tigera-pull-secret \ + --type=kubernetes.io/dockerconfigjson -n tigera-operator \ + --from-file=.dockerconfigjson= +``` + +### Step 5: [Windows only] Apply IPAM Configuration + +Apply the strict affinity IPAM configuration, after Calico `apiserver` is `Available`. + +:::tip + +Status can be checked using `kubectl get tigerastatus`. + +::: + +```bash +kubectl apply -f ./sample-configs/ipam_configuration.yaml +``` + +### Step 6: [Windows only] Apply the kubernetes-services-endpoint Configmap + +```bash +kubectl apply -f - << EOF +kind: ConfigMap +apiVersion: v1 +metadata: + name: kubernetes-services-endpoint + namespace: tigera-operator +data: + KUBERNETES_SERVICE_HOST: "${APISERVER_ADDR}" + KUBERNETES_SERVICE_PORT: "${APISERVER_PORT}" +EOF +``` + +:::tip + +The `APISERVER_ADDR` and `APISERVER_PORT` can be retrieved via `kubectl get endpoints kubernetes -o wide`. + +::: + +### Step 7: [Windows only] Apply the kube-proxy on Windows nodes + +Apply the Windows kube-proxy: + +```bash +curl -L https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml | sed "s/KUBE_PROXY_VERSION//g" | kubectl apply -f - +``` + +Replace `` with the kube-proxy version, e.g. `v1.35.2`. + +:::tip + +The Kubernetes version can be retrieved with `kubectl version -o json | jq -r '.serverVersion.gitVersion'`. + +::: + +:::note + +If the version returned is of the form `v1.35.2+vmware.1`, strip the `+vmware.1` suffix unless a VMware-provided kube-proxy image exists in the `sigwindowstools/kube-proxy` repository. + +::: + +### Step 8: Wait for Core Components + +Monitor the Tigera status until `calico` and `apiserver` report as `Available`: + +```bash +kubectl get tigerastatus +``` + +At this point, all nodes should be in `Ready` state. + +### Step 9: Apply the Tigera License + +```bash +kubectl create -f tigera-license.yaml +``` + +### Step 10: Verify the Installation + +Wait for all remaining components to become available: + +```bash +kubectl get tigerastatus +kubectl get pods -A | grep tigera\|calico +``` + +Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. + +--- + +## Part 3 [FOR AIRGAPPED ENV]: Relocating Bundles to a Private Registry for Air-gapped install + +If the published bundles need to be moved into a customer's private OCI registry (for example, an air-gapped install or a customer who cannot pull from your registry), follow these steps. `make all` produces three bundles and one customer-facing YAML; relocation moves the bundles and updates one reference in the YAML. + +### Prerequisites + +- `imgpkg` and `yq` available locally. +- Network access to the source registry (or a tarball produced from it) and to the customer registry. +- Credentials to push to the customer registry (`imgpkg` will prompt or use the local docker config). + +### Bundles to relocate + +The three bundles correspond to the three Make targets and use the registry paths configured in the Makefile: + +| Bundle | Source path | Default tag | +|---|---|---| +| Package bundle ($[prodname] chart + container images) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}` | `v${CALICO_ENTERPRISE_VERSION}` | +| Repository bundle (Package + PackageMetadata for the package bundle) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}-repo` | `v${REPO_VERSION}` | +| Service bundle (AddonInstall, AddonRepository, AddonRepositoryInstall) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}-svs` | `v${SERVICE_VERSION}` | + +Cross-bundle references (service → repository → package → container images) are stored as digests in each bundle's `.imgpkg/images.yml` lock file. `imgpkg copy` rewrites those lock files automatically when relocating, so no manifests inside the bundles need manual editing. + +### Step 1: Copy the bundles to a tarball (source side) + +Run on a host that has access to the source registry: + +```bash +imgpkg copy --bundle /calico-enterprise:v3.22.2 --to-tar calico-package.tar +imgpkg copy --bundle /calico-enterprise-repo:v3.6.1 --to-tar calico-repo.tar +imgpkg copy --bundle /calico-enterprise-svs:v3.6.1 --to-tar calico-svs.tar +``` + +Transfer the three tarballs along with `calico-svs.yaml` to the destination environment. + +:::tip + +Direct registry-to-registry relocation is also supported. Replace `--to-tar .tar` with `--to-repo /` and skip Step 2. + +::: + +### Step 2: Push the bundles into the customer registry (destination side) + +```bash +imgpkg copy --tar calico-package.tar --to-repo /calico-enterprise +imgpkg copy --tar calico-repo.tar --to-repo /calico-enterprise-repo +imgpkg copy --tar calico-svs.tar --to-repo /calico-enterprise-svs +``` + +After each `imgpkg copy --to-repo` finishes, the relocated bundle's `images.yml` references the customer registry by digest. Container images and cross-bundle references resolve to the customer registry automatically at deploy time. + +### Step 3: Patch `calico-svs.yaml` + +The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-svs.yaml`. Update it to the customer registry: + +```bash +yq eval ' + (select(.kind == "Package") | .spec.template.spec.fetch[0].imgpkgBundle.image) + = "/calico-enterprise-svs:v3.6.1" +' -i calico-svs.yaml +``` + +:::tip + +**Content-pin to digest (optional):** `imgpkg copy --to-repo` prints the destination digest of the relocated bundle on its last line. Substitute `:v3.6.1` with `@sha256:` in the `yq` command above to make the reference immutable. + +::: + +### Step 4: Upload `calico-svs.yaml` to the Supervisor + +Follow the original [Part 1, Build and Publish](#build-and-publish) workflow with the patched `calico-svs.yaml`. From here, [Part 2](#part-2-guest-cluster-deployment) proceeds unchanged. + +### What does *not* work + +- `docker pull` / `docker push` on individual images. Re-tagging can change manifest digests, and the bundles' `images.yml` lock files are not rewritten — kapp-controller fetches will fail with `manifest unknown`. +- Editing `addon_repository.yaml` or `package.yaml` directly. Those files live inside the relocated bundles; their image references are overridden at fetch time by the lock files that `imgpkg copy` already rewrote. Manual edits are unnecessary and won't take effect anyway. +- Skipping any of the three `imgpkg copy` runs. All three bundles must be present in the customer registry — the service bundle references the repository bundle, which references the package bundle. + +--- + +## Part 4 [FOR AIRGAPPED ENV]: Relocate $[prodname] private images to your (or Customer's) private registry + +Follow the steps in [Push $[prodname] images to your private registry](./private-registry/private-registry-regular.mdx#push-calico-enterprise-images-to-your-private-registry) to pull $[prodname] images from Tigera's private image registry to your private registry. + +:::note + +Only steps 1, 2, 3, 4 — i.e. `docker pull`, `docker tag` and `docker push` for Linux only; and the `crane cp` for Windows images — need to be performed. + +::: + +:::caution + +Do not push $[prodname] private images to a public registry. + +::: diff --git a/calico-enterprise_versioned_sidebars/version-3.23-2-sidebars.json b/calico-enterprise_versioned_sidebars/version-3.23-2-sidebars.json index 2bcad9ac35..ac21f59453 100644 --- a/calico-enterprise_versioned_sidebars/version-3.23-2-sidebars.json +++ b/calico-enterprise_versioned_sidebars/version-3.23-2-sidebars.json @@ -66,6 +66,7 @@ "getting-started/install-on-clusters/rke2", "getting-started/install-on-clusters/rancher-ui", "getting-started/install-on-clusters/tkg", + "getting-started/install-on-clusters/vks", "getting-started/install-on-clusters/charmed-k8s", { "type": "category", From 14b3a5b39f0cc49a7004530be0d1c235de6e177e Mon Sep 17 00:00:00 2001 From: Saba Date: Mon, 29 Jun 2026 17:31:21 -0700 Subject: [PATCH 3/6] [DE-4343] Address Copilot review on VKS install page - Fix "Configmap" -> "ConfigMap" (correct Kubernetes kind casing) in the Step 6 heading. - Use kubectl apply -f for the Tigera license (idempotent on re-runs) instead of create -f. - Use grep -E 'tigera|calico' in the verification step (portable; BRE \| isn't on BSD/macOS grep). Copilot's /latest-link comment was already resolved by the prior relative-.mdx-link change. Applied to both the latest and 3.23-2 versioned copies. Co-Authored-By: Claude Opus 4.8 --- .../getting-started/install-on-clusters/vks.mdx | 6 +++--- .../getting-started/install-on-clusters/vks.mdx | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/calico-enterprise/getting-started/install-on-clusters/vks.mdx b/calico-enterprise/getting-started/install-on-clusters/vks.mdx index 463231a1e8..1347e4670f 100644 --- a/calico-enterprise/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise/getting-started/install-on-clusters/vks.mdx @@ -156,7 +156,7 @@ Status can be checked using `kubectl get tigerastatus`. kubectl apply -f ./sample-configs/ipam_configuration.yaml ``` -### Step 6: [Windows only] Apply the kubernetes-services-endpoint Configmap +### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap ```bash kubectl apply -f - << EOF @@ -212,7 +212,7 @@ At this point, all nodes should be in `Ready` state. ### Step 9: Apply the Tigera License ```bash -kubectl create -f tigera-license.yaml +kubectl apply -f tigera-license.yaml ``` ### Step 10: Verify the Installation @@ -221,7 +221,7 @@ Wait for all remaining components to become available: ```bash kubectl get tigerastatus -kubectl get pods -A | grep tigera\|calico +kubectl get pods -A | grep -E 'tigera|calico' ``` Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. diff --git a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx index 463231a1e8..1347e4670f 100644 --- a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx @@ -156,7 +156,7 @@ Status can be checked using `kubectl get tigerastatus`. kubectl apply -f ./sample-configs/ipam_configuration.yaml ``` -### Step 6: [Windows only] Apply the kubernetes-services-endpoint Configmap +### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap ```bash kubectl apply -f - << EOF @@ -212,7 +212,7 @@ At this point, all nodes should be in `Ready` state. ### Step 9: Apply the Tigera License ```bash -kubectl create -f tigera-license.yaml +kubectl apply -f tigera-license.yaml ``` ### Step 10: Verify the Installation @@ -221,7 +221,7 @@ Wait for all remaining components to become available: ```bash kubectl get tigerastatus -kubectl get pods -A | grep tigera\|calico +kubectl get pods -A | grep -E 'tigera|calico' ``` Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. From 2a1e5cf2a4ca2593f39f81626eadfdea9c9c5d8a Mon Sep 17 00:00:00 2001 From: Saba Date: Mon, 29 Jun 2026 18:01:25 -0700 Subject: [PATCH 4/6] [DE-4343] Restructure VKS page to the standard install-page template Reorganize into Big picture / Before you begin / How to / Next steps to match the other "Install on clusters" pages (tkg, openshift, etc.); the Parts 1-4 procedure now lives under How to as ### subsections. Drop the body --- rules (headings already delimit sections), rename per-part "Prerequisites" headings to avoid duplicate slugs, update in-page anchor links to the new slugs, and add a Next steps section. Applied to both the latest and 3.23-2 versioned copies. Co-Authored-By: Claude Opus 4.8 --- .../install-on-clusters/vks.mdx | 91 +++++++++++-------- .../install-on-clusters/vks.mdx | 91 +++++++++++-------- 2 files changed, 102 insertions(+), 80 deletions(-) diff --git a/calico-enterprise/getting-started/install-on-clusters/vks.mdx b/calico-enterprise/getting-started/install-on-clusters/vks.mdx index 1347e4670f..3c6991d061 100644 --- a/calico-enterprise/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise/getting-started/install-on-clusters/vks.mdx @@ -4,9 +4,13 @@ description: Install Calico Enterprise as a Supervisor Service on VMware Kuberne # VMware Kubernetes Service (VKS) -This guide walks through setting up $[prodname] as a Supervisor Service on VKS, and deploying it to guest clusters with optional Windows node support. +## Big picture -## Prerequisites +Install $[prodname] as a Supervisor Service on VMware Kubernetes Service (VKS), and deploy it to guest clusters with optional Windows node support. + +## Before you begin + +This guide requires: - **VM Images**: Ubuntu (and optionally Windows) OVA images registered for a supported TKR version, with the TKR in a healthy state. - **vSphere Resources**: Sufficient capacity to provision guest clusters with at minimum: @@ -15,7 +19,7 @@ This guide walks through setting up $[prodname] as a Supervisor Service on VKS, - 1 Windows worker node (optional, if Windows workloads are required) - **OCI Registry**: An OCI-compliant registry configured in vSphere for storing package manifests and container images. -## Networking Overview +### Networking | Configuration | Dataplane | Encapsulation | BGP | |---|---|---|---| @@ -28,15 +32,17 @@ When Windows nodes are enabled, the encapsulation is automatically set to VXLAN ::: ---- +## How to + +Setting up $[prodname] on VKS has two main phases: package and publish it as a Supervisor Service (Part 1), then deploy it to a guest cluster (Part 2). Parts 3 and 4 apply only to air-gapped environments. -## Part 1: Supervisor Service Setup +### Part 1: Set up the Supervisor Service -### Prerequisites +#### Supervisor Service prerequisites - `helm`, `kbld`, `ytt`, and `imgpkg` binaries available for the host OS architecture in a `./bin` directory. -### Configuration +#### Configuration | Variable | Description | |---|---| @@ -47,24 +53,24 @@ When Windows nodes are enabled, the encapsulation is automatically set to VXLAN | `REPO_VERSION` | Addon repository version | | `SERVICE_VERSION` | Addon service version | -### Build and Publish +#### Build and publish 1. Remove previously generated artifacts: ```bash make clean ``` -2. Build the Carvel package and push to the OCI registry. This creates the Addon package and uploads the artifacts to the OCI registry, also creates the Package and PackageMetadata resources (`calico-svs.yaml`) to be uploaded to the vSphere Supervisor UI: +1. Build the Carvel package and push to the OCI registry. This creates the Addon package and uploads the artifacts to the OCI registry, also creates the Package and PackageMetadata resources (`calico-svs.yaml`) to be uploaded to the vSphere Supervisor UI: ```bash make all ``` -3. In the vSphere Supervisor UI: +1. In the vSphere Supervisor UI: - Create a new Supervisor Service (or add a new version to an existing one). - Upload the generated `calico-svs.yaml` and complete the UI workflow. - Navigate to the service management page and set `registryName` to match the registry name configured in the UI. -### What `make all` Does +#### What `make all` does - Packages the Carvel manifests (AddonConfigDefinition, Package, AddonInstall, etc.). - Pulls and bundles the Tigera Operator Helm chart. @@ -72,18 +78,16 @@ When Windows nodes are enabled, the encapsulation is automatically set to VXLAN - Renders the `crd.projectcalico.org.v1` chart and bundles the $[prodname] CRDs. As of $[prodname] v3.23.0-2.0 the CRDs ship as a separate chart (no longer inside `tigera-operator`); they are applied alongside the operator with the `kapp.k14s.io/disable-original` annotation so the large CRDs (e.g. `installations.operator.tigera.io`) clear kapp's annotation-size limit — the kapp equivalent of the upstream server-side apply step. - Includes the `tigera-operator` namespace and `tigera-elasticsearch` StorageClass in the bundle. ---- - -## Part 2: Guest Cluster Deployment +### Part 2: Deploy to a guest cluster -### Prerequisites +#### Guest cluster prerequisites - **Tigera pull secret** for accessing the Tigera container registry (contact the Tigera team). - **Cluster ResourceSet for tigera pull secret**: The tigera-pull-secret applied as CAPI Cluster ResourceSet in the Supervisor Cluster for all namespaces where guest clusters with $[prodname] will be provisioned. The `./sample-configs/cluster-resourceset-pull-secret.yaml` can be used for this, by replacing the `.dockerconfigjson` value with the value of the private/tigera registry. - **Tigera license file** (contact the Tigera team). - **Dynamic PV provisioning** via a StorageClass, required for Elasticsearch storage. -### Step 1: Apply the AddonConfig (from Supervisor) +#### Step 1: Apply the AddonConfig (from Supervisor) SSH into the Supervisor machine and apply the AddonConfig resource. A sample configuration is provided in `./sample-configs/addon_config.yaml`. @@ -94,7 +98,7 @@ Update the following fields before applying: - `spec.addonConfigDefinitionRef` (name and namespace) - `spec.clusterName` - `spec.values.installation.calicoWindowsDataplane.enabled` (set to `true` for Windows support, `false` otherwise) -- `spec.values.installation.registry` to use $[prodname] images from a private registry (for air-gapped installs). This should be the same registry where $[prodname] images are pushed to using [Part 4](#part-4-for-airgapped-env-relocate-calico-enterprise-private-images-to-your-or-customers-private-registry) of this doc. Optionally `spec.values.installation.imagePath` can be set if the $[prodname] images were pushed to a specific folder within the private registry. +- `spec.values.installation.registry` to use $[prodname] images from a private registry (for air-gapped installs). This should be the same registry where $[prodname] images are pushed to using [Part 4](#part-4-air-gapped-relocate-calico-enterprise-private-images-to-a-private-registry) of this doc. Optionally `spec.values.installation.imagePath` can be set if the $[prodname] images were pushed to a specific folder within the private registry. ::: @@ -108,7 +112,7 @@ All $[prodname] features (`logCollector`, `logStorage`, `manager`, `monitor`, `p kubectl apply -f ./sample-configs/addon_config.yaml ``` -### Step 2: Apply the Cluster Resource (from Supervisor) +#### Step 2: Apply the Cluster Resource (from Supervisor) While still on the Supervisor machine, apply the Cluster resource. A sample is provided in `./sample-configs/cluster.yaml`. @@ -122,7 +126,7 @@ Update the following fields before applying: kubectl apply -f ./sample-configs/cluster.yaml ``` -### Step 3: Retrieve the Guest Cluster Kubeconfig +#### Step 3: Retrieve the Guest Cluster Kubeconfig ```bash kubectl -n get secret -o jsonpath='{.data.value}' | base64 --decode > kubeconfig @@ -134,7 +138,7 @@ From this point onwards, Supervisor SSH access is no longer required. All remain ::: -### Step 4: Create the Tigera Pull Secret +#### Step 4: Create the Tigera Pull Secret ```bash kubectl create secret generic tigera-pull-secret \ @@ -142,7 +146,7 @@ kubectl create secret generic tigera-pull-secret \ --from-file=.dockerconfigjson= ``` -### Step 5: [Windows only] Apply IPAM Configuration +#### Step 5: [Windows only] Apply IPAM Configuration Apply the strict affinity IPAM configuration, after Calico `apiserver` is `Available`. @@ -156,7 +160,7 @@ Status can be checked using `kubectl get tigerastatus`. kubectl apply -f ./sample-configs/ipam_configuration.yaml ``` -### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap +#### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap ```bash kubectl apply -f - << EOF @@ -177,7 +181,7 @@ The `APISERVER_ADDR` and `APISERVER_PORT` can be retrieved via `kubectl get endp ::: -### Step 7: [Windows only] Apply the kube-proxy on Windows nodes +#### Step 7: [Windows only] Apply the kube-proxy on Windows nodes Apply the Windows kube-proxy: @@ -199,7 +203,7 @@ If the version returned is of the form `v1.35.2+vmware.1`, strip the `+vmware.1` ::: -### Step 8: Wait for Core Components +#### Step 8: Wait for Core Components Monitor the Tigera status until `calico` and `apiserver` report as `Available`: @@ -209,13 +213,13 @@ kubectl get tigerastatus At this point, all nodes should be in `Ready` state. -### Step 9: Apply the Tigera License +#### Step 9: Apply the Tigera License ```bash kubectl apply -f tigera-license.yaml ``` -### Step 10: Verify the Installation +#### Step 10: Verify the Installation Wait for all remaining components to become available: @@ -226,19 +230,17 @@ kubectl get pods -A | grep -E 'tigera|calico' Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. ---- - -## Part 3 [FOR AIRGAPPED ENV]: Relocating Bundles to a Private Registry for Air-gapped install +### Part 3 (air-gapped): Relocate bundles to a private registry If the published bundles need to be moved into a customer's private OCI registry (for example, an air-gapped install or a customer who cannot pull from your registry), follow these steps. `make all` produces three bundles and one customer-facing YAML; relocation moves the bundles and updates one reference in the YAML. -### Prerequisites +#### Air-gapped relocation prerequisites - `imgpkg` and `yq` available locally. - Network access to the source registry (or a tarball produced from it) and to the customer registry. - Credentials to push to the customer registry (`imgpkg` will prompt or use the local docker config). -### Bundles to relocate +#### Bundles to relocate The three bundles correspond to the three Make targets and use the registry paths configured in the Makefile: @@ -250,7 +252,7 @@ The three bundles correspond to the three Make targets and use the registry path Cross-bundle references (service → repository → package → container images) are stored as digests in each bundle's `.imgpkg/images.yml` lock file. `imgpkg copy` rewrites those lock files automatically when relocating, so no manifests inside the bundles need manual editing. -### Step 1: Copy the bundles to a tarball (source side) +#### Step 1: Copy the bundles to a tarball (source side) Run on a host that has access to the source registry: @@ -268,7 +270,7 @@ Direct registry-to-registry relocation is also supported. Replace `--to-tar /calico-enterprise @@ -278,7 +280,7 @@ imgpkg copy --tar calico-svs.tar --to-repo /calico-enterp After each `imgpkg copy --to-repo` finishes, the relocated bundle's `images.yml` references the customer registry by digest. Container images and cross-bundle references resolve to the customer registry automatically at deploy time. -### Step 3: Patch `calico-svs.yaml` +#### Step 3: Patch `calico-svs.yaml` The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-svs.yaml`. Update it to the customer registry: @@ -295,19 +297,17 @@ yq eval ' ::: -### Step 4: Upload `calico-svs.yaml` to the Supervisor +#### Step 4: Upload `calico-svs.yaml` to the Supervisor -Follow the original [Part 1, Build and Publish](#build-and-publish) workflow with the patched `calico-svs.yaml`. From here, [Part 2](#part-2-guest-cluster-deployment) proceeds unchanged. +Follow the original [Part 1, Build and publish](#build-and-publish) workflow with the patched `calico-svs.yaml`. From here, [Part 2](#part-2-deploy-to-a-guest-cluster) proceeds unchanged. -### What does *not* work +#### What does *not* work - `docker pull` / `docker push` on individual images. Re-tagging can change manifest digests, and the bundles' `images.yml` lock files are not rewritten — kapp-controller fetches will fail with `manifest unknown`. - Editing `addon_repository.yaml` or `package.yaml` directly. Those files live inside the relocated bundles; their image references are overridden at fetch time by the lock files that `imgpkg copy` already rewrote. Manual edits are unnecessary and won't take effect anyway. - Skipping any of the three `imgpkg copy` runs. All three bundles must be present in the customer registry — the service bundle references the repository bundle, which references the package bundle. ---- - -## Part 4 [FOR AIRGAPPED ENV]: Relocate $[prodname] private images to your (or Customer's) private registry +### Part 4 (air-gapped): Relocate $[prodname] private images to a private registry Follow the steps in [Push $[prodname] images to your private registry](./private-registry/private-registry-regular.mdx#push-calico-enterprise-images-to-your-private-registry) to pull $[prodname] images from Tigera's private image registry to your private registry. @@ -322,3 +322,14 @@ Only steps 1, 2, 3, 4 — i.e. `docker pull`, `docker tag` and `docker push` for Do not push $[prodname] private images to a public registry. ::: + +## Next steps + +**Recommended** + +- [Configure access to the $[prodname] web console](../../operations/cnx/access-the-manager.mdx) +- [Get started with $[prodname] tiered network policy](../../network-policy/policy-tiers/tiered-policy.mdx) + +**Recommended - Networking** + +- For all networking options, see [Determine best networking option](../../networking/determine-best-networking.mdx). diff --git a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx index 1347e4670f..3c6991d061 100644 --- a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx @@ -4,9 +4,13 @@ description: Install Calico Enterprise as a Supervisor Service on VMware Kuberne # VMware Kubernetes Service (VKS) -This guide walks through setting up $[prodname] as a Supervisor Service on VKS, and deploying it to guest clusters with optional Windows node support. +## Big picture -## Prerequisites +Install $[prodname] as a Supervisor Service on VMware Kubernetes Service (VKS), and deploy it to guest clusters with optional Windows node support. + +## Before you begin + +This guide requires: - **VM Images**: Ubuntu (and optionally Windows) OVA images registered for a supported TKR version, with the TKR in a healthy state. - **vSphere Resources**: Sufficient capacity to provision guest clusters with at minimum: @@ -15,7 +19,7 @@ This guide walks through setting up $[prodname] as a Supervisor Service on VKS, - 1 Windows worker node (optional, if Windows workloads are required) - **OCI Registry**: An OCI-compliant registry configured in vSphere for storing package manifests and container images. -## Networking Overview +### Networking | Configuration | Dataplane | Encapsulation | BGP | |---|---|---|---| @@ -28,15 +32,17 @@ When Windows nodes are enabled, the encapsulation is automatically set to VXLAN ::: ---- +## How to + +Setting up $[prodname] on VKS has two main phases: package and publish it as a Supervisor Service (Part 1), then deploy it to a guest cluster (Part 2). Parts 3 and 4 apply only to air-gapped environments. -## Part 1: Supervisor Service Setup +### Part 1: Set up the Supervisor Service -### Prerequisites +#### Supervisor Service prerequisites - `helm`, `kbld`, `ytt`, and `imgpkg` binaries available for the host OS architecture in a `./bin` directory. -### Configuration +#### Configuration | Variable | Description | |---|---| @@ -47,24 +53,24 @@ When Windows nodes are enabled, the encapsulation is automatically set to VXLAN | `REPO_VERSION` | Addon repository version | | `SERVICE_VERSION` | Addon service version | -### Build and Publish +#### Build and publish 1. Remove previously generated artifacts: ```bash make clean ``` -2. Build the Carvel package and push to the OCI registry. This creates the Addon package and uploads the artifacts to the OCI registry, also creates the Package and PackageMetadata resources (`calico-svs.yaml`) to be uploaded to the vSphere Supervisor UI: +1. Build the Carvel package and push to the OCI registry. This creates the Addon package and uploads the artifacts to the OCI registry, also creates the Package and PackageMetadata resources (`calico-svs.yaml`) to be uploaded to the vSphere Supervisor UI: ```bash make all ``` -3. In the vSphere Supervisor UI: +1. In the vSphere Supervisor UI: - Create a new Supervisor Service (or add a new version to an existing one). - Upload the generated `calico-svs.yaml` and complete the UI workflow. - Navigate to the service management page and set `registryName` to match the registry name configured in the UI. -### What `make all` Does +#### What `make all` does - Packages the Carvel manifests (AddonConfigDefinition, Package, AddonInstall, etc.). - Pulls and bundles the Tigera Operator Helm chart. @@ -72,18 +78,16 @@ When Windows nodes are enabled, the encapsulation is automatically set to VXLAN - Renders the `crd.projectcalico.org.v1` chart and bundles the $[prodname] CRDs. As of $[prodname] v3.23.0-2.0 the CRDs ship as a separate chart (no longer inside `tigera-operator`); they are applied alongside the operator with the `kapp.k14s.io/disable-original` annotation so the large CRDs (e.g. `installations.operator.tigera.io`) clear kapp's annotation-size limit — the kapp equivalent of the upstream server-side apply step. - Includes the `tigera-operator` namespace and `tigera-elasticsearch` StorageClass in the bundle. ---- - -## Part 2: Guest Cluster Deployment +### Part 2: Deploy to a guest cluster -### Prerequisites +#### Guest cluster prerequisites - **Tigera pull secret** for accessing the Tigera container registry (contact the Tigera team). - **Cluster ResourceSet for tigera pull secret**: The tigera-pull-secret applied as CAPI Cluster ResourceSet in the Supervisor Cluster for all namespaces where guest clusters with $[prodname] will be provisioned. The `./sample-configs/cluster-resourceset-pull-secret.yaml` can be used for this, by replacing the `.dockerconfigjson` value with the value of the private/tigera registry. - **Tigera license file** (contact the Tigera team). - **Dynamic PV provisioning** via a StorageClass, required for Elasticsearch storage. -### Step 1: Apply the AddonConfig (from Supervisor) +#### Step 1: Apply the AddonConfig (from Supervisor) SSH into the Supervisor machine and apply the AddonConfig resource. A sample configuration is provided in `./sample-configs/addon_config.yaml`. @@ -94,7 +98,7 @@ Update the following fields before applying: - `spec.addonConfigDefinitionRef` (name and namespace) - `spec.clusterName` - `spec.values.installation.calicoWindowsDataplane.enabled` (set to `true` for Windows support, `false` otherwise) -- `spec.values.installation.registry` to use $[prodname] images from a private registry (for air-gapped installs). This should be the same registry where $[prodname] images are pushed to using [Part 4](#part-4-for-airgapped-env-relocate-calico-enterprise-private-images-to-your-or-customers-private-registry) of this doc. Optionally `spec.values.installation.imagePath` can be set if the $[prodname] images were pushed to a specific folder within the private registry. +- `spec.values.installation.registry` to use $[prodname] images from a private registry (for air-gapped installs). This should be the same registry where $[prodname] images are pushed to using [Part 4](#part-4-air-gapped-relocate-calico-enterprise-private-images-to-a-private-registry) of this doc. Optionally `spec.values.installation.imagePath` can be set if the $[prodname] images were pushed to a specific folder within the private registry. ::: @@ -108,7 +112,7 @@ All $[prodname] features (`logCollector`, `logStorage`, `manager`, `monitor`, `p kubectl apply -f ./sample-configs/addon_config.yaml ``` -### Step 2: Apply the Cluster Resource (from Supervisor) +#### Step 2: Apply the Cluster Resource (from Supervisor) While still on the Supervisor machine, apply the Cluster resource. A sample is provided in `./sample-configs/cluster.yaml`. @@ -122,7 +126,7 @@ Update the following fields before applying: kubectl apply -f ./sample-configs/cluster.yaml ``` -### Step 3: Retrieve the Guest Cluster Kubeconfig +#### Step 3: Retrieve the Guest Cluster Kubeconfig ```bash kubectl -n get secret -o jsonpath='{.data.value}' | base64 --decode > kubeconfig @@ -134,7 +138,7 @@ From this point onwards, Supervisor SSH access is no longer required. All remain ::: -### Step 4: Create the Tigera Pull Secret +#### Step 4: Create the Tigera Pull Secret ```bash kubectl create secret generic tigera-pull-secret \ @@ -142,7 +146,7 @@ kubectl create secret generic tigera-pull-secret \ --from-file=.dockerconfigjson= ``` -### Step 5: [Windows only] Apply IPAM Configuration +#### Step 5: [Windows only] Apply IPAM Configuration Apply the strict affinity IPAM configuration, after Calico `apiserver` is `Available`. @@ -156,7 +160,7 @@ Status can be checked using `kubectl get tigerastatus`. kubectl apply -f ./sample-configs/ipam_configuration.yaml ``` -### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap +#### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap ```bash kubectl apply -f - << EOF @@ -177,7 +181,7 @@ The `APISERVER_ADDR` and `APISERVER_PORT` can be retrieved via `kubectl get endp ::: -### Step 7: [Windows only] Apply the kube-proxy on Windows nodes +#### Step 7: [Windows only] Apply the kube-proxy on Windows nodes Apply the Windows kube-proxy: @@ -199,7 +203,7 @@ If the version returned is of the form `v1.35.2+vmware.1`, strip the `+vmware.1` ::: -### Step 8: Wait for Core Components +#### Step 8: Wait for Core Components Monitor the Tigera status until `calico` and `apiserver` report as `Available`: @@ -209,13 +213,13 @@ kubectl get tigerastatus At this point, all nodes should be in `Ready` state. -### Step 9: Apply the Tigera License +#### Step 9: Apply the Tigera License ```bash kubectl apply -f tigera-license.yaml ``` -### Step 10: Verify the Installation +#### Step 10: Verify the Installation Wait for all remaining components to become available: @@ -226,19 +230,17 @@ kubectl get pods -A | grep -E 'tigera|calico' Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. ---- - -## Part 3 [FOR AIRGAPPED ENV]: Relocating Bundles to a Private Registry for Air-gapped install +### Part 3 (air-gapped): Relocate bundles to a private registry If the published bundles need to be moved into a customer's private OCI registry (for example, an air-gapped install or a customer who cannot pull from your registry), follow these steps. `make all` produces three bundles and one customer-facing YAML; relocation moves the bundles and updates one reference in the YAML. -### Prerequisites +#### Air-gapped relocation prerequisites - `imgpkg` and `yq` available locally. - Network access to the source registry (or a tarball produced from it) and to the customer registry. - Credentials to push to the customer registry (`imgpkg` will prompt or use the local docker config). -### Bundles to relocate +#### Bundles to relocate The three bundles correspond to the three Make targets and use the registry paths configured in the Makefile: @@ -250,7 +252,7 @@ The three bundles correspond to the three Make targets and use the registry path Cross-bundle references (service → repository → package → container images) are stored as digests in each bundle's `.imgpkg/images.yml` lock file. `imgpkg copy` rewrites those lock files automatically when relocating, so no manifests inside the bundles need manual editing. -### Step 1: Copy the bundles to a tarball (source side) +#### Step 1: Copy the bundles to a tarball (source side) Run on a host that has access to the source registry: @@ -268,7 +270,7 @@ Direct registry-to-registry relocation is also supported. Replace `--to-tar /calico-enterprise @@ -278,7 +280,7 @@ imgpkg copy --tar calico-svs.tar --to-repo /calico-enterp After each `imgpkg copy --to-repo` finishes, the relocated bundle's `images.yml` references the customer registry by digest. Container images and cross-bundle references resolve to the customer registry automatically at deploy time. -### Step 3: Patch `calico-svs.yaml` +#### Step 3: Patch `calico-svs.yaml` The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-svs.yaml`. Update it to the customer registry: @@ -295,19 +297,17 @@ yq eval ' ::: -### Step 4: Upload `calico-svs.yaml` to the Supervisor +#### Step 4: Upload `calico-svs.yaml` to the Supervisor -Follow the original [Part 1, Build and Publish](#build-and-publish) workflow with the patched `calico-svs.yaml`. From here, [Part 2](#part-2-guest-cluster-deployment) proceeds unchanged. +Follow the original [Part 1, Build and publish](#build-and-publish) workflow with the patched `calico-svs.yaml`. From here, [Part 2](#part-2-deploy-to-a-guest-cluster) proceeds unchanged. -### What does *not* work +#### What does *not* work - `docker pull` / `docker push` on individual images. Re-tagging can change manifest digests, and the bundles' `images.yml` lock files are not rewritten — kapp-controller fetches will fail with `manifest unknown`. - Editing `addon_repository.yaml` or `package.yaml` directly. Those files live inside the relocated bundles; their image references are overridden at fetch time by the lock files that `imgpkg copy` already rewrote. Manual edits are unnecessary and won't take effect anyway. - Skipping any of the three `imgpkg copy` runs. All three bundles must be present in the customer registry — the service bundle references the repository bundle, which references the package bundle. ---- - -## Part 4 [FOR AIRGAPPED ENV]: Relocate $[prodname] private images to your (or Customer's) private registry +### Part 4 (air-gapped): Relocate $[prodname] private images to a private registry Follow the steps in [Push $[prodname] images to your private registry](./private-registry/private-registry-regular.mdx#push-calico-enterprise-images-to-your-private-registry) to pull $[prodname] images from Tigera's private image registry to your private registry. @@ -322,3 +322,14 @@ Only steps 1, 2, 3, 4 — i.e. `docker pull`, `docker tag` and `docker push` for Do not push $[prodname] private images to a public registry. ::: + +## Next steps + +**Recommended** + +- [Configure access to the $[prodname] web console](../../operations/cnx/access-the-manager.mdx) +- [Get started with $[prodname] tiered network policy](../../network-policy/policy-tiers/tiered-policy.mdx) + +**Recommended - Networking** + +- For all networking options, see [Determine best networking option](../../networking/determine-best-networking.mdx). From bf47a150005fc9bad03d96ed2a2b049a51f22943 Mon Sep 17 00:00:00 2001 From: sujeet-kr Date: Thu, 2 Jul 2026 16:50:59 -0400 Subject: [PATCH 5/6] update with simplified VKS install steps --- .../install-on-clusters/vks.mdx | 460 ++++++++++++++---- .../install-on-clusters/vks.mdx | 460 ++++++++++++++---- 2 files changed, 718 insertions(+), 202 deletions(-) diff --git a/calico-enterprise/getting-started/install-on-clusters/vks.mdx b/calico-enterprise/getting-started/install-on-clusters/vks.mdx index 3c6991d061..8b11c322cc 100644 --- a/calico-enterprise/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise/getting-started/install-on-clusters/vks.mdx @@ -12,12 +12,13 @@ Install $[prodname] as a Supervisor Service on VMware Kubernetes Service (VKS), This guide requires: -- **VM Images**: Ubuntu (and optionally Windows) OVA images registered for a supported TKR version, with the TKR in a healthy state. +- **VM Images**: Ubuntu or Photon (and optionally Windows) OVA images registered for a supported TKR version, with TKR in a healthy state. - **vSphere Resources**: Sufficient capacity to provision guest clusters with at minimum: - 1 control plane node - 3 Linux worker nodes - 1 Windows worker node (optional, if Windows workloads are required) - **OCI Registry**: An OCI-compliant registry configured in vSphere for storing package manifests and container images. +- **Tigera pull secret** for accessing the Tigera container registry (contact the Tigera team). ### Networking @@ -34,62 +35,110 @@ When Windows nodes are enabled, the encapsulation is automatically set to VXLAN ## How to -Setting up $[prodname] on VKS has two main phases: package and publish it as a Supervisor Service (Part 1), then deploy it to a guest cluster (Part 2). Parts 3 and 4 apply only to air-gapped environments. - -### Part 1: Set up the Supervisor Service - -#### Supervisor Service prerequisites +Setting up $[prodname] on VKS has two main phases: set it as a Supervisor Service (Part 1) in vSphere (one time), then deploy it to a guest cluster (Part 2). Parts 3 and 4 apply only to air-gapped environments. -- `helm`, `kbld`, `ytt`, and `imgpkg` binaries available for the host OS architecture in a `./bin` directory. +### Part 1: Set up the Supervisor Service (one time cluster admin setup) -#### Configuration - -| Variable | Description | -|---|---| -| `CALICO_ENTERPRISE_VERSION` | $[prodname] version to package | -| `CALICO_ENTERPRISE_SVS` | (Optional) Custom name for the Supervisor Service | -| `ARTIFACTORY_WRITE` | OCI registry for pushing manifests and images | -| `ARTIFACTORY_READ` | OCI registry from which manifests and images are pulled | -| `REPO_VERSION` | Addon repository version | -| `SERVICE_VERSION` | Addon service version | - -#### Build and publish +1. In the vSphere Supervisor UI: + - Create a new Supervisor Service (or add a new version to an existing one). + - Upload Supervisor service Package and PackageMetadata (example below) resources and complete the UI workflow. + - Register the Supervisor Service and set `registryName` to match the registry name configured in the UI (if needed). -1. Remove previously generated artifacts: - ```bash - make clean - ``` +```yaml +--- +apiVersion: data.packaging.carvel.dev/v1alpha1 +kind: Package +metadata: + name: calico-enterprise-svs.tigera.io.3.23.1 +spec: + refName: calico-enterprise-svs.tigera.io + version: 3.23.1 + template: + spec: + fetch: + - imgpkgBundle: + image: quay.io/tigeradev/calico-enterprise-svs:v3.23.1 #Change this to the registry where the carvel package is getting picked up + template: + - ytt: + paths: + - service + - kapp + - kbld: + paths: + - "-" + - .imgpkg/images.yml + - kbld/ + deploy: + - kapp: {} +--- +apiVersion: data.packaging.carvel.dev/v1alpha1 +kind: PackageMetadata +metadata: + name: calico-enterprise-svs.tigera.io +spec: + displayName: "Calico Enterprise SVS" + shortDescription: "Manages Calico Enterprise Addon Repository and Installation" +``` -1. Build the Carvel package and push to the OCI registry. This creates the Addon package and uploads the artifacts to the OCI registry, also creates the Package and PackageMetadata resources (`calico-svs.yaml`) to be uploaded to the vSphere Supervisor UI: - ```bash - make all - ``` +2. Add the tigera-pull-secret ClusterResourceSet to the Supervisor namespace where auto tigera-pull-secret application is desired -1. In the vSphere Supervisor UI: - - Create a new Supervisor Service (or add a new version to an existing one). - - Upload the generated `calico-svs.yaml` and complete the UI workflow. - - Navigate to the service management page and set `registryName` to match the registry name configured in the UI. +```sh +kubectl apply -n -f .yaml +``` -#### What `make all` does +```yaml +# Apply this on the Supervisor cluster for the namespaces where guest clusters will be provisioned with Calico Enterprise. +# The source Secret containing the manifests CRS will mirror +apiVersion: v1 +kind: Secret +metadata: + name: tigera-pull-secret-bootstrap +type: addons.cluster.x-k8s.io/resource-set # CRS-specific type +stringData: + resources.yaml: | + apiVersion: v1 + kind: Namespace + metadata: + name: tigera-operator + labels: + pod-security.kubernetes.io/enforce: privileged + pod-security.kubernetes.io/audit: privileged + pod-security.kubernetes.io/warn: privileged + --- + apiVersion: v1 + kind: Secret + metadata: + name: tigera-pull-secret + namespace: tigera-operator + type: kubernetes.io/dockerconfigjson + data: + .dockerconfigjson: +--- +# The CRS that selects clusters and points at the source +apiVersion: addons.cluster.x-k8s.io/v1beta2 +kind: ClusterResourceSet +metadata: + name: tigera-pull-secret-bootstrap +spec: + strategy: Reconcile + clusterSelector: + matchLabels: + tigera.io/pull-secret: required + resources: + - name: tigera-pull-secret-bootstrap + kind: Secret -- Packages the Carvel manifests (AddonConfigDefinition, Package, AddonInstall, etc.). -- Pulls and bundles the Tigera Operator Helm chart. -- Resolves and bundles all required container images (`tigera-operator`, `tigera-prometheus-operator`, etc.). -- Renders the `crd.projectcalico.org.v1` chart and bundles the $[prodname] CRDs. As of $[prodname] v3.23.0-2.0 the CRDs ship as a separate chart (no longer inside `tigera-operator`); they are applied alongside the operator with the `kapp.k14s.io/disable-original` annotation so the large CRDs (e.g. `installations.operator.tigera.io`) clear kapp's annotation-size limit — the kapp equivalent of the upstream server-side apply step. -- Includes the `tigera-operator` namespace and `tigera-elasticsearch` StorageClass in the bundle. +``` ### Part 2: Deploy to a guest cluster #### Guest cluster prerequisites -- **Tigera pull secret** for accessing the Tigera container registry (contact the Tigera team). -- **Cluster ResourceSet for tigera pull secret**: The tigera-pull-secret applied as CAPI Cluster ResourceSet in the Supervisor Cluster for all namespaces where guest clusters with $[prodname] will be provisioned. The `./sample-configs/cluster-resourceset-pull-secret.yaml` can be used for this, by replacing the `.dockerconfigjson` value with the value of the private/tigera registry. +- **Cluster ResourceSet for tigera pull secret**: The tigera-pull-secret applied as CAPI Cluster ResourceSet in the Supervisor Cluster for all namespaces where guest clusters with $[prodname] will be provisioned. - **Tigera license file** (contact the Tigera team). - **Dynamic PV provisioning** via a StorageClass, required for Elasticsearch storage. -#### Step 1: Apply the AddonConfig (from Supervisor) - -SSH into the Supervisor machine and apply the AddonConfig resource. A sample configuration is provided in `./sample-configs/addon_config.yaml`. +#### Step 1: Apply the AddonConfig :::info @@ -109,42 +158,261 @@ All $[prodname] features (`logCollector`, `logStorage`, `manager`, `monitor`, `p ::: ```bash -kubectl apply -f ./sample-configs/addon_config.yaml +kubectl apply -f ./addon_config.yaml ``` -#### Step 2: Apply the Cluster Resource (from Supervisor) +```yaml +apiVersion: addons.kubernetes.vmware.com/v1alpha1 +kind: AddonConfig +metadata: + name: + namespace: +spec: + addonConfigDefinitionRef: + name: calico-enterprise.tigera.io.3.23.1 + namespace: vmware-system-vks-public + clusterName: + values: + # Every option from tigera-helm-charts/tigera-operator/values.yaml may be + # set here. Anything not specified falls back to the default declared in + # the AddonConfigDefinition schema. The fields below are illustrative; + installation: + enabled: true + imagePullSecrets: + - name: tigera-pull-secret + calicoWindowsDataplane: + enabled: false + # registry: quay.io/ + # imagePath: "" + # controlPlaneReplicas: 3 + apiServer: + enabled: true + # Calico deploys its own bundled Prometheus Operator by default (enabled: true). + # Set this to false ONLY if you run your own external Prometheus Operator -- in + # that case Calico's Monitor still creates its Prometheus/Alertmanager CRs for + # your operator to reconcile. If you want no Prometheus at all, also set + # monitor.enabled: false. + # tigeraPrometheusOperator: + # enabled: false + logCollector: + enabled: true + logStorage: + enabled: true + nodes: + count: 1 + manager: + enabled: true + monitor: + enabled: true + policyRecommendation: + enabled: true +``` -While still on the Supervisor machine, apply the Cluster resource. A sample is provided in `./sample-configs/cluster.yaml`. +#### Step 2: Apply the Cluster Resource Update the following fields before applying: - `metadata.name` and `metadata.namespace` - `spec.topology.variables[].value.cniRef` (name and namespace of the $[prodname] addon) -- Add the label `tigera.io/pull-secret: required` to `metadata.labels` so that tigera-pull-secret will be created for the cluster. +- Add the label `tigera.io/pull-secret: required` to `metadata.labels` so that tigera-pull-secret will be created automatically for the cluster. - Node pool configuration (VM class, storage class, replicas) as needed for your environment ```bash -kubectl apply -f ./sample-configs/cluster.yaml +kubectl apply -f ./cluster.yaml ``` -#### Step 3: Retrieve the Guest Cluster Kubeconfig - -```bash -kubectl -n get secret -o jsonpath='{.data.value}' | base64 --decode > kubeconfig +```yaml +apiVersion: cluster.x-k8s.io/v1beta2 +kind: Cluster +metadata: + name: + namespace: + labels: + tigera.io/pull-secret: required +spec: + clusterNetwork: + pods: + cidrBlocks: + - 100.96.0.0/11 + serviceDomain: cluster.local + services: + cidrBlocks: + - 100.64.0.0/13 + topology: + classRef: + name: builtin-generic-v3.7.0 + namespace: vmware-system-vks-public + controlPlane: + metadata: + annotations: + run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu,os-version=24.04 + replicas: 1 + variables: + - name: node + value: + firewall: + inboundRules: + # BGP + - protocol: tcp + fromPort: 179 + toPort: 179 + # VXLAN + - protocol: udp + fromPort: 4789 + toPort: 4789 + # Typha + - protocol: tcp + fromPort: 5473 + toPort: 5473 + # Wireguard + - protocol: udp + fromPort: 51820 + toPort: 51820 + # Elasticsearch + - protocol: tcp + fromPort: 9200 + toPort: 9200 + # Elasticsearch transport + - protocol: tcp + fromPort: 9300 + toPort: 9300 + # Tigera Manager UI + - protocol: tcp + fromPort: 9443 + toPort: 9443 + # Prometheus + - protocol: tcp + fromPort: 9090 + toPort: 9090 + # Linseed + - protocol: tcp + fromPort: 8444 + toPort: 8444 + - name: bootstrapAddons + value: + cniRef: + name: calico-enterprise + namespace: svc-calico-enterprise-svs-domain-c10 + - name: vmClass + value: best-effort-xlarge + - name: storageClass + value: vsan-esa-default-policy-raid5 + - name: kubernetes + value: + certificateRotation: + enabled: true + renewalDaysBeforeExpiry: 90 + - name: vsphereOptions + value: + persistentVolumes: + defaultStorageClass: vsan-esa-default-policy-raid5 + version: v1.35.0+vmware.2-vkr.4 + workers: + machineDeployments: + - class: node-pool + metadata: + annotations: + run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu,os-version=24.04 + name: node-pool-1 + replicas: 3 + variables: + overrides: + - name: vmClass + value: best-effort-large + - name: node + value: + firewall: + inboundRules: + # BGP + - protocol: tcp + fromPort: 179 + toPort: 179 + # VXLAN + - protocol: udp + fromPort: 4789 + toPort: 4789 + # Typha + - protocol: tcp + fromPort: 5473 + toPort: 5473 + # Wireguard + - protocol: udp + fromPort: 51820 + toPort: 51820 + # Elasticsearch + - protocol: tcp + fromPort: 9200 + toPort: 9200 + # Elasticsearch transport + - protocol: tcp + fromPort: 9300 + toPort: 9300 + # Tigera Manager UI + - protocol: tcp + fromPort: 9443 + toPort: 9443 + # Prometheus + - protocol: tcp + fromPort: 9090 + toPort: 9090 + # Linseed + - protocol: tcp + fromPort: 8444 + toPort: 8444 + - class: node-pool + metadata: + annotations: + run.tanzu.vmware.com/resolve-os-image: os-name=windows,os-version=2022 + name: win-node-pool-1 + # if Windows nodes needed + replicas: 0 + variables: + overrides: + - name: vmClass + value: best-effort-large + - name: node + value: + firewall: + inboundRules: + # BGP + - protocol: tcp + fromPort: 179 + toPort: 179 + # VXLAN + - protocol: udp + fromPort: 4789 + toPort: 4789 + # Typha + - protocol: tcp + fromPort: 5473 + toPort: 5473 + # Wireguard + - protocol: udp + fromPort: 51820 + toPort: 51820 + # Elasticsearch + - protocol: tcp + fromPort: 9200 + toPort: 9200 + # Elasticsearch transport + - protocol: tcp + fromPort: 9300 + toPort: 9300 + # Tigera Manager UI + - protocol: tcp + fromPort: 9443 + toPort: 9443 + # Prometheus + - protocol: tcp + fromPort: 9090 + toPort: 9090 + # Linseed + - protocol: tcp + fromPort: 8444 + toPort: 8444 ``` -:::note - -From this point onwards, Supervisor SSH access is no longer required. All remaining steps can be performed from any machine with access to the guest cluster's API server. - -::: - -#### Step 4: Create the Tigera Pull Secret +#### Step 3: Retrieve the Guest Cluster Kubeconfig -```bash -kubectl create secret generic tigera-pull-secret \ - --type=kubernetes.io/dockerconfigjson -n tigera-operator \ - --from-file=.dockerconfigjson= -``` #### Step 5: [Windows only] Apply IPAM Configuration @@ -157,7 +425,19 @@ Status can be checked using `kubectl get tigerastatus`. ::: ```bash -kubectl apply -f ./sample-configs/ipam_configuration.yaml +kubectl apply -f ./ipam_configuration.yaml +``` + +```yaml +# Required for Windows nodes - ensures IPAM blocks are strictly +# affinitive to their assigned nodes. +# Apply this after calico-node pods are running. +apiVersion: projectcalico.org/v3 +kind: IPAMConfiguration +metadata: + name: default +spec: + strictAffinity: true ``` #### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap @@ -230,82 +510,60 @@ kubectl get pods -A | grep -E 'tigera|calico' Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. -### Part 3 (air-gapped): Relocate bundles to a private registry +### Part 3: Airgapped package registry -If the published bundles need to be moved into a customer's private OCI registry (for example, an air-gapped install or a customer who cannot pull from your registry), follow these steps. `make all` produces three bundles and one customer-facing YAML; relocation moves the bundles and updates one reference in the YAML. +If the published bundles need to be moved into a private OCI registry (for example, an air-gapped install), follow these steps. #### Air-gapped relocation prerequisites - `imgpkg` and `yq` available locally. -- Network access to the source registry (or a tarball produced from it) and to the customer registry. -- Credentials to push to the customer registry (`imgpkg` will prompt or use the local docker config). - -#### Bundles to relocate +- Network access to the source registry (or a tarball produced from it) and to the private registry. +- Credentials to push to the private registry (`imgpkg` will prompt or use the local docker config). -The three bundles correspond to the three Make targets and use the registry paths configured in the Makefile: - -| Bundle | Source path | Default tag | -|---|---|---| -| Package bundle ($[prodname] chart + container images) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}` | `v${CALICO_ENTERPRISE_VERSION}` | -| Repository bundle (Package + PackageMetadata for the package bundle) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}-repo` | `v${REPO_VERSION}` | -| Service bundle (AddonInstall, AddonRepository, AddonRepositoryInstall) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}-svs` | `v${SERVICE_VERSION}` | - -Cross-bundle references (service → repository → package → container images) are stored as digests in each bundle's `.imgpkg/images.yml` lock file. `imgpkg copy` rewrites those lock files automatically when relocating, so no manifests inside the bundles need manual editing. - -#### Step 1: Copy the bundles to a tarball (source side) +#### Step 1: Copy the Supervisor Service (SVS) bundles to a tarball (source side) Run on a host that has access to the source registry: ```bash -imgpkg copy --bundle /calico-enterprise:v3.22.2 --to-tar calico-package.tar -imgpkg copy --bundle /calico-enterprise-repo:v3.6.1 --to-tar calico-repo.tar -imgpkg copy --bundle /calico-enterprise-svs:v3.6.1 --to-tar calico-svs.tar +imgpkg copy --bundle /calico-enterprise-svs:v3.23.1 --to-tar calico-enterprise-svs.tar ``` -Transfer the three tarballs along with `calico-svs.yaml` to the destination environment. - :::tip -Direct registry-to-registry relocation is also supported. Replace `--to-tar .tar` with `--to-repo /` and skip Step 2. +Direct registry-to-registry relocation is also supported. Replace `--to-tar .tar` with `--to-repo /` and skip Step 2. ::: -#### Step 2: Push the bundles into the customer registry (destination side) +#### Step 2: Push the SVS bundle into the private registry (destination side) ```bash -imgpkg copy --tar calico-package.tar --to-repo /calico-enterprise -imgpkg copy --tar calico-repo.tar --to-repo /calico-enterprise-repo -imgpkg copy --tar calico-svs.tar --to-repo /calico-enterprise-svs +imgpkg copy --tar calico-enterprise-svs.tar --to-repo /calico-enterprise-svs ``` -After each `imgpkg copy --to-repo` finishes, the relocated bundle's `images.yml` references the customer registry by digest. Container images and cross-bundle references resolve to the customer registry automatically at deploy time. - -#### Step 3: Patch `calico-svs.yaml` +#### Step 3: Patch Package Resource in `calico-enterprise-svs.yaml` -The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-svs.yaml`. Update it to the customer registry: +The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-enterprise-svs.yaml`. This needs to be updated to the private registry: ```bash yq eval ' (select(.kind == "Package") | .spec.template.spec.fetch[0].imgpkgBundle.image) - = "/calico-enterprise-svs:v3.6.1" + = "/calico-enterprise-svs:v3.23.1" ' -i calico-svs.yaml ``` :::tip -**Content-pin to digest (optional):** `imgpkg copy --to-repo` prints the destination digest of the relocated bundle on its last line. Substitute `:v3.6.1` with `@sha256:` in the `yq` command above to make the reference immutable. +**Content-pin to digest (optional):** `imgpkg copy --to-repo` prints the destination digest of the relocated bundle on its last line. Substituting `:v3.23.1` with `@sha256:` in the `yq` command above will make the reference immutable. ::: -#### Step 4: Upload `calico-svs.yaml` to the Supervisor +#### Step 4: Upload `calico-enterprise-svs.yaml` i.e. Package and PackageMetadata resource to the Supervisor -Follow the original [Part 1, Build and publish](#build-and-publish) workflow with the patched `calico-svs.yaml`. From here, [Part 2](#part-2-deploy-to-a-guest-cluster) proceeds unchanged. #### What does *not* work - `docker pull` / `docker push` on individual images. Re-tagging can change manifest digests, and the bundles' `images.yml` lock files are not rewritten — kapp-controller fetches will fail with `manifest unknown`. -- Editing `addon_repository.yaml` or `package.yaml` directly. Those files live inside the relocated bundles; their image references are overridden at fetch time by the lock files that `imgpkg copy` already rewrote. Manual edits are unnecessary and won't take effect anyway. -- Skipping any of the three `imgpkg copy` runs. All three bundles must be present in the customer registry — the service bundle references the repository bundle, which references the package bundle. +- Not Copying the svs bundle during `imgpkg copy` run. The other 2 bundles are pulled by the svs package. ### Part 4 (air-gapped): Relocate $[prodname] private images to a private registry diff --git a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx index 3c6991d061..8b11c322cc 100644 --- a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx @@ -12,12 +12,13 @@ Install $[prodname] as a Supervisor Service on VMware Kubernetes Service (VKS), This guide requires: -- **VM Images**: Ubuntu (and optionally Windows) OVA images registered for a supported TKR version, with the TKR in a healthy state. +- **VM Images**: Ubuntu or Photon (and optionally Windows) OVA images registered for a supported TKR version, with TKR in a healthy state. - **vSphere Resources**: Sufficient capacity to provision guest clusters with at minimum: - 1 control plane node - 3 Linux worker nodes - 1 Windows worker node (optional, if Windows workloads are required) - **OCI Registry**: An OCI-compliant registry configured in vSphere for storing package manifests and container images. +- **Tigera pull secret** for accessing the Tigera container registry (contact the Tigera team). ### Networking @@ -34,62 +35,110 @@ When Windows nodes are enabled, the encapsulation is automatically set to VXLAN ## How to -Setting up $[prodname] on VKS has two main phases: package and publish it as a Supervisor Service (Part 1), then deploy it to a guest cluster (Part 2). Parts 3 and 4 apply only to air-gapped environments. - -### Part 1: Set up the Supervisor Service - -#### Supervisor Service prerequisites +Setting up $[prodname] on VKS has two main phases: set it as a Supervisor Service (Part 1) in vSphere (one time), then deploy it to a guest cluster (Part 2). Parts 3 and 4 apply only to air-gapped environments. -- `helm`, `kbld`, `ytt`, and `imgpkg` binaries available for the host OS architecture in a `./bin` directory. +### Part 1: Set up the Supervisor Service (one time cluster admin setup) -#### Configuration - -| Variable | Description | -|---|---| -| `CALICO_ENTERPRISE_VERSION` | $[prodname] version to package | -| `CALICO_ENTERPRISE_SVS` | (Optional) Custom name for the Supervisor Service | -| `ARTIFACTORY_WRITE` | OCI registry for pushing manifests and images | -| `ARTIFACTORY_READ` | OCI registry from which manifests and images are pulled | -| `REPO_VERSION` | Addon repository version | -| `SERVICE_VERSION` | Addon service version | - -#### Build and publish +1. In the vSphere Supervisor UI: + - Create a new Supervisor Service (or add a new version to an existing one). + - Upload Supervisor service Package and PackageMetadata (example below) resources and complete the UI workflow. + - Register the Supervisor Service and set `registryName` to match the registry name configured in the UI (if needed). -1. Remove previously generated artifacts: - ```bash - make clean - ``` +```yaml +--- +apiVersion: data.packaging.carvel.dev/v1alpha1 +kind: Package +metadata: + name: calico-enterprise-svs.tigera.io.3.23.1 +spec: + refName: calico-enterprise-svs.tigera.io + version: 3.23.1 + template: + spec: + fetch: + - imgpkgBundle: + image: quay.io/tigeradev/calico-enterprise-svs:v3.23.1 #Change this to the registry where the carvel package is getting picked up + template: + - ytt: + paths: + - service + - kapp + - kbld: + paths: + - "-" + - .imgpkg/images.yml + - kbld/ + deploy: + - kapp: {} +--- +apiVersion: data.packaging.carvel.dev/v1alpha1 +kind: PackageMetadata +metadata: + name: calico-enterprise-svs.tigera.io +spec: + displayName: "Calico Enterprise SVS" + shortDescription: "Manages Calico Enterprise Addon Repository and Installation" +``` -1. Build the Carvel package and push to the OCI registry. This creates the Addon package and uploads the artifacts to the OCI registry, also creates the Package and PackageMetadata resources (`calico-svs.yaml`) to be uploaded to the vSphere Supervisor UI: - ```bash - make all - ``` +2. Add the tigera-pull-secret ClusterResourceSet to the Supervisor namespace where auto tigera-pull-secret application is desired -1. In the vSphere Supervisor UI: - - Create a new Supervisor Service (or add a new version to an existing one). - - Upload the generated `calico-svs.yaml` and complete the UI workflow. - - Navigate to the service management page and set `registryName` to match the registry name configured in the UI. +```sh +kubectl apply -n -f .yaml +``` -#### What `make all` does +```yaml +# Apply this on the Supervisor cluster for the namespaces where guest clusters will be provisioned with Calico Enterprise. +# The source Secret containing the manifests CRS will mirror +apiVersion: v1 +kind: Secret +metadata: + name: tigera-pull-secret-bootstrap +type: addons.cluster.x-k8s.io/resource-set # CRS-specific type +stringData: + resources.yaml: | + apiVersion: v1 + kind: Namespace + metadata: + name: tigera-operator + labels: + pod-security.kubernetes.io/enforce: privileged + pod-security.kubernetes.io/audit: privileged + pod-security.kubernetes.io/warn: privileged + --- + apiVersion: v1 + kind: Secret + metadata: + name: tigera-pull-secret + namespace: tigera-operator + type: kubernetes.io/dockerconfigjson + data: + .dockerconfigjson: +--- +# The CRS that selects clusters and points at the source +apiVersion: addons.cluster.x-k8s.io/v1beta2 +kind: ClusterResourceSet +metadata: + name: tigera-pull-secret-bootstrap +spec: + strategy: Reconcile + clusterSelector: + matchLabels: + tigera.io/pull-secret: required + resources: + - name: tigera-pull-secret-bootstrap + kind: Secret -- Packages the Carvel manifests (AddonConfigDefinition, Package, AddonInstall, etc.). -- Pulls and bundles the Tigera Operator Helm chart. -- Resolves and bundles all required container images (`tigera-operator`, `tigera-prometheus-operator`, etc.). -- Renders the `crd.projectcalico.org.v1` chart and bundles the $[prodname] CRDs. As of $[prodname] v3.23.0-2.0 the CRDs ship as a separate chart (no longer inside `tigera-operator`); they are applied alongside the operator with the `kapp.k14s.io/disable-original` annotation so the large CRDs (e.g. `installations.operator.tigera.io`) clear kapp's annotation-size limit — the kapp equivalent of the upstream server-side apply step. -- Includes the `tigera-operator` namespace and `tigera-elasticsearch` StorageClass in the bundle. +``` ### Part 2: Deploy to a guest cluster #### Guest cluster prerequisites -- **Tigera pull secret** for accessing the Tigera container registry (contact the Tigera team). -- **Cluster ResourceSet for tigera pull secret**: The tigera-pull-secret applied as CAPI Cluster ResourceSet in the Supervisor Cluster for all namespaces where guest clusters with $[prodname] will be provisioned. The `./sample-configs/cluster-resourceset-pull-secret.yaml` can be used for this, by replacing the `.dockerconfigjson` value with the value of the private/tigera registry. +- **Cluster ResourceSet for tigera pull secret**: The tigera-pull-secret applied as CAPI Cluster ResourceSet in the Supervisor Cluster for all namespaces where guest clusters with $[prodname] will be provisioned. - **Tigera license file** (contact the Tigera team). - **Dynamic PV provisioning** via a StorageClass, required for Elasticsearch storage. -#### Step 1: Apply the AddonConfig (from Supervisor) - -SSH into the Supervisor machine and apply the AddonConfig resource. A sample configuration is provided in `./sample-configs/addon_config.yaml`. +#### Step 1: Apply the AddonConfig :::info @@ -109,42 +158,261 @@ All $[prodname] features (`logCollector`, `logStorage`, `manager`, `monitor`, `p ::: ```bash -kubectl apply -f ./sample-configs/addon_config.yaml +kubectl apply -f ./addon_config.yaml ``` -#### Step 2: Apply the Cluster Resource (from Supervisor) +```yaml +apiVersion: addons.kubernetes.vmware.com/v1alpha1 +kind: AddonConfig +metadata: + name: + namespace: +spec: + addonConfigDefinitionRef: + name: calico-enterprise.tigera.io.3.23.1 + namespace: vmware-system-vks-public + clusterName: + values: + # Every option from tigera-helm-charts/tigera-operator/values.yaml may be + # set here. Anything not specified falls back to the default declared in + # the AddonConfigDefinition schema. The fields below are illustrative; + installation: + enabled: true + imagePullSecrets: + - name: tigera-pull-secret + calicoWindowsDataplane: + enabled: false + # registry: quay.io/ + # imagePath: "" + # controlPlaneReplicas: 3 + apiServer: + enabled: true + # Calico deploys its own bundled Prometheus Operator by default (enabled: true). + # Set this to false ONLY if you run your own external Prometheus Operator -- in + # that case Calico's Monitor still creates its Prometheus/Alertmanager CRs for + # your operator to reconcile. If you want no Prometheus at all, also set + # monitor.enabled: false. + # tigeraPrometheusOperator: + # enabled: false + logCollector: + enabled: true + logStorage: + enabled: true + nodes: + count: 1 + manager: + enabled: true + monitor: + enabled: true + policyRecommendation: + enabled: true +``` -While still on the Supervisor machine, apply the Cluster resource. A sample is provided in `./sample-configs/cluster.yaml`. +#### Step 2: Apply the Cluster Resource Update the following fields before applying: - `metadata.name` and `metadata.namespace` - `spec.topology.variables[].value.cniRef` (name and namespace of the $[prodname] addon) -- Add the label `tigera.io/pull-secret: required` to `metadata.labels` so that tigera-pull-secret will be created for the cluster. +- Add the label `tigera.io/pull-secret: required` to `metadata.labels` so that tigera-pull-secret will be created automatically for the cluster. - Node pool configuration (VM class, storage class, replicas) as needed for your environment ```bash -kubectl apply -f ./sample-configs/cluster.yaml +kubectl apply -f ./cluster.yaml ``` -#### Step 3: Retrieve the Guest Cluster Kubeconfig - -```bash -kubectl -n get secret -o jsonpath='{.data.value}' | base64 --decode > kubeconfig +```yaml +apiVersion: cluster.x-k8s.io/v1beta2 +kind: Cluster +metadata: + name: + namespace: + labels: + tigera.io/pull-secret: required +spec: + clusterNetwork: + pods: + cidrBlocks: + - 100.96.0.0/11 + serviceDomain: cluster.local + services: + cidrBlocks: + - 100.64.0.0/13 + topology: + classRef: + name: builtin-generic-v3.7.0 + namespace: vmware-system-vks-public + controlPlane: + metadata: + annotations: + run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu,os-version=24.04 + replicas: 1 + variables: + - name: node + value: + firewall: + inboundRules: + # BGP + - protocol: tcp + fromPort: 179 + toPort: 179 + # VXLAN + - protocol: udp + fromPort: 4789 + toPort: 4789 + # Typha + - protocol: tcp + fromPort: 5473 + toPort: 5473 + # Wireguard + - protocol: udp + fromPort: 51820 + toPort: 51820 + # Elasticsearch + - protocol: tcp + fromPort: 9200 + toPort: 9200 + # Elasticsearch transport + - protocol: tcp + fromPort: 9300 + toPort: 9300 + # Tigera Manager UI + - protocol: tcp + fromPort: 9443 + toPort: 9443 + # Prometheus + - protocol: tcp + fromPort: 9090 + toPort: 9090 + # Linseed + - protocol: tcp + fromPort: 8444 + toPort: 8444 + - name: bootstrapAddons + value: + cniRef: + name: calico-enterprise + namespace: svc-calico-enterprise-svs-domain-c10 + - name: vmClass + value: best-effort-xlarge + - name: storageClass + value: vsan-esa-default-policy-raid5 + - name: kubernetes + value: + certificateRotation: + enabled: true + renewalDaysBeforeExpiry: 90 + - name: vsphereOptions + value: + persistentVolumes: + defaultStorageClass: vsan-esa-default-policy-raid5 + version: v1.35.0+vmware.2-vkr.4 + workers: + machineDeployments: + - class: node-pool + metadata: + annotations: + run.tanzu.vmware.com/resolve-os-image: os-name=ubuntu,os-version=24.04 + name: node-pool-1 + replicas: 3 + variables: + overrides: + - name: vmClass + value: best-effort-large + - name: node + value: + firewall: + inboundRules: + # BGP + - protocol: tcp + fromPort: 179 + toPort: 179 + # VXLAN + - protocol: udp + fromPort: 4789 + toPort: 4789 + # Typha + - protocol: tcp + fromPort: 5473 + toPort: 5473 + # Wireguard + - protocol: udp + fromPort: 51820 + toPort: 51820 + # Elasticsearch + - protocol: tcp + fromPort: 9200 + toPort: 9200 + # Elasticsearch transport + - protocol: tcp + fromPort: 9300 + toPort: 9300 + # Tigera Manager UI + - protocol: tcp + fromPort: 9443 + toPort: 9443 + # Prometheus + - protocol: tcp + fromPort: 9090 + toPort: 9090 + # Linseed + - protocol: tcp + fromPort: 8444 + toPort: 8444 + - class: node-pool + metadata: + annotations: + run.tanzu.vmware.com/resolve-os-image: os-name=windows,os-version=2022 + name: win-node-pool-1 + # if Windows nodes needed + replicas: 0 + variables: + overrides: + - name: vmClass + value: best-effort-large + - name: node + value: + firewall: + inboundRules: + # BGP + - protocol: tcp + fromPort: 179 + toPort: 179 + # VXLAN + - protocol: udp + fromPort: 4789 + toPort: 4789 + # Typha + - protocol: tcp + fromPort: 5473 + toPort: 5473 + # Wireguard + - protocol: udp + fromPort: 51820 + toPort: 51820 + # Elasticsearch + - protocol: tcp + fromPort: 9200 + toPort: 9200 + # Elasticsearch transport + - protocol: tcp + fromPort: 9300 + toPort: 9300 + # Tigera Manager UI + - protocol: tcp + fromPort: 9443 + toPort: 9443 + # Prometheus + - protocol: tcp + fromPort: 9090 + toPort: 9090 + # Linseed + - protocol: tcp + fromPort: 8444 + toPort: 8444 ``` -:::note - -From this point onwards, Supervisor SSH access is no longer required. All remaining steps can be performed from any machine with access to the guest cluster's API server. - -::: - -#### Step 4: Create the Tigera Pull Secret +#### Step 3: Retrieve the Guest Cluster Kubeconfig -```bash -kubectl create secret generic tigera-pull-secret \ - --type=kubernetes.io/dockerconfigjson -n tigera-operator \ - --from-file=.dockerconfigjson= -``` #### Step 5: [Windows only] Apply IPAM Configuration @@ -157,7 +425,19 @@ Status can be checked using `kubectl get tigerastatus`. ::: ```bash -kubectl apply -f ./sample-configs/ipam_configuration.yaml +kubectl apply -f ./ipam_configuration.yaml +``` + +```yaml +# Required for Windows nodes - ensures IPAM blocks are strictly +# affinitive to their assigned nodes. +# Apply this after calico-node pods are running. +apiVersion: projectcalico.org/v3 +kind: IPAMConfiguration +metadata: + name: default +spec: + strictAffinity: true ``` #### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap @@ -230,82 +510,60 @@ kubectl get pods -A | grep -E 'tigera|calico' Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. -### Part 3 (air-gapped): Relocate bundles to a private registry +### Part 3: Airgapped package registry -If the published bundles need to be moved into a customer's private OCI registry (for example, an air-gapped install or a customer who cannot pull from your registry), follow these steps. `make all` produces three bundles and one customer-facing YAML; relocation moves the bundles and updates one reference in the YAML. +If the published bundles need to be moved into a private OCI registry (for example, an air-gapped install), follow these steps. #### Air-gapped relocation prerequisites - `imgpkg` and `yq` available locally. -- Network access to the source registry (or a tarball produced from it) and to the customer registry. -- Credentials to push to the customer registry (`imgpkg` will prompt or use the local docker config). - -#### Bundles to relocate +- Network access to the source registry (or a tarball produced from it) and to the private registry. +- Credentials to push to the private registry (`imgpkg` will prompt or use the local docker config). -The three bundles correspond to the three Make targets and use the registry paths configured in the Makefile: - -| Bundle | Source path | Default tag | -|---|---|---| -| Package bundle ($[prodname] chart + container images) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}` | `v${CALICO_ENTERPRISE_VERSION}` | -| Repository bundle (Package + PackageMetadata for the package bundle) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}-repo` | `v${REPO_VERSION}` | -| Service bundle (AddonInstall, AddonRepository, AddonRepositoryInstall) | `${ARTIFACTORY_READ}/${CALICO_ENTERPRISE_SVS}-svs` | `v${SERVICE_VERSION}` | - -Cross-bundle references (service → repository → package → container images) are stored as digests in each bundle's `.imgpkg/images.yml` lock file. `imgpkg copy` rewrites those lock files automatically when relocating, so no manifests inside the bundles need manual editing. - -#### Step 1: Copy the bundles to a tarball (source side) +#### Step 1: Copy the Supervisor Service (SVS) bundles to a tarball (source side) Run on a host that has access to the source registry: ```bash -imgpkg copy --bundle /calico-enterprise:v3.22.2 --to-tar calico-package.tar -imgpkg copy --bundle /calico-enterprise-repo:v3.6.1 --to-tar calico-repo.tar -imgpkg copy --bundle /calico-enterprise-svs:v3.6.1 --to-tar calico-svs.tar +imgpkg copy --bundle /calico-enterprise-svs:v3.23.1 --to-tar calico-enterprise-svs.tar ``` -Transfer the three tarballs along with `calico-svs.yaml` to the destination environment. - :::tip -Direct registry-to-registry relocation is also supported. Replace `--to-tar .tar` with `--to-repo /` and skip Step 2. +Direct registry-to-registry relocation is also supported. Replace `--to-tar .tar` with `--to-repo /` and skip Step 2. ::: -#### Step 2: Push the bundles into the customer registry (destination side) +#### Step 2: Push the SVS bundle into the private registry (destination side) ```bash -imgpkg copy --tar calico-package.tar --to-repo /calico-enterprise -imgpkg copy --tar calico-repo.tar --to-repo /calico-enterprise-repo -imgpkg copy --tar calico-svs.tar --to-repo /calico-enterprise-svs +imgpkg copy --tar calico-enterprise-svs.tar --to-repo /calico-enterprise-svs ``` -After each `imgpkg copy --to-repo` finishes, the relocated bundle's `images.yml` references the customer registry by digest. Container images and cross-bundle references resolve to the customer registry automatically at deploy time. - -#### Step 3: Patch `calico-svs.yaml` +#### Step 3: Patch Package Resource in `calico-enterprise-svs.yaml` -The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-svs.yaml`. Update it to the customer registry: +The only registry reference outside the bundles is the service-bundle pointer in the `Package` CR inside `calico-enterprise-svs.yaml`. This needs to be updated to the private registry: ```bash yq eval ' (select(.kind == "Package") | .spec.template.spec.fetch[0].imgpkgBundle.image) - = "/calico-enterprise-svs:v3.6.1" + = "/calico-enterprise-svs:v3.23.1" ' -i calico-svs.yaml ``` :::tip -**Content-pin to digest (optional):** `imgpkg copy --to-repo` prints the destination digest of the relocated bundle on its last line. Substitute `:v3.6.1` with `@sha256:` in the `yq` command above to make the reference immutable. +**Content-pin to digest (optional):** `imgpkg copy --to-repo` prints the destination digest of the relocated bundle on its last line. Substituting `:v3.23.1` with `@sha256:` in the `yq` command above will make the reference immutable. ::: -#### Step 4: Upload `calico-svs.yaml` to the Supervisor +#### Step 4: Upload `calico-enterprise-svs.yaml` i.e. Package and PackageMetadata resource to the Supervisor -Follow the original [Part 1, Build and publish](#build-and-publish) workflow with the patched `calico-svs.yaml`. From here, [Part 2](#part-2-deploy-to-a-guest-cluster) proceeds unchanged. #### What does *not* work - `docker pull` / `docker push` on individual images. Re-tagging can change manifest digests, and the bundles' `images.yml` lock files are not rewritten — kapp-controller fetches will fail with `manifest unknown`. -- Editing `addon_repository.yaml` or `package.yaml` directly. Those files live inside the relocated bundles; their image references are overridden at fetch time by the lock files that `imgpkg copy` already rewrote. Manual edits are unnecessary and won't take effect anyway. -- Skipping any of the three `imgpkg copy` runs. All three bundles must be present in the customer registry — the service bundle references the repository bundle, which references the package bundle. +- Not Copying the svs bundle during `imgpkg copy` run. The other 2 bundles are pulled by the svs package. ### Part 4 (air-gapped): Relocate $[prodname] private images to a private registry From c94c4c1519ff426b60df0194eea837a25aa039a4 Mon Sep 17 00:00:00 2001 From: sujeet-kr Date: Thu, 2 Jul 2026 22:39:19 -0400 Subject: [PATCH 6/6] update with changes requested by copilot --- .../install-on-clusters/vks.mdx | 24 +++++++++++-------- .../install-on-clusters/vks.mdx | 24 +++++++++++-------- 2 files changed, 28 insertions(+), 20 deletions(-) diff --git a/calico-enterprise/getting-started/install-on-clusters/vks.mdx b/calico-enterprise/getting-started/install-on-clusters/vks.mdx index 8b11c322cc..d49a91e2b9 100644 --- a/calico-enterprise/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise/getting-started/install-on-clusters/vks.mdx @@ -29,7 +29,7 @@ This guide requires: :::note -When Windows nodes are enabled, the encapsulation is automatically set to VXLAN and BGP is disabled. This is handled internally when `calicoWindowsDataplane` is set to `true` in the AddonConfig. +When Windows nodes are enabled, the encapsulation is automatically set to VXLAN and BGP is disabled. This is handled internally when `calicoWindowsDataplane.enabled` is set to `true` in the AddonConfig. ::: @@ -88,7 +88,7 @@ kubectl apply -n -f .yaml ```yaml # Apply this on the Supervisor cluster for the namespaces where guest clusters will be provisioned with Calico Enterprise. -# The source Secret containing the manifests CRS will mirror +# Source Secret containing the CRS manifests apiVersion: v1 kind: Secret metadata: @@ -413,8 +413,9 @@ spec: #### Step 3: Retrieve the Guest Cluster Kubeconfig +Can be copied from the Supervisor. -#### Step 5: [Windows only] Apply IPAM Configuration +#### Step 4: [Windows only] Apply IPAM Configuration Apply the strict affinity IPAM configuration, after Calico `apiserver` is `Available`. @@ -440,9 +441,12 @@ spec: strictAffinity: true ``` -#### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap +#### Step 5: [Windows only] Apply the kubernetes-services-endpoint ConfigMap ```bash + export APISERVER_ADDR= + export APISERVER_PORT= + kubectl apply -f - << EOF kind: ConfigMap apiVersion: v1 @@ -461,7 +465,7 @@ The `APISERVER_ADDR` and `APISERVER_PORT` can be retrieved via `kubectl get endp ::: -#### Step 7: [Windows only] Apply the kube-proxy on Windows nodes +#### Step 6: [Windows only] Apply the kube-proxy on Windows nodes Apply the Windows kube-proxy: @@ -483,7 +487,7 @@ If the version returned is of the form `v1.35.2+vmware.1`, strip the `+vmware.1` ::: -#### Step 8: Wait for Core Components +#### Step 7: Wait for Core Components Monitor the Tigera status until `calico` and `apiserver` report as `Available`: @@ -493,13 +497,13 @@ kubectl get tigerastatus At this point, all nodes should be in `Ready` state. -#### Step 9: Apply the Tigera License +#### Step 8: Apply the Tigera License ```bash kubectl apply -f tigera-license.yaml ``` -#### Step 10: Verify the Installation +#### Step 9: Verify the Installation Wait for all remaining components to become available: @@ -510,7 +514,7 @@ kubectl get pods -A | grep -E 'tigera|calico' Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. -### Part 3: Airgapped package registry +### Part 3: Air-gapped package registry If the published bundles need to be moved into a private OCI registry (for example, an air-gapped install), follow these steps. @@ -548,7 +552,7 @@ The only registry reference outside the bundles is the service-bundle pointer in yq eval ' (select(.kind == "Package") | .spec.template.spec.fetch[0].imgpkgBundle.image) = "/calico-enterprise-svs:v3.23.1" -' -i calico-svs.yaml +' -i calico-enterprise-svs.yaml ``` :::tip diff --git a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx index 8b11c322cc..d49a91e2b9 100644 --- a/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx +++ b/calico-enterprise_versioned_docs/version-3.23-2/getting-started/install-on-clusters/vks.mdx @@ -29,7 +29,7 @@ This guide requires: :::note -When Windows nodes are enabled, the encapsulation is automatically set to VXLAN and BGP is disabled. This is handled internally when `calicoWindowsDataplane` is set to `true` in the AddonConfig. +When Windows nodes are enabled, the encapsulation is automatically set to VXLAN and BGP is disabled. This is handled internally when `calicoWindowsDataplane.enabled` is set to `true` in the AddonConfig. ::: @@ -88,7 +88,7 @@ kubectl apply -n -f .yaml ```yaml # Apply this on the Supervisor cluster for the namespaces where guest clusters will be provisioned with Calico Enterprise. -# The source Secret containing the manifests CRS will mirror +# Source Secret containing the CRS manifests apiVersion: v1 kind: Secret metadata: @@ -413,8 +413,9 @@ spec: #### Step 3: Retrieve the Guest Cluster Kubeconfig +Can be copied from the Supervisor. -#### Step 5: [Windows only] Apply IPAM Configuration +#### Step 4: [Windows only] Apply IPAM Configuration Apply the strict affinity IPAM configuration, after Calico `apiserver` is `Available`. @@ -440,9 +441,12 @@ spec: strictAffinity: true ``` -#### Step 6: [Windows only] Apply the kubernetes-services-endpoint ConfigMap +#### Step 5: [Windows only] Apply the kubernetes-services-endpoint ConfigMap ```bash + export APISERVER_ADDR= + export APISERVER_PORT= + kubectl apply -f - << EOF kind: ConfigMap apiVersion: v1 @@ -461,7 +465,7 @@ The `APISERVER_ADDR` and `APISERVER_PORT` can be retrieved via `kubectl get endp ::: -#### Step 7: [Windows only] Apply the kube-proxy on Windows nodes +#### Step 6: [Windows only] Apply the kube-proxy on Windows nodes Apply the Windows kube-proxy: @@ -483,7 +487,7 @@ If the version returned is of the form `v1.35.2+vmware.1`, strip the `+vmware.1` ::: -#### Step 8: Wait for Core Components +#### Step 7: Wait for Core Components Monitor the Tigera status until `calico` and `apiserver` report as `Available`: @@ -493,13 +497,13 @@ kubectl get tigerastatus At this point, all nodes should be in `Ready` state. -#### Step 9: Apply the Tigera License +#### Step 8: Apply the Tigera License ```bash kubectl apply -f tigera-license.yaml ``` -#### Step 10: Verify the Installation +#### Step 9: Verify the Installation Wait for all remaining components to become available: @@ -510,7 +514,7 @@ kubectl get pods -A | grep -E 'tigera|calico' Once all items in `tigerastatus` show `Available`, the installation is complete. You can now deploy workloads to validate networking. -### Part 3: Airgapped package registry +### Part 3: Air-gapped package registry If the published bundles need to be moved into a private OCI registry (for example, an air-gapped install), follow these steps. @@ -548,7 +552,7 @@ The only registry reference outside the bundles is the service-bundle pointer in yq eval ' (select(.kind == "Package") | .spec.template.spec.fetch[0].imgpkgBundle.image) = "/calico-enterprise-svs:v3.23.1" -' -i calico-svs.yaml +' -i calico-enterprise-svs.yaml ``` :::tip