diff --git a/telco-core/configuration/reference-crs/optional/cert-manager/README.md b/telco-core/configuration/reference-crs/optional/cert-manager/README.md index 039769cf1..e102d5524 100644 --- a/telco-core/configuration/reference-crs/optional/cert-manager/README.md +++ b/telco-core/configuration/reference-crs/optional/cert-manager/README.md @@ -16,8 +16,9 @@ Cert-manager automates the management and issuance of TLS certificates from vari - `certManagerOperatorgroup.yaml` - Creates the OperatorGroup for cert-manager - `certManagerSubscription.yaml` - Installs the OpenShift cert-manager operator -### Certificate Issuer -- `certManagerClusterIssuer.yaml` - Configures an ACME ClusterIssuer using Let's Encrypt with DNS-01 challenge +### Certificate Issuers + +- `certManagerClusterIssuer.yaml` - **ACME issuer** with DNS-01 challenge (reference recommendation) ### Certificate Resources - `apiServerCertificate.yaml` - Creates a certificate for the API Server endpoint @@ -34,13 +35,22 @@ Before applying these configurations, you must customize the following: 1. **ClusterIssuer** (`certManagerClusterIssuer.yaml`): - Update `email` with your contact email - Configure the appropriate DNS provider for DNS-01 challenge (example shows Route53) - - Add necessary credentials for your DNS provider + - Reference pre-created Secrets for DNS provider credentials via `secretRef` — do not commit credentials in manifests + + > **Note:** Other issuer types (e.g., CA issuer for disconnected environments with existing PKI) are allowable. + > Users may configure their own ClusterIssuer; currently only ACME issuer is provided in the reference. 2. **Certificates** (`apiServerCertificate.yaml` and `ingressCertificate.yaml`): - Update `commonName` and `dnsNames` to match your cluster's domain - Example: Replace `api.example.com` with your actual API endpoint - Example: Replace `*.apps.example.com` with your actual wildcard domain + > **Important:** The reference configuration uses ECDSA P-256 for Certificate resources, which is the recommended algorithm for TLS 1.3. + > While RSA certificates are still supported for authentication in TLS 1.3, RSA key exchange was removed and RSA does not provide Forward Secrecy. + > + > **Note:** Lifecycle-agent currently has limited support for ECDSA certificates (being addressed in lifecycle-agent PR #7610). + > For testing purposes, QE may temporarily use RSA certificates, but production deployments should use ECDSA. + 3. **APIServer Configuration** (`apiServerConfig.yaml`): - Update the `names` field to match your API Server FQDN @@ -61,9 +71,75 @@ After applying these configurations, verify that: - API Server is using the certificate: Test HTTPS connection to API endpoint - Ingress is using the certificate: Test HTTPS connection to any route +## Important: Kubeconfig Trust After API Server Cert Replacement + +> **Note:** When using a non-publicly-trusted issuer, you must complete this kubeconfig update +> *before* applying the APIServer configuration (step 6 in the deployment order above). +> Applying the APIServer configuration first will lock you out. + +> **Warning:** When cert-manager replaces the API server certificate with one signed by a non-publicly-trusted CA, +> existing kubeconfig files become invalid. The embedded `certificate-authority-data` still references +> the original cluster CA and cannot verify the new certificate. All `oc` and API client commands +> will fail with `x509: certificate signed by unknown authority`. + +### Updating kubeconfig + +1. Extract the new root CA certificate: + ```bash + oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/root-ca.crt + ``` + +2. Update your kubeconfig to trust the new CA: + ```bash + oc config set-cluster $(oc config current-context | cut -d/ -f2) \ + --certificate-authority=/tmp/root-ca.crt --embed-certs + ``` + +3. Verify connectivity: + ```bash + oc cluster-info + ``` + +### Best practice for PKI environments + +Generate a root CA once and use it as the root for your PKI (the ACME issuer or CA issuer your clusters will use). Add this root CA to your workstation's system trust store so all certificates issued from it are automatically trusted by workstation tools and browsers. Note that adding the CA to your system trust store does not automatically update existing kubeconfigs with embedded certificate data — see the "Clearing kubeconfig certificate data" section below. + +#### Adding CA to system trust store + +**Red Hat/Fedora/CentOS:** +```bash +sudo cp /tmp/root-ca.crt /etc/pki/ca-trust/source/anchors/ +sudo update-ca-trust +``` + +**Debian/Ubuntu:** +```bash +sudo cp /tmp/root-ca.crt /usr/local/share/ca-certificates/root-ca.crt +sudo update-ca-certificates +``` + +**macOS:** +```bash +sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /tmp/root-ca.crt +``` + +#### Clearing kubeconfig certificate data + +**Important:** `oc` and `kubectl` do NOT fall back to the OS trust store if your kubeconfig contains embedded certificate data (`certificate-authority-data` field). You must clear this field to use the system trust store: + +```bash +# Remove embedded certificate data from current cluster +CLUSTER_NAME=$(oc config view --minify -o jsonpath='{.clusters[0].name}') +oc config unset "clusters.${CLUSTER_NAME}.certificate-authority-data" + +# Verify the OS trust store is now used +oc cluster-info +``` + +After completing these steps, all future clusters using certificates signed by your root CA will be automatically trusted without per-cluster kubeconfig updates. + ## References - [OpenShift Cert-Manager Operator Documentation](https://docs.openshift.com/container-platform/latest/security/cert_manager_operator/index.html) - [Cert-Manager Documentation](https://cert-manager.io/docs/) - [ACME DNS-01 Challenge Configuration](https://cert-manager.io/docs/configuration/acme/dns01/) - diff --git a/telco-core/install/example-standard-clusterinstance.yaml b/telco-core/install/example-standard-clusterinstance.yaml index 5cd9037cb..b7bb9cbe4 100644 --- a/telco-core/install/example-standard-clusterinstance.yaml +++ b/telco-core/install/example-standard-clusterinstance.yaml @@ -31,6 +31,10 @@ spec: common: "core" version: "4.22" region: "zone-1" + # Uncomment to apply KlusterletConfig for cert-manager hub-spoke trust (optional) + # extraAnnotations: + # ManagedCluster: + # agent.open-cluster-management.io/klusterlet-config: "cert-manager-ca-config" clusterNetwork: - cidr: 10.128.0.0/14 hostPrefix: 23 diff --git a/telco-hub/configuration/reference-crs-kube-compare/compare_ignore b/telco-hub/configuration/reference-crs-kube-compare/compare_ignore index 3256e41aa..969abe05e 100644 --- a/telco-hub/configuration/reference-crs-kube-compare/compare_ignore +++ b/telco-hub/configuration/reference-crs-kube-compare/compare_ignore @@ -33,6 +33,10 @@ required/gitops/addPluginsPolicy.yaml # not include the full policy content due to policy templating. required/acm/observabilityRoutePolicy.yaml +# cert-manager config CRs (user-specific, and policies with hub-side templating) +optional/cert-manager/certManagerRootCAExpirationPolicy.yaml +optional/cert-manager/kustomization.yaml + required/gitops/extra-manifests-policy.yaml # ArgoCD files kustomization.yaml diff --git a/telco-hub/configuration/reference-crs-kube-compare/default_value.yaml b/telco-hub/configuration/reference-crs-kube-compare/default_value.yaml index 6fa23b32b..4bd25215e 100644 --- a/telco-hub/configuration/reference-crs-kube-compare/default_value.yaml +++ b/telco-hub/configuration/reference-crs-kube-compare/default_value.yaml @@ -479,6 +479,12 @@ optional_cert_manager_certManagerCertificatePolicy: include: - openshift-ingress - openshift-config + - cert-manager + +optional_cert_manager_certManagerHubCAConfigMap: +- data: + ca-bundle.crt: | + optional_cert_manager_certManagerSubscription: - spec: diff --git a/telco-hub/configuration/reference-crs-kube-compare/metadata.yaml b/telco-hub/configuration/reference-crs-kube-compare/metadata.yaml index 55177d742..2da7d8474 100644 --- a/telco-hub/configuration/reference-crs-kube-compare/metadata.yaml +++ b/telco-hub/configuration/reference-crs-kube-compare/metadata.yaml @@ -31,6 +31,16 @@ parts: allOrNoneOf: - path: optional/cert-manager/apiServerCertificate.yaml - path: optional/cert-manager/apiServerConfig.yaml + - name: cert-manager-hub-spoke-trust + description: |- + Hub-spoke CA trust distribution via KlusterletConfig + allOrNoneOf: + - path: optional/cert-manager/certManagerHubCAConfigMap.yaml + config: + ignore-unspecified-fields: true + - path: optional/cert-manager/certManagerKlusterletConfig.yaml + config: + ignore-unspecified-fields: true - name: cert-manager-monitoring description: |- Certificate monitoring policy for RHACM @@ -38,6 +48,17 @@ parts: - path: optional/cert-manager/certManagerCertificatePolicy.yaml - path: optional/cert-manager/certManagerCertificatePolicyPlacement.yaml - path: optional/cert-manager/certManagerCertificatePolicyPlacementBinding.yaml + - name: cert-manager-root-ca-monitoring + description: |- + Root CA expiration monitoring via PrometheusRule + allOrNoneOf: + - path: optional/cert-manager/certManagerRootCAExpirationPolicy.yaml + config: + ignore-unspecified-fields: true + fieldsToOmitRefs: + - templates + - path: optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml + - path: optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml - name: optional-storage components: - name: local-storage-operator diff --git a/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerCertificatePolicy.yaml b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerCertificatePolicy.yaml index df50e2185..e42c7dc16 100644 --- a/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerCertificatePolicy.yaml +++ b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerCertificatePolicy.yaml @@ -19,7 +19,7 @@ spec: spec: minimumDuration: {{ .spec.minimumDuration | default "720h" }} namespaceSelector: - include:{{- template "unorderedListAllowExtra" (list .spec.namespaceSelector.include (list "openshift-ingress" "openshift-config") ) }} + include:{{- template "unorderedListAllowExtra" (list .spec.namespaceSelector.include (list "openshift-ingress" "openshift-config" "cert-manager") ) }} remediationAction: inform severity: low remediationAction: inform diff --git a/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerHubCAConfigMap.yaml b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerHubCAConfigMap.yaml new file mode 100644 index 000000000..72099bdea --- /dev/null +++ b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerHubCAConfigMap.yaml @@ -0,0 +1,18 @@ +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: cert-manager-hub-ca-bundle + namespace: multicluster-engine + labels: + import.open-cluster-management.io/ca-bundle: "true" + annotations: + argocd.argoproj.io/sync-wave: "-29" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +data: +{{- if .data }} + # Populate with the root CA PEM used by cert-manager to sign hub API/ingress certs. + # Extract from the hub: oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d + ca-bundle.crt: | +{{ index .data "ca-bundle.crt" | trimSuffix "\n" | indent 4 }} +{{- end }} diff --git a/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerKlusterletConfig.yaml b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerKlusterletConfig.yaml new file mode 100644 index 000000000..09cbbdccd --- /dev/null +++ b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerKlusterletConfig.yaml @@ -0,0 +1,16 @@ +--- +apiVersion: config.open-cluster-management.io/v1alpha1 +kind: KlusterletConfig +metadata: + name: {{ .metadata.name | default "cert-manager-ca-config" }} + annotations: + argocd.argoproj.io/sync-wave: "-29" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +spec: + hubKubeAPIServerConfig: + serverVerificationStrategy: UseCustomCABundles + trustedCABundles: + - name: cert-manager-hub-ca-bundle + caBundle: + namespace: multicluster-engine + name: cert-manager-hub-ca-bundle diff --git a/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml new file mode 100644 index 000000000..5a1e716b6 --- /dev/null +++ b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml @@ -0,0 +1,18 @@ +--- +apiVersion: cluster.open-cluster-management.io/v1beta1 +kind: Placement +metadata: + name: root-ca-expiration-placement + namespace: default + annotations: + argocd.argoproj.io/sync-wave: "-11" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +spec: + predicates: + - requiredClusterSelector: + labelSelector: + matchExpressions: + - key: local-cluster + operator: In + values: + - "true" diff --git a/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml new file mode 100644 index 000000000..f9ec1fd7a --- /dev/null +++ b/telco-hub/configuration/reference-crs-kube-compare/optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml @@ -0,0 +1,17 @@ +--- +apiVersion: policy.open-cluster-management.io/v1 +kind: PlacementBinding +metadata: + name: root-ca-expiration-placementbinding + namespace: default + annotations: + argocd.argoproj.io/sync-wave: "-11" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +placementRef: + name: root-ca-expiration-placement + apiGroup: cluster.open-cluster-management.io + kind: Placement +subjects: + - name: root-ca-expiration-monitor + apiGroup: policy.open-cluster-management.io + kind: Policy diff --git a/telco-hub/configuration/reference-crs/optional/cert-manager/README.md b/telco-hub/configuration/reference-crs/optional/cert-manager/README.md index 479abe7f0..46ebfb8e1 100644 --- a/telco-hub/configuration/reference-crs/optional/cert-manager/README.md +++ b/telco-hub/configuration/reference-crs/optional/cert-manager/README.md @@ -16,8 +16,9 @@ Cert-manager automates the management and issuance of TLS certificates from vari - `certManagerOperatorgroup.yaml` - Creates the OperatorGroup for cert-manager - `certManagerSubscription.yaml` - Installs the OpenShift cert-manager operator -### Certificate Issuer -- `certManagerClusterIssuer.yaml` - Configures an ACME ClusterIssuer using Let's Encrypt with DNS-01 challenge +### Certificate Issuers + +- `certManagerClusterIssuer.yaml` - **ACME issuer** with DNS-01 challenge (reference recommendation) ### Certificate Resources - `apiServerCertificate.yaml` - Creates a certificate for the API Server endpoint @@ -34,13 +35,22 @@ Before applying these configurations, you must customize the following: 1. **ClusterIssuer** (`certManagerClusterIssuer.yaml`): - Update `email` with your contact email - Configure the appropriate DNS provider for DNS-01 challenge (example shows Route53) - - Add necessary credentials for your DNS provider + - Reference pre-created Secrets for DNS provider credentials via `secretRef` — do not commit credentials in manifests + + > **Note:** Other issuer types (e.g., CA issuer for disconnected environments with existing PKI) are allowable. + > Users may configure their own ClusterIssuer; currently only ACME issuer is provided in the reference. 2. **Certificates** (`apiServerCertificate.yaml` and `ingressCertificate.yaml`): - Update `commonName` and `dnsNames` to match your cluster's domain - Example: Replace `api.example.com` with your actual API endpoint - Example: Replace `*.apps.example.com` with your actual wildcard domain + > **Important:** The reference configuration uses ECDSA P-256 for Certificate resources, which is the recommended algorithm for TLS 1.3. + > While RSA certificates are still supported for authentication in TLS 1.3, RSA key exchange was removed and RSA does not provide Forward Secrecy. + > + > **Note:** Lifecycle-agent currently has limited support for ECDSA certificates (being addressed in lifecycle-agent PR #7610). + > For testing purposes, QE may temporarily use RSA certificates, but production deployments should use ECDSA. + 3. **APIServer Configuration** (`apiServerConfig.yaml`): - Update the `names` field to match your API Server FQDN @@ -49,8 +59,9 @@ Before applying these configurations, you must customize the following: 1. Deploy operator installation files (NS, OperatorGroup, Subscription) 2. Wait for operator to be ready 3. Deploy the ClusterIssuer -4. Wait for certificates to be issued and secrets created -5. Apply the APIServer and IngressController configurations +4. Deploy the Certificate resources +5. Wait for certificates to be issued and secrets created +6. Apply the APIServer and IngressController configurations ## Certificate Verification @@ -60,9 +71,156 @@ After applying these configurations, verify that: - API Server is using the certificate: Test HTTPS connection to API endpoint - Ingress is using the certificate: Test HTTPS connection to any route +## Important: Kubeconfig Trust After API Server Cert Replacement + +> **Note:** When using a non-publicly-trusted issuer, you must complete this kubeconfig update +> *before* applying the APIServer configuration (step 6 in the deployment order above). +> Applying the APIServer configuration first will lock you out. + +> **Warning:** When cert-manager replaces the API server certificate with one signed by a non-publicly-trusted CA, +> existing kubeconfig files become invalid. The embedded `certificate-authority-data` still references +> the original cluster CA and cannot verify the new certificate. All `oc` and API client commands +> will fail with `x509: certificate signed by unknown authority`. + +### Updating kubeconfig + +1. Extract the new root CA certificate: + ```bash + oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/root-ca.crt + ``` + +2. Update your kubeconfig to trust the new CA: + ```bash + oc config set-cluster $(oc config current-context | cut -d/ -f2) \ + --certificate-authority=/tmp/root-ca.crt --embed-certs + ``` + +3. Verify connectivity: + ```bash + oc cluster-info + ``` + +### Best practice for PKI environments + +Generate a root CA once and use it as the root for your PKI (the ACME issuer or CA issuer your clusters will use). Add this root CA to your workstation's system trust store so all certificates issued from it are automatically trusted by workstation tools and browsers. Note that adding the CA to your system trust store does not automatically update existing kubeconfigs with embedded certificate data — see the "Clearing kubeconfig certificate data" section below. + +#### Adding CA to system trust store + +**Red Hat/Fedora/CentOS:** +```bash +sudo cp /tmp/root-ca.crt /etc/pki/ca-trust/source/anchors/ +sudo update-ca-trust +``` + +**Debian/Ubuntu:** +```bash +sudo cp /tmp/root-ca.crt /usr/local/share/ca-certificates/root-ca.crt +sudo update-ca-certificates +``` + +**macOS:** +```bash +sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /tmp/root-ca.crt +``` + +#### Clearing kubeconfig certificate data + +**Important:** `oc` and `kubectl` do NOT fall back to the OS trust store if your kubeconfig contains embedded certificate data (`certificate-authority-data` field). You must clear this field to use the system trust store: + +```bash +# Remove embedded certificate data from current cluster +CLUSTER_NAME=$(oc config view --minify -o jsonpath='{.clusters[0].name}') +oc config unset "clusters.${CLUSTER_NAME}.certificate-authority-data" + +# Verify the OS trust store is now used +oc cluster-info +``` + +After completing these steps, all future clusters using certificates signed by your root CA will be automatically trusted without per-cluster kubeconfig updates. + +## Hub-Spoke Trust with ACM + +When cert-manager issues certificates for the hub's API server and ingress, managed spokes must trust the cert-manager root CA to maintain connectivity. The reference configuration includes a `KlusterletConfig` and CA ConfigMap to distribute the root CA to spokes automatically. + +> **Requirements:** ACM 2.11 or later. The `serverVerificationStrategy` feature used by the KlusterletConfig was introduced in ACM 2.11 and is not available in earlier versions. + +### Hub-spoke trust files + +- `certManagerHubCAConfigMap.yaml` — ConfigMap in `multicluster-engine` namespace containing the cert-manager root CA, labeled for the import controller +- `certManagerKlusterletConfig.yaml` — KlusterletConfig that switches spoke CA verification from auto-detected leaf cert to the custom root CA bundle + +### Why this is needed + +By default, ACM embeds the hub's leaf serving cert (`CA:FALSE`) in the klusterlet bootstrap kubeconfig. This means every cert rotation requires a ManifestWork update, and a full CA replacement breaks all spokes immediately. The `KlusterletConfig` with `UseCustomCABundles` replaces the leaf cert with the root CA (`CA:TRUE`), so any certificate signed by that root — current or rotated — is automatically trusted. + +### Applying the KlusterletConfig to managed clusters + +#### ZTP deployment (recommended) + +For fully automated ZTP deployments, configure the ManagedCluster annotation via ClusterInstance `extraAnnotations`: + +```yaml +apiVersion: siteconfig.open-cluster-management.io/v1alpha1 +kind: ClusterInstance +metadata: + name: spoke-cluster-name + namespace: spoke-cluster-namespace +spec: + clusterName: spoke-cluster-name + extraAnnotations: + ManagedCluster: + agent.open-cluster-management.io/klusterlet-config: "cert-manager-ca-config" + # ... rest of ClusterInstance spec +``` + +The siteconfig operator automatically applies annotations from `extraAnnotations.ManagedCluster` to the generated ManagedCluster resource, eliminating manual post-deployment steps. + +#### Manual deployment + +For manual cluster imports or retrofitting existing clusters, annotate the managed cluster after deploying the hub-spoke trust files: + +```bash +oc annotate managedcluster \ + agent.open-cluster-management.io/klusterlet-config=cert-manager-ca-config +``` + +### Greenfield (cert-manager before spoke deployment) + +1. Deploy cert-manager on the hub, create the CA, issue hub API/ingress certs +2. Deploy `certManagerHubCAConfigMap.yaml` and `certManagerKlusterletConfig.yaml` +3. Ensure the KlusterletConfig annotation is configured on managed clusters (see [Applying the KlusterletConfig](#applying-the-klusterletconfig-to-managed-clusters) above) +4. Deploy spokes — they register with the root CA in their trust store +5. Cert rotations are seamless with no intervention required + +### Brownfield (cert-manager on existing hub with connected spokes) + +The order matters — distribute the CA **before** replacing the hub certs: + +1. Install cert-manager on the hub, create the CA — but **do not apply certs to the APIServer/IngressController yet** +2. Deploy `certManagerHubCAConfigMap.yaml` with the root CA PEM +3. Deploy `certManagerKlusterletConfig.yaml` +4. Ensure the KlusterletConfig annotation is configured on all managed clusters (see [Applying the KlusterletConfig](#applying-the-klusterletconfig-to-managed-clusters) above) +5. Wait for the import controller to regenerate bootstrap kubeconfigs (check logs for `create a new bootstrap kubeconfig`) +6. **Now** apply the cert-manager certs to the APIServer and IngressController +7. Spokes stay connected because they already trust the root CA + +### Cert rotation + +Once the root CA is in the klusterlet's trust store, cert rotations are seamless. The klusterlet trusts any certificate signed by the root CA regardless of serial number, with no ManifestWork timing dependency. + +## Root CA Expiration Monitoring + +The `certManagerRootCAExpirationPolicy.yaml` creates a PrometheusRule that monitors the root CA certificate expiration using the `certmanager_certificate_expiration_timestamp_seconds` metric: + +- **Warning** at 90 days before expiry +- **Critical** at 30 days before expiry + +This is distinct from the existing `certManagerCertificatePolicy.yaml` which monitors leaf certificate expiration in `openshift-ingress`, `openshift-config`, and `cert-manager` namespaces via ACM CertificatePolicy. Both should be deployed together for comprehensive certificate monitoring. + ## References - [OpenShift Cert-Manager Operator Documentation](https://docs.openshift.com/container-platform/latest/security/cert_manager_operator/index.html) - [Cert-Manager Documentation](https://cert-manager.io/docs/) - [ACME DNS-01 Challenge Configuration](https://cert-manager.io/docs/configuration/acme/dns01/) - +- [Hub-Spoke Trust — Complete Solution](https://gist.github.com/sebrandon1/7265d68c5add6adb1313dce5b695e40d) +- [Hub-Spoke Trust Test Results](https://gist.github.com/sebrandon1/483180614951d23174c4e365a9a02a34) diff --git a/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerCertificatePolicy.yaml b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerCertificatePolicy.yaml index b46e29ca5..de65769a7 100644 --- a/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerCertificatePolicy.yaml +++ b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerCertificatePolicy.yaml @@ -22,6 +22,7 @@ spec: include: - openshift-ingress - openshift-config + - cert-manager remediationAction: inform severity: low remediationAction: inform diff --git a/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerHubCAConfigMap.yaml b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerHubCAConfigMap.yaml new file mode 100644 index 000000000..b9ef3c5ac --- /dev/null +++ b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerHubCAConfigMap.yaml @@ -0,0 +1,16 @@ +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: cert-manager-hub-ca-bundle + namespace: multicluster-engine + labels: + import.open-cluster-management.io/ca-bundle: "true" + annotations: + argocd.argoproj.io/sync-wave: "-29" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +data: + # Populate with the root CA PEM used by cert-manager to sign hub API/ingress certs. + # Extract from the hub: oc get secret root-ca-secret -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d + ca-bundle.crt: | + diff --git a/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerKlusterletConfig.yaml b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerKlusterletConfig.yaml new file mode 100644 index 000000000..372a6e1f0 --- /dev/null +++ b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerKlusterletConfig.yaml @@ -0,0 +1,16 @@ +--- +apiVersion: config.open-cluster-management.io/v1alpha1 +kind: KlusterletConfig +metadata: + name: cert-manager-ca-config + annotations: + argocd.argoproj.io/sync-wave: "-29" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +spec: + hubKubeAPIServerConfig: + serverVerificationStrategy: UseCustomCABundles + trustedCABundles: + - name: cert-manager-hub-ca-bundle + caBundle: + namespace: multicluster-engine + name: cert-manager-hub-ca-bundle diff --git a/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicy.yaml b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicy.yaml new file mode 100644 index 000000000..56ac3bee1 --- /dev/null +++ b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicy.yaml @@ -0,0 +1,59 @@ +--- +# ACM Policy to create a PrometheusRule that alerts on root CA +# certificate expiration. Monitors the cert-manager certificate +# expiration metric and fires warnings at 90 days and critical +# alerts at 30 days before expiry. +apiVersion: policy.open-cluster-management.io/v1 +kind: Policy +metadata: + name: root-ca-expiration-monitor + namespace: default + annotations: + policy.open-cluster-management.io/categories: SC System and Communications Protection + policy.open-cluster-management.io/controls: SC-8 Transmission Confidentiality and Integrity + policy.open-cluster-management.io/standards: NIST 800-53 + argocd.argoproj.io/sync-wave: "-10" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +spec: + disabled: false + policy-templates: + - objectDefinition: + apiVersion: policy.open-cluster-management.io/v1 + kind: ConfigurationPolicy + metadata: + name: root-ca-expiration-prometheusrule + spec: + remediationAction: enforce + severity: high + object-templates: + - complianceType: musthave + objectDefinition: + apiVersion: monitoring.coreos.com/v1 + kind: PrometheusRule + metadata: + name: cert-manager-root-ca-expiration + namespace: cert-manager-operator + spec: + groups: + - name: cert-manager-root-ca + rules: + - alert: CertManagerRootCAExpiringSoon + annotations: + summary: 'Root CA certificate {{`{{ $labels.name }}`}} is expiring soon' + description: 'The root CA certificate {{`{{ $labels.name }}`}} in namespace {{`{{ $labels.namespace }}`}} expires in less than 90 days.' + # Update name="root-ca" if your cert-manager Certificate resource uses a different name + expr: | + certmanager_certificate_expiration_timestamp_seconds{name="root-ca",namespace="cert-manager"} - time() < 90 * 24 * 3600 + for: 1h + labels: + severity: warning + - alert: CertManagerRootCAExpirationCritical + annotations: + summary: 'Root CA certificate {{`{{ $labels.name }}`}} is about to expire' + description: 'The root CA certificate {{`{{ $labels.name }}`}} in namespace {{`{{ $labels.namespace }}`}} expires in less than 30 days. Immediate action required.' + expr: | + certmanager_certificate_expiration_timestamp_seconds{name="root-ca",namespace="cert-manager"} - time() < 30 * 24 * 3600 + for: 1h + labels: + severity: critical + remediationAction: enforce diff --git a/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml new file mode 100644 index 000000000..5a1e716b6 --- /dev/null +++ b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicyPlacement.yaml @@ -0,0 +1,18 @@ +--- +apiVersion: cluster.open-cluster-management.io/v1beta1 +kind: Placement +metadata: + name: root-ca-expiration-placement + namespace: default + annotations: + argocd.argoproj.io/sync-wave: "-11" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +spec: + predicates: + - requiredClusterSelector: + labelSelector: + matchExpressions: + - key: local-cluster + operator: In + values: + - "true" diff --git a/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml new file mode 100644 index 000000000..f9ec1fd7a --- /dev/null +++ b/telco-hub/configuration/reference-crs/optional/cert-manager/certManagerRootCAExpirationPolicyPlacementBinding.yaml @@ -0,0 +1,17 @@ +--- +apiVersion: policy.open-cluster-management.io/v1 +kind: PlacementBinding +metadata: + name: root-ca-expiration-placementbinding + namespace: default + annotations: + argocd.argoproj.io/sync-wave: "-11" + argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true +placementRef: + name: root-ca-expiration-placement + apiGroup: cluster.open-cluster-management.io + kind: Placement +subjects: + - name: root-ca-expiration-monitor + apiGroup: policy.open-cluster-management.io + kind: Policy diff --git a/telco-hub/configuration/reference-crs/optional/cert-manager/kustomization.yaml b/telco-hub/configuration/reference-crs/optional/cert-manager/kustomization.yaml index 78bb41b41..92705eafe 100644 --- a/telco-hub/configuration/reference-crs/optional/cert-manager/kustomization.yaml +++ b/telco-hub/configuration/reference-crs/optional/cert-manager/kustomization.yaml @@ -13,3 +13,8 @@ resources: - certManagerCertificatePolicy.yaml - certManagerCertificatePolicyPlacement.yaml - certManagerCertificatePolicyPlacementBinding.yaml + - certManagerHubCAConfigMap.yaml + - certManagerKlusterletConfig.yaml + - certManagerRootCAExpirationPolicy.yaml + - certManagerRootCAExpirationPolicyPlacement.yaml + - certManagerRootCAExpirationPolicyPlacementBinding.yaml diff --git a/telco-ran/configuration/argocd/example/clusterinstance/example-3node.yaml b/telco-ran/configuration/argocd/example/clusterinstance/example-3node.yaml index bb08fb446..9e9a8fd6a 100644 --- a/telco-ran/configuration/argocd/example/clusterinstance/example-3node.yaml +++ b/telco-ran/configuration/argocd/example/clusterinstance/example-3node.yaml @@ -30,6 +30,10 @@ spec: # ran-group-du-3node-templated.yaml will apply to all clusters with 'group-du-3nc-zone' and 'hardware-type' group-du-3nc-zone: "zone-1" hardware-type: "hw-type-platform-1" + # Uncomment to apply KlusterletConfig for cert-manager hub-spoke trust (optional) + # extraAnnotations: + # ManagedCluster: + # agent.open-cluster-management.io/klusterlet-config: "cert-manager-ca-config" clusterNetwork: - cidr: 1001:1::/48 hostPrefix: 64 diff --git a/telco-ran/configuration/argocd/example/clusterinstance/example-sno.yaml b/telco-ran/configuration/argocd/example/clusterinstance/example-sno.yaml index 8400d5de2..afa57beb5 100644 --- a/telco-ran/configuration/argocd/example/clusterinstance/example-sno.yaml +++ b/telco-ran/configuration/argocd/example/clusterinstance/example-sno.yaml @@ -47,6 +47,10 @@ spec: # ran-group-du-sno-templated.yaml will apply to all clusters with 'group-du-sno-zone' and 'hardware-type' group-du-sno-zone: "zone-1" hardware-type: "hw-type-platform-1" + # Uncomment to apply KlusterletConfig for cert-manager hub-spoke trust (optional) + # extraAnnotations: + # ManagedCluster: + # agent.open-cluster-management.io/klusterlet-config: "cert-manager-ca-config" clusterNetwork: - cidr: 1001:1::/48 hostPrefix: 64 diff --git a/telco-ran/configuration/argocd/example/clusterinstance/example-standard.yaml b/telco-ran/configuration/argocd/example/clusterinstance/example-standard.yaml index d8887bf5f..aacd78c78 100644 --- a/telco-ran/configuration/argocd/example/clusterinstance/example-standard.yaml +++ b/telco-ran/configuration/argocd/example/clusterinstance/example-standard.yaml @@ -30,6 +30,10 @@ spec: # ran-group-du-standard-templated.yaml will apply to all clusters with 'group-du-standard-zone' and 'hardware-type' group-du-standard-zone: "zone-1" hardware-type: "hw-type-platform-1" + # Uncomment to apply KlusterletConfig for cert-manager hub-spoke trust (optional) + # extraAnnotations: + # ManagedCluster: + # agent.open-cluster-management.io/klusterlet-config: "cert-manager-ca-config" clusterNetwork: - cidr: 1001:1::/48 hostPrefix: 64