Skip to content

CNF-22724: Add External Secrets Operator reference configuration #808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
rdiscala wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

CNF-22724: Add External Secrets Operator reference configuration #808

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions telco-core/configuration/core-baseline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,14 @@ policies:
# - path: reference-crs/optional/cert-manager/certManagerOperatorgroup.yaml
# - path: reference-crs/optional/cert-manager/certManagerSubscription.yaml

# External Secrets Operator (optional component - included by default)
- path: reference-crs/optional/external-secrets/esoNS.yaml
- path: reference-crs/optional/external-secrets/esoOperatorgroup.yaml
- path: reference-crs/optional/external-secrets/esoSubscription.yaml
patches:
- spec:
installPlanApproval: Manual
Comment on lines +59 to +61

@imiller0 imiller0 Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although other operators do the same thing, Manual is the value set in the reference-cr. No need for this


Comment thread
coderabbitai[bot] marked this conversation as resolved.
- path: reference-crs/required/networking/sriov/SriovSubscriptionNS.yaml
- path: reference-crs/required/networking/sriov/SriovSubscriptionOperGroup.yaml
- path: reference-crs/required/networking/sriov/SriovSubscription.yaml
Expand Down Expand Up @@ -89,6 +97,11 @@ policies:
policyAnnotations:
ran.openshift.io/ztp-deploy-wave: "6"
manifests:
# External Secrets Operator configuration (optional component - included by default)
- path: reference-crs/optional/external-secrets/esoExternalSecretsConfig.yaml
- path: reference-crs/optional/external-secrets/esoClusterSecretStore.yaml
- path: reference-crs/optional/external-secrets/esoNetworkPolicy.yaml

- path: reference-crs/optional/logging/ClusterLogServiceAccount.yaml
- path: reference-crs/optional/logging/ClusterLogServiceAccountAuditBinding.yaml
- path: reference-crs/optional/logging/ClusterLogServiceAccountInfrastructureBinding.yaml
Expand Down
16 changes: 16 additions & 0 deletions telco-core/configuration/reference-crs-kube-compare/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,22 @@ parts:
allOf:
- path: optional/other/mount_namespace_config_master.yaml
- path: optional/other/mount_namespace_config_worker.yaml
- name: optional-external-secrets
description: |-
External Secrets Operator for secure secret management via external stores
components:
- name: external-secrets-operator
allOrNoneOf:
- path: optional/external-secrets/esoNS.yaml
- path: optional/external-secrets/esoOperatorgroup.yaml
- path: optional/external-secrets/esoSubscription.yaml
- path: optional/external-secrets/esoExternalSecretsConfig.yaml
- path: optional/external-secrets/esoClusterSecretStore.yaml
config:
ignore-unspecified-fields: true
- path: optional/external-secrets/esoNetworkPolicy.yaml
config:
ignore-unspecified-fields: true
- name: optional-cert-manager
description: |-
Cert-manager operator for automated certificate management
Expand Down
21 changes: 21 additions & 0 deletions ...iguration/reference-crs-kube-compare/optional/external-secrets/esoClusterSecretStore.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: ztp-secret-provider

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allow any name

spec:
conditions:
- namespaceSelector:
matchExpressions:
- key: cluster.open-cluster-management.io/managedCluster
operator: Exists
Comment on lines +8 to +11

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted below, allow any non-empty value

provider:
vault:
server: {{ .spec.provider.vault.server | quote }}
path: {{ .spec.provider.vault.path | quote }}
version: {{ .spec.provider.vault.version | quote }}
auth:
tokenSecretRef:
name: {{ .spec.provider.vault.auth.tokenSecretRef.name }}
key: {{ .spec.provider.vault.auth.tokenSecretRef.key }}
namespace: {{ .spec.provider.vault.auth.tokenSecretRef.namespace }}
6 changes: 6 additions & 0 deletions ...ration/reference-crs-kube-compare/optional/external-secrets/esoExternalSecretsConfig.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
apiVersion: operator.openshift.io/v1alpha1
kind: ExternalSecretsConfig
metadata:
name: cluster
spec: {}

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ESO allows configuration to integrate with cert-manager (docs here). Since 4.22 we have cert manager optionally included in RDS. If both ESO and cert-manager are installed is there a required configuration needed here for them to interoperate? Or optional but beneficial?

7 changes: 7 additions & 0 deletions telco-core/configuration/reference-crs-kube-compare/optional/external-secrets/esoNS.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
apiVersion: v1
kind: Namespace
metadata:
annotations:
workload.openshift.io/allowed: management
Comment on lines +5 to +6

@imiller0 imiller0 Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you confirm if the pods in ESO are annotated for workload partitioning? As part of the platform they should be but this is a new operator for Telco use case and likely they have not been added yet. The annotation is:
target.workload.openshift.io/management: '{"effect":"PreferredDuringScheduling"}'

@rdiscala rdiscala Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@imiller0 they are not annotated. Shall I open a request on Jira to the upstream owners?

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this should be part of the management partition, however we should get a baseline of how much CPU and memory the operator requires before making the request (if they are high we need to request they be moderated before moving to the mgmt partition).

name: external-secrets-operator
20 changes: 20 additions & 0 deletions .../configuration/reference-crs-kube-compare/optional/external-secrets/esoNetworkPolicy.yaml
View file
Open in desktop

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does access to the apiServer need to be explicitly allowed here as well? (here and other network policy crs)

Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-secret-store-egress
namespace: external-secrets

@imiller0 imiller0 Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to be external-secrets-operator? Here and other instances of esoNetworkPolicy

spec:
podSelector:
matchLabels:
app.kubernetes.io/name: external-secrets
policyTypes:
- Egress
egress:

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hub includes egress to DNS. Is that needed here as well to allow dns resolution of vault name?

    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: openshift-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: vault
ports:
- port: 8200
protocol: TCP
11 changes: 11 additions & 0 deletions .../configuration/reference-crs-kube-compare/optional/external-secrets/esoOperatorgroup.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
annotations:
operatorframework.io/bundle-unpack-timeout: "10m"

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This annotation should not be included it does not do anything

operatorframework.io/bundle-unpack-min-retry-interval: 10m
name: openshift-external-secrets-operator
namespace: external-secrets-operator
spec:
upgradeStrategy: Default
14 changes: 14 additions & 0 deletions ...e/configuration/reference-crs-kube-compare/optional/external-secrets/esoSubscription.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: openshift-external-secrets-operator
namespace: external-secrets-operator
spec:
channel: stable-v1
name: openshift-external-secrets-operator
source: redhat-operators-disconnected

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically we allow any catalogsource to be used in the comparison:
source: {{ .spec.source }}

sourceNamespace: openshift-marketplace
installPlanApproval: Manual
status:
state: AtLatestKnown
22 changes: 22 additions & 0 deletions telco-core/configuration/reference-crs/optional/external-secrets/esoClusterSecretStore.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: ztp-secret-provider
spec:
# Restrict store usage to RHACM managed-cluster namespaces only
conditions:
- namespaceSelector:
matchExpressions:
- key: cluster.open-cluster-management.io/managedCluster
operator: Exists
Comment on lines +7 to +12

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a core cluster there is no need for BMC/pull-secrets from ESO and there are not any RHACM managed namespaces. In the Core use case ESO will be used exclusively by the user to supply secrets for their use. The CR name here should be placeholder and conditions left empty (for user to fill in). The cluster-compare should allow any non-empty value for the conditions.

provider:
vault:
server: "https://vault.example.com:8200"
path: "secret"
version: "v2"
auth:
tokenSecretRef:
name: vault-token
key: token
namespace: external-secrets-operator
Comment thread
rdiscala marked this conversation as resolved.
6 changes: 6 additions & 0 deletions ...-core/configuration/reference-crs/optional/external-secrets/esoExternalSecretsConfig.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
apiVersion: operator.openshift.io/v1alpha1
kind: ExternalSecretsConfig
metadata:
name: cluster
spec: {}
7 changes: 7 additions & 0 deletions telco-core/configuration/reference-crs/optional/external-secrets/esoNS.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
apiVersion: v1
kind: Namespace
metadata:
annotations:
workload.openshift.io/allowed: management
name: external-secrets-operator
20 changes: 20 additions & 0 deletions telco-core/configuration/reference-crs/optional/external-secrets/esoNetworkPolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-secret-store-egress
namespace: external-secrets
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: external-secrets
policyTypes:
- Egress
egress:

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hub includes egress to DNS. Is that needed here as well to allow dns resolution of vault name?

    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: openshift-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: vault
ports:
- port: 8200
protocol: TCP
10 changes: 10 additions & 0 deletions telco-core/configuration/reference-crs/optional/external-secrets/esoOperatorgroup.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
annotations:
operatorframework.io/bundle-unpack-min-retry-interval: 10m
name: openshift-external-secrets-operator
namespace: external-secrets-operator
spec:
upgradeStrategy: Default
14 changes: 14 additions & 0 deletions telco-core/configuration/reference-crs/optional/external-secrets/esoSubscription.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: openshift-external-secrets-operator
namespace: external-secrets-operator
spec:
channel: stable-v1
name: openshift-external-secrets-operator
source: redhat-operators-disconnected
sourceNamespace: openshift-marketplace
installPlanApproval: Manual
status:
state: AtLatestKnown
9 changes: 9 additions & 0 deletions ...ub/configuration/example-overlays-config/external-secrets/cluster-secret-store-patch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Patch the ClusterSecretStore with your secret store server address.
# Update the server URL and path to match your environment.
- op: replace
path: /spec/provider/vault/server
value: "https://vault.example.com:8200"

- op: replace
path: /spec/provider/vault/path
value: "secret"
23 changes: 23 additions & 0 deletions telco-hub/configuration/example-overlays-config/external-secrets/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../reference-crs/optional/external-secrets/

patches:
# Customize ClusterSecretStore for your secret store backend
- target:
group: external-secrets.io
version: v1
kind: ClusterSecretStore
name: ztp-secret-provider
path: cluster-secret-store-patch.yaml

# Customize NetworkPolicy target for your secret store namespace/port
- target:
group: networking.k8s.io
version: v1
kind: NetworkPolicy
name: allow-secret-store-egress
path: network-policy-patch.yaml
9 changes: 9 additions & 0 deletions telco-hub/configuration/example-overlays-config/external-secrets/network-policy-patch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Patch the NetworkPolicy to target your secret store namespace and port.
# Update the namespace label and port to match your environment.
- op: replace
path: /spec/egress/0/to/0/namespaceSelector/matchLabels/kubernetes.io~1metadata.name
value: "vault"

- op: replace
path: /spec/egress/0/ports/0/port
value: 8200
3 changes: 3 additions & 0 deletions telco-hub/configuration/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ resources:
# if you use ODF, edit and configure storage settings
- example-overlays-config/odf/

# external secrets operator (optional, included by default, customize before deploying)
- example-overlays-config/external-secrets/

# other not optional overlays
- example-overlays-config/gitops/
- example-overlays-config/acm/
Expand Down
22 changes: 22 additions & 0 deletions telco-hub/configuration/reference-crs-kube-compare/metadata.yaml
View file
Open in desktop

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also add the operator label at the bottom under fieldsToOmit.defaults

Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,28 @@ parts:
- path: optional/cert-manager/certManagerCertificatePolicy.yaml
- path: optional/cert-manager/certManagerCertificatePolicyPlacement.yaml
- path: optional/cert-manager/certManagerCertificatePolicyPlacementBinding.yaml
- name: optional-external-secrets
description: |-
External Secrets Operator for secure secret management via external stores
components:
- name: external-secrets-operator
allOrNoneOf:
- path: optional/external-secrets/esoNS.yaml
- path: optional/external-secrets/esoOperatorgroup.yaml
- path: optional/external-secrets/esoSubscription.yaml
- path: optional/external-secrets/esoExternalSecretsConfig.yaml
- path: optional/external-secrets/esoClusterSecretStore.yaml
config:
ignore-unspecified-fields: true
- path: optional/external-secrets/esoNetworkPolicy.yaml
config:
ignore-unspecified-fields: true
- path: optional/external-secrets/esoClusterTemplates.yaml
config:
ignore-unspecified-fields: true
- path: optional/external-secrets/esoNodeTemplates.yaml
config:
ignore-unspecified-fields: true
- name: optional-storage
components:
- name: local-storage-operator
Expand Down
23 changes: 23 additions & 0 deletions ...iguration/reference-crs-kube-compare/optional/external-secrets/esoClusterSecretStore.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
annotations:

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing annotation:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

argocd.argoproj.io/sync-wave: "-30"
name: ztp-secret-provider
spec:
conditions:
- namespaceSelector:
matchExpressions:
- key: cluster.open-cluster-management.io/managedCluster
operator: Exists
provider:
vault:
server: {{ .spec.provider.vault.server | quote }}
path: {{ .spec.provider.vault.path | quote }}
version: {{ .spec.provider.vault.version | quote }}
Comment on lines +16 to +18

@imiller0 imiller0 Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better use of templating :)

auth:
tokenSecretRef:
name: {{ .spec.provider.vault.auth.tokenSecretRef.name }}
key: {{ .spec.provider.vault.auth.tokenSecretRef.key }}
namespace: {{ .spec.provider.vault.auth.tokenSecretRef.namespace }}
31 changes: 31 additions & 0 deletions ...nfiguration/reference-crs-kube-compare/optional/external-secrets/esoClusterTemplates.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
apiVersion: v1
kind: ConfigMap
metadata:
name: eso-cluster-templates-v1
namespace: open-cluster-management
annotations:
argocd.argoproj.io/sync-wave: "-30"
data:
PullSecretExternalSecret: |-
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: "{{ "{{ .Spec.PullSecretRef.Name }}" }}"
namespace: "{{ "{{ .Spec.ClusterName }}" }}"
annotations:
siteconfig.open-cluster-management.io/sync-wave: "1"
spec:
refreshPolicy: OnChange
secretStoreRef:
name: ztp-secret-provider
kind: ClusterSecretStore
target:
name: "{{ "{{ .Spec.PullSecretRef.Name }}" }}"
creationPolicy: Owner
template:
type: kubernetes.io/dockerconfigjson
data:
- secretKey: .dockerconfigjson
remoteRef:
key: clusters/{{ "{{ .Spec.ClusterName }}" }}/pull-secret

@imiller0 imiller0 Jun 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is essentially a "contract" that we are creating which the vault needs to conform to. This needs to be clearly noted in the RDS documentation (thank you for noting clearly in the readme here!)

@rdiscala rdiscala Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I've clarified this further in the documentation. I've also added a section for it in the changes to the reference-design-specifications project, which I will soon push.

property: .dockerconfigjson
8 changes: 8 additions & 0 deletions ...ration/reference-crs-kube-compare/optional/external-secrets/esoExternalSecretsConfig.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
apiVersion: operator.openshift.io/v1alpha1
kind: ExternalSecretsConfig
metadata:
annotations:

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing annotation:
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

argocd.argoproj.io/sync-wave: "-30"
name: cluster
spec: {}
8 changes: 8 additions & 0 deletions telco-hub/configuration/reference-crs-kube-compare/optional/external-secrets/esoNS.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
apiVersion: v1
kind: Namespace
metadata:
annotations:
argocd.argoproj.io/sync-wave: "-45"
workload.openshift.io/allowed: management
name: external-secrets-operator
22 changes: 22 additions & 0 deletions .../configuration/reference-crs-kube-compare/optional/external-secrets/esoNetworkPolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-secret-store-egress
namespace: external-secrets
annotations:
argocd.argoproj.io/sync-wave: "-25"
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: external-secrets
policyTypes:
- Egress
egress:

@imiller0 imiller0 Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to includes egress to DNS here as well

    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: openshift-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP

- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: vault
ports:
- port: 8200
protocol: TCP
Loading
Loading