nad-service-controller

This controller connects a Service resource to secondary pod networks created by Multus.

The controller watches for events associated with Service resources annotated with isim.dev/network: <network-name>. It creates the EndpointSlices resource for the Service to enable traffic to be routed to the pods' secondary network IP addresses, in the <network-name> network.

The service must be:

• headless i.e. spec.clusterIP: None so that traffic is routed directly to the pods' secondary network IP addresses. There is no guarantee that the clusterIP assigned by K8s can reach the secondary network.
• without selector i.e. spec.selector: {} so that the controller can create endpoints that point to the pods' secondary network IP addresses. Endpoints managed by K8s only work in the primary network.

Optionally, if the service has the app.kubernetes.io/name or app labels, only matching pods are included in the endpoint slices. Otherwise, all pods that are part of the secondary network are included.

The states of the pods are reflected in the endpoints' conditions to ensure traffic is routed only to ready pods.

Quick Start

Set up a KinD cluster with all the necessary components:

make cluster

Use ko to build the code, push the image to local registry, and apply the controller manifest to the cluster:

make apply-kind

Alternately, you can get the deploy.yaml manifest from the release page.

Wait till the controller pods are ready in the default namespace:

kubectl wait --for condition=Ready po -lapp.kubernetes.io/name=nad-service-controller

Deploy the test data:

make testdata

Expect to see two sets of workloads labeled app=nginx-basic and app=nginx-nad:

$ kubectl get deploy,svc,po -lapp=nginx-basic
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-basic   2/2     2            2           17s

NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/nginx-basic   ClusterIP   10.96.94.171   <none>        80/TCP,443/TCP   17s

NAME                               READY   STATUS    RESTARTS   AGE
pod/nginx-basic-747cff5f4d-g4cwn   1/1     Running   0          17s
pod/nginx-basic-747cff5f4d-tbrfr   1/1     Running   0          17s

$ kubectl get deploy,svc,po -lapp=nginx-nad
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-nad   2/2     2            2           2m26s

NAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/nginx-nad   ClusterIP   10.96.66.215   <none>        80/TCP,443/TCP   2m26s

NAME                             READY   STATUS    RESTARTS   AGE
pod/nginx-nad-65c7f7c6dc-b4zg8   1/1     Running   0          2m26s
pod/nginx-nad-65c7f7c6dc-jkx5z   1/1     Running   0          2m26s

Workloads with the app=nginx-nad label are connected to a secondary macvlan network named macvlan.

The nad-service-controller creates an EndpointSlice resource for the nginx-nad service, targeting the secondary network IP addresses of the pods in the macvlan network.

$ kubectl get endpointslices -lkubernetes.io/service-name=nginx-nad
NAME              ADDRESSTYPE   PORTS    ENDPOINTS                 AGE
nginx-nad-slice   IPv4          80,443   192.168.2.4,192.168.2.5   5m25s

$ kubectl get endpointslices -lkubernetes.io/service-name=nginx-nad -ojsonpath='{.items[*].endpoints[*].addresses}'
["192.168.2.4"] ["192.168.2.5"]

$ kubectl get po -lapp=nginx-nad -ojson | jq -cr '.items[].metadata.annotations["k8s.v1.cni.cncf.io/network-status"] | fromjson | .[1].ips'
["192.168.2.4"]
["192.168.2.5"]

The netutils-* pods can be used to send client-side traffic to the Nginx services.

For example, the netutils-nad-* pods can reach both the nginx-basic primary network and the nginx-nad secondary network services:

$ kubectl exec netutils-nad-6d66668db6-kxhsj -- curl -s -o /dev/null -w "%{http_code}" nginx-nad
200

$ kubectl exec netutils-nad-6d66668db6-kxhsj -- curl -s -o /dev/null -w "%{http_code}" nginx-basic
200

But the netutils-base-* pods can only reach the nginx-basic primary network service. It can't reach the nginx-nad secondary network service because the pods aren't part of that network:

$ kubectl exec netutils-base-d8dcd5b98-rv9bv -- curl -s -o /dev/null -w "%{http_code}" nginx-basic
200%
$ kubectl exec netutils-base-d8dcd5b98-rv9bv -- curl -s -o /dev/null -w "%{http_code}" --connect-timeout 10 nginx-nad
000command terminated with exit code 28

When done, delete the KinD cluster:

make purge

SBOM And Provenance

The image SBOM is generated by ko in the SPDX format. The release workflow uses actions/attest to create the build provenance.

To attest, use the gh CLI:

gh attestation verify oci://ghcr.io/ihcsim/nad-service-controller --repo ihcsim/nad-service-controller

Development

Development requires the following tools:

• go
• golangci-lint
• kubectl
• kind - for cluster testing
• ko - for image building and pushing to a registry

Go code targets:

# build the src
make build

# run unit tests
make test

# run main with go run
make run

# use golangci-lint to check for linting issues
make lint

Working with OCI images:

# use ko to build and push the image to a local registry
make image-local

# use ko to apply the Deployment resource YAML to the cluster using the image
# built with ko, targeting your current context cluster
make apply

# use ko to delete the Deployment resource from the cluster in your current 
# kubeconfig context
make delete

To recompile the GitHub agentic workflows:

make compile-aw

Release

Create a new tag using main as the target branch:

tag_version=<tag_version>

git tag -a "${tag_version}" main -m "${tag_version} release"

Manually trigger the `release` GHA workflow.