AWS EKS layer (Terraform):
- VPC
- EKS cluster and EKS addons
- Karpenter
- AWS Load Balancer Controller IAM
- ArgoCD
- GitOps Bridge — cluster Secret + root Application
Core layer (ArgoCD at argocd/applications/core/):
- Traefik ingress controller
- AWS Load Balancer Controller (Helm release; IAM stays in TF)
- NVIDIA GPU Operator
- Kube-prometheus-stack + Loki + Promtail for Monitoring
- Piraeus Operator for Linstor tests
- Istio for LLM inference smart routing
- Hashicorp Vault + Bank Vaults Operator — secrets management
MLOps layer (ArgoCD at argocd/applications/mlops/):
- JupyterLab (CUDA/LLM) — image built in-cluster via BuildKit Job, deploy via manual-sync Argo Application
- JupyterHub — multi-user Jupyterhub
- Argo Workflows - CI for ML pipelines
- LLM-D stack — multi-model LLM inference (vLLM + EPP smart pod routing)
- LiteLLM — unified OpenAI API over the LLM-D model gateways
- OpenClaw — autonomous AI agent (cluster agent example)
Each core component is gated by an enable_<addon> flag in terraform/main.tf.
Flags are published as labels on the ArgoCD cluster Secret; each ApplicationSet in argocd/applications/core/ filters on its own label. Set a value to false and Argo prunes the addon.
- Deployment
- Delete infrastructure
- Bank Vaults Operator
- JupyterLab example with GPU
- Piraeus Operator tests for Linstor
- JupyterHub deployment
- Argo Workflows deployment
- LLM-D deployment
- OpenClaw deployment
- Terraform — creates VPC, EKS, Karpenter, ArgoCD, cluster Secret, root Application.
- ArgoCD picks up the root Application → recursively discovers
argocd/applications/core/andargocd/applications/apps/. - ApplicationSets materialise child Applications that install Traefik, ALB controller, etc.
- Traefik comes up, NLB gets provisioned by ALB controller, you map the NLB IP in
/etc/hosts.
cd terraform
terraform init -upgrade
terraform applyDuring the first minutesthe ArgoCD UI is not yet reachable via argocd.local.
Access the UI via port-forward:
k -n argocd port-forward svc/argocd-server 8080:80
# open http://localhost:8080k -n traefik get svc traefik \
-o jsonpath='{.status.loadBalancer.ingress[0].hostname}' \
| xargs dig +shortPick any one of the returned IPs and add:
<IP> argocd.local argo-workflows.local grafana.local jupyter.local litellm.local openclaw.local vault.localk -n argocd get secret argocd-initial-admin-secret \
-o jsonpath="{.data.password}" | base64 -d; echoLogin to CLI and add the GitOps repo (if not public):
argocd login argocd.local:443
argocd repo add https://github.com/silazare/argocd-infra-example.git \
--username silazare --password github_pat_xxxxx
argocd repo add ghcr.io --type helm --name stable --enable-oci# delete ArgoCD root application
# remove stuck application sets
for kind in applications applicationsets; do
for name in $(kubectl -n argocd get $kind -o name); do
kubectl -n argocd patch $name --type=json \
-p='[{"op":"remove","path":"/metadata/finalizers"}]' 2>/dev/null
done
done
# deleted Karpenter CRD to fix terraform destroy stuck
terraform state rm helm_release.karpenter_crd
terraform destroyThree independent pieces make up the stack (wired by the bank-vaults ApplicationSet):
| Component | Source | Role |
|---|---|---|
vault-operator |
Helm chart, values in argocd/helm-values/bank-vaults/ | Kubernetes operator. Watches the Vault CR and reconciles the Vault StatefulSet + Configurer Job. |
vault-secrets-webhook |
Helm chart | Mutating admission webhook. Injects a secret-fetch sidecar into pods annotated with vault.security.banzaicloud.io/*. Independent of the operator — works on its own. |
Vault CR |
Manifest in argocd/manifests/bank-vaults/ | Declarative spec of the Vault cluster itself (+ SA and RBAC). Read by the operator. |
The Vault CR pins 2 container images:
image: hashicorp/vault— upstream HashiCorp Vault server. Independent release cycle.bankVaultsImage: bank-vaults/bank-vaults— the bank-vaults CLI. Runs as sidecar in each Vault pod and as the Configurer Job. Handles init/unseal (keys stored in a K8s Secret) and applies everything underexternalConfig:(policies, auth methods, secrets engines,startupSecrets) through the Vault API.
- This setup is non-production, because unseal keys are stored in the same cluster and k8s secrets, consider KMS
- The unseal keys and root token are managed by the Bank-Vaults operator.
- There are 5 key shares created, with a threshold of 3 required to unseal Vault.
- The unseal information is stored as Kubernetes Secrets in the "vault" namespace.
- The secrets managed by Vault are stored in the Raft storage, which is persisted on the Kubernetes PersistentVolumes (
gp3). - Each Vault pod will have its own PersistentVolume, and Raft ensures that the data is replicated across these volumes for high availability.
Wait until Vault will be synced.
Login to Vault UI at http://vault.local and retrieve root token:
k -n vault get secret vault-unseal-keys -o jsonpath="{.data.vault-root}" | base64 -dLogin to Vault CLI:
export VAULT_ADDR=http://vault.local
export VAULT_SKIP_VERIFY=true
vault status
export VAULT_TOKEN="xxxxx"
vault kv get secret/openclaw/envhttps://medium.com/@sinan.ozel_23433/iac-for-generative-ai-llm-jupyterlab-on-kubernetes-a33d31841a27 https://www.jimangel.io/posts/nvidia-rtx-gpu-kubernetes-setup/
Build runs as a Job in buildkit namespace and pushes image, layer cache to jupyterlab-llm-cache repo. Update branch/tag inside build-job.yaml if needed.
Edit the tag in files (keep them in sync), then build + push + sync:
- mlops/jupyterlab-llm/build-job.yaml —
--output=...:<NEW_TAG> - argocd/manifests/jupyterlab-llm/jupyterlab-llm-pod.yaml —
image: ...:<NEW_TAG>
k replace --force -f mlops/jupyterlab-llm/build-job.yamlApplication is not auto-synced — image must exist in ECR before first sync. Trigger sync manually:
argocd app sync jupyterlab-llmSandbox for a Piraeus / LINSTOR / DRBD persistent-storage stack Settings at argocd/helm-values/linstor-cluster/values.yaml
Three placement modes, one StorageClass per replica count:
| Manifest | StorageClass | Placement | What it proves |
|---|---|---|---|
| mlops/hdd1-test-sts.yaml | linstor-hdd-1r (autoPlace=1) |
1 diskful replica on a storage node | Provisioning + ext4 + Retain reclaim works; PV survives Pod recreate on the same node |
| mlops/hdd2-test-sts.yaml | linstor-hdd-2r (autoPlace=2) |
2 diskful replicas across storage nodes | Synchronous DRBD replication; Pod can come back on either replica node |
| mlops/diskless-test-sts.yaml | linstor-hdd-2r |
2 diskful on storage NG + 1 diskless DRBD client on karpenter ubuntu node |
Compute / storage separation pattern — the bare-metal target shape where GPU nodes mount data over the network from CPU storage nodes |
# Satellites + storage pools
k -n piraeus-datastore exec deploy/linstor-controller -- linstor node list
k -n piraeus-datastore exec deploy/linstor-controller -- linstor storage-pool list
# Apply any of the test STS and watch the resource list
k apply -f mlops/linstor/hdd2-test-sts.yaml
k -n piraeus-datastore exec deploy/linstor-controller -- linstor resource list
# Live DRBD state on a specific satellite
k -n piraeus-datastore get pod -l app.kubernetes.io/component=linstor-satellite -o wide
k -n piraeus-datastore exec <satellite-pod> -- drbdadm statusSandbox for multi-user JupyterHub via the zero-to-jupyterhub (z2jh) chart
| Component | What it is | Deploys |
|---|---|---|
hub |
JupyterHub control plane — auth + per-user pod spawning (KubeSpawner) | hub Deployment + state DB on gp3 |
proxy |
configurable-http-proxy — routes /hub and /user/<name> traffic |
proxy Deployment + proxy-public Service (ClusterIP) |
singleuser |
Template for each user's notebook server (JupyterLab UI, jupyter_server backend) |
Per-user pod + dynamic 10Gi gp3 PVC, spawned on first login |
scheduling |
Cost/packing: dedicated scheduler, pod priority, warm placeholders | user-scheduler Deployment, PriorityClass, user-placeholder pods |
cull |
Idle-culler — stops servers idle >1h so Karpenter scales in | culler service inside the hub (no separate pod) |
prePuller |
Pre-pulls the singleuser image so spawns are fast | pre-upgrade Job (hook) + continuous DaemonSet on ubuntu nodes |
User management — edit the lists in hub.config.Authenticator, then helm upgrade:
allowed_users— who may log in (the user slicing);admin_users— subset with the admin panel.- Usernames must be DNS-safe (lowercase) — they become the PVC name
claim-<username>and the/home/jovyanowner.
All users share the single host — routing is path-based inside the proxy (CHP), not per-user ingress:
- Log in at
jupyter.local> redirected to/user/<username>/lab(that user's own pod). - Admins reach other users' servers via
/hub/admin> Access Server
Each user gets one PVC claim-<username> (template claim-{username}{servername}), backed by a 10Gi gp3 volume mounted at /home/jovyan. The Hub never deletes a PVC — server shutdown detaches the volume but keeps the data:
login → Hub creates PVC claim-<username> > CSI creates PV (real disk)
→ PV mounted into the pod as /home/jovyan
server shutdown → pod deleted, PV detached
→ PVC + PV REMAIN (data lives on)
re-login → Hub sees the existing PVC > mounts the same PV
Two independent flows — access vs data.
- Revoke access (safe, instant) — remove from
allowed_users/admin_users, thenhelm upgrade. PVC stays, data intact. - Stop a running server — admin UI
/hub/admin> Stop, ork -n jupyterhub delete pod jupyter-<username>(pod only, leaves the PVC). - Handle data — back up BEFORE deleting.
Workflow engine for ML pipelines, used as an in-cluster CI that deploys Helm releases in order. Installed via ArgoCD (argo/argo-workflows chart.
| Component | What it is | Deploys |
|---|---|---|
controller |
Workflow controller — reconciles Workflow CRs into step pods |
controller Deployment in argo, namespaced (singleNamespace: true) |
server |
Argo UI + API (argo-server), auto --namespaced on argo |
server Deployment + ClusterIP :2746 + Traefik Ingress argo-workflows.local |
crds |
Workflow / CronWorkflow / etc. CRDs | minified CRDs as plain chart templates (crds.full: false — no pre-install hook Job, no network pull); Application uses ServerSideApply=true |
workflow SA + RBAC |
identity step pods run as | SA argo-workflow + Role in ns argo |
extraObjects (UI) |
UI login identity | SA argo-ui + long-lived token Secret + RoleBinding to argo-argo-workflows-admin, scoped to ns argo |
extraObjects (deploy) |
cross-ns Helm deployer | wf-helm-deployer ClusterRole + mlops-pipelines Namespace + argo-workflow-helm RoleBinding (grants the workflow SA Helm rights in the target ns) |
Architecture: control-plane and CI workflows both live in argo. Pipelines deploy into target namespaces (mlops-pipelines) via cross-namespace RBAC, not by running steps there — adding a target is one Namespace + one RoleBinding in extraObjects, no controller-scope change.
UI at http://argo-workflows.local (Traefik, HTTP). No SSO — client mode, paste an SA bearer token. The UI lands on ns argo at login (where the workflows are) — no Forbidden.
# Login token, paste WHOLE output (incl. "Bearer ") into the client-auth field
echo "Bearer $(k -n argo get secret argo-ui.service-account-token \
-o jsonpath='{.data.token}' | base64 -d)"# basic test
argo submit -n argo --serviceaccount argo-workflow --watch \
https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml
# cross-ns Helm deploy
argo submit -n argo --watch mlops/argo-wf/helm-deploy-test.yaml
helm -n mlops-pipelines listServe small LLMs (start Qwen/Qwen2.5-0.5B-Instruct) with smart pod routing — pick the vLLM pod with the warmest KV-cache, not round-robin. Installed via ArgoCD, gated by enable_llm_d, in namespace llm-d.
DATA — real bytes (▼ spine) CONTROL — decides, no bytes (┄►)
─────────────────────────── ────────────────────────────────
client
│ 1 · POST /v1/chat/completions
▼
Service <rn>-gateway-istio :80 ◄┄ provisioned by ┄ Gateway CR (gatewayClassName: istio, :80)
│
▼
Envoy (Istio Gateway, managed) ◄┄ 2 · routed by ┄ HTTPRoute "/" (backend = InferencePool)
│ 3 · ext-proc gRPC :9002 ⇄ EPP ┄► EPP reads InferencePool, scores pods
│ (kv-cache / queue / prefix) → 4 · returns "use pod X"
▼ 5 · forward to chosen pod :8000
vLLM decode pod :8000 (the EPP-picked one, GPU node)
│
▼ tokens stream back up the same path to client
Only the ▼ spine carries bytes (client → Envoy → decode pod → back). Everything on the
┄► side is logic, not traffic — and the InferencePool is a config object with no IP it just names the pod set (selector) + the picker (EPP).
Step by step:
- (data) client sends
POST /v1/chat/completionsto the Gateway Service<resourceName>-gateway-istio:80. - (control) the
HTTPRoutematches/and points its backend at this model'sInferencePool; the managed Envoy + Service were provisioned by istiod from theGatewayCR. - (control) because the backend is an
InferencePool, the Gateway's Envoy consults the EPP via ext-proc gRPC:9002— "which pod?". - (control) EPP (llm-d endpoint-picker) reads the
InferencePool's pods, scores them (KV-cache hit / queue depth / prefix match), returns the best one. - (data) the Gateway's Envoy forwards the request to the chosen vLLM decode pod
:8000; tokens stream back up the same spine to the client.
| Piece | What it is | From |
|---|---|---|
| router + EPP | Envoy proxy + endpoint-picker + the InferencePool |
upstream Helm chart llm-d-router-standalone v0.9.0 (OCI), shared values in argocd/helm-values/llm-d-router/ |
| modelserver | the vLLM decode Deployment + weight-cache PVC + SA |
our chart argocd/charts/llm-d-modelserver/ |
| wiring | one ArgoCD App per model (matrix: clusters × model list) | argocd/applications/mlops/llm-d-recipe.yaml |
The router and the modelserver find each other by labels (llm-d.ai/guide + llm-d.ai/model) and by name (InferencePool name = the model's resourceName). Keep those in sync.
One entry in the list.elements of llm-d-recipe.yaml — nothing else:
- resourceName: phi-4-mini # short unique id
model: microsoft/Phi-4-mini-instruct
modelLabel: Phi-4-mini-instruct
gpu: "1"
replicas: "1"
maxModelLen: "8192"The entry point is the router Service <resourceName>-epp:80 (Envoy → EPP → decode pod).
# full path through the router (Envoy -> EPP smart pick -> vLLM)
k -n llm-d port-forward svc/qwen-qwen2-5-0-5b-instruct-epp 8082:80
# list served models
curl -s localhost:8082/v1/models | jq
# chat completion — stream so the EPP can compute per-token latency
curl -sN localhost:8082/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","stream":true,
"messages":[{"role":"user","content":"Hello, who are you?"}],"max_tokens":64}'
curl -s localhost:8082/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"Hello, who are you?"}],"max_tokens":64}' \
| jq -r '.choices[0].message.content'
#### 2nd model
k -n llm-d port-forward svc/phi-4-mini-instruct-epp 8083:80
# list served models
curl -s localhost:8083/v1/models | jq
curl -sN localhost:8083/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"microsoft/Phi-4-mini-instruct","stream":true,
"messages":[{"role":"user","content":"Hello, who are you?"}],"max_tokens":64}'
curl -s localhost:8083/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"microsoft/Phi-4-mini-instruct","messages":[{"role":"user","content":"Hello, who are you?"}],"max_tokens":64}' \
| jq -r '.choices[0].message.content'The external edge <resourceName>-gateway-istio:80 (Istio Gateway Envoy → EPP smart pick → decode pod).
k get gateway,httproute -A
k -n llm-d get httproute qwen-qwen2-5-0-5b-instruct-route \
-o jsonpath='{.status.parents[*].conditions[?(@.type=="ResolvedRefs")]}'
# full path through the Istio Gateway (Envoy -> EPP smart pick -> vLLM)
k -n llm-d port-forward svc/qwen-qwen2-5-0-5b-instruct-gateway-istio 8082:80
# list served models
curl -s localhost:8082/v1/models | jq
# chat completion with SSE stream merge
curl -s localhost:8082/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"Hello, who are you?"}],"max_tokens":64}' \
| jq -r '.choices[0].message.content'
#### 2nd model
k -n llm-d port-forward svc/phi-4-mini-instruct-gateway-istio 8083:80
# list served models
curl -s localhost:8083/v1/models | jq
# chat completion with SSE stream merge
curl -s localhost:8083/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"microsoft/Phi-4-mini-instruct","messages":[{"role":"user","content":"Hello, who are you?"}],"max_tokens":64}' \
| jq -r '.choices[0].message.content'OpenAI-compatible endpoint for all models — LiteLLM maps an alias (qwen-0.5b, phi-4-mini)
to that model's Istio Gateway Service (argocd/helm-values/litellm/values.yaml).
export LITELLM_KEY=sk-f46d79880b7d7125d3b503f6783a0380# list model aliases
curl -s http://litellm.local/v1/models -H "Authorization: Bearer $LITELLM_KEY" | jq
# both models through the SAME endpoint, switched by alias
curl -s http://litellm.local/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_KEY" -H 'Content-Type: application/json' \
-d '{"model":"qwen-0.5b","messages":[{"role":"user","content":"Hello, who are you?"}],"max_tokens":64}' \
| jq -r '.choices[0].message.content'
curl -s http://litellm.local/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_KEY" -H 'Content-Type: application/json' \
-d '{"model":"phi-4-mini","messages":[{"role":"user","content":"Hello, who are you?"}],"max_tokens":64}' \
| jq -r '.choices[0].message.content'If Qwen runs 2 decode pods. The EPP prefix-cache scorer should stick
prefix-sharing requests to the SAME pod (cache hits), while distinct prefixes spread
by queue/kv-cache load. Metrics come from the llm-d-vllm PodMonitor + llm-d-epp
# long shared prefix — same system prompt every request, unique question at the tail
PREFIX=$(python3 -c "print('You are a meticulous assistant. ' * 100)")
# 1) warm-up + affinity check: 40 requests with the SAME prefix
for i in $(seq 40); do
curl -s http://litellm.local/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_KEY" -H 'Content-Type: application/json' \
-d "{\"model\":\"qwen-0.5b\",\"messages\":[{\"role\":\"system\",\"content\":\"$PREFIX\"},{\"role\":\"user\",\"content\":\"Question $i: what is $i+$i?\"}],\"max_tokens\":16}"
done
# 2) control group: 40 requests with RANDOM prefixes (no cache to hit)
for i in $(seq 40); do
RAND=$(openssl rand -hex 32)
curl -s http://litellm.local/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_KEY" -H 'Content-Type: application/json' \
-d "{\"model\":\"qwen-0.5b\",\"messages\":[{\"role\":\"system\",\"content\":\"$RAND\"},{\"role\":\"user\",\"content\":\"hi\"}],\"max_tokens\":16}"
done
# 3) Check which pod did EPP pick per request
# distribution
sum by (pod) (increase(vllm:request_success_total{model_name="Qwen/Qwen2.5-0.5B-Instruct"}[10m]))
# hit rate
sum by (pod) (rate(vllm:prefix_cache_hits_total[5m]))
/
sum by (pod) (rate(vllm:prefix_cache_queries_total[5m]))
# TTFT per pod
histogram_quantile(0.95, sum by (pod, le) (rate(vllm:time_to_first_token_seconds_bucket[5m])))Autonomous AI agent (openclaw/openclaw) — gateway + Control UI, shell/browser tools, optional messaging channels.
| Component | What it is | Deploys |
|---|---|---|
init-config |
seeds openclaw.json (JSON5, from ConfigMap) onto the PVC; configMode merge/overwrite |
init container |
init-skills |
installs ClawHub skills onto the PVC (list empty by default) | init container |
main |
OpenClaw gateway :18789 (HTTP + WebSocket), reads secrets via envFrom |
Deployment (1 replica, Recreate) + ClusterIP + Traefik Ingress openclaw.local |
chromium |
headless browser for the browser tool, CDP localhost:9222 |
sidecar |
data |
agent state — config, sessions, workspace, skills | 10Gi gp3 RWO PVC at /home/node (HOME, so ~/.openclaw is owned by UID 1000, not by the root-owned volume root) |
OpenClaw does not scale horizontally, keep one replica.
Cluster access is read-only: SA openclaw is bound to the built-in view ClusterRole plus get/list on nodes, PVs, StorageClasses, metrics, Karpenter and ArgoCD CRDs. No Secrets, no exec, no writes. Prometheus and Loki are reachable over plain HTTP inside monitoring.
Check the effective allowlist: k -n openclaw exec deploy/openclaw -c main -- node dist/index.js approvals get.
k create ns openclaw
k -n openclaw create secret generic openclaw-env \
--from-literal=OPENCLAW_GATEWAY_TOKEN=$(openssl rand -hex 32) \
--from-literal=ANTHROPIC_API_KEY=sk-ant-xxx \
--from-literal=TELEGRAM_BOT_TOKEN=placeholderOpen http://openclaw.local, enter the gateway token in Settings, click Connect. Approve the pairing request from the pod:
# gateway token
k -n openclaw get secret openclaw-env -o jsonpath='{.data.OPENCLAW_GATEWAY_TOKEN}' | base64 -d; echo
# list pending devices
k -n openclaw exec deploy/openclaw -c main -- node dist/index.js devices list
# approve the device
k -n openclaw exec deploy/openclaw -c main -- node dist/index.js devices approve <REQUEST_ID>