This folder contains Terraform and GitOps (ArgoCD) artifacts to deploy the Private AI Code Assistant on Azure Red Hat OpenShift (ARO) with an NVIDIA H100 GPU node for LLM inference.
Developer (DevSpaces / OpenCode)
│
│ HTTPS (cluster-internal, self-signed TLS)
▼
MaaS / RHCL Gateway (HTTPS + per-namespace API key)
│ Chat: /v1/chat/completions
▼
Guardrails proxy + TrustyAI (enabled by default)
│ Optional Semantic Router (off in the ARO overlay)
▼
llm-d Gateway → workload Service (EPP disabled by default)
▼
vLLM Replica N (KServe LLMInferenceService, port 8000 HTTPS)
│ Upstream vLLM via vllm.image (v0.19.0)
│ Tool calling: --enable-auto-tool-choice --tool-call-parser=qwen3_xml
│ Reasoning: --reasoning-parser=qwen3
▼
Qwen/Qwen3.6-35B-A3B-FP8
│ FP8 quantized, 35B total / 3B active MoE
▼
NVIDIA H100 NVL 94GB HBM3
The supplied ARO values deploy one vLLM replica with EPP disabled:
- MaaS / RHCL —
maas-default-gatewayauthenticates per-namespace API keys. - Chat —
/v1/chat/completionsgoes through guardrails by default, then llm-d. Semantic Router is an optional hop. - Local/non-chat requests — Continue tab
/local/v1and other/v1paths go to llm-d, skipping guardrails and Semantic Router. - llm-d HTTPRoute — forwards directly to
qwen3-coder-kserve-workload-svc:8000(HTTPS).
EPP, InferencePool, and InferenceModel resources render only with epp.enabled=true. The current catch-all route still targets the workload Service; enabling those resources does not change its backend.
| Component | Version |
|---|---|
| Azure Red Hat OpenShift (ARO) | 4.20.15 |
| Kubernetes | v1.33.6 |
| RHCOS | 9.6.20260217-1 (Plow) |
| CRI-O | 1.33.9 |
| Operator | Version | Channel |
|---|---|---|
| Red Hat OpenShift AI (RHOAI) | 3.3.1 | stable |
| NVIDIA GPU Operator | 26.3.1 | v26.3 |
| Node Feature Discovery (NFD) | 4.20.0 | stable |
| Red Hat DevSpaces | 3.27.1 | stable |
| DevWorkspace Operator | 0.40.1 | fast |
| Red Hat OpenShift GitOps (ArgoCD) | 1.15.4 | latest |
| Red Hat OpenShift Serverless | 1.37.1 | stable |
| Red Hat Service Mesh | 3.3.3 | stable |
| Component | Version |
|---|---|
| NVIDIA Kernel Driver | 550.144.03 |
| CUDA Toolkit (in container) | 12.9 |
| CUDA Compat Libs | 575.57.08 |
| GPU Hardware | NVIDIA H100 NVL 94 GB HBM3 |
Driver update note: If the GPU node is reprovisioned with NVIDIA driver 580+ (CUDA 13.0), set
VLLM_ENABLE_CUDA_COMPATIBILITY=0and updateLD_LIBRARY_PATHto remove compat libs. See Troubleshooting.
| Component | Version | Notes |
|---|---|---|
| vLLM | 0.19.0 (upstream) | vllm.image on LLMInferenceService — see Why Upstream vLLM |
| PyTorch | 2.10.0+cu129 | Bundled with vLLM v0.19.0 |
| Transformers | 4.57.6 | Required >=5.1 for Qwen3.6 |
| Model | Qwen/Qwen3.6-35B-A3B-FP8 | 35B total / 3B active MoE, FP8, 256K ctx (native max) |
| Serving | KServe LLMInferenceService |
Sole NVIDIA serving path (vllm.image) |
| Gateway | Data Science Gateway | Gateway API + HTTPRoute (TLS) |
| EPP | Disabled by default | Optional prefix-cache + queue-depth scorer resources |
| Envoy Proxy | v1.33.2 (distroless) | Optional EPP sidecar; not deployed by default |
| InferencePool | GAIE v1 (GA CRD) | Optional pod discovery + EPP reference |
| InferenceModel | GAIE v1alpha2 | Optional model-to-pool mapping |
| Tool | Version |
|---|---|
| Terraform | 1.9.8 |
| Azure CLI | 2.85.0 |
| oc CLI | 4.21.5 |
RHOAI 3.3.1 bundles registry.redhat.io/rhaiis/vllm-cuda-rhel9 based on vLLM
~0.13 with transformers <5.x. The Qwen3.6-35B-A3B-FP8 model uses the
Qwen3_5MoeForConditionalGeneration architecture class, which requires:
transformers >=5.1— the tokenizer and config classes for Qwen3.5-MoE are not present in older versionsvLLM >=0.18— native support for the Qwen3.5-MoE architecture, including DeepGEMM FP8 MoE kernels and FlashAttention v3 on H100- CUDA 12.9 toolkit — vLLM v0.19.0 ships with PyTorch 2.10 compiled against
CUDA 12.9. The host NVIDIA driver is 550 (CUDA 12.4), so
VLLM_ENABLE_CUDA_COMPATIBILITY=1bridges the gap using CUDA compat libraries (575.57.08)
The chart pins upstream vLLM via vllm.image on the LLMInferenceService
(sole NVIDIA serving path). Cluster enableLLMInferenceServiceTLS=true is required
so KServe mounts /var/run/kserve/tls and the chart’s HTTPS probes / --ssl-*
args match.
Note: Upstream
vllm/vllm-openaiis unsupported by Red Hat. When RHOAI ships with vLLM >= 0.19 and transformers >= 5.1, pointvllm.imageat the bundled runtime image instead.
Agentic coding tools (OpenCode, Claude Code, Cline) require vLLM to correctly
parse tool calls and reasoning tokens. Misconfiguration causes </think> token
leaks in output and silently dropped tool calls.
| Model Family | --tool-call-parser |
--reasoning-parser |
Min vLLM | Notes |
|---|---|---|---|---|
| Qwen3.6 / Qwen3.5 (MoE) | qwen3_xml |
qwen3 |
0.19.0 | XML tool calls + <think> blocks |
| Qwen2.5 | hermes |
(none) | 0.13+ | Hermes JSON format, no thinking mode |
| DeepSeek R1 / V3 | hermes |
deepseek_r1 |
0.18+ | JSON tool calls + <think> blocks |
| Llama 3.x (Instruct) | llama3_json |
(none) | 0.15+ | Native JSON tool calling |
| Mistral / Mixtral | mistral |
(none) | 0.14+ | Mistral tool call format |
-
Never use
--tool-call-parser=hermeswith Qwen3.x — Hermes expects JSON tool calls; Qwen3.x emits XML<tool_call>tags. The parser silently fails and</think>tokens leak into thecontentfield. -
Always set
--reasoning-parserfor thinking models — Without it, vLLM has no way to separate<think>...</think>from content. The raw tokens appear in the response and break downstream tool-call parsing in clients. -
Use
tool_choice: "auto"in client requests — The"required"path in vLLM usesTypeAdapter(list[FunctionDefinition]).validate_json()which only handles JSON. Qwen3.x XML tool calls silently fail withtool_calls: []. -
Avoid
--tool-call-parser=qwen3_coderwith streaming — Known bug in vLLM 0.19.x where the streaming extractor fails across the</think>→<tool_call>boundary. Fixed in 0.20+. Useqwen3_xmlinstead.
Use Terraform model_variant to select a curated model preset:
model_variant = "qwen3.6" # or "qwen3.8"Custom model IDs and parsers belong in Helm values: model.id, vllm.toolCallParser, and vllm.reasoningParser on pca-ai-serving; keep modelId in pca-devspaces aligned. The qwen3.8 preset overrides the model ID and tool parser, so use the default passthrough preset for custom values.
After deployment, run the smoke test from inside the cluster:
curl -sk $GATEWAY_URL/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
"messages": [{"role":"user","content":"List files in /tmp"}],
"tools": [{"type":"function","function":{"name":"list_files","description":"List directory","parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"]}}}],
"tool_choice": "auto",
"max_tokens": 200
}'Expected: Response has "finish_reason": "tool_calls" with a populated
tool_calls array and reasoning in the reasoning field (not in content).
Failure indicators:
</think>appearing incontent→ missing--reasoning-parsertool_calls: []with XML incontent→ wrong--tool-call-parsertool_calls: []silently → usingtool_choice: "required"(switch to"auto")
| Tool | Version | Purpose |
|---|---|---|
terraform |
>= 1.4.6 | Infrastructure provisioning |
az (Azure CLI) |
>= 2.50 | Azure authentication and ARO management |
oc (OpenShift CLI) |
>= 4.19 | Cluster interaction and GitOps bootstrap |
jq |
>= 1.6 | JSON processing in the GPU MachineSet script |
Your Azure account needs:
- Contributor or Owner on the target subscription
- User Access Administrator (for role assignments created by
az aro create)
Register the ARO resource providers if not already registered:
az provider register --namespace Microsoft.RedHatOpenShift --wait
az provider register --namespace Microsoft.Compute --wait
az provider register --namespace Microsoft.Storage --wait
az provider register --namespace Microsoft.Authorization --waitRequest quota for Standard_NC40ads_H100_v5 in your target region before
deployment. The H100 VM requires 40 vCPUs.
az vm list-usage --location australiaeast -o table | grep -i "NC40ads"- A Red Hat account with an active OpenShift subscription
- Pull secret from console.redhat.com/openshift/install/pull-secret
| Component | Specification |
|---|---|
| Platform | Azure Red Hat OpenShift (ARO) |
| OpenShift version | 4.20.15 |
| Azure region | Australia East (australiaeast) |
| Master nodes | 3x Standard_D8s_v5 |
| Worker nodes | 3x Standard_D8s_v5 |
| GPU nodes | 1x Standard_NC40ads_H100_v5 (NVIDIA H100 NVL 94 GB) |
| Storage class | managed-csi (Azure Managed Disk CSI) |
az login
az account set --subscription "<your-subscription-id>"cd PCA_Deployment_ARO/terraform/
cp terraform.tfvars.example terraform.tfvarsEdit terraform.tfvars:
| Variable | Description |
|---|---|
subscription_id |
Your Azure subscription ID |
pull_secret |
Red Hat pull secret (single-line JSON string) |
cluster_name |
Cluster name (default: aro-pca-aue) |
location |
Azure region (default: australiaeast) |
huggingface_token |
HuggingFace token (required for local Qwen; also required if semantic_router_enabled = true) |
semantic_router_enabled |
Optional. Leave unset so the ARO overlay keeps SR off. Set true only with huggingface_token plus extras JSON if you want a split |
gitops_repo_url |
Your fork of the Private_AI_Coding_Assistant repo |
terraform init
terraform plan -out=aro-plan.tfplan
terraform apply aro-plan.tfplanTerraform provisions: Resource Group, VNet, Subnets, ARO Cluster (~35-45 min), GPU MachineSet, OpenShift GitOps, and ArgoCD App-of-Apps.
az aro list-credentials --name aro-pca-aue --resource-group aro-pca-aue-rg
az aro show --name aro-pca-aue --resource-group aro-pca-aue-rg --query consoleProfile.url -o tsv
oc login <API_URL> --username=kubeadmin --password=<PASSWORD>Configure developer identities in pca-platform-config's devspaces.instances and keep usernames/namespaces aligned with pca-devspaces's devspaces list. The ARO workspace overlay targets Dev1 / dev1-devspaces and Dev2 / dev2-devspaces. Supply non-empty passwords for the demo HTPasswd IDP, or use your cluster's existing IDP.
Sync the charts, then log in to the DevSpaces dashboard as each user and start the pre-created workspace. Helm renders DevWorkspaces with started: false.
Alternative: Factory URL (self-service). Users can also create their own workspace by navigating to the DevSpaces factory URL — no admin script needed:
https://<devspaces-url>/#https://github.com/manujoy7/Private_AI_Coding_Assistant.git
DevSpaces reads devfile.yaml from the repo root. That factory devfile uses direct llm-d access with key EMPTY; prefer chart-managed workspaces for MaaS authentication and chat guardrails.
# Check all operators
oc get csv -A | grep -v Succeeded
# Check GPU node
oc get nodes -l nvidia.com/gpu.present=true
# Check model serving
oc get inferenceservice -n ai-serving
oc get servingruntime -n ai-serving
# Check AI Gateway
oc get gateway,httproute -n ai-serving
# Check DevSpaces — workspaces are in the configured namespaces
oc get devworkspace -A
# Test direct llm-d access (escape hatch; skips MaaS and guardrails)
GATEWAY_SVC="llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local"
curl -sk https://${GATEWAY_SVC}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
"messages": [{"role": "user", "content": "Write a Python hello world"}],
"max_tokens": 100
}'ArgoCD manages the platform using the unified Helm charts at the repo root. Both ROSA and ARO use the same charts/ directory; ARO-specific overrides are in values-aro.yaml within each chart.
private-coding-assistant/ ← repo root
├── charts/ # Unified Helm charts (ROSA + ARO)
│ ├── pca-app-of-apps/ # Root ArgoCD AppProject + child Applications
│ ├── pca-operators/ # Wave 1: Operator Subscriptions
│ │ ├── values.yaml
│ │ └── values-aro.yaml # ARO overrides (NFD required, no Serverless)
│ ├── pca-platform-config/ # Wave 2: Namespace, RBAC, DSC, CheCluster
│ │ ├── values.yaml
│ │ ├── values-aro.yaml # ARO overrides (managed-csi storage class)
│ │ └── charts/pca-mcp/ # Optional: OpenShift MCP server
│ ├── pca-ai-serving/ # Wave 3: LLMInferenceService, llm-d, MaaS front door
│ │ ├── values.yaml
│ │ ├── values-aro.yaml # ARO overrides (model, storage, EPP disabled, …)
│ │ ├── charts/pca-guardrails/ # TrustyAI guardrails proxy (enabled by default)
│ │ └── charts/pca-observability/ # Grafana + Langfuse/OTel Collector (enabled on ARO)
│ ├── pca-devspaces/ # Wave 4: OpenCode or Continue/Roo/Cline workspaces + API keys
│ │ ├── values.yaml
│ │ └── values-aro.yaml
│ └── pca-benchmarks/ # Wave 5: GuideLLM sweep (enabled in values-aro.yaml)
│ ├── values.yaml
│ └── values-aro.yaml
│
└── PCA_Deployment_ARO/ ← You are here
├── README.md
├── Dockerfile.opencode # Custom OpenCode image with pre-installed CLI
├── devfile.yaml # DevSpaces factory URL devfile
├── terraform/ # Azure infrastructure provisioning
│ ├── main.tf # RG, VNet, subnets, ARO cluster, GPU MachineSet
│ ├── gitops-bootstrap.tf # OpenShift GitOps operator + App-of-Apps bootstrap
│ ├── variables.tf # Input variables with defaults
│ ├── versions.tf # Provider versions
│ ├── outputs.tf # Credential retrieval commands
│ └── terraform.tfvars.example
└── scripts/
├── create-gpu-machineset.sh # Post-cluster H100 node provisioning
├── deploy-full-stack.sh # Full stack deployment script
├── post-terraform-fullstack.sh # Post-terraform automation
├── setup-devspaces-users.sh # HTPasswd IDP + DevWorkspace provisioning
└── validate.sh # Post-deployment validation
Registered in redhat-ods-applications, makes the H100 GPU visible in the
RHOAI dashboard when deploying models. Defines CPU (4-16), Memory (40-120Gi),
and GPU (1x nvidia.com/gpu) resource bounds.
| Field | Value |
|---|---|
| Image | vllm/vllm-openai:v0.19.0 (vllm.image) |
| Entrypoint | python3 -m vllm.entrypoints.openai.api_server |
| Model load | Local PVC path /model-cache (storage-initializer; HF_HUB_OFFLINE=1) |
| Protocol | HTTPS on :8000 (--ssl-certfile / --ssl-keyfile under /var/run/kserve/tls) |
| Workload Service | qwen3-coder-kserve-workload-svc:8000 |
| Cache | PVC-backed (/model-cache) for model weights and JIT kernels |
| Probes | Startup / readiness / liveness with scheme: HTTPS |
vLLM server args (from templates/llminferenceservice.yaml):
| Arg | Purpose |
|---|---|
--port=8000 |
Listen port |
--model=/model-cache |
Local path after storage-initializer |
--served-model-name=… |
OpenAI API model name (defaults to model.id) |
--trust-remote-code |
Required for Qwen3.5-MoE architecture |
--enable-prefix-caching |
KV cache reuse for shared prefixes within vLLM |
--enable-auto-tool-choice |
Allow model to decide when to use tools (required by OpenCode/Roo Code) |
--tool-call-parser=qwen3_xml |
Parse Qwen3.x XML tool calls into OpenAI tool_calls |
--ssl-certfile / --ssl-keyfile |
Serve TLS on :8000 (cluster enableLLMInferenceServiceTLS=true) |
Environment variables handle non-root container constraints:
| Variable | Value | Purpose |
|---|---|---|
HF_HOME |
/model-cache |
HuggingFace cache on PVC |
HF_HUB_OFFLINE |
1 |
Always on for LLMIS local-path load (storage-initializer populates the PVC) |
TRITON_CACHE_DIR |
/model-cache/triton-cache |
Triton MoE kernel cache on PVC |
XDG_CACHE_HOME |
/model-cache/xdg-cache |
General cache on PVC |
HOME |
/tmp |
Writable home for non-root user |
VLLM_ENABLE_CUDA_COMPATIBILITY |
0 |
Disabled — host driver 580 (CUDA 13.0) natively supports CUDA 12.9 toolkit |
LD_LIBRARY_PATH |
/usr/local/cuda/lib64:/usr/lib64 |
Standard CUDA library paths (no compat libs needed with driver 580) |
DG_JIT_CACHE_DIR |
/model-cache/deep-gemm |
DeepGEMM MoE kernel JIT cache on PVC — saves ~5 min on restart |
VLLM_CACHE_ROOT |
/model-cache/vllm-cache |
torch.compile AOT cache on PVC — saves ~30s on restart |
- Mode: KServe RawDeployment via
LLMInferenceService(no Knative/Serverless dependency) - Resources per replica: 8-16 CPU, 80-120Gi RAM, 1x NVIDIA GPU
- Toleration:
nvidia.com/gpu=present:NoSchedule - Scaling:
spec.replicas: 1in the chart (raise for more GPU nodes)
Combined vLLM args (chart command / args):
python3 -m vllm.entrypoints.openai.api_server \
--port=8000 \
--model=/model-cache \
--served-model-name=Qwen/Qwen3.6-35B-A3B-FP8 \
--trust-remote-code \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser=qwen3_xml \
--reasoning-parser=qwen3 \
--tensor-parallel-size=1 \
--max-model-len=32000 \
--ssl-certfile=/var/run/kserve/tls/tls.crt \
--ssl-keyfile=/var/run/kserve/tls/tls.key
Tool calling is required for OpenCode's agentic features. Without
--enable-auto-tool-choiceand--tool-call-parser, OpenCode requests fail with:"auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set.
100Gi managed-csi PVC stores HuggingFace model weights (~35GB), Triton JIT
kernels, DeepGEMM warmup artifacts (DG_JIT_CACHE_DIR), and torch.compile AOT
cache (VLLM_CACHE_ROOT). Survives pod restarts to avoid re-downloading the
model and re-compiling kernels (~17 min saved on warm restart — from 19.5 min
→ 2.5 min cold start with populated caches).
HF_HUB_OFFLINE is always 1: the storage-initializer populates the PVC before
vLLM starts, so runtime does not call Hugging Face Hub. Keep the PVC across
upgrades for a warm cache.
Gateway API Gateway with HTTPS listener (self-signed TLS) and HTTPRoute.
The catch-all HTTPRoute forwards requests directly to the LLMIS workload Service. EPP is disabled in the supplied ARO values. IDEs normally reach this gateway through the MaaS front door; direct access is the escape hatch.
Cluster-internal endpoint:
https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1
EPP and its Envoy sidecar are optional chart resources, disabled by default. Their configuration is described below; the current catch-all HTTPRoute targets the workload Service rather than EPP.
| Component | Image |
|---|---|
| EPP | registry.redhat.io/rhoai/odh-llm-d-inference-scheduler-rhel9 (RHOAI-bundled) |
| Envoy | envoyproxy/envoy:distroless-v1.33.2 |
Scheduling algorithm (configurable via EndpointPickerConfig):
queue-scorer(weight 2) — routes to replicas with shorter queuesprefix-cache-scorer(weight 3) — routes similar prompts to the same replica for KV cache reuse, minimizing redundant computation
ExtProc flow:
- Envoy receives the inference request on port 8081
- Envoy calls EPP via gRPC ExtProc (localhost:9002)
- EPP queries InferencePool for available vLLM pods
- EPP scores each pod using queue depth + prefix cache hit metrics
- EPP returns
x-gateway-destination-endpointheader with optimal pod IP - Envoy forwards request to the selected pod using ORIGINAL_DST cluster
When epp.enabled=true, selects vLLM pods by label serving.kserve.io/inferenceservice: qwen3-coder
and forwards traffic to port 8000. Automatically discovers new replicas when
the LLMInferenceService scales up.
When epp.enabled=true, maps model name Qwen/Qwen3.6-35B-A3B-FP8 to qwen3-coder-inference-pool. This automation deploys only one local model at a time.
The default OpenCode workspace runs VS Code in the browser with the private model configured through MaaS. OpenCode is available through its VS Code extension and Web UI; type: continue provides Continue, Cline, and Roo Code instead.
| Mode | How to Access | Description |
|---|---|---|
| VS Code Extension | Ctrl+Esc in editor |
Opens OpenCode TUI in a split terminal panel. Context-aware — shares current editor selection. File reference shortcut: Alt+Ctrl+K. Extension sst-dev.opencode auto-installed via DEFAULT_EXTENSIONS env var (official CheCode mechanism — .vsix downloaded in postStart, then installed by the editor at startup). |
| Browser Web UI (in-IDE) | VS Code: F1 → "Simple Browser: Show" → http://localhost:4096 |
Opens the OpenCode Web UI inside an editor tab; use the workspace's OpenCode password when prompted. |
| Browser Web UI (external) | Direct route URL from DevSpaces dashboard | Full Web UI in a separate browser tab. The chart starts it with the password from opencode-web-password. |
Users authenticate through the configured cluster IDP. For demo HTPasswd, configure matching users/passwords in pca-platform-config as described in Step 5.
| User | Dashboard Login |
|---|---|
Dev1 |
DevSpaces URL with Dev1 credentials |
Dev2 |
DevSpaces URL with Dev2 credentials |
Helm renders stopped DevWorkspaces in devspaces[].namespace. The ARO workspace overlay uses dev1-devspaces and dev2-devspaces, without random suffixes. Keep the platform namespace/user list aligned with those workspace entries.
Users log in to the dashboard and start their pre-created workspace so the controller can stamp the creator identity. ArgoCD ignores changes to spec.started, allowing users to start and stop their workspace.
IDEs use MaaS with per-namespace API keys. Direct llm-d access is an escape hatch (aiGateway.escapeHatchToLlmd=true).
| Config | Value |
|---|---|
| Provider | OpenAI-compatible (vLLM) |
| Base URL | https://maas-default-gateway-data-science-gateway-class.openshift-ingress.svc.cluster.local/v1 |
| Model | Qwen/Qwen3.6-35B-A3B-FP8 |
| API Key | Per-namespace key from the pca-maas-apikey Secret |
| TLS | Self-signed cert (NODE_TLS_REJECT_UNAUTHORIZED=0) |
| Extension | sst-dev.opencode (auto-installed via DEFAULT_EXTENSIONS env var — see CheCode docs) |
| Web UI Port | 4096 (auto-started via postStart; access via VS Code Simple Browser at http://localhost:4096) |
| Web UI Auth | Password-protected; startup loads OPENCODE_SERVER_PASSWORD from the namespace's opencode-web-password Secret |
The workspace uses a custom container image built from the Red Hat Universal Developer Image (UDI) with OpenCode pre-installed and pre-configured:
| Component | Detail |
|---|---|
| Base image | registry.redhat.io/devspaces/udi-rhel8:latest |
| OpenCode binary | Copied to /usr/local/bin/opencode (not symlinked — see troubleshooting) |
| Config | ~/.config/opencode/opencode.json — points to MaaS at workspace startup |
| Auth | ~/.local/share/opencode/auth.json — bake-time EMPTY replaced at startup with the namespace's pca-maas-apikey key |
| Build namespace | opencode-build |
| ImageStream | devspaces-opencode:latest |
| Rebuild | oc start-build devspaces-opencode -n opencode-build |
Important: The binary is copied to
/usr/local/bininstead of symlinked from~/.local/binbecause the DevSpaces runtime overlay overwrites the latter directory at container start.
GuideLLM sweep results for Qwen3.6-35B-A3B-FP8 on H100 NVL:
| Workload | Prompt Tokens | Output Tokens | Peak Throughput (tok/s) | Sync Latency (s) | Sync TTFT (ms) |
|---|---|---|---|---|---|
| Code Completion | 256 | 128 | 4,512 | 0.70 | 36 |
| Code Generation | 1,024 | 512 | 12,790 | 2.78 | 83 |
| Code Review | 4,096 | 1,024 | 16,133 | 5.59 | 157 |
| File Generation | 8,192 | 2,048 | 13,976 | 11.15 | 208 |
Full results: testresults_h100.md · A100 sweep: testresults.md · Summary: docs/benchmarks.md
| Aspect | ROSA (AWS) | ARO (Azure) |
|---|---|---|
| GPU VM | e.g. g6e (L40S) via RHCS machine pool |
GPU MachineSet (e.g. H100 NVL / A100 family) after cluster create |
| Storage class | gp3-csi |
managed-csi |
| Charts | charts/ + values-rosa.yaml |
Same charts/ + values-aro.yaml |
| Cloud values | Terraform sets gitops.cloud=rosa |
Terraform sets gitops.cloud=aro |
| NSG / network | AWS security groups | ARO-managed subnets (no customer NSG on master/worker) |
| vLLM image | Chart / RHOAI default path for the ROSA overlay | Upstream vLLM (see Why Upstream vLLM v0.19.0) |
| Tool / reasoning parsers | Set in ROSA values-rosa.yaml / LLMIS args for the chosen model |
See Model Tool Calling & Reasoning Configuration (e.g. qwen3_xml + qwen3 for Qwen3.6) |
Azure GPU choice depends on quota and model size (A100 80 GB vs H100 NVL 94 GB). Prefer a SKU with native FP8 and enough VRAM for your context window; see docs/benchmarks.md.
For detailed infrastructure sizing, model comparison, and total cost of ownership
analysis, see assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v3.md.
Key findings:
| Finding | Detail |
|---|---|
| Concurrent users per L40S | 17 developers at 64K context (Qwen 3.6 35B-A3B) |
| Cost per developer | $15–41/mo at 50–500 developers (3yr commitment, ROSA on AWS) |
| Peak concurrency | ~20% of team size (65% online × 25% active × 1.2× buffer) |
| Throughput | ~42 tok/s per user at 17 concurrent on L40S (exceeds 30 tok/s minimum) |
| KV cache efficiency | ~10 KB/token (DeltaNet) vs 48–80 KB/token (standard transformers) |
| Cold start (warm PVC) | ~2.5 min with DG_JIT_CACHE_DIR + VLLM_CACHE_ROOT on PVC |
| Cold start (fresh) | ~19.5 min (includes HF download + JIT compilation) |
Recommended GPU tiers:
- L40S (48 GB) — Qwen 3.6 35B-A3B, single-GPU instance, $15–39/dev/mo
- H100 (80 GB) — Qwen3-Coder-Next 80B, single-GPU instance, $17–52/dev/mo
- H200 (141 GB) — Large teams (500+) only; AWS requires 8-GPU instances
The chart renders one vLLM replica. Additional replicas require GPU capacity and a persisted GitOps change; live patches below are temporary under ArgoCD self-healing.
# 1. Scale GPU MachineSet to N nodes
oc scale machineset <infra_id>-gpu-h100 -n openshift-machine-api --replicas=N
# 2. Wait for nodes to be Ready
oc get nodes -l nvidia.com/gpu.present=true -w
# 3. Update LLMInferenceService replicas
oc patch llminferenceservice qwen3-coder -n ai-serving --type merge \
-p '{"spec":{"replicas": N}}'
# 4. Check workload pods (EPP is disabled by default)
oc get pods -n ai-serving -l serving.kserve.io/inferenceservice=qwen3-coderThe current route targets the workload Service. EPP resources and scoring are not part of the default deployment.
Only for a custom deployment with EPP enabled and its gateway routing configured:
oc scale deploy/llm-d-epp -n ai-serving --replicas=2oc patch llminferenceservice qwen3-coder -n ai-serving --type merge \
-p '{"spec":{"replicas": 0}}'
oc scale machineset <infra_id>-gpu-h100 -n openshift-machine-api --replicas=0This automation deploys one local model at a time. Concurrent local models behind the same AI Gateway are unsupported. Use Terraform model_variant (qwen3.6 or qwen3.8) to switch models.
az aro delete --name aro-pca-aue --resource-group aro-pca-aue-rg --yes
az group delete --name aro-pca-aue-rg --yes --no-waitvLLM pod stuck in startup (DeepGEMM warmup): First launch compiles ~2,785 DeepGEMM MoE kernels via JIT (~10-15 min). Subsequent restarts are fast when using PVC-backed cache. The startup probe allows up to 60 minutes.
GPU node taint preventing pod scheduling:
The H100 node has taint nvidia.com/gpu=present:NoSchedule. The
LLMInferenceService includes the matching toleration. If deploying custom
pods, add the toleration.
CUDA driver version mismatch:
If the GPU node has been updated to NVIDIA driver 580+ (CUDA 13.0), set
VLLM_ENABLE_CUDA_COMPATIBILITY=0 and remove compat libs from LD_LIBRARY_PATH
(use /usr/local/cuda/lib64:/usr/lib64 instead of /usr/local/cuda/compat:...).
On original deployments with driver 550 (CUDA 12.4), vLLM v0.19.0 still needs
CUDA 12.9 toolkit support. Set VLLM_ENABLE_CUDA_COMPATIBILITY=1 and include
the compat libs (575.57.08) in LD_LIBRARY_PATH. If you see CUDA errors on
driver 550, verify LD_LIBRARY_PATH includes /usr/local/cuda/compat.
AI Gateway returns 503 or 504:
Check the guardrails proxy and vLLM workload pods, then inspect pca-maas-front-door and llm-d-gateway-route in ai-serving. The default deployment has no EPP pod or InferencePool to diagnose.
EPP pod in CrashLoopBackOff (only when explicitly enabled):
Check the EPP config version (apiVersion: inference.networking.x-k8s.io/v1alpha1).
Ensure RBAC includes inferenceobjectives and leases. Check that the
qwen3-coder-inference-pool InferencePool exists.
OpenCode "auto tool choice requires --enable-auto-tool-choice" error:
OpenCode sends tool_choice: "auto" for agentic features. vLLM rejects these
requests unless both --enable-auto-tool-choice and --tool-call-parser are set
in the LLMInferenceService args (charts/pca-ai-serving/values-aro.yaml /
templates/llminferenceservice.yaml). Use qwen3_xml for Qwen3.x models.
Fix via GitOps (preferred) or patch the LLMIS container args, then roll the pod.
Model download slow or failing:
The model-cache PVC persists downloads across restarts. If HuggingFace is rate-limited,
set HF_TOKEN in the container environment or the hf-token secret.
OpenCode binary not found in PATH inside DevSpaces workspace:
The DevSpaces runtime overlay filesystem overwrites ~/.local/bin at container
start, removing any symlinks placed there during the image build. The fix (already
applied in the Dockerfile) copies the binary to /usr/local/bin/opencode instead
of symlinking from ~/.local/bin. If you still see opencode: command not found,
rebuild the image: oc start-build devspaces-opencode -n opencode-build.
DevSpaces dashboard shows 0 workspaces for a user:
Check that devspaces.instances in platform-config and devspaces in the workspace chart agree on the username and namespace. Verify that the DevWorkspace exists there and the user has its RoleBinding. Users should start chart-created workspaces from the dashboard so the controller stamps the correct creator identity.
OpenCode VS Code extension not auto-installed:
The extension is installed via the DEFAULT_EXTENSIONS env var — the only reliable
auto-install mechanism in CheCode/DevSpaces. The postStart command downloads the
.vsix from Open VSX to /tmp/opencode-ext/, and the DEFAULT_EXTENSIONS env var
tells CheCode to install it at editor startup. Other mechanisms that do NOT work:
vscode-extensions-config.yamlwithrecommendations— only shows the extension in the sidebar, does not auto-installche-code.eclipse.org/vscode-extensionsdevfile attribute — unreliable, often silently ignored by the DevWorkspace controller
If the extension is missing, check: (1) the postStart download succeeded
(ls /tmp/opencode-ext/), (2) the DEFAULT_EXTENSIONS env var is set in the
container, (3) the workspace was fully restarted (not just reconnected). To force
reinstall: delete the workspace and recreate it.
OpenCode Web UI shows blank page or password popup in browser:
The OpenCode Web UI uses absolute asset paths (/assets/...). When served through
the che-gateway path-prefix routing (e.g., /dev1/opencode-dev1/4096/), assets
fail to load because the browser resolves them against the domain root. Do NOT
set urlRewriteSupported: true on the opencode-web endpoint — this causes
path-prefix stripping which breaks asset loading.
The chart deliberately enables password authentication. Retrieve the namespace's opencode-web-password Secret and use that password when prompted; keep the startup command that exports OPENCODE_SERVER_PASSWORD.
The correct configuration for the opencode-web endpoint:
endpoints:
- name: opencode-web
targetPort: 4096
exposure: public
protocol: https
# postStart loads OPENCODE_SERVER_PASSWORD from opencode-web-passwordFor in-IDE access (recommended): use VS Code Simple Browser → http://localhost:4096.
OpenCode Web UI (port 4096) not starting automatically:
The postStart command requires the opencode binary to be in PATH. If the
image was built with the old symlink approach, the binary won't be found. Rebuild
the image with the /usr/local/bin copy fix. To start manually in the meantime:
export PATH="/home/user/.opencode/bin:$PATH"
NS=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
export OPENCODE_SERVER_PASSWORD=$(oc get secret opencode-web-password -n "$NS" \
-o jsonpath='{.data.password}' | base64 -d)
nohup opencode web --port 4096 --hostname 0.0.0.0 > /tmp/opencode-web.log 2>&1 &