Skip to content

Commit 0f1d431

Browse files
committed
docs: align guides and inference architecture with the code
Correct assistant choices, gateway authentication, EPP defaults, guardrails behavior, and deployment guidance. Refresh the inference diagram and its content source, with larger text and a taller layout.
1 parent 5f61a92 commit 0f1d431

10 files changed

Lines changed: 667 additions & 448 deletions

File tree

‎PCA_Deployment_ARO/README.md‎

Lines changed: 62 additions & 142 deletions
Large diffs are not rendered by default.

‎PCA_Deployment_ROSA/README.md‎

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
Fully reproducible deployment of an enterprise-grade private AI code assistant on Red Hat OpenShift (ROSA HCP), using **Terraform** for infrastructure provisioning and **ArgoCD (GitOps)** for all on-cluster components.
44

55
The end result is a ROSA HCP cluster running:
6-
- **Qwen/Qwen3.6-35B-A3B-FP8** served via KServe + llm-d with intelligent EPP routing
7-
- **OpenShift Dev Spaces** with pre-configured VS Code extensions (Roo Code, Continue, Cline) consuming the self-hosted model
8-
- All inference traffic stays **cluster-internal** (zero external egress)
6+
- **Qwen/Qwen3.6-35B-A3B-FP8** served via KServe + llm-d; EPP is disabled by default
7+
- **OpenShift Dev Spaces** with OpenCode by default; `type: continue` provides Continue, Cline, and Roo Code
8+
- Inference stays **cluster-internal** unless external Semantic Router backends are configured
99

1010
---
1111

@@ -66,11 +66,11 @@ private-coding-assistant/ ← repo root (fork and clone this)
6666
│ ├── pca-app-of-apps/ # Root ArgoCD AppProject + child Applications
6767
│ ├── pca-operators/ # Wave 1: Operator Subscriptions (RHOAI, GPU, DevSpaces, NFD, RHCL, …)
6868
│ ├── pca-platform-config/ # Wave 2: Namespace, RBAC, secrets, DSC, CheCluster
69-
│ │ ├── charts/pca-guardrails/ # Optional: TrustyAI guardrails proxy
7069
│ │ └── charts/pca-mcp/ # Optional: OpenShift MCP server
7170
│ ├── pca-ai-serving/ # Wave 3: LLMInferenceService, llm-d Gateway, RHCL AI Gateway
72-
│ │ └── charts/pca-observability/ # Optional: Grafana + Langfuse/OTel Collector
73-
│ ├── pca-devspaces/ # Wave 4: DevWorkspaces + Roo/Continue/Cline + API keys
71+
│ │ ├── charts/pca-guardrails/ # TrustyAI guardrails proxy (enabled by default)
72+
│ │ └── charts/pca-observability/ # Grafana + Langfuse/OTel Collector (enabled on ROSA)
73+
│ ├── pca-devspaces/ # Wave 4: OpenCode or Continue/Roo/Cline workspaces + API keys
7474
│ └── pca-benchmarks/ # Wave 5: GuideLLM sweep (disabled by default)
7575
│
7676
└── PCA_Deployment_ROSA/ ← You are here
@@ -503,7 +503,7 @@ helm template pca-root charts/pca-app-of-apps \
503503

504504
### Step 10: Monitor ArgoCD Sync
505505

506-
ArgoCD now automatically syncs all four waves. This takes **15-30 minutes** (longer if GPU nodes are still scaling).
506+
ArgoCD now automatically syncs all five waves. This takes **15-30 minutes** (longer if GPU nodes are still scaling).
507507

508508
#### 10a. Open the ArgoCD dashboard
509509

@@ -645,33 +645,33 @@ echo "Console: $(terraform output -raw cluster_console_url)"
645645

646646
### Wave 2 — Platform Configuration (`pca-platform-config`)
647647

648-
- Namespace: `ai-serving` (default; configurable via `aiNamespace` in values)
648+
- Namespace: `ai-serving` (`namespace` in chart values; align workspace `aiServingNamespace` and ArgoCD destinations when changing it)
649649
- DataScienceCluster and DSCI with KServe enabled
650650
- CheCluster instance in `openshift-devspaces`
651651
- NVIDIA ClusterPolicy and NFD instance
652652
- OAuth HTPasswd IDP with developer users
653653
- HuggingFace token secret
654-
- Optional: `pca-guardrails` (TrustyAI guardrails proxy) and `pca-mcp` (OpenShift MCP server)
654+
- Optional: `pca-mcp` (OpenShift MCP server)
655655

656656
### Wave 3 — AI Serving (`pca-ai-serving`)
657657

658658
- **PVC**: 100Gi `model-cache` on `gp3-csi` (persists model weights across restarts). Chart always sets `HF_HUB_OFFLINE=1` for LLMIS local-path load (storage-initializer populates the PVC before vLLM starts).
659659
- **LLMInferenceService** (`qwen3-coder`): `Qwen/Qwen3.6-35B-A3B-FP8` via `vllm.image` (API name matches `model.id`) with vLLM args:
660660
- `--tool-call-parser qwen3_xml --reasoning-parser qwen3`
661-
- `--max-model-len 32000 --gpu-memory-utilization 0.90`
661+
- `--max-model-len 32000`; GPU memory utilization uses vLLM's default unless overridden
662662
- `--enable-prefix-caching --kv-cache-dtype fp8`
663663
- HTTPS on :8000 (`--ssl-*` + probe `scheme: HTTPS`; requires cluster `enableLLMInferenceServiceTLS=true`)
664664
- Workload Service: `qwen3-coder-kserve-workload-svc:8000`
665-
- EPP scorer weights: queue=2, kv-cache=2, prefix-cache=3
666-
- **llm-d Gateway + HTTPRoute**: cluster-internal llm-d Gateway with EPP routing
665+
- **llm-d Gateway + HTTPRoute**: routes directly to the vLLM workload Service; EPP resources are disabled by default
667666
- **MaaS / RHCL Gateway** (`maas-default-gateway`): HTTPS front door with per-developer API key auth (AuthPolicy)
668667
- **HardwareProfile**: GPU hardware profile definition
669668
- **ServingRuntime** (optional): vLLM Neuron runtime template for Inferentia/Trainium only
670-
- **pca-observability** (optional): Grafana dashboards + optional Langfuse/OTel Collector
669+
- **pca-guardrails**: TrustyAI guardrails proxy (enabled by default)
670+
- **pca-observability**: Grafana dashboards + Langfuse/OTel Collector (enabled in the ROSA overlay)
671671

672672
### Wave 4 — Developer Workspaces (`pca-devspaces`)
673673

674-
- Roo Code, Continue, and Cline ConfigMaps per namespace (pre-configured with model endpoint)
674+
- OpenCode configuration at workspace startup; Roo Code, Continue, and Cline ConfigMaps for `type: continue` workspaces
675675
- Per-namespace API key secrets for RHCL AI Gateway authentication
676676
- DevWorkspaces with postStart auto-install of extensions
677677
- Global DevSpaces ConfigMaps in `openshift-devspaces`
@@ -683,19 +683,19 @@ echo "Console: $(terraform output -raw cluster_console_url)"
683683

684684
### Extension Pre-Configuration Details
685685

686-
Each extension connects to the self-hosted model endpoint. Here is exactly what is pre-populated:
686+
OpenCode is the default; `type: continue` installs all three extensions below per workspace. IDEs use MaaS with per-namespace API keys. Direct llm-d access is an escape hatch (`aiGateway.escapeHatchToLlmd=true`).
687687

688688
| Setting | Value |
689689
|---------|-------|
690-
| **Model Endpoint (Base URL)** | `https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1` |
690+
| **Model Endpoint (Base URL)** | `https://maas-default-gateway-data-science-gateway-class.openshift-ingress.svc.cluster.local/v1` |
691691
| **Model ID** | `Qwen/Qwen3.6-35B-A3B-FP8` |
692-
| **API Key** | `EMPTY` (no auth required — cluster-internal) |
692+
| **API Key** | Per-namespace key from the `pca-maas-apikey` Secret |
693693

694694
| Extension | Auto-Configured? | How | What the User Sees |
695695
|-----------|------------------|-----|---------------------|
696-
| **Roo Code** | Yes | ConfigMap auto-mounted into workspace | Profile named **"Private AI Assistant"** is pre-selected with all modes (architect, code, ask, debug) pointed at the model. Streaming is disabled (required for tool calling). |
697-
| **Continue** | Yes (dev1 only) | `~/.continue/config.yaml` written by postStart script | Model named **"Private AI Assistant"** appears in the chat panel. Tab autocomplete (named **"Private AI Autocomplete"**) is also pre-configured. |
698-
| **Cline** | Installed only (dev1 only) | Extension is installed but requires **manual UI setup** (Cline stores settings in VS Code `globalState`, not in files) | User must open Cline settings and enter: Provider = OpenAI Compatible, Base URL / Model ID / API Key from the table above. |
696+
| **Roo Code** | Yes | Provider profiles ConfigMap | Profile **"llm-d-qwen3-coder"** selects the model for all configured modes; streaming is disabled. |
697+
| **Continue** | Yes | `~/.continue/config.yaml` ConfigMap | **"Private model (llm-d)"** and **"Private model Autocomplete (llm-d)"** are configured for chat and tab autocomplete. |
698+
| **Cline** | Provider files supplied | Provider ConfigMaps + postStart copy | OpenAI-compatible URL, model ID, and API key are supplied for each workspace. |
699699

700700
---
701701

@@ -725,10 +725,10 @@ Each extension connects to the self-hosted model endpoint. Here is exactly what
725725
| vLLM Argument | Value | Purpose |
726726
|---------------|-------|---------|
727727
| `--max-model-len` | `32000` (`tokens.total`) | Context window (tokens) |
728-
| `--gpu-memory-utilization` | `0.90` | GPU memory fraction |
728+
| `--gpu-memory-utilization` | Unset unless overridden | Uses vLLM's default GPU memory fraction |
729729
| `--enable-prefix-caching` | — | KV-cache reuse across requests |
730730
| `--enable-auto-tool-choice` | — | Tool/function calling support |
731-
| `--tool-call-parser` | `qwen3_coder` | Tool call format parser |
731+
| `--tool-call-parser` | `qwen3_xml` | Tool call format parser for the default Qwen3.6 model |
732732
| `--reasoning-parser` | `qwen3` | Reasoning extraction |
733733
| `--kv-cache-dtype` | `fp8` | Memory-efficient KV-cache |
734734

@@ -909,7 +909,7 @@ oc get gateway,httproute -n ai-serving
909909
| ROSA HCP | 4.21.7 | Terraform (RHCS provider) |
910910
| RHOAI | 3.3 | OperatorHub (stable-2.19) |
911911
| KServe | 0.15 | Managed by RHOAI |
912-
| llm-d EPP | 0.4 | Managed by RHOAI |
912+
| llm-d EPP | Disabled by default | Optional chart resources |
913913
| vLLM | 0.13.0+rhai11 | RHOAI runtime image |
914914
| Service Mesh | 3.2+ | OperatorHub (stable) |
915915
| NVIDIA GPU Operator | 26.3 | OperatorHub (v26.3) |

‎README.md‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Private AI Code Assistant on Red Hat OpenShift
22

3-
Deploy a private self-hosted AI coding assistant on OpenShift so developers get AI-powered IDEs while no code leaves your environment
3+
Deploy a self-hosted AI coding assistant on OpenShift, with private inference unless you enable external model backends.
44

55
## Table of Contents
66

@@ -30,26 +30,26 @@ Teams want IDE-integrated AI coding help without sending proprietary source to p
3030

3131
This quickstart deploys:
3232

33-
- **OpenShift Dev Spaces** workspaces with OpenCode (Continue, Cline, Roo Code optional)
34-
- **On-cluster inference** with vLLM behind **llm-d** intelligent routing — default model `Qwen/Qwen3.6-35B-A3B-FP8` (you can change it to any open-source model you want)
35-
- An optional **RHCL AI Gateway** front door with per-developer API keys
33+
- **OpenShift Dev Spaces** workspaces offering OpenCode (default), Continue, Cline, or Roo Code
34+
- **On-cluster vLLM inference** through **llm-d** — default model `Qwen/Qwen3.6-35B-A3B-FP8`, configurable for compatible models
35+
- **MaaS / RHCL AI Gateway** with per-developer API keys, enabled by default
3636
- GitOps Helm charts (ROSA/ARO) or Helm-only install on an existing cluster
3737

3838
### Features
3939

4040
- Source and inference stay inside your OpenShift network boundary when configured that way
4141
- OpenAI-compatible `/v1` API for IDE extensions
42-
- Prefix-cache–aware routing via llm-d Endpoint Picker (EPP)
42+
- vLLM prefix caching; EPP off by default
4343
- Multi-developer namespaces with ConfigMap-prewired extensions
44-
- Optional observability (Grafana; Langfuse/OTel), guardrails, and MCP
44+
- Grafana/guardrails by default; ROSA/ARO Langfuse/OTel; optional MCP
4545

4646
### Solution stack
4747

4848
| Layer | Technology |
4949
|-------|------------|
5050
| IDE | OpenShift Dev Spaces |
5151
| Auth front door | MaaS / RHCL (`maas-default-gateway`) |
52-
| Routing | llm-d + Gateway API + EPP |
52+
| Routing | llm-d + Gateway API; optional EPP |
5353
| Serving | vLLM / KServe `LLMInferenceService` |
5454
| Platform | Red Hat OpenShift AI, NVIDIA GPU Operator, NFD |
5555
| Delivery | Unified Helm charts in `charts/` (ArgoCD or `make`) |
@@ -58,6 +58,8 @@ This quickstart deploys:
5858

5959
Developers → Dev Spaces → maas-default-gateway → (optional guardrails / Semantic Router) → llm-d → vLLM on NVIDIA GPU.
6060

61+
Illustrative inference traffic flow:
62+
6163
![Inference traffic flow from Dev Spaces through RHCL AI Gateway to llm-d and vLLM](docs/images/architecture-traffic-flow.svg)
6264

6365
Full diagrams: [docs/architecture.md](docs/architecture.md).
@@ -158,7 +160,7 @@ make ai-serving-undeploy-existing-openshift DELETE_NAMESPACE=1
158160
|-----|--------|
159161
| [docs/architecture.md](docs/architecture.md) | Stack, traffic path, diagrams |
160162
| [docs/requirements.md](docs/requirements.md) | Hardware, software, permissions |
161-
| [docs/ide-and-extensions.md](docs/ide-and-extensions.md) | Dev Spaces, Continue / Cline / Roo |
163+
| [docs/ide-and-extensions.md](docs/ide-and-extensions.md) | Dev Spaces, OpenCode / Continue / Cline / Roo Code |
162164
| [docs/models-and-routing.md](docs/models-and-routing.md) | vLLM, llm-d, RHCL |
163165
| [docs/semantic-router.md](docs/semantic-router.md) | Official vLLM SR hop, extras, inject |
164166
| [docs/models-and-hardware.md](docs/models-and-hardware.md) | Changing models and accelerators |
@@ -177,7 +179,7 @@ make ai-serving-undeploy-existing-openshift DELETE_NAMESPACE=1
177179
## Tags
178180

179181
- **Title:** Private AI Code Assistant on Red Hat OpenShift
180-
- **Description:** Deploy a private self-hosted AI coding assistant on OpenShift so developers get AI-powered IDEs while no code leaves your environment
182+
- **Description:** Deploy a self-hosted AI coding assistant on OpenShift, with private inference unless you enable external model backends.
181183
- **Industry:** Media and IT services
182184
- **Product:** OpenShift AI
183185
- **Use case:** AI coding assistant, private inference

‎assets/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,22 +4,22 @@
44

55
| File | Purpose |
66
|------|---------|
7-
| `Dockerfile.opencode` | Custom DevSpaces image with OpenCode v1.15.6 pre-installed and configured for private AI Gateway |
7+
| `Dockerfile.opencode` | Custom DevSpaces image with OpenCode v1.15.13 pre-installed and configured for private AI Gateway |
88
| `devfile.yaml` | DevSpaces workspace template — appears on the DevSpaces dashboard as a launchable sample |
99
| `build-opencode-image.sh` | Script to build the custom image on any OpenShift cluster |
1010

1111
## Image Details
1212

1313
- **Base**: `registry.redhat.io/devspaces/udi-rhel8:latest` (Red Hat Universal Developer Image)
14-
- **Added**: OpenCode v1.15.6 CLI, pre-configured for the cluster-internal llm-d AI Gateway
14+
- **Added**: OpenCode v1.15.13 CLI, pre-configured for the cluster-internal llm-d AI Gateway
1515
- **Endpoint**: `https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1`
1616
- **Model**: `Qwen/Qwen3.6-35B-A3B-FP8`
1717
- **SHA**: `sha256:dcb6b1cee00115467e6c5432930416a7e8b868b6a8e5ee3075ce6823ff7e2e98`
1818

1919
## Building on a New Cluster
2020

21-
The GitOps manifests in `argocd/04-devspaces/` automatically create a BuildConfig
22-
and trigger the image build. For manual builds:
21+
The Helm chart in `charts/pca-devspaces/` automatically creates a BuildConfig
22+
and triggers the image build. For manual builds:
2323

2424
```bash
2525
./build-opencode-image.sh

‎deploy_existing_openshift/README.md‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,7 @@ oc patch datasciencecluster default-dsc -n redhat-ods-applications --type merge
100100
101101
### Upgrading: guardrails moved to ai-serving
102102
103-
If you previously deployed guardrails under `pca-platform-config`, upgrade **ai-serving before platform-config** so guardrails are recreated in the AI namespace before the old platform release prunes them. ArgoCD wave order (platform wave 2, ai-serving wave 3) already does this; for `make ai-serving-deploy-existing-openshift`, run that target before any platform-config upgrade that removes the old subchart.
103+
ArgoCD and `make ai-serving-deploy-existing-openshift` deploy **platform-config before ai-serving**. Migrating guardrails from an older platform-config release requires a separate staged upgrade; the normal deployment order does not perform that migration.
104104
105105
### Demo DevSpaces (`dev-user1..N`)
106106
@@ -126,7 +126,7 @@ make devspace-deploy-existing-openshift DEV_USER=dev-user2
126126
127127
> **OpenCode build:** `opencode-build` is cluster-singleton infrastructure owned by the first Helm release. The Makefile auto-detects whether it already exists and suppresses `opencodeBuild` for subsequent runs. If deploying manually with `helm upgrade --install`, pass `--set opencodeBuild.enabled=false` from the second developer onward.
128128
>
129-
> **Do not uninstall** the release that owns `opencode-build` while other OpenCode workspaces exist — doing so deletes the BuildConfig and ImageStream (the Namespace is kept), breaking image pulls for all running workspaces. Recovery: delete the empty namespace (`oc delete namespace opencode-build`), then re-run any opencode devspace deploy without `--set opencodeBuild.enabled=false` to recreate the full build infrastructure.
129+
> **Shared build resources:** Helm uninstall preserves the `opencode-build` Namespace, BuildConfig, ImageStream, and image-puller RoleBindings (`helm.sh/resource-policy: keep`).
130130
131131
## Parameters
132132
@@ -151,8 +151,7 @@ make devspace-deploy-existing-openshift DEV_USER=dev-user2
151151
| `--set opencodeBuild.enabled=false` | 2nd+ opencode developer — avoids Helm ownership conflict on the shared `opencode-build` namespace (Makefile detects this automatically) |
152152
| `--set pca-observability.langfuse.enabled=true` | Opt in Langfuse (+ OTel) with AI serving |
153153
| `--set guardrails.enabled=false` | Disable guardrails subchart on ai-serving (default is enabled) |
154-
| `--set aiGateway.escapeHatchToLlmd=true` | Skip MaaS; IDEs call llm-d Gateway directly |
155-
| `--set maas.enabled=false` | Disable MaaS HTTPRoute / AuthPolicy (IDEs fall back to llm-d) |
154+
| `--set aiGateway.escapeHatchToLlmd=true` | On DevSpaces, skip MaaS; IDEs call llm-d Gateway directly |
156155
| `--set maas.hostname=<Gateway/Route host>` | Pin HTTPRoute, AuthConfig, and EnvoyFilter to the public MaaS host (`oc get gateway maas-default-gateway -n openshift-ingress`). Empty default is Istio catch-all `*:443`. Do not commit the cluster hostname in values files. |
157156
| `--set semanticRouter.enabled=true,global.semanticRouter.enabled=true` | Deploy the Semantic Router hop (pins local Qwen until extras exist). Prefer `SEMANTIC_ROUTER_ENABLED=true` so Make sets both flags |
158157
@@ -198,7 +197,7 @@ Grafana (boards B/C — latency, KV/GPU) deploys by default with AI serving via
198197
199198
| Flag | Default | What you get |
200199
|------|---------|--------------|
201-
| `grafana.enabled` | `true` | 1-pod Grafana + boards B/C. Boards A/D when Langfuse is on |
200+
| `pca-observability.grafana.enabled` | `true` | 1-pod Grafana + boards B/C. Boards A/D when Langfuse is on |
202201
| `pca-observability.langfuse.enabled` | `false` | Langfuse + OTel Collector; wires vLLM OTLP in the same release |
203202
| `pca-observability.langfuse.ioCapture` | `full` | When Langfuse is on: store full prompt/completion via vLLM middleware (async). Set `metadata` for tokens/latency only |
204203
@@ -222,7 +221,7 @@ oc get secret pca-langfuse-credentials -n $AI_NAMESPACE -o jsonpath='{.data.init
222221
```bash
223222
make ai-serving-deploy-existing-openshift HF_TOKEN=hf_xxx MCP_ENABLED=true \
224223
HELM_ARGS='--set pca-observability.langfuse.enabled=true'
225-
make devspace-deploy-existing-openshift N=2 MCP_ENABLED=true
224+
make devspace-deploy-existing-openshift N=2 TYPE=continue MCP_ENABLED=true
226225
```
227226
228227
---
@@ -233,15 +232,17 @@ MCP gives AI coding extensions (Continue, Roo Code) live read-only access to clu
233232
234233
### Deploy with MCP enabled from the start
235234
236-
Pass `MCP_ENABLED=true` to both the AI serving and devspace make targets:
235+
Pass `MCP_ENABLED=true` to both targets and `TYPE=continue` for automatic Continue/Roo Code wiring. The default OpenCode configuration has no MCP entry.
237236
238237
```bash
239238
make ai-serving-deploy-existing-openshift HF_TOKEN=hf_xxx MCP_ENABLED=true
240-
make devspace-deploy-existing-openshift N=2 MCP_ENABLED=true
239+
make devspace-deploy-existing-openshift N=2 TYPE=continue MCP_ENABLED=true
241240
```
242241
243242
### Enable MCP on an already-running deployment
244243
244+
Automatic IDE wiring applies to `type: continue` workspaces:
245+
245246
```bash
246247
make mcp-enable AI_NAMESPACE=<ai-ns> DEV_USER=dev-user1
247248
```
@@ -298,10 +299,12 @@ This target:
298299
299300
The DevWorkspace is created with `started: false`.
300301
301-
### Step 2 — Trigger the image build (first time only)
302+
### Step 2 — Wait for the image build (first time only)
303+
304+
The BuildConfig's `ConfigChange` trigger starts the build automatically. Follow its logs:
302305
303306
```bash
304-
oc start-build devspaces-opencode -n opencode-build --follow
307+
oc logs -f bc/devspaces-opencode -n opencode-build
305308
```
306309
307310
The build installs OpenCode CLI (version pinned via `opencodeBuild.opencodeVersion` in values) and stubs `xdg-open` to prevent crashes in headless environments.

0 commit comments

Comments
 (0)