Sample/ai gateway observability

samples/ai-gateway-observability/.gitattributes

@@ -0,0 +1,8 @@
+# Keep the shell scripts LF-only in the working tree on every platform.
+# Git on Windows often converts checked-out files to CRLF, and bash refuses a script
+# whose shebang line ends in CR: "/usr/bin/env: 'bash\r': No such file or directory".
+# Windows users run this sample inside WSL2, where that conversion would break it.
+*.sh text eol=lf
+*.yaml text eol=lf
+*.yml text eol=lf
+*.toml text eol=lf

samples/ai-gateway-observability/.github/workflows/ci.yml

@@ -0,0 +1,55 @@
+name: AI Gateway Observability Sample PR Check
+
+on:
+  workflow_dispatch:
+  pull_request:
+    branches:
+      - main
+    paths:
+      - 'samples/ai-gateway-observability/**'
+
+permissions:
+  contents: read
+
+jobs:
+  pr-check:
+    runs-on: ubuntu-24.04
+    defaults:
+      run:
+        working-directory: samples/ai-gateway-observability
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - name: Install dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y jq
+
+      - name: Shell syntax check
+        run: bash -n setup.sh load.sh test.sh teardown.sh
+
+      - name: Validate dashboard JSON and WireMock mappings
+        run: |
+          python3 -c "import json,glob; [json.load(open(f)) for f in glob.glob('observability/*.json') + glob.glob('wiremock/mappings/*.json')]"
+
+      - name: Start the gateway and observability stack
+        run: ./setup.sh
+
+      - name: Generate traffic
+        run: ./load.sh 45
+
+      - name: Assert metrics and traces are flowing
+        run: ./test.sh
+
+      - name: Dump container logs on failure
+        if: failure()
+        run: |
+          docker ps -a
+          (cd wso2apip-ai-gateway-1.2.0 && docker compose --profile metrics --profile tracing logs --tail 200) || true
+
+      - name: Teardown
+        if: always()
+        run: ./teardown.sh --clean

samples/ai-gateway-observability/.gitignore

@@ -0,0 +1,6 @@
+# The gateway distribution downloaded and extracted by setup.sh. The zip is already
+# covered by the repository's root .gitignore; the extracted directory is not, and it
+# holds generated secrets (listener key, AES key, api-platform.env) that must never be
+# committed. ./teardown.sh --clean removes both.
+wso2apip-ai-gateway-*/
+wso2apip-ai-gateway-*.zip

samples/ai-gateway-observability/README.md

@@ -0,0 +1,154 @@
+# AI Gateway: Metrics and Tracing with a Ready-Made Dashboard
+
+This sample runs the WSO2 AI Gateway with its full observability stack switched on
+(Prometheus, Grafana, an OpenTelemetry collector and Jaeger), pointed at two LLM
+proxies backed by a mock model. Generate a minute of traffic, and you get a live
+dashboard showing request rate, latency and errors per proxy, plus a complete trace of
+any single request through the gateway. No OpenAI API key, no cloud account, nothing
+to configure by hand.
+
+## Prerequisites
+
+- Docker with the Compose plugin
+- `curl` (or `wget`), `unzip`, `jq` and `openssl`
+
+On Windows, run these from a WSL2 shell with Docker Desktop's WSL integration enabled.
+
+## Getting started
+
+```bash
+./setup.sh
+```
+
+1. Downloads and extracts the AI Gateway distribution.
+2. Enables the Prometheus endpoints and tracing, and provisions the Grafana dashboard.
+3. Starts a WireMock container standing in for the OpenAI API.
+4. Starts the gateway together with Prometheus, Grafana, Jaeger and the OTel collector.
+5. Waits for the gateway to report healthy, then puts the mock on its network.
+6. Registers two proxies, `assistant-proxy` and `support-proxy`, each with an inbound
+   API key. Only the provider behind `support-proxy` has a token budget.
+
+Credentials, certificates and the environment file the stack needs are generated in
+step 4, so there is nothing to configure beforehand.
+
+```bash
+./load.sh
+```
+
+1. Checks the gateway is running, and stops if it is not.
+2. Sends a request every 0.25 seconds for 60 seconds, alternating between the two
+   proxies. Pass a different duration as `./load.sh 120`.
+3. Cycles through a fixed pattern every ten requests: one answered slowly, one failing
+   upstream, one with an invalid key, and seven ordinary ones. The same every run.
+4. Counts every response by status code and prints the totals.
+
+Then open the two URLs the scripts print:
+
+| URL | What you see |
+|-----|--------------|
+| <http://localhost:3000> | **Grafana**: the AI Gateway Overview dashboard, live (admin / admin) |
+| <http://localhost:16686> | **Jaeger**: pick the `router` service, open any trace |
+
+## What to look for
+
+**In Grafana**, the dashboard opens on the AI Gateway Overview: four tiles showing peak
+values, and six charts.
+
+- *Request rate per proxy*: how much traffic each proxy is handling.
+- *Gateway processing time per route*: how long the gateway itself takes, as a typical
+  time (p50) and a slow-tail time (p95).
+- *End-to-end latency*: how long the whole call takes, including the model.
+- *Responses by status class*: how many requests succeeded (2xx) against how many were
+  rejected (4xx) or failed (5xx).
+- *Policy rejections by policy*: requests the gateway blocked, and which rule blocked
+  them. Bad keys throughout, the token budget from part-way through the run.
+- *Upstream failures*: requests the model backend itself failed.
+
+**In Jaeger**, open a trace to see one request broken into its steps. The charts show
+totals across all traffic; a trace shows where a single slow request lost its time.
+
+## Verify from the terminal
+
+`test.sh` checks the pipeline end to end: the metrics endpoints respond, Prometheus is
+scraping all three of them, both proxies report per-proxy metrics, Grafana loaded the
+dashboard, and Jaeger stored traces.
+
+```bash
+./test.sh
+```
+
+Expected output:
+
+```
+══════════════════════════════════════════════════
+ Pre-flight checks
+══════════════════════════════════════════════════
+[INFO] Checking gateway health at http://localhost:9094/health ...
+[PASS] Gateway is healthy.
+
+══════════════════════════════════════════════════
+ Test 1: Metrics endpoints respond
+══════════════════════════════════════════════════
+[PASS] Gateway controller: HTTP 200 (http://localhost:9011/metrics)
+[PASS] Policy engine: HTTP 200 (http://localhost:9003/metrics)
+[PASS] Envoy router: HTTP 200 (http://localhost:9901/stats/prometheus)
+...
+[PASS]  Observability pipeline is working end to end.
+```
+
+## How it works
+
+```
+  ./load.sh ──► Gateway :8080 ──► mock LLM (WireMock)
+                     │
+        ┌────────────┴────────────┐
+        │                         │
+   Prometheus scrapes        gateway pushes
+   metrics every 15s         traces (OpenTelemetry)
+        │                         │
+        ▼                         ▼
+   Grafana :3000             Jaeger :16686
+```
+
+- **Metrics** answer how much traffic, how fast, and how often broken. Token usage is
+  not among them; that goes to analytics (Moesif), not metrics.
+- **Traces** answer where a single request spent its time.
+
+## What's running
+
+| Container | Role | Port |
+|-----------|------|------|
+| `gateway-controller` | Control plane, where proxies are registered | 9090 (API), 9011 (metrics) |
+| `gateway-runtime` | Envoy router + policy engine, where traffic flows | 8080 (HTTP), 9901 (Envoy admin), 9003 (metrics) |
+| `mock-llm-openai` | WireMock standing in for the OpenAI API | 8082 |
+| `prometheus` | Scrapes and stores the metrics | 9092 |
+| `grafana` | Charts them | 3000 |
+| `otel-collector` | Receives spans from the gateway | 4317 / 4318 |
+| `jaeger` | Stores and displays traces | 16686 |
+
+## Send your own request
+
+```bash
+curl -X POST http://localhost:8080/assistant/chat/completions \
+  -H "Content-Type: application/json" \
+  -H "api_key: demo-assistant-key" \
+  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hello"}]}'
+```
+
+`demo-assistant-key` is the inbound key `setup.sh` registered on the proxy; the gateway
+rejects requests without it. Watch the request land on the dashboard, then find its
+trace in Jaeger. The support proxy is at `/support/chat/completions` with
+`demo-support-key`.
+
+Ports, credentials, keys and traffic duration are all environment variables at the top
+of `setup.sh` and `load.sh`. Override any of them before running.
+
+## Teardown
+
+```bash
+./teardown.sh            # delete the proxies, stop the containers, drop the volumes
+./teardown.sh --clean    # also remove the extracted distribution and the zip
+```
+
+`--clean` makes the next `./setup.sh` download and set up the gateway from scratch,
+instead of reusing the copy already on disk.

samples/ai-gateway-observability/additional-config.toml

@@ -0,0 +1,24 @@
+# Merged into the distribution's configs/config.toml by setup.sh.
+#
+# [controller.metrics] and [policy_engine.metrics] expose the Prometheus endpoints the
+# Grafana panels are built from. [tracing] exports OpenTelemetry spans to the
+# otel-collector container, which forwards them to Jaeger.
+
+[controller.metrics]
+enabled = true
+port = 9091
+
+[policy_engine.metrics]
+enabled = true
+port = 9003
+
+[tracing]
+enabled = true
+endpoint = "otel-collector:4317"
+insecure = true
+service_version = "1.0.0"
+batch_timeout = "1s"
+max_export_batch_size = 512
+# Sample every request. Fine for a demo; production deployments normally sample
+# a small fraction of traffic instead.
+sampling_rate = 1.0

samples/ai-gateway-observability/llm-provider-budgeted.yaml

@@ -0,0 +1,34 @@
+apiVersion: gateway.api-platform.wso2.com/v1
+kind: LlmProvider
+metadata:
+  name: mock-openai-provider-budgeted
+spec:
+  displayName: Mock OpenAI Provider (token budget)
+  version: v1.0
+  template: openai
+  context: /openai/budgeted
+  upstream:
+    # Same mock backend as llm-provider.yaml — the only difference is the token
+    # budget below. The support proxy points here so that, part-way through
+    # load.sh, it starts returning 429 while the assistant proxy keeps serving.
+    # That contrast is what makes the per-proxy panels worth looking at.
+    url: http://mock-llm-openai:8080/v1
+    auth:
+      type: api-key
+      header: Authorization
+      value: Bearer local-mock-key
+  accessControl:
+    mode: deny_all
+    exceptions:
+      - path: /chat/completions
+        methods: [POST]
+  policies:
+    - name: token-based-ratelimit
+      version: v1
+      paths:
+        - path: /chat/completions
+          methods: [POST]
+          params:
+            totalTokenLimits:
+              - count: 300
+                duration: "1m"

samples/ai-gateway-observability/llm-provider.yaml

@@ -0,0 +1,21 @@
+apiVersion: gateway.api-platform.wso2.com/v1
+kind: LlmProvider
+metadata:
+  name: mock-openai-provider
+spec:
+  displayName: Mock OpenAI Provider
+  version: v1.0
+  template: openai
+  context: /openai/latest
+  upstream:
+    # WireMock stands in for api.openai.com — see wiremock/mappings/.
+    url: http://mock-llm-openai:8080/v1
+    auth:
+      type: api-key
+      header: Authorization
+      value: Bearer local-mock-key
+  accessControl:
+    mode: deny_all
+    exceptions:
+      - path: /chat/completions
+        methods: [POST]

samples/ai-gateway-observability/llm-proxy-assistant.yaml

@@ -0,0 +1,19 @@
+apiVersion: gateway.api-platform.wso2.com/v1
+kind: LlmProxy
+metadata:
+  name: assistant-proxy
+spec:
+  displayName: Assistant Proxy
+  version: v1.0
+  context: /assistant
+  provider:
+    id: mock-openai-provider
+  policies:
+    - name: api-key-auth
+      version: v1
+      paths:
+        - path: /chat/completions
+          methods: [POST]
+          params:
+            key: api_key
+            in: header

samples/ai-gateway-observability/llm-proxy-support.yaml

@@ -0,0 +1,20 @@
+apiVersion: gateway.api-platform.wso2.com/v1
+kind: LlmProxy
+metadata:
+  name: support-proxy
+spec:
+  displayName: Support Proxy
+  version: v1.0
+  context: /support
+  provider:
+    # The budgeted provider — this proxy runs out of tokens under load.
+    id: mock-openai-provider-budgeted
+  policies:
+    - name: api-key-auth
+      version: v1
+      paths:
+        - path: /chat/completions
+          methods: [POST]
+          params:
+            key: api_key
+            in: header