Investigate infrastructure failures and radio anomalies, run remediation playbooks, and track incident outcomes in connected dashboards.
Deploy on your own OpenShift cluster → Single-cluster deployment guide
Network operations teams need to investigate infrastructure failures and degraded radio service. This quickstart provides two workflows: Network investigates OpenShift workload alerts, while RAN (radio access network) detects unusual radio performance measurements. Both combine incident evidence with reference documents to explain likely causes and proposed actions.
Operators can follow diagnosis and remediation in separate, linked dashboards, reducing the need to gather evidence across tools. Ansible Automation Platform (AAP) executes configured playbooks, and ServiceNow records Network incidents that need escalation.
This guide covers deployment with real integrations. For contributing, local development, simulated incidents, and mock services, use the contributor guide. Mocks and automation bypasses are for development only.
Workload alerts and radio measurements feed separate workflows. Shared AI services support diagnosis; AAP runs remediation, ServiceNow handles Network escalation, and dashboards show outcomes.
See the architecture guide for deployment topology and service details.
The existing Network sizing baseline is listed below. A combined Network/RAN minimum has not been established; reserve additional capacity for RAN services and any locally hosted predictor.
- CPU: 8+ vCPU cores for Network.
- Memory: 32+ GB RAM for Network.
- Storage: 100+ GB with dynamic volume provisioning. The bundled Kafka uses the
gp3-csistorage class by default. - GPU: One NVIDIA A10G, A100, L40S, or equivalent for local Granite model serving. An external model endpoint avoids this local GPU requirement.
- Network: Access from hub pods to container registries, Helm chart repositories, model endpoints, AAP, and ServiceNow.
These are documented prerequisites. Exact tested versions and a combined Network/RAN sizing profile have not been recorded.
| Requirement | Purpose |
|---|---|
| OpenShift 4.21+ and OpenShift AI 3.4+ | Application and AI platform; aligned with the topology guide |
Llama Stack operator with LlamaStackDistribution available |
Model access and document retrieval |
oc matching the cluster, Helm 3+, GNU Make, Git, Bash, Python 3 |
Deployment tools |
| OpenAI-compatible language-model endpoint and credentials | Network and RAN diagnosis and chat |
RAN predictor accepting POST /v1/detect |
Anomaly detection; see the predictor guide |
| AAP 2.5+, its operator, and a configured controller | Playbook execution |
| Ansible Lightspeed enabled in AAP | Generate remediation playbooks |
| ServiceNow instance and an integration account with incident access | Network escalation and incident updates |
Provide workload logs through OpenShift Logging and a producer for RAN measurements. The operational input setup covers logging prerequisites, Kafka client certificates, and the measurement format.
Use cluster-admin or equivalent permissions for operator installation and cluster-scoped access rules.
Set up AAP on the hub cluster: Follow the AAP installation guide for OpenShift to install the operator and deploy the platform. Enable the controller in the AnsibleAutomationPlatform resource with spec.controller.disabled: false.
Create an AAP token: Follow Adding tokens in the AAP authentication guide. Create a personal access token with Write scope, copy its value when shown, and set AAP_TOKEN in .env.
Complete the integration setup guide to prepare the AAP project, lightspeed-runner template, target-cluster credentials, and Lightspeed. For RAN, provision working playbooks and matching ran-* templates using the remediation guide; deployment does not install these on a real controller.
For multiple clusters, use the multi-cluster guide, including its OpenShift AI 3.4, ACM 2.15+, GitOps, and edge logging prerequisites. Each edge cluster requires OpenShift 4.21+, 16+ GB RAM, and TLS-secured Kafka connectivity to the hub.
Operator needs depend on the topology: single-cluster (CLUSTER_COUNT=1) or hub-spoke (CLUSTER_COUNT>=2).
| Operator | Single-cluster | Multi-cluster hub | Multi-cluster spokes |
|---|---|---|---|
| OpenShift AI (Llama Stack) | Required | Required | - |
| Ansible Automation Platform | Required | Required | - |
| OpenShift GitOps | Recommended | Required | - |
| Advanced Cluster Management (with multicluster-engine) | Not used | Required | - |
| Cluster Logging | Optional | Required | Required |
| Loki operator | Optional | Required | - |
- OpenShift AI: enable the Llama Stack component in the
DataScienceCluster(llamastackoperator=Managed) soLlamaStackDistributionis available. - OpenShift GitOps: single-cluster uses Argo CD for the edge chart and falls back to a Helm install when GitOps is missing; hub-spoke requires Application and ApplicationSet APIs for edge fan-out.
- Cluster Logging and Loki operator: single-cluster leaves log forwarding and the hub LokiStack off by default (
CLUSTER_LOG_FORWARDER_ENABLED=false,ENABLE_LOKISTACK=false); enable both settings to require these operators. Hub-spoke enables both by default: the hub aggregates logs in a LokiStack, and each spoke needs the Cluster Logging operator (stable-6.x channel) so its edgeClusterLogForwarderreconciles and ships logs to hub Kafka. - Spokes: no operators beyond Cluster Logging; edge workloads are delivered through ACM and GitOps.
Install order and verification commands for the hub are in the multi-cluster guide.
The following steps install Network and RAN on one hub cluster, with edge workloads on that same cluster. Use dedicated namespaces for NAMESPACE and EDGE_NAMESPACE: cleanup deletes both namespaces and their data.
-
Clone the repository.
git clone https://github.com/rh-ai-quickstart/ai-driven-network-remediation.git cd ai-driven-network-remediation -
Configure the deployment. Create
.envif it does not already exist:test -e .env || cp .env.example .env
Edit
.envand supply these values. Replace example endpoints and credentials before continuing.Setting Value NAMESPACE,EDGE_NAMESPACEDedicated namespaces; defaults are hubanddark-noc-edgeCLUSTER_COUNT1ADNR_LLM_ID,ADNR_LLM_URL,ADNR_LLM_TOKENModel ID, OpenAI-compatible URL ending in /v1, and API tokenADNR_DETECT_INFERENCE_URLRAN predictor URL including /v1/detectADNR_DETECT_TOKENPredictor token, or empty if authentication is not required ENABLE_AAP_MOCK,ENABLE_SERVICENOW_MOCKBoth false; deployment defaults otherwise enable development mocksAAP_TOKENAAP OAuth2 token; leave AAP_SECRET_NAMEunset for this procedureAAP_NAMESPACEController namespace; defaults to aapENABLE_LIGHTSPEEDtrue; deployment discovers the Lightspeed URL and token when unsetLIGHTSPEED_VERIFY_SSLtrue; use a Lightspeed endpoint with a trusted certificate as described in the setup guideSERVICENOW_INSTANCE_URL,SERVICENOW_API_KEYServiceNow base URL and the API key from make servicenow-bootstrap; used whenENABLE_SERVICENOW_MOCK=falseHELM_EXTRA_ARGSUncomment and customize the real-integration override example in .env.examplefor RAN's AAP endpointFRONTEND_AUTH_ENABLEDtrueto require OpenShift login for both dashboardsKeep
ENABLE_NETWORK_REMEDIATION=trueandENABLE_TELCO_ORAN=true. Customize the real-integration example in.env.example; see integration endpoints for controller addresses and ServiceNow credentials.Real ServiceNow setup: Run
make servicenow-bootstraponce with your PDI admin credentials to provision the integration account and an API key. Then setSERVICENOW_INSTANCE_URLandSERVICENOW_API_KEY, and setENABLE_SERVICENOW_MOCK=false. See scripts/servicenow-bootstrap/README.md for details.Current limitation: Authenticated ServiceNow incident counts are unavailable in the Network dashboard through this deployment configuration. The escalation integration is configured separately.
Quote values containing spaces or shell special characters. Configure operational inputs for workload logs and radio measurements so the workflows receive incidents.
-
Log in to the hub cluster and deploy. From the repository root, run in Bash:
set -a && source .env && set +a make deploy
Reload
.envafter editing it or opening a new shell; Make uses the exported values. For hub-spoke deployments, follow the topology deployment guide. -
Verify deployment and open the dashboards. Use the namespace values loaded from
.envin the previous step.oc get pods -n "$NAMESPACE" oc get pvc -n "$NAMESPACE" oc get pods -n "$EDGE_NAMESPACE" oc get route hub-frontend -n "$NAMESPACE" -o jsonpath='https://{.spec.host}{"\n"}' oc get route hub-ran-frontend -n "$NAMESPACE" -o jsonpath='https://{.spec.host}{"\n"}'
Long-running pods should be Ready, setup jobs may be Completed, and volume claims should be Bound. Open each printed URL and sign in with OpenShift. The Network dashboard shows infrastructure incidents and escalation status; the RAN dashboard shows radio anomalies, diagnoses, and remediation outcomes.
For an incident from your monitored environment, follow its diagnosis in the dashboard, confirm any launched job in AAP, and verify the workload or radio condition has recovered. Confirm escalated Network incidents in ServiceNow.
If incidents or diagnoses do not appear, follow the troubleshooting checks.
From the repository root, load the same .env used for deployment and run:
set -a && source .env && set +a
make teardown- Network operations: Investigate infrastructure incidents and track response outcomes.
- Radio access networks: Detect unusual radio measurements and explain likely causes.
- AI-assisted diagnosis: Combine incident evidence with reference documents.
- Automation: Run remediation playbooks and escalate incidents that need attention.