Multivendor Agentic RCA for Cloud RAN
An on-demand 5G RAN and core sandbox on OpenShift, a live PTP timing-fault injection, and a governed agent that finds the root cause by collecting evidence from every vendor's layer.
For Red Hat solution architects. Source: github.com/dkypuros/multivendor-agentic-rca
1The problem
A cell goes down. In a multivendor RAN that one symptom can come from:
- the RAN software (one vendor)
- the cloud platform (another)
- the timing network (a third)
- the NIC silicon and firmware (a fourth)
Each vendor's tools see only their own layer. The NOC sees "cell unavailable".
2The idea
Click any box to open its code on GitHub.
3Demo flow
| Click | What you see | Code |
|---|---|---|
| Start RAN Slice | Pod scales 0→1, 32 NFs boot, cell ACTIVE, UE registers (5G-AKA), 3/3 GTP-U echoes | launch_slice.py, ue_sim.py |
| Inject Fault | PTP FREERUN (−50 ms), cell locked over O1, O-DU alarm lossOfRealTimeSynchronization | ptp_bridge.py |
| Run RCA | CAPIF token, 4 MCP tool calls, OC-TimingDegraded, 2/3 → HOLD, TMF688 event | nep_orchestrator.py |
| Heal Timing | PTP LOCKED, cell ACTIVE, alarm cleared | sandbox_controller.py |
| Run RCA again | 0/3 → NO ACTION: the verdict follows the evidence | example run |
| Stop RAN Slice | Pod scales 1→0, CPU and memory released | ran-slice.yaml |
Full presenter script with talk tracks: demo walkthrough (about 20 minutes).
4Block diagram
Click any box to open its code on GitHub. Manifests: deploy/openshift.
- Dashed frame: one OpenShift project. A NetworkPolicy lets pods talk only to each other; the Route to the console is the only way in.
- Left column, top to bottom: the presenter's browser, the console that drives the demo, and the RAN slice it scales between 0 and 1 pods.
- Middle: the console injects and heals timing faults on
ptp-bridge, and asks the NEP agent to run an RCA. - Right: the agent's path: token from CAPIF, tool calls through the gateway, one MCP server per plane, and an optional LLM for the narrative.
5Sequence (high level)
- Columns are components; time runs downward. Each arrow is a call, labeled with what it does or returns.
- Four phases: Start (a UE attaches), Inject (timing breaks, the cell is locked), RCA (token, tool calls, verdict), then Heal, re-check and Stop.
- Compare the two RCA results:
2/3 → HOLDduring the fault and0/3 → NO ACTIONafter heal. The verdict follows the evidence.
Full call-by-call version: sequence-detailed.pdf
6Components and fidelity
| Component (code) | Role | How real |
|---|---|---|
| ran-slice | 5G core + O-RAN split RAN, 32 NFs in one pod | Real protocol logic in Python; no PHY/RF |
| ue_sim | UE attach, PDU session, GTP-U echo | Real MILENAGE 5G-AKA and NAS flow |
| ptp-bridge | PTP state and CloudEvents, fault injection | Software model of ptp4l |
| O-DU alarm | TS 28.532 alarm over O1 | Raised by the O-DU, after the console's protective lock |
| capif | TS 29.222 onboarding and scoped token | Real flow; unsigned lab tokens |
| mcp-gateway | Scope check, routes tool calls to planes | Real; not a security boundary |
| NIC plane | Intel E810 timestamp counters | Emulated, never counted |
| O-Cloud plane | ACM ManagedCluster health | Real only with ACM; else "unavailable" (how) |
| nep-orchestrator | Evidence, policy, TMF688 event | Deterministic decision; LLM optional (how) |
Terms: glossary · concept pages. More: architecture · APIs, config and ports · deployment · security notes
7How the decision is made
- Each plane returns evidence plus signals (e.g.
ptp_offset_exceeded,du_sync_loss_alarm). - A fixed route table maps signals to a fault class (e.g.
OC-TimingDegraded). - Corroboration counts real planes only. The bar is 3.
- Verdicts:
APPLY (auto-eligible),HOLD(needs a human),NO ACTION. - In the demo: RAN + PTP agree, NIC is emulated → 2/3 → HOLD. Nothing runs automatically.
- The LLM explains the audit record. It never makes the decision.
8Try it
You need an OpenShift 4.x cluster, oc logged in, and git. Repo: github.com/dkypuros/multivendor-agentic-rca
1. Get the code
git clone https://github.com/dkypuros/multivendor-agentic-rca.git cd multivendor-agentic-rca
2. Deploy (run from the repo folder)
oc new-project multivendor-rca oc apply -k deploy/openshift # Deployments, Services, RBAC, NetworkPolicies, Route, BuildConfig oc start-build mvrca --from-dir=. --follow # builds the image from your local clone oc rollout restart deployment -l app.kubernetes.io/part-of=multivendor-rca
3. Open the console and check it
oc get route ran-sandbox -o jsonpath='https://{.spec.host}{"\n"}'
python3 tests/smoke_route.py https://$(oc get route ran-sandbox -o jsonpath='{.spec.host}') # ALL GREEN
Then follow the 20-minute demo walkthrough. Details and troubleshooting: deployment guide.
No cluster?
pip install pyyaml python3 tests/e2e_local.py # runs Start / Inject / RCA / Heal / Stop on a laptop
Lab only: the console has no login. Share its URL deliberately.
9Reference: the lab
The single-node OpenShift cluster these demos run on, as inventoried on 29 Sep 2026. For reference only; the demo itself needs far less.
Hardware (one server)
| CPU | 2 × Intel Xeon 6767P (Granite Rapids), 256 threads |
| CPU features | Intel AMX (INT8, BF16, FP16, FP8), SGX, TDX |
| Accelerators | Intel QAT 402xx (8 endpoints), Intel DLB 2.5 (8 devices) |
| Memory | about 512 GB |
| Local disk | about 1.9 TB |
| Network | Broadcom NetXtreme-E, dual port. No Intel E810, which is why the NIC plane is emulated |
| GPU | None: LLM inference runs on CPU with AMX |
Platform
| OpenShift | 4.22.1, single node |
| OS | RHEL CoreOS 9.8, kernel 5.14.0-687 |
| Runtime | CRI-O 1.35 |
| Storage | NFS (default class), MinIO S3 |
| Demo image | UBI 9 Python 3.12 |
Used by this demo
| OpenShift | 4.22.1: runs every component (Deployments, Route, NetworkPolicy, builds) |
| Red Hat OpenShift AI | 3.4.0: Llama 3.1 8B Instruct (w8a8) on vLLM for CPU. Optional LLM narrator, off by default |
Timing comes from the demo's own software PTP model, not the PTP Operator. See the glossary.
On the lab cluster (not used by the demo)
PTP Operator 4.22 · O-Cloud Manager 4.22.1 · Topology Aware Lifecycle Manager 4.22.1 · Node Feature Discovery 4.22 · Intel Device Plugins 0.36 · Argo CD 0.18 · Advanced Cluster Security 4.11 · Service Mesh 3.4 · Serverless 1.37 · Pipelines 1.23 · Connectivity Link 1.4 · build of Keycloak 26.4 · Quay 3.14 · Streams for Apache Kafka 3.2 · Logging 6.6 + Loki · Tempo 0.22 · build of OpenTelemetry 0.158 · Cluster Observability 1.5 · cert-manager 1.19 · sandboxed containers 1.13 · build of Trustee 1.2 · Intel TDX DCAP 0.1 · Kernel Module Management 2.6
10Reference: Red Hat and Intel, better together for RCA
- Top row, left to right: the steps of an AI assistant that answers "why did this cell go down?". The query is the alarm plus the evidence the agent collected.
- Solid boxes are in this demo: the evidence query, optional LLM narration (Llama 3.1 8B on vLLM, running on the lab's Xeon 6 CPUs), and the response. Dashed boxes are the natural next step: embed, retrieve and rerank runbooks and vendor knowledge bases to ground the answer.
- Bottom bands: every step runs on the same stack. Red Hat OpenShift AI serves the models on the same OpenShift that runs the RAN; Intel Xeon 6 AMX runs the matrix math on the CPU.
- Why it matters at the edge: cell sites and regional data centers rarely have GPUs. Running retrieval and a small LLM on Xeon keeps the assistant next to the network it explains.
- Still true: the decision stays with the deterministic policy. Retrieval and the LLM improve the explanation, not the verdict.
See the lab's hardware on the previous slide. Related: mcp-multivendor-tutorial (vendor knowledge agents over MCP) · connect an LLM