Beyond Log Search: What We Learned Building a RAG-Based Incident Diagnosis System

The story revolves around the development and evaluation of a RAG-based incident diagnosis system designed to improve incident response times by leveraging knowledge retrieval. The system was built by combining incident tickets, runbooks, postmortems, and live alerts from various sources, including ServiceNow, PagerDuty, Confluence, GitHub, and Prometheus/Dynatrace.

The authors emphasize that the majority of incident resolution time is spent on knowledge retrieval rather than remediation, suggesting that the issue lies not in fixing the problem but in finding relevant information.

The authors outline five layers of the system: data sources, ingestion and indexing, retrieval, LLM reasoning, and feedback loop. They note that most of the engineering effort went into the data sources, ingestion, and indexing layers, particularly in creating chunking strategies tailored to different document types. The retrieval layer involved embedding chunks in a FAISS index for approximate nearest-neighbor search and re-ranking candidates before passing them to the LLM.

The LLM reasoning layer produced structured outputs, including ranked root causes, confidence scores, remediation steps, and links back to source documents.

The system was tested against 2,400 annotated incident scenarios, outperforming several baseline methods, including a standalone LLM, a BM25 keyword-retrieval baseline, a supervised ML classifier, and a rule-based expert system. The RAG framework achieved 87.3% root-cause identification accuracy, compared to 71.8% for the best baseline.

Diagnosis time for all incidents dropped by 59%, with the most significant improvement seen in P1 critical incidents, which reduced from an average of 48.2 minutes of manual search to just 19.8 minutes. The ablation study revealed that removing the re-ranking step or the semantic chunking strategy led to the largest drops in accuracy.