BLACKBOX: Giving Incident Response a Memory
Production incidents rarely arrive as isolated problems. A single alert can be connected to a recent deployment, a downstream dependency, an overloaded database, an unfamiliar service owner, or a similar incident from months ago. Th e challenge for on-call engineers is not just finding an answer. It is connecting all of that context quickly and explaining why the answer is trustworthy. That is…
BLACKBOX is an AI-powered incident investigation tool designed to help SRE and on-call teams quickly understand the context behind incidents. It uses a knowledge graph called FalkorDB to connect services, dependencies, alerts, deployments, runbooks, people, and previous postmortems. When an incident occurs, BLACKBOX traverses the graph to identify relationships and uncover the likely root cause, blast radius, relevant experts, and an explanation that engineers can act upon.
The investigation results are then stored back into FalkorDB as structured findings, creating a feedback loop for future incidents. The graph representation allows BLACKBOX to perform multi-hop reasoning, asking questions like which services depend on an alerting service, which recent deployment is connected to the failure, and who owns the code involved in the suspected root cause.
This enables BLACKBOX to explain not just what it believes, but how it reached that conclusion. BLACKBOX also distinguishes between "cold" investigations, which start from scratch and explore the graph, and "warm" investigations that can leverage previous knowledge and findings. The tool provides a real-time interface for engineers to see the investigation process, graph traversal, and reasoning steps, fostering better collaboration during incident response.
The language model assists in selecting tools, interpreting results, summarizing evidence, and producing clear explanations, but the graph remains the authoritative source of operational truth. If the language model is unavailable, BLACKBOX can fall back to a deterministic planner to still deliver explainable findings. The key takeaway is that agent memory is most effective when it is structured and connected to the operational environment, allowing future investigations to build upon previous knowledge.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
