Designing an AI incident agent for 3 a.m.: making memory visible
Picture the person who uses an incident tool. It's the middle of the night. An alert just woke them up. Customers are affected, a manager is asking for updates, and they are trying to read logs with half-open eyes. That person doesn't need a clever chatbot. They need an answer they can trust in seconds, and proof they can check. Our team built DejaVu , an AI on-call agent that remembers how past…
DejaVu is an AI incident agent designed to provide answers and proof for engineers handling incidents during off-hours. The agent leverages past incident data stored in Hindsight, an open-source agent memory system, to recall similar incidents and provide relevant information. The design of DejaVu focuses on making the invisible memory visible and useful for the stressed engineer dealing with the incident.
The first design element is a live memory map that displays past incidents as dots, grouped by type and sized by resolution time. When DejaVu recalls incidents for a new error, the matching dots light up, providing a visual representation of past incidents and their relevance without requiring the engineer to read anything.
Next, the design includes a comparison feature that runs the same model with and without memory, demonstrating the significant improvement in confidence when using memory. This feature clearly illustrates the value of memory to users, technical or not.
Proof is provided under every claim by displaying a plain-words sentence explaining the confidence level, along with details about matching past incidents and their resolutions. Each recalled incident is accompanied by a card containing information such as the service, incident details, match strength, resolution time, and reasons for the match. This approach ensures that the agent's confidence claims are backed by evidence.
To facilitate feedback from the engineer, DejaVu offers two simple buttons: Fix worked and Fix failed. These buttons save the outcome to Hindsight in real-time, ensuring that the learning curve remains honest. The interface remains unblocked during the learning process, allowing engineers to focus on resolving the incident.
The design also emphasizes calm colors and minimal alerts, with red only being used for real alerts and failed fixes. This approach helps maintain a low-stress environment for the engineer. Additionally, the design integrates seamlessly into existing team workflows, such as importing past postmortems into Hindsight, sending alert notifications through PagerDuty or Grafana webhooks, and posting fix plans to team communication channels.
The final design considerations include a user-friendly interface with calming colors, a downloadable postmortem generator, a learning curve chart to showcase the improvement in time-to-fix, and friendly error messages during high demand. These small details contribute significantly to the overall user experience and the successful adoption of the AI incident agent.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
