OnCall Memory:Building an Incident Response Agent That Learns From Production

OnCall Memory is an incident-response website aimed at teaching AI assistants about an organization's past incidents. Production incidents often reflect recurring patterns, but typical AI assistants lack knowledge of previous experiences when responding to new alerts. OnCall Memory addresses this by providing two perspectives during incident resolution: one without memory and one with hindsight memory.

The hindsight memory aspect of the website enables it to draw from relevant historical incidents, offering context that a regular AI assistant would not have. The system is built around a dataset of realistic production incidents from a fictional fintech company called Northwind Pay. Each incident contains information about symptoms, logs, root cause, resolution steps, outcome, and the effectiveness of attempted fixes.

Using this data, the website can retain, recall, and reflect on past incidents to enhance future diagnoses. The architecture of the website consists of a lightweight frontend (HTML, CSS, and JavaScript) and a backend (FastAPI) that manages incident analysis, memory operations, and language model requests. Groq provides the language-model layer, while Hindsight handles persistent memory.

Configuration separate from application code allows for better security management. After generating a suggested fix, the engineer can indicate whether the fix was successful or not, and provide the actual solution. This feedback becomes another memory, allowing the system to learn and improve over time. In total, OnCall Memory's memory bank contains 139 memories.

Through this approach, the website helps to create an incident-response assistant that doesn't forget past incidents, ultimately leading to better diagnoses and resolutions.