Your Next Incident Shouldn’t Start From Zero: Building Recall

An incident investigation concludes with a green dashboard, a summary written, and the ticket marked as resolved. Weeks later, another engineer encounters similar symptoms and searches for previous experience. The challenge lies in determining whether that history is still relevant. The Recall project aims to address this gap by creating an incident memory command center.

It seeks to capture useful experience from one investigation and apply it to the next. The key is deciding how that historical context applies to the current problem.

When an engineer starts an investigation, they input symptoms such as the affected service, environment, severity, symptoms, and relevant logs. The backend saves this information locally and requests an analysis. If a provider call fails, the incident remains accessible for later reference. Recall then utilizes Hindsight to retrieve relevant historical context from past incidents and sends the current evidence and retrieved memories to a model using Groq.

The model generates a response containing a suspected root cause, investigation steps, recommended actions, verification steps, and references to the historical sources used.

Recall distinguishes between a hypothesis and a confirmed diagnosis. For example, a previous incident suggesting a connection pool issue may prompt an engineer to check pool utilization, but it does not guarantee that today's slowdown has the same cause. The system does not execute infrastructure commands; instead, it provides guidance for the engineer to investigate further.

To ensure accurate history application, Recall employs two storage systems: SQLite and Hindsight. SQLite holds structured state information about the investigation, such as incident status, updates, saved analysis, and recorded outcomes. Hindsight, on the other hand, makes previous incident information available for retrieval across different investigations.

When an engineer records evidence or outcome details, Recall can send a readable incident document to the memory bank. This document includes symptoms, actions taken, outcome, lessons learned, and labels human-reported evidence to differentiate it from the model's interpretation.

The comparison view feature allows engineers to analyze the same incident using two approaches: current evidence alone or current evidence combined with retrieved historical memory. This side-by-side analysis helps identify what memory contributed to the investigation and whether it suggested a more specific check, surfaced an earlier failed attempt, changed the suspected cause, or provided useful historical context.

The feature emphasizes that memory analysis is an inspection tool rather than a benchmark proving superior results.

Validation checks are implemented to ensure the reliability of AI responses. Pydantic validates structured model responses before accepting them, and the application verifies cited source IDs against the actual memories supplied to the model. If a response fails validation or cites an unknown source, the application prompts a correction instead of generating an alternative diagnosis.

Retrieved text and incident fields are treated as evidence rather than instructions, and selected risky action patterns are flagged for review to catch any unsupported claims.

While the system aims to enhance the incident investigation process with memory, it acknowledges the importance of maintaining human oversight. Verification still plays a crucial role in the workflow, allowing engineers to question suggestions, incorporate contradictory evidence, and adapt their approach as needed.

Looking ahead, the Recall project plans to evaluate its effectiveness through repeatable incident scenarios, assess the impact of retrieved history on investigation steps, improve the presentation of evidence and outcomes, and handle situations where earlier hypotheses prove incorrect. Ultimately, the goal remains to make the most of the team's previous experience when facing new incidents.