Building IncidentMind: An AI Incident Investigation Assistant That Learns From the Past

Incident response often becomes a laborious task because engineers repeatedly investigate similar failures. Building IncidentMind aimed to address this issue by creating an AI-powered incident investigation assistant that can learn from past incidents. The core idea is to capture the knowledge gained from resolving an incident and retain it for future use.

When an incident occurs, IncidentMind guides engineers through a workflow to submit and investigate the problem. The context includes details like the affected service, incident ID, severity level, recent deployments, configuration changes, symptoms, error logs, and traces. Using this context, the investigation agent creates a structured investigation report, including evidence, potential root causes, and recommended actions.

Once the incident is resolved, engineers document the confirmed root cause, resolution steps, runbook, lessons learned, and future prevention measures. This information can then be stored in Hindsight memory, a separate layer within the IncidentMind system.

During testing, an authentication incident was simulated. The affected service, Authentication API, experienced intermittent logout issues and HTTP 401 errors after a deployment introduced JWT validation middleware and a centralized secrets provider. The logs indicated inconsistencies in signing-key versions across instances.

Rather than treating the logs as isolated errors, IncidentMind considered the surrounding incident context. The investigation agent analyzed this information and identified the signing-key inconsistency as the likely cause - some authentication instances were using the new signing-key version while others were still using the previous version. This structured investigation helped engineers quickly pinpoint the root cause.

After confirming the root cause, engineers resolved the issue by synchronizing signing-key versions, restarting affected instances, and verifying login, token refresh, and existing-session validation. This resolution was documented along with a runbook and lessons learned, which were then stored in the memory layer for future reference.

By distinguishing between application state persistence (SQLite) and operational experience retention (Hindsight), IncidentMind can effectively leverage past incident knowledge to aid future investigations. The system can recall relevant past incidents based on specific criteria such as service or severity, providing valuable insights to engineers when tackling new problems.