What 104 Real Postmortems Taught My Agent That I Couldn't Have Written Myself

Building a helpful incident-response agent begins with selecting the right data to remember. The author chose real postmortems from various organizations, such as Slack, Cloudflare, GitHub, AWS, Datadog, CircleCI, and LaunchDarkly. These incidents contain important details, like vendor-specific information and failed remediation attempts, that are hard to generate artificially.

However, the author acknowledges the limits of this approach, as they didn't perform a direct comparison between real and hand-written data. Instead, they used the OpenSRE incident dataset, which contained 104 real postmortems, and retained 104 of them for the agent's memory bank, while holding out 10 for later testing. To enhance the agent's memory, the author implemented two strategies: re-ranking recalled memories based on the presence of "trap" actions, and instructing the model to avoid certain actions when trap actions are mentioned in the retrieved context.

When the agent was tested with both memory and without memory, it showed improved performance with memory, correctly identifying the root cause in 9 out of 10 cases. However, the author also points out the limitations of the chosen dataset, such as its limited diversity and the potential for retrieval noise. Despite these limitations, the author emphasizes the importance of choosing data that records failed fixes, holding some data out before seeding, and keeping a clear label for grading.