What 104 Real Postmortems Taught My Agent That I Couldn't Have Written Myself
When I started building an incident-response agent, the first decision wasn't about models or frameworks. It was about what the agent should remember. I chose real postmortems, and this post explains why, what the data looked like once it was in memory, and what I can and can't claim about it. Why real incidents An SRE agent is only useful if its advice matches how outages actually unfold. Real…
Building a helpful incident-response agent begins with selecting the right data to remember. The author chose real postmortems from various organizations, such as Slack, Cloudflare, GitHub, AWS, Datadog, CircleCI, and LaunchDarkly. These incidents contain important details, like vendor-specific information and failed remediation attempts, that are hard to generate artificially.
However, the author acknowledges the limits of this approach, as they didn't perform a direct comparison between real and hand-written data. Instead, they used the OpenSRE incident dataset, which contained 104 real postmortems, and retained 104 of them for the agent's memory bank, while holding out 10 for later testing. To enhance the agent's memory, the author implemented two strategies: re-ranking recalled memories based on the presence of "trap" actions, and instructing the model to avoid certain actions when trap actions are mentioned in the retrieved context.
When the agent was tested with both memory and without memory, it showed improved performance with memory, correctly identifying the root cause in 9 out of 10 cases. However, the author also points out the limitations of the chosen dataset, such as its limited diversity and the potential for retrieval noise. Despite these limitations, the author emphasizes the importance of choosing data that records failed fixes, holding some data out before seeding, and keeping a clear label for grading.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
