AI handles incidents, engineers lose touch with their systems

When I worked as an SRE at LinkedIn in 2012, I engineered a system capable of self-healing and learning from past incidents. Although AI capabilities at the time were far from today's standards, this concept has now become a reality. Modern AI tools can perform a range of tasks: inspecting alerts, formulating hypotheses, analyzing telemetry, correlating recent deployments, and even remedying the issues themselves.

While I appreciate the utility of these systems, I do harbor a significant concern – we may be distancing ourselves from our systems. As these tools grow more proficient at resolving everyday incidents, human responders could lose valuable practice. When faced with an ambiguous, high-severity incident that automation cannot handle, the engineers responsible for intervention may find themselves unprepared.

These AI-enhanced incident response tools, often referred to as "AI SREs," offer numerous benefits. Their effectiveness is especially evident when they manage routine incidents during off-hours, sparing responders from unnecessary wake-ups. However, routine incidents also play a crucial role in developing intuition about how systems behave and fail.

As AI encounters unprecedented issues it cannot resolve, engineers may have to step in with less hands-on experience than they would have had before. Lisanne Bainbridge, a human-factors researcher in her 1983 paper "The Ironies of Automation," highlighted this paradox. She argued that automation reduces operators' opportunities for routine practice while still holding them accountable for novel situations.

Consequently, operators might require enhanced skills and additional training than they would have needed pre-automation. Looking ahead, I anticipate that the average Mean Time To Resolution (MTTR) for incidents will decrease due to AI-assisted response, but the resolution time for complex incidents will likely increase, as responders lose touch with their systems and struggle to investigate.

The aviation industry provides a relevant analogy. Modern aircraft rely heavily on automation, yet pilots remain responsible for managing abnormal conditions such as engine failures, unreliable instruments, rejected takeoffs, stalls, and other irregularities. These events are extremely rare, with modern turbine engines experiencing fewer than one in-flight shutdown per 100,000 engine flight hours.

That infrequency means a commercial pilot may go their entire career without experiencing such a situation outside a simulator. Yet when these rare failures occur, pilots must react swiftly and accurately. For instance, in the TransAsia Airways Flight 235 incident, the right engine's propeller autofeathered just after takeoff, leading to a crash 117 seconds later.

Airlines regularly subject pilots to simulator training for rare emergencies, mandating recurrent training or proficiency checks every six months, including scenarios like engine failure during takeoff. While software incidents rarely endanger lives, this should not deter us from refining our skills. The technology that initially created the problem can also help address it.

At Rootly, an incident management firm where I now work, we have collaborated with Uptime Labs to implement this approach through realistic incident simulations. Engineers assume the role of incident commander during a simulated e-commerce outage, utilizing observability tools while coordinating with LLM-powered stakeholders in Slack.

The experience feels authentic – responders must investigate the issue while maintaining clear communication with executives and customer support. Through this process, they hone essential skills such as interpreting incomplete information, communicating effectively, coordinating team members, and executing the response. However, practice and observation alone are insufficient.

Responders can glean insights from observing AI agents, understanding their decision-making process, the signals examined, and the evidence supporting their diagnosis. Yet, hands-on practice remains irreplaceable. I spent over half a decade building a software engineering school around progressive education, without relying on teachers.

Instead, students undertook projects working on broken infrastructure, learning to diagnose and repair it. When Dropbox shared feedback from graduates, stating they were still inexperienced in troubleshooting, I developed projects requiring students to diagnose and fix broken infrastructure. For most practical skills, hands-on education surpasses passive instruction significantly.

As LLMs assume more responsibilities, engineering teams risk accumulating comprehension debt – a widening gap between their understanding of how systems function and their ability to respond effectively. To mitigate this, engineers should engage regularly with the systems they oversee, tackle unfamiliar failures, practice under pressure, and rehearse the coordination and communication required during SEV0 incidents.

Tabletop exercises and chaos engineering are well-established practices, but their importance has grown in the era of LLMs. Bainbridge suggested giving operators regular hands-on control and utilizing simulation to prevent skill decay, particularly as automation becomes more successful. The irony of automation is that as it becomes more powerful, humans may become less prepared for moments when it inevitably fails.