AI handles incidents, engineers lose touch with their systems

In 2012, the author, an SRE at LinkedIn, crafted a self-healing system that learned from past incidents. Though AI was not as advanced as it is today, the concept has now become a reality. These tools can now inspect alerts, generate hypotheses, analyze telemetry, correlate recent deployments, and even execute fixes autonomously.

While these advancements are impressive, there is a significant concern: we might be distancing ourselves from our systems. As these AI-assisted incident response tools, commonly referred to as "AI SREs," become increasingly proficient in handling routine incidents, human responders may be losing valuable practice. When confronted with an unforeseen, high-severity incident that these tools cannot resolve, engineers will have to take over with less hands-on experience than they previously had.

Lisanne Bainbridge, a human-factors researcher, coined this paradox in her 1983 paper, "The Ironies of Automation," which highlights that automation not only reduces operators' opportunities to practice routine work but also leaves them solely responsible for new and abnormal situations. Consequently, operators must be trained more rigorously and receive additional training compared to earlier periods.

The author predicts that, although the average mean time to resolution (MTTR) for incidents will decrease due to AI-assisted response, the resolution time for complex incidents will increase as responders lose touch with their systems and struggle to investigate. To draw a parallel, the aviation industry offers insight. While modern aircraft automation manages most of the flying process, pilots remain responsible for handling rare situations such as engine failures, unreliable instruments, rejected takeoffs, and stalls.

These scenarios are extremely uncommon, with modern turbine engines experiencing fewer than one in-flight shutdown per 100,000 engine flight hours. However, when such a failure occurs, pilots must react swiftly and accurately. For instance, during TransAsia Airways Flight 235, the right engine's propeller autofeathered shortly after takeoff, and the crew misidentified the issue, leading to the aircraft stalling and crashing only 117 seconds after the first warning.

Commercial pilots are required to undergo recurrent training or proficiency checks every six months, which may include scenarios such as an engine failure during takeoff. Despite the fact that most software incidents do not pose a threat to human life, it is crucial not to neglect the refinement of our skills. In fact, the technology that initially created the issue can also aid in its resolution.

Rootly, an incident management company where the author currently works, has partnered with Uptime Labs to apply this concept through realistic incident simulations. Engineers assume the role of incident commander during a simulated e-commerce outage, employing observability tools while coordinating with LLM-powered stakeholders in Slack.

The experience feels genuine, as responders must investigate the underlying issues while maintaining the response's organization and interacting with the CEO, customer support, and other stakeholders. Although most software incidents do not result in fatalities, it is essential to perfect our craft. In the author's previous role, they built a software engineering school based on progressive education, emphasizing learning by doing rather than passive instruction.

When Dropbox's graduates expressed concerns about their lack of troubleshooting skills, the author introduced projects that required students to diagnose and repair broken infrastructure. They believe that hands-on education surpasses passive instruction by a significant margin. As LLMs take over more aspects of our work, engineering teams risk developing comprehension debt – a widening gap between how their systems function and how well responders comprehend them.

To counteract this issue, engineers should frequently interact with the systems they oversee, manage unfamiliar failures, practice working under pressure, and rehearse the coordination and communication required during a SEV0 incident. Regular hands-on control and simulation are not new concepts, but they have become even more crucial in the era of LLMs.

Bainbridge recommended giving operators regular hands-on control and using simulation to prevent skill decay. This is the irony of automation: the more successful it becomes, the less prepared humans may be when the system inevitably fails.