Presentation: When Incidents Refuse to End
Vanessa Huerta Granda explains how marathon incidents expose the gap between work as imagined and work as done. Drawing from real-world scenarios, she shares how complex outages reveal organizational fragility, human limits, and system interdependencies—and why incident response requires structured endurance, humane rotations, and holistic cross-functional coordination. By Vanessa Huerta Granda
Title: Unending Incidents: Lessons from Marathon Outages
Vanessa Huerta Granda, an Engineering Manager at Enova, discusses the challenges of marathon incidents and how they reveal the gaps between our expectations and reality in incident response. Drawing from her experience as a Resilience Engineering team leader and her work on the Production Incident process, she shares insights into the human limits, system interdependencies, and the need for structured endurance, humane rotations, and holistic cross-functional coordination in these complex scenarios.
Granda begins by emphasizing that incidents rarely follow the neat timeline one might imagine in training or post-mortems. Real-world incidents often resemble a chaotic, tangled mess, with multiple questions arising: "What is happening? Who owns this? How is this connected? Did we actually fix it?" This turbulence is where the lessons of incident response truly resonate, both for the technical aspects of our systems and for the people behind them.
Granda, who has been on call for years and has led incident response efforts, shares her personal journey in incident management. From the early days of handling on-call rotations to leading retrospectives and scaling escalation programs, she has witnessed the evolution of incident management across various organizations. She highlights the shift from quick fixes to achieving lasting resilience, drawing from her experience with long-running incidents that span sprints and marathons.
Granda stresses the importance of understanding the impact of incidents on both systems and teams. She argues that these extended incidents offer valuable insights into the fragility of our organizational structures and the human elements involved. Her unique perspective as an industrial engineer, who bridges technical and non-technical perspectives, allows her to communicate complex concepts effectively to a diverse audience.
Throughout her presentation, Granda emphasizes the need for structured endurance in incident response, highlighting the importance of humane rotations and cross-functional coordination. She shares anecdotes of how she has managed incidents, from reminding responders to take breaks to ensuring that executives are kept informed without overwhelming them.
Her experience has shown her that incident management is not just about fixing technical issues but about fostering resilience, communication, and adaptability within the entire organization.
Written by urgent.news from InfoQ's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
