How STPA Helps Find Unknown Unknowns Before They Cost You Millions

Complex systems often fail not due to a single broken component, but rather from the interactions between seemingly healthy parts. A retry mechanism may amplify load rather than aid recovery; an autoscaler might make decisions based on stale metrics; two automated systems could optimize for different local goals and inadvertently conflict in production. In such scenarios, there may be no obvious broken component or a clearly wrong decision made by an engineer. The issue lies in the gaps between the parts.

Most engineering teams excel at identifying known risks through architecture reviews, incident analyses, threat modeling exercises, and building dashboards and alerts. However, these practices share a common limitation: they primarily focus on risks that can be explicitly described. What about the unknown unknowns—expensive surprises hiding within system behavior and assumptions that were not documented?

Systems-Theoretic Process Analysis (STPA) addresses this gap. STPA treats systems as networks of control loops rather than collections of independent components. Rather than solely questioning "what could break and how," STPA examines what control actions could lead to undesirable outcomes if they are missing, incorrect, timetimed improperly, or maintained excessively.

This approach has proven effective in safety-critical industries like aviation, nuclear energy, and defense, where system failures can have severe consequences. In recent years, technology sectors have begun exploring the application of STPA to software engineering. For a deeper understanding of STPA's principles and methodology, refer to the STPA handbook.

In distributed systems, control loops are ubiquitous. Examples include deployment pipelines, incident response, and human operator actions. An autoscaler control loop, for instance, involves determining scaling actions based on CPU usage metrics. If usage exceeds a certain threshold, the system scales up, while usage below a specific level triggers scaling down. If the thresholds are not met, no action is taken.

STPA prompts a crucial question: "What control actions could result in loss if they are missing, incorrect, mistimed, or applied for too long?" This differs from traditional risk analysis, which primarily concerns itself with identifying potentially failing components. By focusing on the safety of control actions, STPA shifts the analysis towards when a valid control action might become unsafe.