How Complex Systems Fail

Complex systems, including transportation, healthcare, and power generation, are inherently hazardous by their very nature. While the frequency of hazard exposure may be altered, the processes within these systems are fundamentally hazardous. This hazard drives the development of various defenses against accidents, encompassing technical, human, and organizational components.

These defenses, such as backup systems, training, and policies, create multiple layers of protection that typically prevent catastrophic failures. However, when seemingly minor failures align, they can create opportunities for systemic accidents. In reality, there are more potential failure points than actual accidents, with most initial failures being blocked by safety measures.

The complexity of these systems means that flaws are often present but remain insignificant until they combine to cause failure. Complex systems operate as "broken systems," functioning despite numerous flaws due to redundancy and human ingenuity. Following accidents, it is common to note that the system had a history of prior near-misses.

Despite this, naive perceptions of system performance often hinder the recognition of these degraded conditions. Complex systems are dynamic, with continuous changes in components and their replacements. The potential for catastrophic failure is always present due to the system's inherent nature. No single factor can be isolated as the root cause of an accident; instead, multiple contributors must combine to create the circumstances for failure.

This complexity makes it nearly impossible to implement isolated "root cause" evaluations. The presence of hindsight bias, resulting from knowledge of the outcome, often leads to inaccurate assessments of practitioner performance during accidents. This bias is not unique to technical fields but is a universal cognitive trait affecting all human cognition about past events and their causes.

Practitioners balance the demands for production with the risk of incipient failure, a dynamic aspect of system operation that is often overlooked by outsiders. After accidents, the emphasis shifts towards defense against failure, and practitioner actions may be perceived as blunders or deliberate disregard for impending failure. However, all practitioner actions are essentially gambles, taken in the face of uncertain outcomes.