How Complex Systems Fail

A while back, I came across an interesting post written by a medical doctor researching the reasons why patient care systems fail. The author, Dr. Richard Cook, breaks the reasons down into 18 different categories.

I believe a lot of it is applicable to software systems.

I've copied it below, but you can also read it here.

Richard I. Cook, MD
Cognitive Technologies Labratory
University of Chicago

  1. Complex systems are intrinsically hazardous systems.

All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by the own nature. The frequency of hazard exposure can sometimes be changed but the processes involved in the system are themselves intrinsically and irreducibly hazardous. It is the presence of these hazards that drives the creation of defenses against hazard that characterize these systems.

2. Complex systems are heavily and successfully defended against failure

The high consequences of failure lead over time to the construction of multiple layers of defense against failure. These defenses include obvious technical components (e.g. backup systems, ‘safety’ features of equipment) and human components (e.g. training, knowledge) but also a variety of organizational, institutional, and regulatory defenses (e.g. policies and procedures, certification, work rules, team training). The effect of these measures is to provide a series of shields that normally divert operations away from accidents.

3. Catastrophe requires multiple failures – single point failures are not enough.

The array of defenses works. System operations are generally successful. Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident. Each of these small failures is necessary to cause catastrophe but only the combination is sufficient to permit failure. Put another way, there are many more failure opportunities than overt system accidents. Most initial failure trajectories are blocked by designed system safety components. Trajectories that reach the operational level are mostly blocked, usually by practitioners.

4. Complex systems contain changing mixtures of failures latent within them.

The complexity of these systems makes it impossible for them to run without multiple flaws being present. Because these are individually insufficient to cause failure they are regarded as minor factors during operations. Eradication of all latent failures is limited primarily by economic cost but also because it is difficult before the fact to see how such failures might contribute to an accident. The failures change constantly because of changing technology, work organization, and efforts to eradicate failures.

5. Complex systems run in degraded mode.

A corollary to the preceding point is that complex systems run as broken systems. The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously.

6. Catastrophe is always just around the corner.

Complex systems possess potential for catastrophic failure. Human practitioners are nearly always in close physical and temporal proximity to these potential failures – disaster can occur at any time and in nearly any place. The potential for catastrophic outcome is a hallmark of complex systems. It is impossible to eliminate the potential for such catastrophic failure; the potential for such failure is always present by the system’s own nature.

7. Post-accident attribution to a ‘root cause’ is fundamentally wrong.

Because overt failure requires multiple faults, there is no isolated ‘cause’ of an accident. There are multiple contributors to accidents. Each of these is necessarily insufficient in itself to create an accident. Only jointly are these causes sufficient to create an accident. Indeed, it is the linking of these causes together that creates the circumstances required for the accident. Thus, no isolation of the ‘root cause’ of an accident is possible. The evaluations based on such reasoning as ‘root cause’ do not reflect a technical understanding of the nature of failure but rather the social, cultural need to blame specific, localized forces or events for outcomes. 1

1 Anthropological field research provides the clearest demonstration of the social construction of the notion of ‘cause’ (cf. Goldman L (1993), The Culture of Coincidence: accident and absolute liability in Huli, New York: Clarendon Press; and also Tasca L (1990), The Social Construction of Human Error, Unpublished doctoral dissertation, Department of Sociology, State University of New York at Stonybrook)

8. Hindsight biases post-accident assessments of human performance.

Knowledge of the outcome makes it seem that events leading to the outcome should have appeared more salient to practitioners at the time than was actually the case. This means that ex post facto accident analysis of human performance is inaccurate. The outcome knowledge poisons the ability of after-accident observers to recreate the view of practitioners before the accident of those same factors. It seems that practitioners “should have known” that the factors would “inevitably” lead to an accident. 2 Hindsight bias remains the primary obstacle to accident investigation, especially when expert human performance is involved.

2 This is not a feature of medical judgements or technical ones, but rather of all human cognition about past events and their causes.

9. Human operators have dual roles: as producers & as defenders against failure.

The system practitioners operate the system in order to produce its desired product and also work to forestall accidents. This dynamic quality of system operation, the balancing of demands for production against the possibility of incipient failure is unavoidable. Outsiders rarely acknowledge the duality of this role. In non-accident filled times, the production role is emphasized. After accidents, the defense against failure role is emphasized. At either time, the outsider’s view misapprehends the operator’s constant, simultaneous engagement with both roles.

10. All practitioner actions are gambles.

After accidents, the overt failure often appears to have been inevitable and the practitioner’s actions as blunders or deliberate willful disregard of certain impending failure. But all practitioner actions are actually gambles, that is, acts that take place in the face of uncertain outcomes. The degree of uncertainty may change from moment to moment. That practitioner actions are gambles appears clear after accidents; in general, post hoc analysis regards these gambles as poor ones. But the converse: that successful outcomes are also the result of gambles; is not widely appreciated.

11. Actions at the sharp end resolve all ambiguity.

Organizations are ambiguous, often intentionally, about the relationship between production targets, efficient use of resources, economy and costs of operations, and acceptable risks of low and high consequence accidents. All ambiguity is resolved by actions of practitioners at the sharp end of the system. After an accident, practitioner actions may be regarded as ‘errors’ or ‘violations’ but these evaluations are heavily biased by hindsight and ignore the other driving forces, especially production pressure.

12. Human practitioners are the adaptable element of complex systems.

Practitioners and first line management actively adapt the system to maximize production and minimize accidents. These adaptations often occur on a moment by moment basis. Some of these adaptations include: (1) Restructuring the system in order to reduce exposure of vulnerable parts to failure. (2) Concentrating critical resources in areas of expected high demand. (3) Providing pathways for retreat or recovery from expected and unexpected faults. (4) Establishing means for early detection of changed system performance in order to allow graceful cutbacks in production or other means of increasing resiliency.

13. Human expertise in complex systems is constantly changing

Complex systems require substantial human expertise in their operation and management. This expertise changes in character as technology changes but it also changes because of the need to replace experts who leave. In every case, training and refinement of skill and expertise is one part of the function of the system itself. At any moment, therefore, a given complex system will contain practitioners and trainees with varying degrees of expertise. Critical issues related to expertise arise from (1) the need to use scarce expertise as a resource for the most difficult or demanding production needs and (2) the need to develop expertise for future use.

14. Change introduces new forms of failure.

The low rate of overt accidents in reliable systems may encourage changes, especially the use of new technology, to decrease the number of low consequence but high frequency failures. These changes maybe actually create opportunities for new, low frequency but high consequence failures. When new technologies are used to eliminate well understood system failures or to gain high precision performance they often introduce new pathways to large scale, catastrophic failures. Not uncommonly, these new, rare catastrophes have even greater impact than those eliminated by the new technology. These new forms of failure are difficult to see before the fact; attention is paid mostly to the putative beneficial characteristics of the changes. Because these new, high consequence accidents occur at a low rate, multiple system changes may occur before an accident, making it hard to see the contribution of technology to the failure.

15. Views of ‘cause’ limit the effectiveness of defenses against future events.

Post-accident remedies for “human error” are usually predicated on obstructing activities that can “cause” accidents. These end-of-the-chain measures do little to reduce the likelihood of further accidents. In fact that likelihood of an identical accident is already extraordinarily low because the pattern of latent failures changes constantly. Instead of increasing safety, post-accident remedies usually increase the coupling and complexity of the system. This increases the potential number of latent failures and also makes the detection and blocking of accident trajectories more difficult.

16. Safety is a characteristic of systems and not of their components

Safety is an emergent property of systems; it does not reside in a person, device or department of an organization or system. Safety cannot be purchased or manufactured; it is not a feature that is separate from the other components of the system. This means that safety cannot be manipulated like a feedstock or raw material. The state of safety in any system is always dynamic; continuous systemic change insures that hazard and its management are constantly changing.

17. People continuously create safety.

Failure free operations are the result of activities of people who work to keep the system within the boundaries of tolerable performance. These activities are, for the most part, part of normal operations and superficially straightforward. But because system operations are never trouble free, human practitioner adaptations to changing conditions actually create safety from moment to moment. These adaptations often amount to just the selection of a well-rehearsed routine from a store of available responses; sometimes, however, the adaptations are novel combinations or de novo creations of new approaches.

18. Failure free operations require experience with failure.

Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure. More robust system performance is likely to arise in systems where operators can discern the “edge of the envelope”. This is where system performance begins to deteriorate, becomes difficult to predict, or cannot be readily recovered. In intrinsically hazardous systems, operators are expected to encounter and appreciate hazards in ways that lead to overall performance that is desirable. Improved safety depends on providing operators with calibrated views of the hazards. It also depends on providing calibration about how their actions move system performance towards or away from the edge of the envelope.

Scoping Developer Tools

I've always been a Kubernetes Maximalist, but recently I've been thinking about the secondary effects of API and technical decisions.

The argument goes, developers don't need to know how to use Kubernetes. Data scientists shouldn't need to know how to build a container.

Both can be true. Infrastructure standardization, but at the same time limited API surface for the end-user.

One interesting example of this is an open-source tool for data analysts called dbt. Don't be thrown off by the relatively small numbers of GitHub stars or issues – this is a widely used tool amongst data analysts – they just aren't always on GitHub.

dbt is successful for many reasons, but the one I find most interesting is that it is mainly a SQL + YAML tool. This reduces the API surface, and most importantly, the skills needed by an end-user drastically. It's much easier to find a data analyst that only knows SQL than it is a data scientist who knows Python or a data engineer who deals with infrastructure.

Configuration, environment setup, and testing could have been done in Python (the language the tool is written in), but instead is done in YAML.

Where else can you intentionally scope the API to tailor it towards a specific kind of technical user?

Bash/Pulumi+YAML for DevOps Engineers / SREs
With some scripting plus YAML, DevOps engineers can orchestrate cloud infrastructure without being fundamental programmers. Still controversial, but if companies can save money by accessing a much larger talent pool, they will. Take a look at this Reddit thread, DevOps becomes "no-code"?

Markup languages (HTML, CSS, Markdown) / JavaScript -> Technical Writers / Frontend Engineer
Harder to see this one play out as most of the markup is moving towards JavaScript rather than away from it (CSS-in-JavaScript, JSX instead of HTML), but this might be a better abstraction for no-code website builders than learning new paradigms.

In fact, you might even argue that static site builders (Gatsby, Hugo, etc.) do something like this by having HTML, CSS, and a templating engine like Handlebars.

Templating or configuration files as mechanisms for removing scope. This can be enabling for complicated workflows that might otherwise require multiple skills that are difficult (and expensive) to find in the real world. This is the version of low-code that makes the most sense to me: removing unnecessary "skill sprawl" that's embedded into many of our tools today.

IDE as a Platform

Developers live in their integrated developer environments (IDEs). They provide a place to browse, refactor, debug, and search code. They usually do autocomplete and integrate with version control systems. IDEs perform many of the same functions they did nearly 40 years ago, but look completely different.

The first IDE was TurboPascal in 1983. It was written by 23-year-old Anders Hejlsberg at the computer company Borland. Hejlsberg would go on to design both C# and TypeScript at Microsoft later in his career.

Some other notable IDEs over the years

  • Microsoft Visual Studio (1997)
  • Eclipse (IBM, 2001)
  • IntelliJ IDEA (JetBrains, 2001)
  • Xcode (Apple, 2003)
  • TextMate (2004)
  • Sublime Text (2008)
  • Xamarin (2011, acquired by Microsoft in 2016)
  • Atom (GitHub, 2014)
  • Microsoft Visual Studio Code (2015)

The first step in becoming a platform is extensibility. IDEs have always leaned heavily on the idea of integration – an all-in-one environment for the entire code lifecycle. Even the first IDEs were extensible in some way or another – through plugins, addons, or packages. Some IDEs integrated deeply within an ecosystem – Visual Studio could theoretically be used for many languages, but excelled at languages that targeted the CLR and the .NET ecosystem.

The second wave towards platformization came as a reaction to the heavyweight language-specific IDEs. Their total addressable market was only the users of a specific language or two. Sublime Text (or TextMate), arguably started a trend of lightweight general-purpose editors that were extremely extensible through a marketplace or package manager. Developers would just install the tools needed for the languages they worked with and the version control systems they needed. Third-party developers could find some distribution through these marketplaces to reach developers.

Visual Studio Code (VSCode) took this to the logical extreme – an open-source editor with a vast marketplace of high-quality extensions. The top extension – Python language tools – has over 44 million installs. But it's not just python, developers love VSCode for every language.

The latest wave towards platformization is coming with run-anywhere VSCode. Like I wrote about in Everything is a Browser, the line is blurring between desktop software and web browsers. There are many ways to run VSCode in the browser –,,, or There are still a few things to figure out – how do you handle high filesystem I/O smoothly? lets you edit local files. But the move to the browser and simpler-than-installing workflow can be really powerful. In the browser, you can assume an always-connected developer and use cloud resources well beyond any laptop can offer. This brings IDEs one step closer to becoming a platform.