The Tarpit

Oct 10, 2021

In 1986, Fred Brooks made the prediction that in the future we wouldn't find any magical solution for what he called essential complexity in software construction. You might remember Brooks from The Mythical Man-Month set of essays on project management and software engineering.

Brooks' argument boiled down to a running analogy that I love:

As you get better at running, it never gets easier. You just get faster.

Brooks makes a distinction between essential and accidental complexity. Following the running analogy, accidental complexity is avoidable toil: not having running shoes or proper gear, not fueling or hydrating correctly, or bad form. We can avoid this kind of complexity in software – using higher-level languages, more powerful machines, and better tools. But, we can't avoid essential complexity – communication between team members, understanding all possible states of the program, and extending programs to new functions without exposing new side effects.

In No Silver Bullet – Essence and Accident in Software Engineering, Brooks explains some of the proposed solutions to essential complexity, and why they won't materialize. On graphical programming:

software is very difficult to visualize. Whether we diagram control flow, variable scope nesting, variable cross-refrencing, data blow, hierarchical data structures, or whatever, we feel only one dimension of the intricately interlocked software elephant. If we superimpose all the diagrams generated by the many relevant views, it is difficult to extract any global overview.

Other silver bullets that Brooks believed wouldn't materialize include program verification, AI / expert systems, and object-oriented programming.

What does Brooks think you should do? Buy software – the best kind of software is not writing software, build software incrementally, and hire great designers.

On another interesting tangent, Brooks breaks down what he believes are the four components that make building software hard: complexity, conformity, changeability, and invisibility. 30 years later, a paper called Out of the Tar Pit took a critical look at how the ideas of Brooks had evolved over the years and made an argument that complexity is the only significant one.

The authors of the Tar Pit paper go on to categorize their four components of complexity: state, code volume, and control flow. The authors make some interesting distinctions early on but then get into more practical suggestions like functional programming, immutability, and other techniques. The authors come to some obvious solutions: write less code, and when you do write code, always aim for simplicity. But the paper is worth a skim regardless.