It was never “Agile” vs “Waterfall”

This post by Brendan Best is from Red Plaid Barn

A pointless argument, full of sound and fury, signifying nothing

Plenty of research supports the idea binary thinking is deeply entrenched in how our brains work. Experts in Psychology, Evolutionary Biology, Sociology, and Philosophy offer explanations why binary thinking is as common as it is. These include evolutionary: There’s not a lot of room for nuance when a sabre tooth tiger is heading towards you. Cognitively simplifying your choices, improves your chance of living long enough to pass on your genes. Sociological explanations include class and geographically determined exposure to differences in culture, linguistics, and gender expression as factors. No matter how or why we humans gravitate towards understanding our world as a series of binary, opposed, pairs, it’s hard to deny that it’s extremely everywhere. It shows up in almost every aspect of human endeavour. Think of how often people argue over something as trivial as yea or nay to pineapple as a pizza topping. It’s depressingly common for people and politicians to take all-or-nothing positions on complex questions like public health responses to COVID-19.

That’s life on the Serengeti for ya

It seems reasonable to suggest that binary thinking is as persistent as it is because it carries some advantages. Binary thinking benefits us by simplifying decision making, and reducing complexity. There are instances where this is helpful. It avoids the pitfalls of “choice paralysis“, and its sibling “analysis paralysis“. Certainty and decisiveness can be their own reward, at least part of the time. The downside is that it increases the risk of flawed conclusions and limits our ability to identify other, more nuanced approaches. Where psychologists describe this behaviour as “binary thinking”, philosophers and logicians are likely to refer the logical fallacy of a “false dichotomy”. False dichotomies are a problem because they force acceptance of one approach as the only alternative to an undesirable extreme. Obscuring nuance and avoiding complexity negatively impacts our depth of understanding and impairs our ability reason. Over the last twenty-five years, one of the most persistently unhelpful threads in software has been “Agile vs Waterfall”. It’s a question invoking the same stiff-necked intransigence as debates over pineapple on pizza. Or worse it evokes more serious subjects like public health responses to COVID-19. Characteristic of binary thinking, the perennial “debate” looks at the problem from one angle, and only one angle. As often as not, it’s not an especially useful angle, either.

Under the waterfall

The words “Waterfall’ and “Agile” are used and abused as proxies for “Things I Hate About My Job” or “Things I Love About My Job”. This supposed dichotomy between “Waterfall” and “Agile” is emblematic of a difference not of methodologies. Instead it’s one of philosophies governing our professional lives. In this context we see “Waterfall,” its linear path through the phases, maligned as bureaucratic, too rigid or even just too old fashioned. It belongs to a bygone era. It’s from a time of ironclad plans and sequential processes. It hearkens back to an era when change met resistance and deviation from the path represented a risk to avoid or to mitigate. It epitomizes the idea of professionalism as dogged, (dogmatic?) adherence to the best practices the received wisdom has to offer.

A new hope…

“Agile” advocates promote its promises of new hope —grounded in adaptability, collaboration, and iterative progress. Ultimately it is a mindset, one that embraces change as a natural and necessary aspect of the development process. It shuns the rigidity of its predecessor in favour of a dynamic, customer-centric approach. To advocates of “Agile,” it represents more than an approach. It is an ethos empowering teams to respond to the unavoidable evolution of requirements as part of the work. There is always the potential for overselling its advantages to apply everywhere, all the time. Then “Agile” becomes the hammer in search of a nail, or magic pixie dust. In short order, we saw the term Agile co-opted as a quick fix for every woe of software development. Commodification reduced “Agile” to a buzzword. Poorly applied, “Agile” and “Scrum” are a recipe for chaotic un-planning of work, performative and unhelpful ceremonies and frustration, This usually results in software development teams longing for the predictability and familiarity of “Waterfall”. It may be broken, but it’s broken in ways we can predict. In practice “Waterfall” and “Agile,” are less approaches to development than they are ideological proxies. Two terms that intentionally or not, obscure material differences with formalist ones. If we look closely, we see that the differences aren’t between Waterfall or Agile. The differences are between favouring Scientific Management, Command and Control structures and embracing Initiative, Autonomy and Mutual Accountability. Everything else is window dressing.

Sweat, Tears, and Taylorism: Industrial efficiency, scientific management and its risks

While working at U.S. manufacturer Bethlehem Steel Frederick Taylor observed that the managers knew little, if anything about how jobs performed under their oversight were actually performed¹. A one time industrial apprentice, Taylor’s career saw him promoted to gang-boss, foreman, and eventually chief engineer. Taylor’s professional interests reflected his experience. His interests centred on optimizing efficiency and minimizing waste. To achieve this he focused much of his attention on principles that are the foundation of scientific management to this day:

• Use the scientific method² to determine the “one best way” to do the job, to this day we hear the term “best practices”
• Decompose work tasks to the smallest reasonable units and assign jobs based on individual strengths and aptitudes. Following this, assess which workers are most capable of each specific job and train them to work at peak efficiency.
• Monitor workers‘ productivity and efficiency, correct sub-optimal performance with further instruction or training when required.
• Separate the work tasks, identifying which responsibilities are managers’ and which belong to workers’. Managers plan and train. Workers implement according to that planning and training. You may have encountered modern-day proponents of this as “Think / Do”.

These principles make sense in the context of consistent and predictable processes. The are especially effective when we can identify risks in advance, where a successful business or product outcome is effectively static. In software development projects that take less than a month to complete and address familiar and well understood needs . These principles apply with different degrees of commitment accordingly.

Not going anywhere

To this day scientific management and its inheritors have proponents in almost every field of human activity. Evidence suggests many contexts still exist where these principles successfully apply . It’s hard to imagine that they wouldn’t continue to exert so much influence, otherwise. The influence can be see in the way companies of every size and industry, sports organizations, non-profits, and even political parties operate. Some scholars dispute whether or not it was Taylor’s intention to focus on how management manages, that is controls the flow of work (and information), for optimum efficiency. Whatever Taylor’s intentions, that is how most working people experience the principles of scientific management. It is noteworthy that it’s also an approach that due to its tendency towards hierarchy nature the horizontal flow of information. For example the intermediation of managers between workers in different disciplines represents an extra-step and introduces an opportunity for miscommunication, without adding value in most cases. It also favours the flow of information from the top down. This risks disincentivizing workers from raising concerns about quality, efficiency, safety etc. depending on the availability of managers or managers’ openness to hearing unwelcome information.

The Challenger Shuttle Disaster

One of the proverbial examples of the consequences of insufficient flow of information is the 1986 Challenger explosion, that resulted in the deaths of seven NASA astronauts and contributed greatly to the eventual end of the Space Shuttle Program. For most of us, most of the time it’s rare that how we operate has consequences of life and death, or of the scope of the NASA Space Shuttle Program. Many of us do experience the result of poor information flow. In my experience hierarchies and top down management lend themselves to poor information feed back loops, hampering decision making and increasing risk. Scientific management is intentionally or not, based on principles of command and control.

Breaking out of the binary, the power of trust, accountability and autonomy

It’s possible to look at the persistence of Waterfall and it’s non- or anti- agile approaches in the context of how well they dovetail with organizational resource planning, enterprise budget allocations which often occur as much as year in advance. Waterfall’s emphasis on early, detailed planning is a similar approach. The question of whether of not the stage gated separation of requirements gathering, planning and execution into discrete blocks of effort, scheduled in advance produces optimal results from the product, consumer or business outcome perspective is secondary to how well this approach fits into a larger organizational culture. Hierarchies match well with hierarchies, they often present as mutually intelligible fractals. I think that the above is supported by evidence. It is also limiting because it is resigned to judgement about hierarchy vs agility verging on the moral. I think it would be more helpful to accept that there are different types of operations and tasks. Some will be more efficiently and effectively executed according to a clearly defined, stage-gated plan, and some that are better suited to just enough planning, short iterations and rapid feedback cycles. I think we can look at two poles of kinds of work, deterministic or non-deterministic.

Deterministic versus non-deterministic

Deterministic projects or operations are those where each successive state is determined by the preceding state. The existence of such a project or operation of any greater complexity than installing a WordPress theme is debatable. There are nonetheless plenty of types of operation where the work can be predicted with a high degree of confidence. In cases where the work is familiar, templated or near to it a conventional waterfall approach is possibly more efficient, since iterative, planning and deciding at the latest responsible moment approaches favoured by agilism, are going to represent additional meetings and overhead, offering little value. Non-deterministic projects or operations are ones where the all the factors affecting an outcome are difficult or even impossible to predict earlier, and sometimes are only knowable after the fact. These are suited to the kinds of risk mitigation that doing just enough work in a product cycle to test a hypothesis, and confirm that we’re adding value. In this context an Agile approach, Lean, XP, Scrum, or Kanban is likely to produce the most valuable outcome, in the shortest possible time. Experience tells me, deterministic projects are smaller, shorter than three months, partly because it’s very unlikely that any project or operation that is longer than that will remain unaffected by outside events or forces, such as changes in software platforms, frameworks, third party integrations, competition, and so on. Either of these cases are subject to a factor that doesn’t fall cleanly on one side or the other of Agile vs Waterfall, and that is the relationship of the workers to the work.

More Taylorism or why it was never “Agile” vs “Waterfall”

This brings us back to Taylorism, the question of command and control versus autonomy. Despite claims to the contrary a Waterfall/Stage Gate Project doesn’t have to be run according to a strict command and control model. I have seen plenty of times where the PM facilitates the horizontal communication across stages, disciplines and groups. It may require a greater emphasis on working that trust and autonomy into staged work than it does in an Agile context. But no one who has participated in an Agile project thinks that communications, accountability and autonomy don’t require constant attention in an Agile context as well. When we say mutual accountability, it describes an approach where all team members equally share in the teams’ outcomes. We use autonomy to mean that team members are responsible for deciding what to build, how to build it, and how to deliver it to market. These factors along with organizational and team transparency, clear organizational and team objectives, and measures for success or failure are more important to the success of any team, project or operation than whether it’s run “Agile” or “Waterfall”.

1. Editorial comment: It’s hard not to feel that in a large number of workplaces that little has changed in that respect in over 100 years. ︎
2. There are many definitions of “the scientific method” for our purposes the scientific method can be summed up as “Do an experiment, record its outcome faithfully and objectively and make that record available for doubters.” see The New Scientist https://www.newscientist.com/definition/the-scientific-method/ ︎