At the Boston ACM Agile Bazaar June 18 meeting, Ken Schwaber presented a slide from a recent Forrester survey that showed that in 2008 the use of agile development methodologies overtook the use of traditional waterfall methodologies. Here is Ken’s slide:
Three thoughts struck me about this chart:
1. The crossover to agile in many areas of development and in the U.S. actually occurred much much earlier. In the last several years, those areas would be considered the drivers of innovation, for example, Web 2.0, social media, web services, etc. In many of these areas, agile processes are so ingrained that no one would even think of not doing agile development; the question doesn’t even come up.
I’m purposely using a broad scope of the term “agile” beyond any particular formalism, such as Scrum, to include formal and informal development processes that are modern derivatives of spiral iterative empirical development methodologies, that share many of the same objectives and methods, and thus that are antithetical to a waterfall methodology.
2. Though from the chart, a lot of development appears still to use a waterfall methodology, you don’t hear much about these projects, and you don’t hear them being seriously touted as innovative. Can anyone any longer justify using waterfall for a modern development effort? Is waterfall methodology simply obsolete?
3. The trajectory of the agile adoption curve is clear. However, if the drive to innovate invariably leads to a drive to adopt agile processes, can you innovate without being agile? Or stated conversely, do projects necessarily fail to innovate when they continue to use or are forced to use elements of a waterfall process?
Long before the term “agile” was coined, developers of innovations embraced empirical incremental methods and shunned waterfall methods.
In my experience, I have always found that true innovation requires some variation of an agile process. I’m hard pressed to find any interesting use for waterfall or its core elements except in project management education for providing a historical background. Perhaps, I am being stark, but innovative and interesting appear to be inconsistent with waterfall.
Unfortunately, in a setting or organization that hasn’t already gelled around agile, you still too often run into extreme resistance, both active resistance and passive resistance, to implementing core attributes of an agile process. This can be especially troublesome when forming a new organization to pursue a new innovation. The resistance to agile principles by those purposely or unknowingly seeking to perpetuate waterfall methods can hamper and thwart the project, and can set up the project for failure. Resistance can also result in a denial to recognize the success of an agile-run project, by characterizing the success as a fluke rather than an outcome of the agile methodology. There are repeated examples of precisely such scenarios. Hopefully, now that agile has overtaken waterfall, such scenarios will become less frequent and eventually disappear.
Returning to Ken’s talk, he discussed and gave several examples of how people and projects fail to apply agile methods properly, specifically with respect to Scrum methodology. The title of his talk was “Flaccid Scrum – A New Pandemic?”. Ken is one of the originators of Scrum methodology. His bio and the talk are described on the Boston ACM Agile Bazaar website, where Ken’s slides are available. Further, information about Ken and his efforts can be found on the Scrum Alliance website and Ken’s website.
- – -
For another comparison of transition of interest to agile from waterfall development, here is a look at Google Search Insights data. Admittedly, search results is a leading indicator.
In this chart of worldwide search, agile overtook waterfall in early 2005. The current search ratio is >2 to 1.
Side note: The curves combine the search data for “development” and “model”. The dominant search terms are “agile development” and “waterfall model”. However, “waterfall development” search results are significant as well; “agile model” is not particularly significant, but is added for symmetry.
Here is a chart of the U.S. search data:
In the U.S., agile overtook waterfall perhaps in 2003 or at least by 2004, the earliest data available from Google. The current search ratio is ~3.5 to 1. The U.S. chart compared to the world chart supports the notion that the transition occurred earlier in the U.S.
Both of these search charts support the clear transition to agile processes and the repudiation of waterfall processes.