Have you ever thought about how Agile history has evolved and the broad archaeology behind what direction it is going in? I recently responded to a question about Extreme Programming (XP) on a discussion forum and I made the comment that I thought that the usage of XP has been largely superseded by Scrum. Someone responded to my comment and said that he didn’t agree that Scrum has displaced XP. To really understand what is going on, its useful to pretend to be amateur archaeologists and step back and look at the history of how Agile has evolved.
Image by Makalu from Pixabay
What is Archaeology and How Does it Relate to Agile?
Archaeology is an interesting area. Here’s now “archaeology” is defined:
” The study of human history and prehistory through the excavation of sites and the analysis of artifacts and other physical remains.”
https://www.lexico.com/en/definition/archaeology
I recently traveled in Central America, saw a number of different Mayan ruins, and it was very interesting to see how the Mayan civilization evolved and declined over hundreds of years.
- The Mayans had a very extensive and elaborate civilization that was in its prime from about 200BC to about 950AD. It began to significantly decline about 950AD in the southern areas of Central America (Belize, Nicarauga, and Honduras) while it continued to some extent in northern areas of Central America (like Yucatan) until those areas were decimated by the Spanish Conquistadors in the 1500’s.
- The Mayans built massive temples and cities with an elaborate infrastructure that almost totally disappeared into jungle undergrowh until the full extent of it began to be discovered in the 1800’s by explorers such as John Lloyd Stephens. Explorers like John Lloyd Stephens began to discover how extensive the Mayan civilzation was by clearing away some of the jungle overgrowth that had completely covered up a lot of the very elaborate Mayan ruins.
There have been many studies of Mayan archaeology since that time that have broken down the evolution into eras. Here’s a good one from the Canadian Museum of History. The essence of archaeology is to look back at an evolution like this that takes place over a long period of years to better understand the patterns of growth and the factors that influenced the growth and decline of a civilization.
Agile “Archaeology”
We shouldn’t have to wait hundreds of years to look back at how Agile has evolved to understand how Agile has evolved. If we begin to clear away the “jungle” of various methodologies, frameworks, principles, and practices, we can begin to see how the underlying evolution has progressed over the years and the general direction it is headed. If we step back and look at the history of Agile, we can begin to better understand the patterns and trends that are going on.:
Agile History Overview
I’m not going to attempt to capture every single detail related to the evolution of Agile history. I only intend to capture what I think are the most important elements of this evolution and show how they fit into periods of evolution. Here’s a brief summary of the periods that I have used to describe this evolution:
Early Software Development Period
1965 – Software Crisis of the 1960’s
“Software engineering was spurred by the so-called software crisis of the 1960s, 1970s, and 1980s, which identified many of the problems of software development. Many projects ran over budget and schedule… The software crisis was originally defined in terms of productivity, but evolved to emphasize quality.”
Project Management Period
Prior to this period of time, software development used a very adhoc, unstructured approach. There was not a well-defined and organized approach for managing the development and testing processes of the software to ensure that the software:
- Met design requirements within cost and schedule goals
- Was well-designed to appropriate standards of coding and architecture
- Was well-tested to ensure that it was free of major defects
Prior to this period, there were some disastrous projects and there was a lot of pressure to improve the way software was developed. The pressure to improve the software development process came particularly from the US government defense department. The Defense Department was beginning to launch some very large-scale development porjects that had to be reliable and should be relatively close to their cost and schedule budgets.
1969 – PMI Founded
In 1969,, Jim Snyder and others created the Project Management Institute. “It goes back to the 1960s, when project management became a discipline in aerospace, construction and defense industries. It was from these industries that the PMI seed was planted.” This was a recognition of the need for some amount of project management discipline in successfully planning nad managing large, complex projects.
1970 – Waterfall Model
In 1970, Dr. Winston Royce published his famous paper entitled “Managing the Development of Large Software Systems“. This paper defined what has become commonly known as “Waterfall”. Prior to this period of time, the development of software was largely unstructured and there were huge cost and schedule overruns. The primary emphasis of “Waterfall” was to try to bring some level of control and predicatbility over project costs and schedules. However, even Dr. Winston Royce who documented the initial Waterfall process in his 1970 paper recognized the limitations in this process.
- It heavily emphasized planning and control and was very inflexible which made it difficult to implement changes that might be necessary while the project was in progress. It also had a tendency to stifle creativity and innovation
- It was bureaucratic and documentation-centric which impacts the cost and schedule for implementing the project
- There was very limited contact with the potential users and users did not typically see the overall solution until the very end of the project so there was very little opportunity for feedback and inputs as the project was in progress. The impact of that is that many projects may have met their cost and schedule goals but failed to deliver an appropriate level of business value
In spite of all of these limitations, its important to recognize that the Waterfall model was a major improvement over what came before it. Prior to the Waterfall model, software development used a very adhoc, unstructured approach. There was not a well-defined and organized approach for managing the development and testing processes of the software.
Manufacturing Influence Period
Agile was heavily influenced by new developments in the manufacturing industry which included:
1978 – Toyota Production System
Taiichi Ohno publishes the book “Toyota Production System – Beyond Large-Scale Production” describing LEAN, Lean Manufacturing and Kanban. “Toyota Motor Corporation’s vehicle production system is a way of making things that is sometimes referred to as a “lean manufacturing system,” or a “Just-in-Time (JIT) system…” – History of Kanban
1986 – New New Product Development Game
Two Japanese business experts published an article in the Harvard Business Review entitled “New New Product Development Game” that is considered to be the early foundation for the thinking behind Scrum – The New New Product Development Game
1990 – Lean Manufacturing
“The thought process of lean was thoroughly described in the book The Machine That Changed the World (1990) by James P. Womack, Daniel Roos, and Daniel T. Jones. In a subsequent volume, Lean Thinking (1996), James P. Womack and Daniel T. Jones distilled these lean principles even further to five principles.”
2007 – Kanban
Kanban originated in a manufacturing environment (See Toyota Production System). In 2007, David Andersen described the software development process that resulted in the Kanban board, Kanban card and the majority of the Kanban policies that have been applied to software development
Early Developer-centric Period
Agile started out in the early 1990’s as a revolution by developers against traditional, plan-driven Waterfall development processes. The common denominator of many of these processes was a heavy emphasis on development processes and a rejection of the need for project-level planning and control. Some of the early Agile development processes in this period included:
1991 – Crystal Clear
“Crystal Clear” was a very early Agile methodology developed by Alistair Cockburn that is no longer used. It is one of the most lightweight, adaptable approaches to software development Crystal Clear
1996 – Extreme Programming (XP)
XP was defined by Kent Beck and others. “Extreme Programming (XP) is an agile software development framework that aims to produce higher quality software, and higher quality of life for the development team. XP is the most specific of the agile frameworks regarding appropriate engineering practices for software development.” – Agile Alliance
Solution-level Period
During this period of time, it was recognized that an emphasis on development level practices only was not sufficient to develop well-integrated solutions to meet important business requirements. During this period of time, the emphasis shifted from simply focusing on development practices to developing overall solutions. Important developments during this period included:
1995 – Scrum
Jeff Sutherland and Ken Schwaber first defined Scrum as an Agile framework. “Scrum is an agile process framework for managing complex knowledge work, with an initial emphasis on software development, although it has been used in other fields and is slowly starting to be explored for other complex work, research and advanced technologies
1996 – Rational Unified Process
Rational acquired the Objectory Process that had been written by Ivar Jacobson and company and renamed it “Rational Unified Process” (RUP). The Rational Unified Process (RUP) is an iterative software development process framework. The initial implementation of RUP was heavily based on Rational tools.
1997 – Feature-Driven Development
Feature-driven development (FDD) is an iterative and incremental software development process that was developed by Jeff Deluca. “It is a lightweight or Agile method for developing software. These practices are driven from a client-valued functionality (feature) perspective.
1999 – Enterprise Unified Process
The EUP is a simplified version of Rational Unified Process Developed by Scott Ambler. It is a more general process that is not dependent on Rational tools and “whereas the RUP defines a software development lifecycle, the EUP extends it to cover the entire information technology (IT) lifecycle”
Agile Definition Period
The proliferation of different methodlogies that evolved during the 1990’s created a need to define some common ground that unified all of these methodologies.
2001 – Agile Manifesto
“Representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes convened” to define a common set of values and principles to unify Agile.
Enterprise-level Period
Agile has significant implications on an enterprise that are essential to implement it successfully. Many prior implementations of Agile were built around simple, small projects. As Agile started to grow into larger and more complex enterprise-level projects, there was an increased recognition of the need for addressing enterprise-level frameworks that better-defined how Agile might be integrated with a business for large, complex projects.
2011 – Scaled Agile Framework
SAFe is an enterprise-level framweork developed by Dean Leffingwell. “The Scaled Agile Framework, or SAFe, methodology is an agile framework for development teams built on three pillars: Team, Program, and Portfolio. SAFe is designed to give a team flexibility and to help manage some of the challenges larger organizations have when practicing agile.
2012 – Disciplined Agile Delivery
DAD is a hybrid, enterprise-level framework developed by Scott Ambler. “Disciplined Agile Delivery (DAD) is a people-first, learning-oriented hybrid agile approach to IT solution delivery. It has a risk-value delivery lifecycle, is goal-driven, is enterprise aware, and is scalable.”
2013 – Managed Agile Development Framework
The Managed Agile Development Framework is a hybrid, project-level Agile approach that was first defined by Chuck Cobb in 2013 in his book “Managed Agile Development: Making Agile Work for Your Business” It provides a way of blending a plan-driven project management approach with an Agile development process.
Overall Summary
It’s easy to get lost in the “jungle” of software development and see just a collection of different methodologies that might even seem to be in conflict with each other. By stepping back and looking at the broad trends that have been going on, we can get a better understanding of what’s going on in this evolution and how these various methodologies fit into a pattern.
Over all this period of time, there have been some significant changes; and, as with any significant change, the pendulum often swings back-and-forth before it settles into the middle.
Related Articles
Check out the following related articles on the “Understanding Agile”:
- Agile History and Archaeology
- What is Agile? How Would You Define Agile? What Does Agile Mean?
- Is Agile Just a Development Process?
- Agile and Six Sigma – Are They Complementary to Each Other?
- What is the Real Essence of Agile? What Are the Real Advantages?
- Mixing Lean and Agile – Is Lean in Conflict with Agile?
- What’s the Future of Agile? Is There Something Else Coming Next?
- Product Development Flow and Agile
- What Is the Agile Scrum Master Role?
Additional Resources
Resources for Agile Project Management Online Training.