Many idealistic Agile practitioners propose the idea of Emergent Design:
“With emergent design, a development organization starts delivering functionality and lets the design emerge. Development will take a piece of functionality A and implement it using best practices and proper test coverage and then move on to delivering functionality B. Once B is built, or while it is being built, the organization will look at what A and B have in common and refactor out the commonality, allowing the design to emerge. This process continues as the organization continually delivers functionality. At the end of an agile release cycle, development is left with the smallest set of the design needed, as opposed to the design that could have been anticipated in advance.”
Based on my 20+ years of experience with Software Development, I’m convinced that Emergent Design does not work. Below I discuss its main problems, illustrated with some nice pictures.
1st Problem: Software Systems Change
Emergent Design may be enough to create an initial design that will be right until it is changed. If a software design is not planned to support change, it will deteriorate very fast. As illustrated on the picture on the right, it may be challenging to keep the design coherent even for very simple systems. Since software systems always change, a design that does not support change is simply wrong.
2nd Problem: Software Systems Grow
Emergent Design may be enough to create an initial design that will be right until the system grows. If a software design is not planned to support growth, its complexity will increase very fast. As illustrated on the picture on the right, the initial design may become a disaster when one of the components grows. Since software systems always grow, a design that does not support growth is simply wrong.
3rd Problem: Dependencies are Important
Emergent Design is based on ignoring the big picture and designing only for the lower-level tasks (user stories). This may work well when these tasks are completely independent from each other, but in most software systems the dependencies are very important. If individual tasks are designed and implemented in the wrong order, ignoring their dependencies, the result may be a total mess, as illustrated on the picture on the right.
4th Problem: Coordination is Essential
Emergent Design is based on the individual work of software developers, ignoring the need of a higher-level shared architecture and team-level cooperation. Each programmer is responsible for his own low-level tasks, which he tries to implement with minimal coordination with other team members. As illustrated on the picture on the right, this lack of coordination may cause a total deadlock.
The Solution: Define Clear Interfaces
The only solution to all the problems above is to define very clear interfaces among the modules in a software system. Of course, this definition of the interfaces must be done much before the developers start the implementation of the individual components. At the component level the low-level design may emerge, but at the system level there must be a pre-defined shared architecture. I give more information about my proposed approach for software design in my posts about Adaptable Design Up Front.
What do you think? Please share your experience in the comments below.
Effective Software Design 
Great post, Hayim!
100% agreed, some design must be made upfront. That’s the role of a software architect.
Thanks, Vladik!
By the way, the notion of clear interfaces resembles a lot DDD’s bounded contexts – http://martinfowler.com/bliki/BoundedContext.html
RE “4th Problem: Coordination is Essential” – do you mean explicit design-time coordination or explicit and machine-assisted run-time coordination (as in your picture) or both of them?
I mean coordination among team members during the design and implementation of the software system. For example, if there is no proper coordination, a developer may be unable to complete his work because he must use an API that was not yet defined.
The point around API’s and API definition / governance is particularly apropos. In my experience Agile and Emergent Design are increasingly used (in conjunction with REST) as excuses to avoid any kind of up front API thought and relying on Swagger based tooling to (at best) produce some as built docs.
Whereas it only seems to take minimal agreement up front between parties to define what they should be building towards to both increase the independence of the development concerns as well as flush out massive amounts of issues (in both implementation and understanding misalignment) that otherwise impede progress for multiple sprints to come.
Hi, Hayim.
I tend to agree with the issues you have raised in the article, and it is already practiced out there, mostly as Architectural guidlines or standards that comes from system architects, which then detailed later on by emergent deigns of the teams, of course aligned by very good coordination & communication between teams.
You can see for example how the SAFe model describe it: http://www.scaledagileframework.com/agile-architecture/
So, I would say emergent design is good, but maybe not by itself alone. There should be long-term, high-level, modullar architectural guidance as well, to keep it scalable, flexible and so on…
I would say, in a wider view, this goes to other product lifcycle aspects, such as:
1. Planning –> there should be a roadmap, long-term vision & intent, which detailed later on by User Stories per iteration…
2. Documentation –> There should be technical documentation for a product, they just start with a “big picture” and detailed incrementaly along the way during product development…
Hi Hayim,
It’s good article that highlights the major drawbacks of the Agile approach. My current understanding Agile serves the purpose of collaborating a collective mind by making the process more controlled and visible for the management. There is even a ‘lighter incarnation’ of it nowadays call Kaban methodology mainly applicable to ‘Work in Progress’ phase.
.
I tend to say slightly differently – nothing is fundamentally wrong with the tool itself like the hummer in the IPhone era. What the more likely makes a difference is – how we consume them for the relevan tasks. It’s applicable to the DevOps process when the project itself is stable enough. There is no voting (51/49) based decision should dominate when a project architect looking for more radical change in the project structure, unless it’s considered be too risky for the business perspective. It’s more about luck to have great leadership, trust and responsibility that is never substituted by the collective opinion. I believe the upfront ideas and design will ever stay crucial even often based on well defined templates.
I agree Hayim…great post!
In my experience agile approaches tend do focus on “things” that makes it possible for anyone, with no experience or long experience, to participate in a agile teams. If software engineering was hard before it is suddenly easy…just call it agile and your home free. You can skip the design and everything that is important when it comes to actually deliver something with business value. The biggest problem in my opinion is the lack of scenarios in the “userstory” approach. The scenarios is important for being able to understand the business problems and from them you can do the proper skeleton design of the solution in front. This is a must to be able to deliver a solution with a predictable lifecycle and behaviour and a real value for the business. Don’t get me wrong..i like the agile approach with agile teams and the measuring progress through, for example, scrum boards etc. What I don´t like is the lack of proper software engineering.
I don’t know the definitions behind these methodologies, but totally agree with the article. Design never “emerges”, because that is a top-down thing, something that controls the actions.
Understanding emerges, and that brings needs for refactoring: changing the control – if there is anything to refactor, not only local solutions to a pile of “tasks” handled independently.
To have anything more than that useless heap, you need an initial design. To support efficient refactor, that design should also be represented in the implementation (interfaces, module separation).
Two of my favorite quotes: “You don’t need plans to follow. You need plans to have at least something to diverge from.” “Failing to plan is planning to fail.”
… and maybe you will like this too: https://www.linkedin.com/pulse/writing-source-code-evil-lorand-kedves
(copied from LinkedIn)
Agile and Emergent Design are great and a reality.
BUT…
When talking on architecture one relates to the bigger picture. In order to get to the bigger picture you need more than just a few user stories.
At the time the Unified Process (today Open/Agile Unified process) proposed that a large project transitions through 4 different phases of maturity; Inception, Elaboration, Construction and Transition. For the sake of this discussion i propose to call a project the that part of the a system that provides full operational functionality to the customer; that is the set of epics/user stories that only after all of them have been delivered, the system becomes fully operational.
The goal of inception is to successfully complete the PoC, The goal of elaboration is to reach a sound architecture. These phases are typically done with a small highly skilled team and consumes ~ 25% of the planned development effort. The goal of construction is to actually build the system (spending 70 % of the effort) where as transition focuses on rendering the system fully operational.
Therefore sprints during inception and elaboration are focused on architecture, on getting the confidence that the design allows development of the system and on getting feedback from the customer on the correct understanding and implementation of the needs.
Based on experience, these sprints (because of their architecture/design goals, should be longer sprints, let’s say sprints of 4 weeks. Sprints during construction, that is initiated on system parts whose architecture/design has stabilized can be short ones, sprints of let’s say 2 weeks.
So during construction the design is only adapted conforming to the architecture rules, guidelines and concepts that have been defined/designed during the initial 2 phases.
Very good post.
Unfortunately I have seen the result of emergent design too many times during my soon 30 year of experience and the result is too often a mess.
One problem is that refactoring never happens in practice, and as we, i.e. my current employment organization, are bound to be “infinite” backwards compatible in our truck product line development – we are just not allowed to make any kind of refactoring as it will have real cost impact on the aftermarket.
Secondly, and perhaps the biggest problem, as the next project (and project management team) are bound to deliver just their specific feature scope (in time, budget, etc) there is no willingness in a succeeding project to correct what another predecessor project ruined. In our case the line organization that should own the elements of the product line is not strong enough (and some might not even have proper competence) and do not really dare to NOT accept a solution delivered by a project. The balance between the owning organization and the project organization is unbalanced. Very much related to this is that the line organization does not really have any budget for owning the product line but all budgets are owned by the project. So this is also a budget distribution problem.
My experience is that if you are doing big design mistakes in your first projects you will have to live with them until you phase out the solution completely. Much of this is related to cultural stuff but of course also to pure knowledge. Unfortunately a good designer is not always the same as being a good programmer.
The pictures are great.
It does seem a little unfair to tag Agile with the 4th problem – communication is essential.
Most Agile methods put a great emphasis on communication within a team.
Scrum for example requires a short team meeting each day.
Pingback: TDD: Almost, but Not Quite, Entirely Like Test-Driven Development " | "LeadingAgile
read https://www.martinfowler.com/articles/designDead.html (long but insightful)
Yep, I was reminded of that article too. Very much worth reading.
Pingback: TDD: Almost, but Not Quite, Entirely Like Test-Driven Development " | "
Excellent article, precise, well argued, obviously backed up by sound practical experience, and therefore largely irrefutable (inevitably, some commentators tried nonetheless). I see “emergent design” as an euphemism for “lack of concept”, and a recipe for disaster, especially with large projects that require maintainability over many years.