Software developers following Agile methodologies often ask themselves how to adapt the traditional Design phase to an iterative software lifecycle. The basic question is how much design should be done before starting the implementation.
There seems to be a consensus that it is not possible to do all the design before starting to code, called Big Design Up Front (BDUF), because at the beginning of the project the requirements are usually incomplete or partially defined. Another reason to avoid BDUF is our understanding that even the requirements that were defined are likely to change.
Some people have proposed the idea of Emergent Design:
“With emergent design, a development organization starts delivering functionality and lets the design emerge. Development will take a piece of functionality A and implement it using best practices and proper test coverage and then move on to delivering functionality B. Once B is built, or while it is being built, the organization will look at what A and B have in common and refactor out the commonality, allowing the design to emerge. This process continues as the organization continually delivers functionality. At the end of an agile release cycle, development is left with the smallest set of the design needed, as opposed to the design that could have been anticipated in advance.”
I personally do not believe in this concept of Emergent Design. I think it may work only when implementing small projects or prototypes. My experience is that the development of large software systems requires a lot of effort to keep the design consistent when successive changes are made. It is a fallacy to imagine that the design will tend to naturally re-organize and re-structure itself in an orderly fashion. The reality is that software tends to break, to become more complex and deteriorate over time.
This divergence from the desired order is called Software Entropy:
“The second law of thermodynamics, in principle, states that a closed system’s disorder cannot be reduced, it can only remain unchanged or increase. A measure of this disorder is entropy. This law also seems plausible for software systems; as a system is modified, its disorder, or entropy, always increases.”
Lehman has also captured this behavior in two of his Laws of Software Evolution:
- A computer program that is used will be modified.
- When a program is modified, its complexity will increase, provided that one does not actively work against this.
A famous article that discusses the complexity of software systems is the Big Ball of Mud. It says:
“No one really goes into any project blindly. The groundwork must be laid, the infrastructure must be decided upon, tools must be selected, and a general direction must be set. A focus on a shared architectural vision and strategy should be established early.
Unbridled, change can undermine structure. Orderly change can enhance it. Change can engender malignant sprawl, or healthy, orderly growth.
The biggest risk associated with Piecemeal Growth is that it will gradually erode the overall structure of the system, and inexorably turn it into a Big Ball of Mud.”
So the question is: How do you design a system to support orderly changes that actually enhance its structure? My answer to this is Adaptable Design Up Front.
Effective Software Design 
I found many good articles from you Hayim. I agreed with most of what you wrote and slightly differed on a few points.
This article is an exception. I totally agree with all the points made. Of course, as usual I have come comments and extensions but they are all in harmony with what is said.
If so much of wisdom on software requirements and design—Emergent Design, Software Entropy, Laws of Software Evolution—-is available, it is surprising how “Extreme and Agile Approaches” are still talked about.
The very idea that pieces of requirements can be picked up “at random” and be used to define and build functionality piece meal is ridiculous (to be polite). Furthermore, the argument that such fragmented and disjoint pieces of functionality will merge into a harmonious system is appalling and infuriating.
Now here are some reinforcing comments on the main points of your article.
BDUF: I do not think anyone is prescribing BDUF. The very fact that one talks of solution architecture, high-level design first is a clear indication that the proponents are well aware of all the incompleteness and changes in requirements. In spite of such uncertainties in requirements, well selected architecture and high-level design based on whatever is known of requirements do cope with new aspects of requirements without having to redo architecture and high-level design ab-initio or de-novo. This is demonstrated in many branches of engineering including software engineering (not convincingly though).
EMERGENT DESIGN: One can be creative but not absurd. Software Entropy and Laws of Software Evolution provide all the reasons why the concept of Emergent Design is ridiculous (though it is obvious and there is no need for reasons!).
I have not read Adaptable Design Up Front (ADUF) but good Architecture and High-Level Design are indeed ADUF. One can enhance “Adaptable” characteristic of ADUF which in software should be very much easier because of the very nature of software: non-material and non-physical.
19JUN13
Thanks for your comments, Putcha. I agree that the apparent success of Agile methods in small projects cannot be directly translated to complex software systems. I intend to write more about ADUF, explaining how it differs from other approaches, when it should be used and giving more concrete examples.
Hayim:
I copied the above in your discussion but it is not there now. It does not matter much but I am wondering why!
It seems there are many interpretations of “emergent”, will you please define the word as such and also as applied to “design”.
20JUN13
Putcha, I started identical discussions in several groups. Your original comment was in the “Object Oriented Analysis and Design Users Group”. I’m very happy with the feedback I get from these discussions. Thanks.
Pingback: Emergence versus Evolution | Form Follows Function
Pingback: Big Design Up Front versus Emergent Design | Random Acts of Architecture
I think reading this article a year ago would have saved our architecture group a great deal of pain and hassle.
ADUF is almost exactly where we ended up, but there was a lot of effort, and even a little light conflict, while coming to these conclusions.
One thing we have found is that ADUF takes a very high degree of coordination with our dev leads and dev team. While this may seem obvious, I’ve found that the “Adaptive” part can be driven heavily by how well (or not) the developers are able to work with the design. The trouble I ran into was the perception of “The Architect said to do it this way, so we make it work”, which oftentimes just made things worse.
Now, every time I hand a design to the dev team we talk it over. What each part means, even if I think it’s obvious. How each part relates to the acceptance criteria of a user story. Expectations surrounding the design, that sort of thing. The next 2-3 days of the sprint, I always follow up, both reading the checked-in code (code is checked in daily, preferably more often) and talking to the developers. The dev lead and I are in almost constant communication. I rarely consider the design “finished” before the end of the first week of the sprint.
This may sounds like it risks work that needs to be redone, and it does. Especially since the development team is in India and I’m normally asleep when they’re at work. And it certainly does give rise to a lot of meetings. But the high level of communication makes sure that when we need to adapt, we do so quickly and efficiently.
Thanks for your comment, Matt. It is interesting to consider the consequences of ADUF for the coordination between the SW architect and the development team. I agree that by defining an extensible design we are placing more responsibility on the developers.
It is not well designed code that emerges over time. What emerges is a clarity about the right design. The authors must frequently refactor. If you’re comfortable with refactoring then it should not be a big effort – it’s really just about reorganizing blocks of code (not rewriting them!). Unit testing keeps you from breaking things.
Thanks for your comment, Pzul. I believe that what you say may apply to low-level design, but not to high-level design (or architecture). My proposal of Adaptable Design Up Front (ADUF) is intended to address the high level design.
Interesting article, but main arguments are actually flawed.
1. Entropy effect is only applicable if nobody cares about design and architecture. It would be same as saying: a city or a country as a extremely large and complex system will end up in chaos very fast because of entropy. You forgot the people factor and misinterpreted most important part in Lehman statement (“…provided that one does not actively work against this.”).
2. “I personally do not believe in this concept of Emergent Design.” Did you actually try it or at least talked to people who delivered large system without BUFD and emergent design.
4. I don’t think anybody makes it black and white as you suggest in your article. There are always a number of architectural decisions with huge impact everyone takes before development starts.
So, at the end. I’m kind of left with no real arguments from you what problem do you actually solve with ADUF. This problem should supposedly occur in so many XP and Scrum projects worldwide where architecture and design emerges most of the time. In reality, Scrum teams usually deliver better quality and architecture with some sort of emergent design than non-Scrum projects.
Thanks for your comments, Viktor. In my article I’ve defined Emergent Design as an equivalent of No Design Up Front (NDUF), and it’s clear that you define Emergent Design as something the follows the initial architectural decisions. Therefore, what emerges is the lower level design and not the architecture.
I’m sure that many projects are successful using a combination of architectural definitions followed by emergent low-level design, which some people call Just Enough Design Up Front. But the problem is how to define systematically the components of this initial architecture.
In my approach of ADUF I try to answer the question of how much design should be done up front. My answer in summary is to capture the essential entities through domain modeling and then add mechanisms to make this design extensible and adaptable. I describe that in more details in my other posts about ADUF, and I will appreciate your comments there:
https://effectivesoftwaredesign.com/category/adaptable-design/
I also plan to continue writing about ADUF, and comments like yours help me to understand which points need clarification.
Your initiative to shine light on architecture and design is certainly welcome. In general I would suggest to use less expensive words to reach Agile audience. E.g. those few crucial decisions taken usually during Product Vision Box Workshop you call “architecture”. The problem after this is that there is suddenly high-level and low-level design and so on. It’s not easy to discuss this stuff when you introduce many new terms. I’m interested in reasons why, which problems do you solve and is that really worth the trouble.
One concrete thing: Why do you think one should define all components in the beginning? I’m even trying to prevent premature unnecessary definition of components when an Agile project starts. Why? It tend to push everyone into more complex architecture based on assumptions. The cost of splitting stuff into more components afterwards is lower, as long as team is disciplined to always apply proper (as you might say “low level”) design practices.
I’ll try to find some time to read your other articles, because they are probably worth reading. 🙂
Thanks, Viktor. Actually, I don’t think it is necessary to define all the components in the beginning. But I do think we should define the components that belong to the “closed” aspects of the system, following the Open/Closed Principle, as I explain in my post:
Please note that in these “closed aspects” there should not be “architecture based on assumptions”, as you say.
Pingback: Attention Agile Programmers: Project Management is not Software Engineering | Effective Software Design
Pingback: The End of Agile: Death by Over-Simplification | Effective Software Design
Pingback: Emergence versus Evolution | Iasa Global
Pingback: “Emergence versus Evolution” on Iasa Global Blog | Form Follows Function
Pingback: Re-Post: The End of Agile: Death by Over-Simplification | Youry's Blog
Pingback: On Agile Architecture, Emergent Design and Framework-Based Design | Effective Software Design
Your post reminds me of something I recently heard Juval Lowy say in an interview: to design software (or to allow a design to emerge) against the particular customer requirements (as opposed to consciously designing against, what he calls, “points of volatility”) is ‘insane’ in the sense that it always ends in brittle, tangled software. Yet we keep doing it.
It makes me wonder whether we have made a religion of Agile (or, at any rate, of a certain understanding of it) whereby we are selectively inattentive to defects committed in its name. To believe that a team of developers – of varying abilities – coding against a stream of individual requirements will gradually give birth to a robust, extensible system architecture is to my mind more a matter of faith than of actual experience: I’ve never seen it happen.
Good architecture has always, in my experience at least, been the result of conscious design and never the by-product of coding against requirements.
Thanks for your comment, David. In the following post I explain that most people who sincerely believe they are doing “Emergent Design” are actually doing Framework-Based Design:
Pingback: Links emergent architectures | The Architect's Cornershop
Pingback: Talk: Adaptable Design Up Front (slides + video) | Effective Software Design
Pingback: [翻譯] 敏捷已死 | NDark MSN Live Space
Pingback: Why Project Management is not Software Engineering? - Smardart