In the last decade we have seen a decrease of interest in Software Design and a sharp increase of interest in Agile methods. The first graph below compares the trends between Software Design and Agile. The second graph compares the trends between Design Patterns and Scrum. They are very similar. And in my personal opinion, they are very sad.
I am a firm believer in the importance of Software Design. In particular, I strongly believe that Software Design is essential in the context of Agile and Lean software development. In a series of previous posts I defined my Adaptable Design Up Front approach that is focused exactly on supporting Agility: having a software architecture that embraces changes through adaptation.
Thus I am trying to understand this phenomenon that has caused so many software developers to lose their interest in Software Design and get so enthusiastic about Lean and Agile methods. I think one of the reasons is the current predominance of Indefinite Optimism.
Definite vs. Indefinite Optimism
The difference between Definite and Indefinite Optimism is described by Peter Thiel in his book “Zero To One, Notes on Startups, Or How to Build the Future“.
Definite Optimist: has concrete plans for the future and strongly believes that the future will be better than today.
“To a definite optimist, the future will be better than the present if he plans and works to make it better.”
Indefinite Optimist: has great expectations about the future but lacks any design or plan for how to make such a future a reality.
“To an indefinite optimist, the future will be better, but he doesn’t know how exactly, so he won’t make any specific plans.”
Indefinite Optimism, Agile and Lean
If we go back to the graphs above, in my opinion it is clear that they depict a transition in software development from definite to indefinite optimism. Agile and Lean methods are characterized by the lack of planning and design, by the emphasis on the process and by a tendency to postpone decisions.
For example, the popular YAGNI principle from eXtreme Programming instructs developers to avoid implementing things that they are not sure if they will need. In practice, programmers adopt the optimistic guess that they will be able to implement anything when it is really needed. Then, they not only avoid the implementation of new features, they also avoid preparing for these features. In other words, YAGNI causes people to avoid designing for extensibility.
In another example, the Last Responsible Moment principle from Lean Software Development instructs developers to postpone their decisions as much as possible. In practice, programmers adopt the optimistic guess that this Last Responsible Moment will always happen in the future. Then, they not only avoid making premature decisions, they also avoid thinking about the consequences of these decisions. In other words, the Last Responsible Moment causes people to treat decisions as if they were independent of each other, and not part of a cohesive design.
Finally, I am sincerely amazed about the absolute emphasis the Agile community places on the process, focusing on project management instead of software engineering. Apparently having a Sprint retrospective to discuss how to improve the process is considered by many Agile practitioners to be much more important than conducting proper code an design reviews. In the words of Thiel:
“Arguing over process has become a way to endlessly defer making concrete plans.”
In recent years it became almost a consensus that startups should adopt the Lean Feedback Loop, based on evolution through experimentation and incremental improvements. However I think that this evolution requires a software architecture that supports extensibility and adaptability. In the absence of such an Adaptable Design, the only alternative are endless mutations in the form of changes and patches. To conclude with a final quote by Thiel:
“Darwinism may be a fine theory in other contexts, but in startups, intelligent design works best.”
What do you think? Please share your experience in the comments below.
Effective Software Design 
Excellent post! In my experience (I haven’t tried your Adaptable Design so I can’t speak to that), both ways can be pretty painful.
I was on a project where the planning and up-front design was so rigid and trying to change anything was so bureaucratic that we ended up producing software no one wanted or liked and the project was shutdown after a year in the market.
On the other hand, I’ve been part of an “Agile” project that reached “Sprint 324” (don’t ask how they got to that number) at which point we were basically fixing and hacking in features that were painful to implement so far into the project, but could have been done easily had we sat down and designed the system in the beginning.
I think the biggest thing our industry lacks is a sense of pragmatism. Every methodology or process that comes along is now “the one true thing” and everything else is stupid and outdated. In Lean, it’s all about not knowing what the customer or market truly wants, so make the Minimum Viable Product and adapt to your customers.
I’m fine with that line of thinking from a Business standpoint, but I think it’s incompatible with the practice of Software Development. If there can be huge, sweeping changes to a product to the point where it might not even solve the original problem the customer had, then neither Agile or Waterfall will really work very well. Agile still has to make some sort of assumptions on what you’re actually building.
What I’ve been seeing as a way to shoehorn Lean into the Software Development world is the idea of “Disposable Software”. Basically, you build enough to solve the business problems right now, but throw the code away at some point in the near future if it’s too expensive to change to meet the new requirements.
I should point out that this isn’t a formal approach and that I don’t agree with it. Also, it’s usually born out of a developer inheriting some bad code (which reached the state of being bad because they had to hack changes and new features into it) and then wanting to do a rewrite so they can code it “properly”.
Anyways, I will stop ranting. I love the blog and look forward to more posts. I would love to hear your thoughts.
Thanks for your comments, Don. I believe that many Agile success stories are based on the fact that developers are using very powerful frameworks and platforms. Thus, in this case, they don’t need to invest in design, because they are adopting the design that was already created by someone else. I think that most problems appear when there is no such an existing design that may be adopted. I discuss that in this post:
Well said. Thanks.
See an example (out of many) at http://www.malotaux.nl/designlogcase2
In order to move to Zero Defects, we have to “design+review, code+review, upon which testing doesn’t find issues”. See http://malotaux.nl/doc.php?id=99 or the presentation video at http://tinyurl.com/nn5gf7c
I asked a firmware programmer to review his firmware. He sent me code. I said: “I can review your code, but what’s the point. As soon as we have reviewed the design, there will be so many changes that now looking at your code is a waste of time.”
He didn’t really know what I meant by ‘design’. Every time I sent him a design suggestion, a week later he said: “I did it differently”. I said “That’s ok, show me your design, so that I can understand what you did.” He sent me code. It’s not easy to teach design to people who never have learnt about it, and still think it’s just overhead.
Every time I see people stuck in their code (in some cases for months!), I drag them to the whiteboard asking to explain the design: “Show me a picture!” Because they don’t have a design it may take some time, but usually within an hour they see the bottleneck that was holding them back and within a week the problem is solved.
I hear so many programmers claim that software development is so complex. I think the complexity is in the first place caused by the programmers themselves rather that by the problem to solve.
Lack of design causes incomprehensibility, which they explain as ‘complex’.
Thanks for your comments and links, Niels. Your stories remind me of the concepts of Essential and Accidental Complexity as defined by Fred Brooks:
https://en.wikipedia.org/wiki/No_Silver_Bullet
I cannot contest Fred Brooks saying “There is no silver bullet.”
On the other hand, there are a lot of magic bullets that work wonders, but which are often ignored, or unknown by some.
Remember Cobb’s paradox: “We know why projects fail; we know how to prevent their failure—so why do they still fail?”
I agree with the core message that software design is necessary and that a modular architecture is a prerequisite for agile/lean software development. Nevertheless, in my experience this is not a problem of the methods but their implementation.
YAGNI just states that you should only develop what you really need but for this to be able to work you need a very adaptive architecture. With respect to last responsible moment principle – it does not mean that you should not do anything until this moment. The very opposite is true – you should try to work on different possible scenarios including design that fits best and then decide at the time you need to for the best possible approach.
Again, it all depends on the implementation of these methods but in general agile and/or lean software development methods put a strong focus on an adaptable architecture and software design although the approach to reach it might be looking slightly different than it did in the waterfall model.
Thanks for your comments, Roland. But in my experience it is not true that Agile projects “put a strong focus on an adaptable architecture”. I’ve seen a focus on Simplicity and Emergent Design. But very often this focus results in simplistic design decisions and Big Balls of Mud:
Loved your post!
The post is well written, but I can’t join the overwhelming cheers. Any philosophy – be it agile, lean or the flying spaghetti monster PoV – can be abused.
Being able to evolve your design is an approach that holds a lot more business sense, in my opinion.
Why should the company invest time on an area of the service that may not survive its first exposure to the market? Instead, invest in techniques that will allow us to evolve good design.
Design is not the goal, a sustained satisfaction of the customer is. I strongly argue that proper design is one of the ways to achieve this, while also contributing to maintaining a proud and professional workforce, but let’s not mix means with goals. Coding is a means-to-an-end, not the end.
I am a strong believer in great architecture, superb design, professional coding, high coverage and automatic testing. But, I’m also a strong believer in understanding what they serve. Agile, in my opinion, does that better than the way we used to do it just a few decades ago.
As for the trends, it’s nice – but so is this trend:
– https://www.google.com/trends/explore#q=Agile%2C%20Waterfall&cmpt=q&tz=Etc%2FGMT-2 (agile vs waterfall)
I mean no disrespect, and I do enjoy reading your posts, but I think this one’s title is a bit too sensational for its own good.
Thanks for your comments, Erez. I don’t really think we disagree, and I’m certainly not a proponent of Waterfall or BDUF. I think the key here is your sentence “invest in techniques that will allow us to evolve good design”. In my experience most Agile developers are not doing that: they are not using such techniques, and they don’t even have a “good design” to start with. The results are always increasing Technical Debt and endless Refactorings, instead of proper evolution.
Agree. Not BDUF, but DUF.
Very good article, thanks for that because I sometimes consider myself as a strange one when I fell that some Agile managers are too optimistic. And now I meet this article so it’s not my observation only 😉 Agile has its weaknesses (for example Kanban, as interestingly written here: http://kanbantool.com/kanban-library/kanban-results/when-kanban-fails ) so forgetting about them is quite frivolous to me.