The End of Agile: Death by Over-Simplification

hypeThere is something basically wrong with the current adoption of Agile methods. The term Agile was abused, becoming the biggest ever hype in the history of software development, and generating a multi-million dollar industry of self-proclaimed Agile consultants and experts selling dubious certifications. People forgot the original Agile values and principles, and instead follow dogmatic processes with rigid rules and rituals.

But the biggest sin of Agile consultants was to over-simplify the software development process and underestimate the real complexity of building software systems. Developers were convinced that the software design may naturally emerge from the implementation of simple user stories, that it will always be possible to pay the technical debt in the future, that constant refactoring is an effective way to produce high-quality code and that agility can be assured by following strictly an Agile process. We will discuss each one of these myths below.

Myth 1: Good design will emerge from the implementation of user stories

Agile development teams follow an incremental development approach, in which a small set of user stories is implemented in each iteration. The basic assumption is that a coherent system design will naturally emerge from these independent stories, requiring at most some refactoring to sort out the commonalities.

However, in practice the code does not have this tendency to self-organize. The laws governing the evolution of software systems are that of increasing entropy. When we add new functionality the system tends to become more complex. Thus, instead of hoping for the design to emerge, software evolution should be planned through a high-level architecture including extension mechanisms.

Myth 2: It will always be possible to pay the technical debt in the future

The metaphor of technical debt became a popular euphemism for bad code. The idea of incurring some debt appears much more reasonable than deliberately producing low-quality implementations. Developers are ready to accumulate technical debt because they believe they will be able to pay this debt in the future.

However, in practice it is not so easy to pay the technical debt. Bad code normally comes together with poor interfaces and inappropriate separation of concerns. The consequence is that other modules are built on top of the original technical debt, creating dependencies on the simplistic design decisions that should be temporary. When eventually someone decides to pay the technical debt, it is already too late: the fix became too expensive.

Myth 3: Constant refactoring is an effective way to produce code

Refactoring became a very popular activity in software development; after all it is always focused on improving the code. Techniques such as Test-Driven Development (TDD) allow refactoring to be performed at low risk, since the unit tests automatically indicate if some working logic has been broken by code changes.

However, in practice refactoring is consuming an exaggerated amount of the efforts invested in software development. Some developers simply do not plan for change, in the belief that it will always be easy to refactor the system. The consequence is that some teams implement new features very fast in the first iterations, but at some point their work halts and they start spending most of their efforts in endless refactorings.

Myth 4: Agility can be assured by following an Agile process

One of the main goals of Agility is to be able to cope with change. We know that nowadays we must adapt to a reality in which system requirements may be modified unexpectedly, and Agile consultants claim that we may achieve change-resilience by adhering strictly to their well-defined processes.

However, in practice the process alone is not able to provide change-resilience. A software development team will only be able to address changing system requirements if the system was designed to be flexible and adaptable. If the original design did not take in consideration the issues of maintainability and extensibility, the developers will not succeed in incorporating changes, not matter how Agile is the development process.

Agile is Dead, Now What?

If we take a look at the hype chart below, it is sure that regarding Agile we are after the “peak of inflated expectations” and getting closer to the “trough of disillusionment”.

gartner-hype-cycle

Several recent articles have proclaimed the end of the Agile hype. Dave Thomas wrote that “Agile is Dead”, and was immediately followed by an “Angry Developer Version”. Tim Ottinger wrote “I Want Agile Back”, but Bob Marshall replied that “I Don’t Want Agile Back”. Finally, what was inevitable just happened: “The Anti-Agile Manifesto”.

Now the question is: what will guide Agile through the “slope of enlightenment”?

In my personal opinion, we will have to go back to the basics: To all the wonderful design fundamentals that were being discussed in the 90’s: the SOLID principles of OOD, design patterns, software reuse, component-based software development. Only when we are able to incorporate these basic principles in our development process we will reach a true state of Agility, embracing change effectively.

Another question: what will be the next step in the evolution of software design?

In my opinion: Antifragility. But this is the subject for a future post

What about you? Did you also experience the limitations of current Agile practices? Please share with us in the comments below.

About Hayim Makabee

Veteran software developer, currently working as a Research Engineer for Yahoo! Labs. Co-founder of the International Association of Software Architects in Israel.
This entry was posted in Agile, Refactoring, Software Evolution and tagged , , . Bookmark the permalink.

22 Responses to The End of Agile: Death by Over-Simplification

  1. Matt Osbun says:

    “People forgot the original Agile values and principles, and instead follow dogmatic processes with rigid rules and rituals.”

    And that’s the meat of the problem. You can do a great implementation of the Agile principles using Scrum. You can also use Scrum to manage a project badly. I’ve seen a lot of people trying to apply some sort of transitive property to that and say that Agile ends up in poorly managed projects or poor quality code.

    Agile is a tool, and it’s not one that’s always used well. It’s not the hammer’s fault if you hold the wrong end.

  2. Scott Whitmire says:

    The funny thing about myth #4 is that the original Agile people were as anti-process as they come. They thought talent, common sense, and real software engineering would cause the design to evolve correctly. In practice, they were mostly right, but they followed a fairly well-defined process within each sprint (but they will never admit that).

    • Blake Hansard says:

      I agree completely. One of the other fundamental issues with this approach is that it assumes that the vast majority of software engineers are talented individuals with common sense and are real software engineers. While I have been fortunate to work with many of these types of individual, the majority have been inexperienced in software engineering. They have been exposed or trained on legacy code that often promoted bad software design, been limited in exposure to maintenance style roles, or been raised in development shops with limited exposure to real world production problems. It also indicates that a user is thinking in terms of how the story should best be told. In many cases, this is also not the case. In many of the “Agile” shops I have seen, users are either focused on how they currently use a system or are Subject Matter Experts that know what needs to be done, but often not how the information is presented to the ultimate user. Again these are not always the case, but they are also not rare occurrences.

  3. Rony Atoun says:

    Very well written article which does indeed capture the challenges facing Agile at large scales. I think that you will find that in smaller projects some of these issues are much less pronounced. In my opinion it raises two fundamental questions:
    1. If so many implementations are struggling, are these challenges inherent to Agile? To scrum?
    2. Is there a contradiction between Agility and good software engineering?
    I think that the answer to #1 is yes as far as Scrum is concerned. and the answer to #2 is no.

  4. Rade Radumilo says:

    I was (with my team) using Agile in the days when I didn’t even knew that something named Scrum even existed. Our Product Owners were our clients, the length of the sprint was determined by the government demanding reports from our clients (Banks) on monthly basis. The lawmaker introduced a new set of regulations to be implemented on short notice. First version of solution was consisted of a simple Data Warehouse and 4 (four) stored procedures doing complex calculations and reporting. Three years later it was an elaborate DW with a huge set of reports and complex calculations, custom tailored for each client. Throughout that entire period, both the lawmaker and the clients changed the requests constantly, and always on a very short notice. Also, the clients had some discretion in interpretation of some of the regulations in the way that suited their internal procedures. This resulted in need to have a tailored designed for each individual client.
    We were making estimates, gantt charts and time sheets for upper management, but we might just as well had given them NBA player records. Nobody looked at those, because we were in direct communication with clients, and upper management paid no attention as long as customers were happy.
    After I had ventured on another path in my career, I was introduced to Scrum and realised what I had been doing (and coached my team into doing) all along.
    Agile is not about sprints, product backlogs and planning poker. Those rituals are important, for you to have a measuring scale, but that’s not Agile. Agile is about solving your customers problems and providing them with software solutions (if you’re using Agile in software) that will enable them to create value.
    The Anti Agile Manifesto summarises it all: Some people think they know what Agile is because they know how to draw burndown charts, and know how to calculate velocity but they are actually clueless.

  5. I’m one of those Agile/Scrum Consultants at Scrum.org. And I have to admit that your article makes a good point. Agility was a counter-reaction to heavy plan-driven Software development processes ten, twenty years ago. As a matter of facts, the truth lays always somewhere in the middle and not at the extreme positions. Each Project has its own optimum somewhere in the middle. I’m working in the area of embedded real-time system and we do
    - work with user stories, but with more text than just “As A … I Want … So That”. The Inventor of user stories never thought that they will be reduced to this template. There is more behind. See http://www.scrumcrazy.com.
    - measure technical debt and care about it.
    - refactoring, but create an upfront architecture and initial design respecting SOLID, design principles, common sense and known requirements for more than a Sprint.
    - respect the fact that in our industry initial architecture is important to design custom made hardware and mechanics and there are fewer changes than when programming a Webpage.
    I think that agility requires good develpers. If you have good developers in your Team, they anyway follow kind of an agile process, whether they know Scrum or not. If you are not blessed with smart developers, you will have a problem anyway, whether you are following Scrum or not. In my experience, the key is not the process, but the – good – people.
    We are focussing at Scrum.org with our Professional Scrum Developer Trainings to improve the knowledge of developers.

  6. Bob Crawford says:

    I think companies use the terms associated with Agile and Scrum, but do not use the original philosophies. In a large corporate enviroment it’s very difficult to follow the principles of Agile/Scrum. There are too many individual teams that work independently and iteratively and the true nature of a cross functional team is missed, but they still call it agile. The handoffs (milestones) are passed from one team to another. Using the terms associated with agile does not mean that you have cross-functional team that is collaborative, self-managed or self-directed.

  7. I am convinced the problem is not inherent in the Agile “concepts”, “process”, “ideas”, “manifesto” or whatever you want to call it. The problem is that these “myths” have been misused and abused as excuses by poor cowboy-developers to justify bad code practices.

    The concept of technical debt was not supposed to be a reason to write poor code thinking it is OK since “somebody” will pay the debt later. In my opinion it was more a tool to arrive to a compromise between the development team and the sponsors so that a feature might be finished in a sprint provided that time would be allocated to fix the shortcuts taken as a new story with a high priority on the backlog.

    The idea of refactoring was definitely not intended to be used as an excuse to write poor code by saying “it will be refactored later” it was more definitely an idea with two objectives:
    1. As you are writing the unit tests and making the them pass afterwards the resulting code would most definitely not be the best or cleanest so it would be refactored RIGHT AWAY using the tests as aids so that nothing would break during the refactoring step.
    2. Instead of writing complex code implementing it with all the patterns under the sun just because it “may be required later”. The idea was more in the lines of you DO NOT over engineer the solution but STILL write clean code. When the new requirement comes so that IT MAKES SENSE to implement those patterns and add complexity to the solution THEN you refactor the code accordingly.

    I am still a strong believer in Agile but I do agree that it will die if we still allow for irresponsible, sloppy, lazy and careless developers to keep writing poor code thinking that someone in the next two hundred years will pay the technical debt by refactoring the spaghetti that they are creating.

    • Mark Taylor says:

      ” provided that time would be allocated to fix the shortcuts taken as a new story with a high priority on the backlog.”

      and what color is the sky in your world?

      In all seriousness, the Agile and Scrum approaches taken by many companies in the last decade have exactly that goal – intentionally taking those shortcuts to avoid dealing with that long term planning and technical debt thinking they are actually getting more done. Scrum, for them, is simply a method to get more (but sloppy) results from fewer developers and look good on their quarterly corporate reports.

      Eventually it bites every one of them in the *you know what* though as that debt cascades and catches up until it can no longer be ignored, sometimes with disastrous results.

      That does not seem to be an issue with the methodology itself, however, but with the intent and/or implementation.

  8. pmatt says:

    Constant refactoring is an effective way to produce code. – Once upon on a time – refactoring was know by another name – “Rework” and it was always frowned upon

    • sergeromero says:

      This is completely false and exactly the abuse and miss interpretation of the methodology.

      If you think that refactoring is the same as rework, then you are doing it wrong. Again refactoring is necessary in the two situations I described above, improving the code IMMEDIATELY after the unit tests passed or modifying the code (which should already be clean, unit tested and simple to modify) so it can adapt to the new requirements.

  9. interstar says:

    This is all very well, but the agile movement came out of a generation of people who had seen all the “self-discipline” and “design proper interfaces” and “design for flexibility” that was advocated in the 80s and early 90s. And seen it FAIL.

    They had already experienced the reality that they had designed what they thought the future required. And then the future came along and caught them out. Again and again.

    Until you can demonstrate you have a crystal ball to give you accurate ideas of what tomorrow’s new requirements are going to be, then it’s pointless to say “agile’s emphasis on flexibly *reacting* to new requirements should be replaced by up-front *preparation* for the new requirements”. Because we’ve all seen exactly how THAT turned out.

    Of course, like all good ideas, the more widely agile gets disseminated the more it gets misunderstood and abused. But that is NOT a symptom of the idea being bad. That’s just what happens when ideas get widely adopted.

  10. Matt Osbun says:

    A few expanded thoughts on the matter. Great stuff, by the way. Really made me think about the problem.

    http://www.mattosbun.com/2014/03/agile-isnt-dead.html

  11. In have written a response to your thoughts on my blog where I give my opinions to your myths. So far only part1 is written.

    http://www.continuouslearning.se/2014/03/resurrection-of-agile.html

    @continlearning

  12. Pingback: Weekly Reading Material – 27/03/2014 | Can I change this later???

  13. Pingback: Is scrum agile? | Neuron Blast

  14. Kate says:

    I agree and find your article very interesting. You bring up many problems we have encountered as a development team using scrum. I have sum up my experience with scrum in my blog.

    http://www.neuronblast.com/2014/03/30/is-scrum-agile/

  15. Mike Frenette says:

    It is probably a good time to step back, take the best ingredients of Agile, the best of ingredients of legacy methods, stir them all together and bake a new … well… cake. But, enough of that analogy before we get into comments like “They want to have their cake and eat it too.” and “If they have no bread, let them eat cake”. :)

    There are some fabulous aspects to Agile that I hope we carry with us as we climb the slope of enlightenment to the plateau of productivity (and quality as defined by our customers). These include continual client involvement; prioritization and re-prioritization by the business; extremely high visibility of the work to be done and the progress being made against that work; scope flexibility in cases where schedule and budget are critical; daily short team meetings for great communication, issue identification (but not solving) and esprit de corps; the end of analysis paralysis as we try to document all requirements to the nth degree up front; frequently produced product and/or deliverables; team self-management; and, the realization by some that the rusty old iron triangle of scope, time and cost should have another element called value – again, as defined by our customers).

    What can we take from our traditional methods? Solid architectural design, fitting pieces into a well defined architectural puzzle as we move along; quality ASSURANCE as in “do it right the first time” in addition to quality CONTROL (remember Deming’s famous saying “You can’t test quality in”); solid management of our projects; and, a focus on what we need to (and are) delivering.

    So while we deal with the fact that many of us may have had inflated expectations while we searched for the silver bullet to solve all of our software development woes . . . … well, there is no silver bullet – only intelligence, discipline and good practices.

  16. PlatinumJC says:

    Thank you for sharing this article as it gives me new perspective!

  17. sj487069 says:

    Whether agile ends or is dead depend heavily on us, doesn’t it? We are fully in charge. And better you do not have to rely on external consultants but you should have, in addition, internal experts which pair with the externals. Especially because they run different business cases … ;-)

    Agile is not covering all. In general, you can combine agile with everything, especially with SW Craftsmanship, SOLID, Clean Code, etc.

    Nobody prevents you from doing architectural work up-front, or to do the right user stories in the order to build up the right architecture. When your up-front trip on your architecture went into the wrong direction, you may prepare the next functional user story with some architectural ones.

    Refactoring has to be done while implementing the user story in a Sprint. If the story does not fit into the Sprint, it is simply too big. Right, it is not the most effective way to produce code. But is there an alternative? E.g. big up-front design and architectural work? Only if you have a calm market with less changes, I think. This I have never been faced with in my long history as a SW developer … But I have heared there are such places …

    Maybe we have not all to go back to the basics. Only the ones who do not have always the principles and values in mind while applying Scrum, XP practices and so.

    Taleb’s book about Anti-fragility is a nice one and it would be worthwhile to talk about. I can recommend everyboedy to read it. I like a lot of the ideas there. But I do not think it is the next big thing compared to agile. Why do you? Please let me know :-)

  18. Brad says:

    Good article. Agile became too much hype. It needs to be viewed as a set of techniques within a more disciplined, systematic framework. Especially in the context of complex applications and systems.

    Nothing replaces good, comprehensive requirements understanding and good holistic design and data modelling when building complex systems. And that isn’t done with an interactive design approach. Development yes, not design.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s