There is something basically wrong with the current adoption of Agile methods. The term Agile was abused, becoming the biggest ever hype in the history of software development, and generating a multi-million dollar industry of self-proclaimed Agile consultants and experts selling dubious certifications. People forgot the original Agile values and principles, and instead follow dogmatic processes with rigid rules and rituals.
But the biggest sin of Agile consultants was to over-simplify the software development process and underestimate the real complexity of building software systems. Developers were convinced that the software design may naturally emerge from the implementation of simple user stories, that it will always be possible to pay the technical debt in the future, that constant refactoring is an effective way to produce high-quality code and that agility can be assured by following strictly an Agile process. We will discuss each one of these myths below.
Myth 1: Good design will emerge from the implementation of user stories
Agile development teams follow an incremental development approach, in which a small set of user stories is implemented in each iteration. The basic assumption is that a coherent system design will naturally emerge from these independent stories, requiring at most some refactoring to sort out the commonalities.
However, in practice the code does not have this tendency to self-organize. The laws governing the evolution of software systems are that of increasing entropy. When we add new functionality the system tends to become more complex. Thus, instead of hoping for the design to emerge, software evolution should be planned through a high-level architecture including extension mechanisms.
Myth 2: It will always be possible to pay the technical debt in the future
The metaphor of technical debt became a popular euphemism for bad code. The idea of incurring some debt appears much more reasonable than deliberately producing low-quality implementations. Developers are ready to accumulate technical debt because they believe they will be able to pay this debt in the future.
However, in practice it is not so easy to pay the technical debt. Bad code normally comes together with poor interfaces and inappropriate separation of concerns. The consequence is that other modules are built on top of the original technical debt, creating dependencies on the simplistic design decisions that should be temporary. When eventually someone decides to pay the technical debt, it is already too late: the fix became too expensive.
Myth 3: Constant refactoring is an effective way to produce code
Refactoring became a very popular activity in software development; after all it is always focused on improving the code. Techniques such as Test-Driven Development (TDD) allow refactoring to be performed at low risk, since the unit tests automatically indicate if some working logic has been broken by code changes.
However, in practice refactoring is consuming an exaggerated amount of the efforts invested in software development. Some developers simply do not plan for change, in the belief that it will always be easy to refactor the system. The consequence is that some teams implement new features very fast in the first iterations, but at some point their work halts and they start spending most of their efforts in endless refactorings.
Myth 4: Agility can be assured by following an Agile process
One of the main goals of Agility is to be able to cope with change. We know that nowadays we must adapt to a reality in which system requirements may be modified unexpectedly, and Agile consultants claim that we may achieve change-resilience by adhering strictly to their well-defined processes.
However, in practice the process alone is not able to provide change-resilience. A software development team will only be able to address changing system requirements if the system was designed to be flexible and adaptable. If the original design did not take in consideration the issues of maintainability and extensibility, the developers will not succeed in incorporating changes, not matter how Agile is the development process.
Agile is Dead, Now What?
If we take a look at the hype chart below, it is sure that regarding Agile we are after the “peak of inflated expectations” and getting closer to the “trough of disillusionment”.
Several recent articles have proclaimed the end of the Agile hype. Dave Thomas wrote that “Agile is Dead”, and was immediately followed by an “Angry Developer Version”. Tim Ottinger wrote “I Want Agile Back”, but Bob Marshall replied that “I Don’t Want Agile Back”. Finally, what was inevitable just happened: “The Anti-Agile Manifesto”.
Now the question is: what will guide Agile through the “slope of enlightenment”?
In my personal opinion, we will have to go back to the basics: To all the wonderful design fundamentals that were being discussed in the 90’s: the SOLID principles of OOD, design patterns, software reuse, component-based software development. Only when we are able to incorporate these basic principles in our development process we will reach a true state of Agility, embracing change effectively.
Another question: what will be the next step in the evolution of software design?
In my opinion: Antifragility. But this is the subject for a future post…
What about you? Did you also experience the limitations of current Agile practices? Please share with us in the comments below.
Effective Software Design 
“People forgot the original Agile values and principles, and instead follow dogmatic processes with rigid rules and rituals.”
And that’s the meat of the problem. You can do a great implementation of the Agile principles using Scrum. You can also use Scrum to manage a project badly. I’ve seen a lot of people trying to apply some sort of transitive property to that and say that Agile ends up in poorly managed projects or poor quality code.
Agile is a tool, and it’s not one that’s always used well. It’s not the hammer’s fault if you hold the wrong end.
This right here is the true story. Agile doesn’t fail. People fail. People who refuse to abandon their preconceptions and dogmas. People who pick up a tablespoon and call it a pitchfork. I’ve never seen a *true* implementation of Scrum fail. I’ve never seen any sort of implementation of waterfall succeed. 30 years and counting.
A silly comment to suggest waterfall doesn’t work. Agile is to be better, more responsive to
Change, faster, etc. and if executed properly, it can be. But to suggest waterfall doesn’t work – or have never seen it work – is a statement on the organizations you’ve worked for, not the methodology. Millions of projects have successfully used waterfall. That isn’t what is
Being argued in any intelligent ‘agile vs waterfall’ discussion.
The funny thing about myth #4 is that the original Agile people were as anti-process as they come. They thought talent, common sense, and real software engineering would cause the design to evolve correctly. In practice, they were mostly right, but they followed a fairly well-defined process within each sprint (but they will never admit that).
I agree completely. One of the other fundamental issues with this approach is that it assumes that the vast majority of software engineers are talented individuals with common sense and are real software engineers. While I have been fortunate to work with many of these types of individual, the majority have been inexperienced in software engineering. They have been exposed or trained on legacy code that often promoted bad software design, been limited in exposure to maintenance style roles, or been raised in development shops with limited exposure to real world production problems. It also indicates that a user is thinking in terms of how the story should best be told. In many cases, this is also not the case. In many of the “Agile” shops I have seen, users are either focused on how they currently use a system or are Subject Matter Experts that know what needs to be done, but often not how the information is presented to the ultimate user. Again these are not always the case, but they are also not rare occurrences.
Very well written article which does indeed capture the challenges facing Agile at large scales. I think that you will find that in smaller projects some of these issues are much less pronounced. In my opinion it raises two fundamental questions:
1. If so many implementations are struggling, are these challenges inherent to Agile? To scrum?
2. Is there a contradiction between Agility and good software engineering?
I think that the answer to #1 is yes as far as Scrum is concerned. and the answer to #2 is no.
I was (with my team) using Agile in the days when I didn’t even knew that something named Scrum even existed. Our Product Owners were our clients, the length of the sprint was determined by the government demanding reports from our clients (Banks) on monthly basis. The lawmaker introduced a new set of regulations to be implemented on short notice. First version of solution was consisted of a simple Data Warehouse and 4 (four) stored procedures doing complex calculations and reporting. Three years later it was an elaborate DW with a huge set of reports and complex calculations, custom tailored for each client. Throughout that entire period, both the lawmaker and the clients changed the requests constantly, and always on a very short notice. Also, the clients had some discretion in interpretation of some of the regulations in the way that suited their internal procedures. This resulted in need to have a tailored designed for each individual client.
We were making estimates, gantt charts and time sheets for upper management, but we might just as well had given them NBA player records. Nobody looked at those, because we were in direct communication with clients, and upper management paid no attention as long as customers were happy.
After I had ventured on another path in my career, I was introduced to Scrum and realised what I had been doing (and coached my team into doing) all along.
Agile is not about sprints, product backlogs and planning poker. Those rituals are important, for you to have a measuring scale, but that’s not Agile. Agile is about solving your customers problems and providing them with software solutions (if you’re using Agile in software) that will enable them to create value.
The Anti Agile Manifesto summarises it all: Some people think they know what Agile is because they know how to draw burndown charts, and know how to calculate velocity but they are actually clueless.
“Agile is about solving your customers problems and providing them with software solutions (if you’re using Agile in software) that will enable them to create value.”
This statement is also true of “waterfall”, and any other form of software development you can think of. Agile officially just doesn’t document much of anything or involve much planning, whereas other types of development requires lots of planning documents upfront (Interface design, class design, etc).
That’s an interesting assessment of Agile. My experience has been quite different. In waterfall projects, the business always (yes, always) finds TONS of defects and “missed” requirements during UAT, despite the overwhelming plethora of finely-crafted documents. Our Agile projects, on the other hand, typically catch missed requirements and defects as they go, with frequent business feedback. Also, there is a TON of planning, from the ongoing refinement of the backlog to the daily development efforts and conversations with the business partners. Those teams that follow the values and principles of Scrum and Agile succeed quite nicely, sometimes brilliantly, while those who shortcut the process (avoid planning, cowboy it up, etc.) tend to struggle.
And, of course, teams that are staffed with inexperienced or incompetent engineers absolutely struggle. The difference being, on a waterfall project, the struggles can be swept under the rug and the consequences are not realized until UAT. Agile methodologies really do expose weaknesses in teams! (I’m sorry, “areas of opportunity”!)
That’s been my experience, anyhow. I had to laugh out loud (literally) when I read that Agile doesn’t “involve much planning”. Heh, we plan the crap outta stories!
I’m one of those Agile/Scrum Consultants at Scrum.org. And I have to admit that your article makes a good point. Agility was a counter-reaction to heavy plan-driven Software development processes ten, twenty years ago. As a matter of facts, the truth lays always somewhere in the middle and not at the extreme positions. Each Project has its own optimum somewhere in the middle. I’m working in the area of embedded real-time system and we do
– work with user stories, but with more text than just “As A … I Want … So That”. The Inventor of user stories never thought that they will be reduced to this template. There is more behind. See http://www.scrumcrazy.com.
– measure technical debt and care about it.
– refactoring, but create an upfront architecture and initial design respecting SOLID, design principles, common sense and known requirements for more than a Sprint.
– respect the fact that in our industry initial architecture is important to design custom made hardware and mechanics and there are fewer changes than when programming a Webpage.
I think that agility requires good develpers. If you have good developers in your Team, they anyway follow kind of an agile process, whether they know Scrum or not. If you are not blessed with smart developers, you will have a problem anyway, whether you are following Scrum or not. In my experience, the key is not the process, but the – good – people.
We are focussing at Scrum.org with our Professional Scrum Developer Trainings to improve the knowledge of developers.
Having good people on the team is the key.
I think companies use the terms associated with Agile and Scrum, but do not use the original philosophies. In a large corporate enviroment it’s very difficult to follow the principles of Agile/Scrum. There are too many individual teams that work independently and iteratively and the true nature of a cross functional team is missed, but they still call it agile. The handoffs (milestones) are passed from one team to another. Using the terms associated with agile does not mean that you have cross-functional team that is collaborative, self-managed or self-directed.
I am convinced the problem is not inherent in the Agile “concepts”, “process”, “ideas”, “manifesto” or whatever you want to call it. The problem is that these “myths” have been misused and abused as excuses by poor cowboy-developers to justify bad code practices.
The concept of technical debt was not supposed to be a reason to write poor code thinking it is OK since “somebody” will pay the debt later. In my opinion it was more a tool to arrive to a compromise between the development team and the sponsors so that a feature might be finished in a sprint provided that time would be allocated to fix the shortcuts taken as a new story with a high priority on the backlog.
The idea of refactoring was definitely not intended to be used as an excuse to write poor code by saying “it will be refactored later” it was more definitely an idea with two objectives:
1. As you are writing the unit tests and making the them pass afterwards the resulting code would most definitely not be the best or cleanest so it would be refactored RIGHT AWAY using the tests as aids so that nothing would break during the refactoring step.
2. Instead of writing complex code implementing it with all the patterns under the sun just because it “may be required later”. The idea was more in the lines of you DO NOT over engineer the solution but STILL write clean code. When the new requirement comes so that IT MAKES SENSE to implement those patterns and add complexity to the solution THEN you refactor the code accordingly.
I am still a strong believer in Agile but I do agree that it will die if we still allow for irresponsible, sloppy, lazy and careless developers to keep writing poor code thinking that someone in the next two hundred years will pay the technical debt by refactoring the spaghetti that they are creating.
” provided that time would be allocated to fix the shortcuts taken as a new story with a high priority on the backlog.”
and what color is the sky in your world?
In all seriousness, the Agile and Scrum approaches taken by many companies in the last decade have exactly that goal – intentionally taking those shortcuts to avoid dealing with that long term planning and technical debt thinking they are actually getting more done. Scrum, for them, is simply a method to get more (but sloppy) results from fewer developers and look good on their quarterly corporate reports.
Eventually it bites every one of them in the *you know what* though as that debt cascades and catches up until it can no longer be ignored, sometimes with disastrous results.
That does not seem to be an issue with the methodology itself, however, but with the intent and/or implementation.
“That does not seem to be an issue with the methodology itself, however, but with the intent and/or implementation.”
My point exactly!
The fact is those companies would be taking shortcuts no matter what. I’ve been in software engineering for decades, and long before agile software development companies were promising everything and the moon by a certain date, then pushing their employees to get it done no matter what by the date promised to clients or customers, regardless of what shortcuts or bad implementations were required. Part of the point of genuine agile software development is that the development team is supposed to push back, and not take on a piece of work too big to be completed. The fact is that if they are rushing to squeeze something in because the business or a customer is demanding it, then they’re doing it wrong.
Constant refactoring is an effective way to produce code. – Once upon on a time – refactoring was know by another name – “Rework” and it was always frowned upon
This is completely false and exactly the abuse and miss interpretation of the methodology.
If you think that refactoring is the same as rework, then you are doing it wrong. Again refactoring is necessary in the two situations I described above, improving the code IMMEDIATELY after the unit tests passed or modifying the code (which should already be clean, unit tested and simple to modify) so it can adapt to the new requirements.
This is all very well, but the agile movement came out of a generation of people who had seen all the “self-discipline” and “design proper interfaces” and “design for flexibility” that was advocated in the 80s and early 90s. And seen it FAIL.
They had already experienced the reality that they had designed what they thought the future required. And then the future came along and caught them out. Again and again.
Until you can demonstrate you have a crystal ball to give you accurate ideas of what tomorrow’s new requirements are going to be, then it’s pointless to say “agile’s emphasis on flexibly *reacting* to new requirements should be replaced by up-front *preparation* for the new requirements”. Because we’ve all seen exactly how THAT turned out.
Of course, like all good ideas, the more widely agile gets disseminated the more it gets misunderstood and abused. But that is NOT a symptom of the idea being bad. That’s just what happens when ideas get widely adopted.
A few expanded thoughts on the matter. Great stuff, by the way. Really made me think about the problem.
http://www.mattosbun.com/2014/03/agile-isnt-dead.html
In have written a response to your thoughts on my blog where I give my opinions to your myths. So far only part1 is written.
http://www.continuouslearning.se/2014/03/resurrection-of-agile.html
@continlearning
Pingback: Weekly Reading Material – 27/03/2014 | Can I change this later???
Pingback: Is scrum agile? | Neuron Blast
I agree and find your article very interesting. You bring up many problems we have encountered as a development team using scrum. I have sum up my experience with scrum in my blog.
http://www.neuronblast.com/2014/03/30/is-scrum-agile/
It is probably a good time to step back, take the best ingredients of Agile, the best of ingredients of legacy methods, stir them all together and bake a new … well… cake. But, enough of that analogy before we get into comments like “They want to have their cake and eat it too.” and “If they have no bread, let them eat cake”. 🙂
There are some fabulous aspects to Agile that I hope we carry with us as we climb the slope of enlightenment to the plateau of productivity (and quality as defined by our customers). These include continual client involvement; prioritization and re-prioritization by the business; extremely high visibility of the work to be done and the progress being made against that work; scope flexibility in cases where schedule and budget are critical; daily short team meetings for great communication, issue identification (but not solving) and esprit de corps; the end of analysis paralysis as we try to document all requirements to the nth degree up front; frequently produced product and/or deliverables; team self-management; and, the realization by some that the rusty old iron triangle of scope, time and cost should have another element called value – again, as defined by our customers).
What can we take from our traditional methods? Solid architectural design, fitting pieces into a well defined architectural puzzle as we move along; quality ASSURANCE as in “do it right the first time” in addition to quality CONTROL (remember Deming’s famous saying “You can’t test quality in”); solid management of our projects; and, a focus on what we need to (and are) delivering.
So while we deal with the fact that many of us may have had inflated expectations while we searched for the silver bullet to solve all of our software development woes . . . … well, there is no silver bullet – only intelligence, discipline and good practices.
Thank you for sharing this article as it gives me new perspective!
Whether agile ends or is dead depend heavily on us, doesn’t it? We are fully in charge. And better you do not have to rely on external consultants but you should have, in addition, internal experts which pair with the externals. Especially because they run different business cases … 😉
Agile is not covering all. In general, you can combine agile with everything, especially with SW Craftsmanship, SOLID, Clean Code, etc.
Nobody prevents you from doing architectural work up-front, or to do the right user stories in the order to build up the right architecture. When your up-front trip on your architecture went into the wrong direction, you may prepare the next functional user story with some architectural ones.
Refactoring has to be done while implementing the user story in a Sprint. If the story does not fit into the Sprint, it is simply too big. Right, it is not the most effective way to produce code. But is there an alternative? E.g. big up-front design and architectural work? Only if you have a calm market with less changes, I think. This I have never been faced with in my long history as a SW developer … But I have heared there are such places …
Maybe we have not all to go back to the basics. Only the ones who do not have always the principles and values in mind while applying Scrum, XP practices and so.
Taleb’s book about Anti-fragility is a nice one and it would be worthwhile to talk about. I can recommend everyboedy to read it. I like a lot of the ideas there. But I do not think it is the next big thing compared to agile. Why do you? Please let me know 🙂
Good article. Agile became too much hype. It needs to be viewed as a set of techniques within a more disciplined, systematic framework. Especially in the context of complex applications and systems.
Nothing replaces good, comprehensive requirements understanding and good holistic design and data modelling when building complex systems. And that isn’t done with an interactive design approach. Development yes, not design.
Agile is the reaction to constrainted development. Being agile does not mean just strictly follow a methodology, Agile is a way to be, a philosophy and Agile methodologies without agileness are the anti agile.
So in my opinion the first question to ask ourselves is: am i Agile?
2nd question. Are you high?
Quote: “To all the wonderful design fundamentals that were being discussed in the 90’s” – really 90’s, I learned this in the 80’s in the Motorola Software Developers Training courses! Cranking out very high quality, highly maintainable software is not that difficult, with good requirements analysis; believe it or not, SW actually follows a set of mathematics, very predictable. You can predict how good or bad the SW is or will be!
“SW actually follows a set of mathematics, very predictable”
Sure, until users get their hands on it.
Pingback: Re-Post: The End of Agile: Death by Over-Simplification | Youry's Blog
> instead of hoping for the design to emerge, software evolution should be planned through a high-level architecture including extension mechanisms.
Not my experience. You’re right that entropy increases unless you actively work against that. My answer is simple: actively work against that! But it’s very hard to know which pieces need to be fixed up front and which don’t; in my experience on such a system, the decisions that were fixed at design time were the wrong decisions, and crippled the resulting system’s ability to adapt to changing requirements.
> Bad code normally comes together with poor interfaces and inappropriate separation of concerns. The consequence is that other modules are built on top of the original technical debt, creating dependencies on the simplistic design decisions that should be temporary. When eventually someone decides to pay the technical debt, it is already too late: the fix became too expensive.
That’s part of the metaphor: debt accumulates interest, and if you don’t pay off at least the interest each month, the debt balloons.
> However, in practice refactoring is consuming an exaggerated amount of the efforts invested in software development. Some developers simply do not plan for change, in the belief that it will always be easy to refactor the system. The consequence is that some teams implement new features very fast in the first iterations, but at some point their work halts and they start spending most of their efforts in endless refactorings.
Citation needed. I have worked in a wide variety of organizations following a wide variety of practices. Organizations that did more design up-front needed more, not less refactoring later on – we had to spend extra time removing or refactoring “design for change” in parts of the code that never changed, or “architecture” that overcomplicated simple code.
> If the original design did not take in consideration the issues of maintainability and extensibility, the developers will not succeed in incorporating changes, not matter how Agile is the development process.
Simply not true. I have direct experience of doing this. As long as you have the source code, you can change a system. Even if it’s a horrible ball of mud, it’s entirely possible to turn a bad system into a better one, a little at a time. And Agile is the best way I have found to do this.
Totally agree ! Biggest problems I’ve encountered in software development were due to over-engineering, while trying to solve potential, non-existent problems.
In my development, the nature of re-factoring usually focuses on removing the useless, and streamlining logic.
To finish, a quote:
“A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.”
– Antoine de Saint-Exupry
This quote, I find, fits my view of perfect code design.
As for the methodology, I find Agile servers my approach best. However, I have found myself questioning Scrum whenever it’s overly rigid nature coupled with overly zealous scrum master results in counter productive behaviors. The focus is often lost and transferred from tangible results to pretty charts. However, this happens in any over-bureaucratized environment regardless of the overall methodology. I just find it happens less often in Agile. As for Scrum specifically, I find it can be counter productive, and many of it’s assumptions are very unrealistic.
Perfection is achieved not when there is nothing left to add, but when there is nothing left to take away. (The part “a designer knows he has achieved” is not in St-Ex’s text, which reads: Il semble que la perfection soit atteinte non quand il n’y a plus rien à ajouter, mais quand il n’y a plus rien à retrancher. p.59 in: Antoine de Saint-Exupéry, Terre des hommes, Le Livre de Poche, 1939, 243 p.) Cheers.
It’s amazing to me that nobody has mentioned retrospectives at all in this post or in this thread. If you want to fix agile or have a better process, the retrospective is where you start and how you get better.
It’s like A/B testing, you try a test, you gather the data, and you go with the winner. You then repeat that process.
People seem to be treating whatever they call agile as some stuck process that never changes or gets better. They then get fed up and throw away agile because it sucks and then reinvent a new thing that they will think is better only to decide they hate it because it is stuck in one spot and so on and so forth.
The best way to get better is not to decry some new set of principles every 6 months. It’s to work to continuously improve your process over time. Sometimes that is big changes, sometimes it is small. To make that work as a team requires a lot more communication and a bit less code.
The only team I’ve seen do agile really well is fanatical about retrospectives and improving the process. Every team that avoids retrospectives fails at agile.
Thanks for the post. Even though the original principles were simple and noble, it’s a relief to see the industry and dogma that built up around it, which was obvious nonsense to anyone not getting paid to propagate it, finally being put to death.
My experiences with Agile have been almost universally bad. At the time it was hurting me, I blogged about it: http://mentalagility.wordpress.com/
Pingback: The End of Agile: Death by Over-Simplification - weigandtLabs
Pingback: The end of Agile – death by over-simplification | systems perestroika – éminence grise
Pingback: Object oriented principles – SOLID | No fluff, just stuff...that matters.
Um, no – I disagree with the concept of this article (although, honestly, not with the concept of over-simplification. Agile is hard…). Let me tell you why…
Myth 1: Good design will emerge from the implementation of user stories
No, we actually do not say this. Good design will emerge from having a base design, then trying to develop working software. If I cannot deliver working software (that is, meeting a business need), my design was wrong…and I have to change it. I can change it as soon as I discover I screwed-up, or when we try to hand-over the project to the customer (when it is way too late to change). So, pay me now., or pay me later…
Myth 2: It will always be possible to pay the technical debt in the future
NO NO NO NO NO – I amazed constantly amazed at how many people (who consider themselves “good developers”) believe we can ignore technical debt until “later” – later when? If you are in the code, and see something that sucks I have a radical suggestion – FIX IT NOW. There is always (and needs to be) a tension between the business (“give me more features”) and technology (“let me design and develop this to be foolproof / better / great / gold plated”). This is a good tension, but only if we work as a team to solve business and technical problems.
Myth 3: Constant refactoring is an effective way to produce code
See above…Refactoring is a way of producing QUALITY code, not FUNCTIONAL code. Refactoring is important to me, because when I see something in the code that can be improved, fixed, made better, I own making it better. Jeez, are we really still discussing this?
Myth 4: Agility can be assured by following an Agile process
Well, frankly I agree with this. Now what do I say as a witty comeback?
Look, we have tried for a very long time (in my experience since the 1970’s – but I know people going back further than this) to have a unified process – “Just follow the process and it will be OK” – which quite simply does not work. Developing and delivering software is not factory work – put bolt 2 into hole 3 – it is creative work, and no single process can take into account the way people work together, the way organizations want to work, the needs of customers, the needs of the organization, yada yada yada. Agile is not a single process – it is the connection of living, thinking, complicated people.
So I intended this to be a 3-line response. Fail.
Agile is complex – but many “coaches” want to make it look easy, simple, and foolproof. It is not. But is it dead? Well, not if you care about your people, and focusing on a partnership between business and technology. Of course, if you want o follow a rigid process, follow PMBOK…it is as bad (and , honestly, as good) as any other way – especially if you want a “coding factory” 🙂
So one size does not fit all; a hammer can be used on many different nail sizes but only to a point. There are still many different hammers for different specialized purposes (from claw hammer to Ball-peen hammer and from rubber to steal). There is nothing at all wrong with Agile -as it was defined in the original manifesto and in the context of the original problem space. In fact, there was nothing wrong with the Waterfall model – given it’s original context. As George Box so aptly put it, “essentially, all models are wrong, but some are useful. Agile, Scrum, Test Driven Design, Technical Debt, and even Design Patterns are useful.
Scrum is a useful model for reducing friction about roles and responsibilities and provides a limited vocabulary to improve consistency. It works for many small teams. But, Scrum does not scale to adequately manage the integration of business rule changes across a portfolio of application. Scaled Agile Framework (SAFe) is a new model that attempts to address many of the short comings of Agile as a small team-oriented approach to development. But even SAFe does not address how architecture, process, and standards need to come together to build or integrate changes into a portfolio of applications. So here is my model for addressing this problem space. It’s the Franchise Model.
The Franchise Model: The idea is simple. Centralize those things that development teams want to use in common to improve integration, communication, and development efficiency. The franchise will own the overall architecture, the platform(s), and the tools used across teams. These are the artifacts that are best maintained by a central team. The franchisee brings a talented team with domain knowledge and leverages the assets provided by the Franchise to develop and maintain applications. Franchise teams use what the Franchise has to offer because it helps them to be more productive than doing it on their own. The two roles, franchisor and franchisee, operate to mutually leverage each other. Each brings unique value to the other.
Let me provide an example. The organization says it needs a Document and Records Management capability. This capability may be used in several different applications. The Franchise selected the products, stands up the infrastructure, defines the APIs that applications will use, defines the integration and system test environments, and provides the Platform, and the Handbook on how to consistently use the Platform, to the Franchisee. The Franchisee, as a Scrum team, gathers business requirements, builds the application, and deploys the application into the Franchisor managed integration environment and then to production.
There is a lot more to the Franchise Model but I trust you get the point. I have seen this work. It scales to large organizations with dozens of development teams from different contractors. It reduces cost, development time, and improves product quality faster than a portfolio of independent teams. It’s use of Scrum and System Integration leverages the strengths of both while minimizing the downside of each.
If you want to hear more about it, let me know. I’d be happy to share our experience.
This sounds really interesting. Is there any other documentation on this approach?
This is one of the most valuable and insightful articles on Agile I have read in a long time. Over-simplification has become a real problem in the Agile community today. Agile has become a hype.
I agree wholeheartedly with this perspective: “In my personal opinion, we will have to go back to the basics: To all the wonderful design fundamentals that were being discussed in the 90’s: the SOLID principles of OOD, design patterns, software reuse, component-based software development. Only when we are able to incorporate these basic principles in our development process we will reach a true state of Agility, embracing change effectively.”
Sensible reply. I’ve noticed that the posts favoring this piece tend to be simple, based in experience, and easy to relate to. Those opposed long and mired in dogmatic explainations.. Occam’s Razor seems to apply.
The signs showed up early. At the peak of the hype, you could see clues of what Hayim describes.
I wrote this http://queue.acm.org/detail.cfm?id=1281893 on the flight back from the agile conference, on the summer 2006, the year we founded Agile Vancouver and started our own series of little local events.
The original agile community was acutely aware of the hype and some slippage, and discussed it, briefly: http://philippe.kruchten.com/2011/02/13/the-elephants-in-the-agile-room/
The context in which software is developped is very variable. We are looking too much for universally applicable solutions. Simple recipes. Retrospectives are not enough to get us out of the rut. Agile is not a solution (or any of the named methods). Even the agile manifesto shows its age. We must constantly adapt to new circumstances and context.
Pingback: The End of Agile? Or The Next Step in the Evolution of Work and Team Mgmt?
Pingback: DevOps Is The New Agile | Secure Software Development
Pingback: Is ‘Agile’ forgetting to value individuals and interactions over processes and tools? | MATT BURRELL
After 6 years of working on “agile/SCRUM” teams for large software development projects, all I can say is AMEN and AMEN! Agree 1,000%. Please, please, please flush agile/SCRUM down the toilet of over-hyped, over-simplified and ineffective software development approaches. Oh, and fire every “agile coaches” and “certified scrum master” on the planet.
Yes. I agree Agile proponents have come up with terms like technical debt and refactoring and present it as if it’s something new. It’s calling something else a different name. Technical Debt is cost/expense. Refactoring is rewriting. Something like that. You get the idea.
I almost think some of this politicall left wing propaganda. OH we have to “progress” and come up with some kind of change that’s unproven and then by edict declare it’s better than the old way. How many agile projects out there can prove beyoond a doubt that they saved 10%, 20%, 30%…by having used Agile and not some antequated spiral (which is incremental) development process.
And software has always been done in interative approaches. Agile mandates it in a time box which imho makes it a rigid process by definition. I have worked my last 2 jobs that use Agile and SCRUM. They think they are doing Agile and SCRUM when we are just playing Agile and SCRUM. You dont work down bugs in your clearquest database and get more out of that by having SCRUM becuase most team members task are unrelated to all the others. WHen I go to SCRUM, I can’t think of one time where I benefited from hearing what someone else is working on. That kind of stuff just doesnt matter until you go in software integration mode which usually means you will be in the lab on real hardware and what also usually means the mgmt will take any daily meetings you might have been having and stop them so you can be in the lab MORE and get MORE done. Good developers know what to do WE dont need daily meetings which is nothing more than micromanagement so the managers get a daily log of what we worked on. Good managers dont want that either. They want their engineers happy and they if they have good ones just getting the F*** out of the way and letting them “git ir done” works the best way. SCRUM makes good software engineers UNHAPPY. I know a slew of good software engineers and we all say (know) SCRUM is pure bs and it’s micromanagement and we are not happy about it.
YOu would think that for the last 30 years or so because Agile hasnt been used every Software project was a failure, they lost money and their process was all WRONG. Gimme a Break.
Sensible reply. I’ve noticed that the posts favoring this piece tend to be simple, based in experience, and easy to relate to. Those opposed long and mired in dogmatic explainations.. Occam’s Razor seems to apply.
Pingback: Is ‘Agile’ forgetting to value individuals and interactions over processes and tools? - MATT BURRELL
Pingback: Are You a Pragmatic Programmer? | New Repo
You know, some people mistakenly think of the terms ‘Scrum’ and ‘Agile’ as synonymous. You can ‘scrum’ any methodology, and for me, this in itself lends to more effective, continuous and open communication (“Hey! Meet up once a day for 15 minutes!” – revelation?).
I felt many pangs of familiarity with this article; having ‘Agile’ imposed on my team by a large parent organization can prevent an enthusiastic uptake, but uptake we did for 4 solid years – and this was true get-down-on-your-knees-and-pray-to-the-god-of-flexibility stuff; meticulous planning, meticulous post mortem analysis and meticulous (or is that ridiculous) blocking of anything that may disrupt the sacred sprint. I am the Scrum Master – hear me roar!
Burn-downs were useful to pinpoint how off-track things were going, but it got to the point where a failed sprint was just normal and down to ‘pollution by chaos’ and/or ‘sure, it wasn’t planned but we had to estimate this hot potato and move XYZ out to make up for it’ (much to the chagrin of the Product Owner, who was rarely able to attend planning as he also had a full-time job).
Regardless, we gave it our all. We used JIRA (quite a fan of this) to overarch our incredible new-world process and gradually began to accept that, as a software company that deals with both R&D and the ever-popular maintenance tasks, the gospel proclaimed by the Church of Agile wasn’t the ideal solution.
What did we do? We didn’t abandon it all – we still plan, we still review and we still drag the Product Owners kicking and screaming to the demonstration sessions, but in the end we’ve reverted to more of a KANBAN system. There’s no real term for it; there’s a prioritized backlog, but we invoke a never-ending ‘Treadmill’ ‘sprint’ that displays the activity in the room. Reports are now fixated on case longevity, and each morning we review the sticky issues to see if anyone can help move them through to ‘done-done-done’.
Paradoxically, we’re more agile than ever.
Yes, I agree. SOme/Most think Agile and SCRUM are one and the same. Agile is the process (2 to 4 week time boxed sprints, etc, and SCRUM, back log grooming). SCRUM is the 15 minute stand up on a daily basis.
I have worked 2 jobs that use Agile. THey didnt know what they were doing. I am using is now. Most of the work I do is maintenance work on a huge system. MOst problems/bug reports are in Clearquest and they are unrelated to other team members bug that they are working. Hence, we dont need to do SCRUM.. It’s a waste of time.
WE also use JIRA. It’s ok. But this place also has clearquest. Funny thing is all the information in clearquest, a bug report database if you will, has to be copied/entered intoa a JIRA making it a duplication of effort. WHen we plan our 2 week sprint, every story/task in JIRA has been copies out of clearquest. WHen we complete the task, we log this in JIRA and we also log it in clearquest.
Frankly, I think this place is using Agile/SCRUM and JIRA so they get a daily track of what engineers are doing. It’s a good piece of documention for them when the military (the customer) is griping toward the end of a program wondering why things are a ilttle behind if they are at all. The mgmt can whip out the JIRA data and tell and show the customer just how much time is spent on the effort and what exactly we wroked on , on a minute by minute basis.
Agile is ok is that’s what you think you want. SCRUM is not needed when your team is working on software matters that are not related to your piece of software. It’s more of a distraction and an impedent to progress than something that makes things go faster and better.
On defense programs, the military customer drives what your process is and what has to be tracked/traced. They need to either get rid of clearquest and go all out Agile or get rid of Agile. I honestly think they will realize that Agile does not give them much benefit when it comes to cost savings, delivering on time and quality. Hence, Agile will get pushed aside.
Pingback: Are You a Pragmatic Programmer? | Coding Insider
Agile cannot fail; it can only be failed. (just like socialism and conservatism)
Once, a long time ago in a galax…(no not that one). Around 20 years ago I had to manage a team to support a lottery with weekly TV shows. A very creative customer, but nearly always forgot to inform us (the IT droids) when some changes where made in the show that will effect the software used to support CRM, drawing, collections, and paying the prices.
We developed some principles: every developer or operator had to understand end to end what we are supposed to support, there is a short and direct interaction with the end user, you work on quality (first time right) by fast and direct feedback, you understand the impact on the user as well as operations, you are not afraid to enter the end user domain (go to the call center, and the die hard’s even picked up the phone to experience their own “solution”).
Being agile became the way to survive, we had high stress levels, although no one broke. Some people left defining us being idiots. But as long as you were able to distinct between urgent and important you would make the right choices and was appreciated.
I dragged this with me in my career: a newspaper, an internet provider and since then several telco’s.. But there is a difference between being agile and calling agile. It is the mindset as well as the behavior.
Pingback: Do SOLID design principles make code slow? | Effective Software Design
Pingback: Are You a Pragmatic Programmer? | New Coding Arts
Pingback: Sales-Driven Development |
Wow, sorry cant give yu a long interesting of complicated answer, i just know if I caught any of my coaches or scrum masters advocating the behavior outlined they would be looking for a new job.
Seems you have had some ugly experiences, sadly poorly run projects exist in all models. I have seen some nasty Agile and RUP and Waterfall projects. I hope you see a well run agile project sooner rather than later.
This is from the user point of view. Agile/scrum is an ILLUSION. Software released today is filled with bugs and deficiencies due to the use of agile/scrum framework. Even the gamers complain the video game software today is more buggy than in the 1990s and early 2000s.
Agile/scrum creates users who become BEGGARS. The new system released which replaced the old system was delivered in piecemeal with bugs so the users are left with massive workarounds due to missing features and software bugs. Therefore, the users end up as BEGGARS begging the agile team to include features via their user stories. To the agile team, the user stories are just a to-do list that guarantee their job security. To the users, the user stories are stories of their PAIN and SUFFERING due to the deployment of a deficient and defective system.
The true Agile manifesto of the development team (from the user’s point of view).
1. Delivery over quality. Delivery of partial working software over delivery of full working software with high quality and performance. (The team feels good about delivering every 4 weeks even though the software is deficient and defective. This is an illusion of team productivity. Sadly, even company CEOs and executives embrace this illusion of team productivity believing that their organization is productive.)
2. Development over planning. High sense of self-accomplishment by development team over sense of delivering overall customer value. (The team feels good about delivering something even if the delivery is of no value to the user but the illusion that something of value was delivered is what matters to the team’s self-esteem.)
3. Heroism and recognition of development team (by creating a desperate situation for users) over reliability of software. (Because of bugs, the software is not reliable and creates fires. The team feels like a hero (like a firefighter) after an emergency bug fix when a user calls frantically about software errors. Everytime a feature is finally added in the next release, the team is hailed a hero because the users no longer need to do workarounds. Again, an illusion that the team solved a problem, when in fact they created the problem.
4. Job security of development team (via backlog and piecemeal delivery of features) over optimized user processes (via complete availability of all features). (The team is guaranteed a job for life. As long as the team does not deliver all the features of the software, the huge backlog means continuous work and job security.)
AGILE RESULTS for users are a buggy and incomplete software.
Users are left with workarounds on bugs. Users waste a lot of time on workarounds due to missing features that are no longer in the new system but were in the old system. The next time you see the backlog list of user stories, those user stories are in fact, stories of the PAIN and SUFFERING of users wasting massive amount of their time on workarounds due to a buggy and incomplete system.
Agile software development is not the root cause of buggy or incomplete software. The existence of buggy and incomplete software long before its advent negates your assertion.
Features are never complete. Ever. Anyone who has spent any time in software development knows that. This has nothing to do with agile software development, which in fact has a far better chance of delivering what users actually want and use, as well as delivering it earlier, to a higher standard, rather than later.
Reliability is a key part of agile software development. You are most definitely not supposed to compromise reliability to finish quickly. In fact it is the older methods of software development management and planning that try to cram in as many features as possible by a deadline regardless of whether they’ve been done well.
Customer value is a core element of agile software development.
Pingback: On the Real Danger of Quick-and-Dirty Programming | Effective Software Design
Pingback: Is Agile dead? I don’t think so… part I | Now I am the Master
Even of the signers of the Agile manifesto said that Agile is dead (at least in its essence taken to current practice) See https://www.youtube.com/watch?v=a-BOSpxYJ9M
Pingback: Agile Software Design
What you said had been well hinted and clarified in the 7 books of Extreme Programming 16 years ago. However, today people would just read 16-page Scrum guide and get a certificate to become a master leading a team of developers. An inevitable fate in capitalism with incompetent management.
Pingback: [翻譯] 敏捷已死 | NDark MSN Live Space
Pingback: Agile reborn in new Avatars! | Agile Gnostic
I just don’t understand how it makes sense in many situations. I was on a project were the goal was to merge 2 legacy database driven systems with some similarities and differences as well as add new functionality. Well that takes some time studying the legacy systems at the database level by me a database guy because a traditional business analyst might not pick up on the same things as me. Doing user stories driven by the users does not necessarily give you the info you need in the order that you need it to create an intelligent database design. There is high likely hood you later run in to things that can no longer be addressed gracefully because so much code has now been written against a database design that was based on incomplete information.
User stories don’t come users.
By that I mean not directly, and in the scenario you describe they’d be coming from the business because the analysis of a database system you describe would lend itself perfectly to agile software development.
You’re not describing actual well-implemented agile software development. Certainly the development team does not start randomly implementing the software with no concern for how things will fit together and they definitely don’t intentionally allow technical debt to accumulate. All software will always develop some technical debt because we are not perfect. In agile software development they seek opportunities to eliminate it when they find it.
Agile software development does not eliminate or ignore good design principles, so there is nothing to go back to. Its all still there.
I find that when you have 200+ user stories spread across 20+ sprints, they somehow do not add up properly, but you may not realise until sprint 20 because the detail is not discussed at the start of the project (like non-Agile projects). Perhaps this is down to having an ineffective team involved in producing user stories – but the issue to me seems to be that business analysts are free to write user stories that clash with previously delivered ones and it is the technical team who are responsible for stopping this or pointing it out. It certainly does not make for good software development anyway.
Completely understand that this is probably a case of Agile done badly, but it seems too easy to do it badly at least for less traditional (ironically) software projects.
In my case it is more of a case of using existing software than developing software. Often it really is a case of “what color do you want for that paragraph of text” and I just cannot get over the overhead of having a sprint to plan activities like this, daily stand-ups and demos of the work. You could literally condense 5 sprints into a few days if you just wrote down the simple set of requirements on a napkin.
To some extent or in some cases “Welcoming change” is a very bad thing as it means decision makers can make bad decisions knowing that they change their mind. They are not pressured into being decisive even though their very existence is to be decisive and make one good decision instead of infinite bad decisions. If this were Civil Engineering, people would die based on bad decisions!
Remember: “Nothing concentrates the mind like the prospect of being hanged at dawn.”
Just a general feeling that when requirements are quite simple, Agile makes it more confusing!
Pingback: Weekly watercooler #97 – Four Kitchens
The PragDave link is broken, here’s the new link to the same article:
https://pragdave.me/blog/2014/03/04/time-to-kill-agile.html
The key issue with agile comes from having the wrong personnel. Agile requires GOOD developers and product folks who have a clear concept of what the product should look like an do, not set in stone mind you but at least a matrix of possibilities. Without very good developers the code becomes brittle and inflexible, thus causing the issue of technical debt. A well conceived foundation will minimize this until a significant deviance from the initial concept comes into the picture. At the end of the day you are trying to evolve a product that does something well defined. If you start building an elephant and decide you really needed a monkey, throw it out and start again besides both being animals the commonality is less than the differences and there are totally different design constraints. The overhead of the standard software life cycle is a wast of efforts in most cases as the products evolve, as others have stated scrum, agile, etc is instituting a framework, rules, and process around common sense. Agile is a misused term and often projects utilizing an “agile” methodology are anything but. I think the key to agile is good developers and eliminating a lot of the process that adds overhead and slows down the velocity of change. The principal issue is the overhead and rigid processes were added to protect products from incompetent engineers, with competent engineers it just works.
If you have competent engineers and good developers – however you define those terms – why is any process needed at all? They will always do the right thing most of the time and in fact they may actually work better in Waterfall, since they can run forward really fast without getting bogged down by mindless sprints boundaries and useless meetings (stand-up and retros and half-baked demos). The whole idea of process is that you ensure success even if the conditions are less than ideal.
Excellent post and equally good comments.
Few additional issues and questions:
1) If you start development knowing only 5-10% of requirements (sorry user stories) – how can you even be sure if you design is correct? And the chance that you design will survive the future stories are probably in range of 5-10% only. So why waste time and effort on doing something that has only 10% success rate?
2) Usually business want to know how much the software or enhancement cost BEFORE committing $$$. How will you come up that using agile?
3) Every project has a chance of getting cancelled due to budget and changing priorities. Why commit say 10 devs working – instead of just 1-2 BA, particularly if the business has not fully thought through the end stage? Seems counterproductive.
There are others objections – but would prefer to write a blog on this site.
Pingback: Agile Principles Aren’t About Process. They’re a Shift in Values and Mindset. – 7pace Blog
Discussing while Agile is doomed to fail is would lead to a very long discussion. There are simply too many reasons. However, not all is bad in Agile and it can be a good approach for some specific projects. We need to set things straight again and to move forward. You find many answers in the AB6 Framework, an advanced course for Analysts (directly downloadable: http://bit.ly/2S9aGN3)
I know this is old but I find myself getting more and more frustrated with “agile” and “scrum”. The endless cycle of meetings and ceremonies which now are growing with yet more garbage being added.. (behavior driven development and other buzz words) Since I do Consulting a lot, I constantly hear “the only time agile is to good is because they were doing it wrong “. The problem is it seems as everyone must be doing it wrong because I have yet to see a place where everyone is like “yeah I love agile, please let’s find ways to incorporate everywhere as we are doing it right!”.. Or here is a fun one… Let’s spend an hour to debate on agile terms or the riveting topic of why you should or should not use spikes.. Whatever I used to love about software development is long gone and now replaced with a variation of the misery that is agile everywhere. (Kind of like the proliferation of the awesome open offices around the world so we can collaborate so much better!!! …. gag)
Very well stated. Consistent with what I’ve witnessed as well. Agile morphed from a set of solid principles to exactly what is described here. Became an excuse for skipping a design phase as well as an excuse for bad coding practices. And over- hyped to an amazing extent.
Agile Software Development was a hype from day one. That time itself many people said that was a restatement of good practices already available. NASA’s spaceshuttle software built from 1977 to 1980 was completed in a series of 17 iterations. This was way back anyone talked about Agile ! Very often than not Agile enthusiasts blame the old “waterfall model” and call for incremental development! But was there a “waterfall model” ever especially after 1980 ? No ! What we had in the punch card days was a well defined design document which helped to minimize punched card errors! The aim was to reduce the time consumed for correcting silly punching mistakes. But all these ended as soon as interactive terminals were available for compiling and correcting programs !
In all projects I worked with Agile meant having to stand 15 minutes every day for Scrum Meetings! Scrum helped the project leader with zero knowledge of the system to conduct meetings! Because he said nothing as all input came form the participants. In a closed small project where members understood each other’s work, these meetings did help sometimes. But these were exceptions. In most Scrum meetings the participants were not following what others were doing! This was true when half the participants were not programmers at all ! In most meetings those who could talk well stole the show leaving no time for others to say anything! Real programmers considered these Scrum meetings a nuisance and some skipped the Monday’s meetings!
“Technical debt” became a euphemism for “errors” rather than unimplemented features. Catching an exception and making a correction for a batch program stoppage was a 15 minutes job in olden days. I have done it for mainframe and Java. But Agile postponed it as a “technical debt”! A wrong database definition could have been corrected before a single row was inserted in production. No! Postponed as “technical debt”!
In the modern software development there are no “errors”! No drawbacks either! Because all will be fixed as techinal debt in the next release!
Quite recently I came across some wonderful lectures on clean code. There what struck me was the emphasis on short self-documenting functions with no Java comments! Well, way back in 1984 I wrote some ADS/Online dialogs with IDMS database which did exactly that ! That time I was blamed for not writing huge documentation (no word, no powerpoint then!). My own defence then was that the program so small that anyone could read and understand what it was doing! Believe it or not it is still in production !
Pingback: There's Nothing Less Agile Than Scrum - Marcus Technical Services
Pingback: There’s Nothing Less Agile Than Scrum
Agile threw away all the lessons learned in the past and ignored many essential principles. It has been designed with an intent that is way too low, local, simplified: How to get a software feature for demand as fast as possible in production. Unfortunately, in corporate environments, the challenge of conceiving , from the diagnosis and analysis on, and development of larger software systems is way more complex than this starting point and many more aspects must be taken into account.
Check out this more serious free and very visual book/course:
The Analyst’s Red Book (TARB)
http://bit.ly/2Z5h77G
Click to access The-Analysts-Red-Book–Axel-Vanhooren.pdf
http://bit.ly/2Z5h77G
Pingback: There’s Nothing Less ‘Agile’ Than Scrum – Marcus Technical Services
Pingback: Xamarin.Forms: Half the Cost; Double the Speed – Marcus Technical Services
Pingback: This New Design is Raising Quite a “Flapp” – Marcus Technical Services
Pingback: Taking Control of Variable Lifecycle – Marcus Technical Services
Pingback: Agile Software Design - Software Consultants | Moove It Blog