Indefinite Optimism: the Problem with the Agile and Lean Mindsets

In the last decade we have seen a decrease of interest in Software Design and a sharp increase of interest in Agile methods. The first graph below compares the trends between Software Design and Agile. The second graph compares the trends between Design Patterns and Scrum. They are very similar. And in my personal opinion, they are very sad.

design vs agile

patterns vs scrum

I am a firm believer in the importance of Software Design. In particular, I strongly believe that Software Design is essential in the context of Agile and Lean software development. In a series of previous posts I defined my Adaptable Design Up Front approach that is focused exactly on supporting Agility: having a software architecture that embraces changes through adaptation.

Thus I am trying to understand this phenomenon that has caused so many software developers to lose their interest in Software Design and get so enthusiastic about Lean and Agile methods. I think one of the reasons is the current predominance of Indefinite Optimism.

Definite vs. Indefinite Optimism

zero to oneThe difference between Definite and Indefinite Optimism is described by Peter Thiel in his book “Zero To One, Notes on Startups, Or How to Build the Future“.

Definite Optimist: has concrete plans for the future and strongly believes that the future will be better than today.

“To a definite optimist, the future will be better than the present if he plans and works to make it better.”

Indefinite Optimist: has great expectations about the future but lacks any design or plan for how to make such a future a reality.

“To an indefinite optimist, the future will be better, but he doesn’t know how exactly, so he won’t make any specific plans.”

Indefinite Optimism, Agile and Lean

If we go back to the graphs above, in my opinion it is clear that they depict a transition in software development from definite to indefinite optimism. Agile and Lean methods are characterized by the lack of planning and design, by the emphasis on the process and by a tendency to postpone decisions.

For example, the popular YAGNI principle from eXtreme Programming instructs developers to avoid implementing things that they are not sure if they will need. In practice, programmers adopt the optimistic guess that they will be able to implement anything when it is really needed. Then, they not only avoid the implementation of new features, they also avoid preparing for these features. In other words, YAGNI causes people to avoid designing for extensibility.

In another example, the Last Responsible Moment principle from Lean Software Development instructs developers to postpone their decisions as much as possible. In practice, programmers adopt the optimistic guess that this Last Responsible Moment will always happen in the future. Then, they not only avoid making premature decisions, they also avoid thinking about the consequences of these decisions. In other words, the Last Responsible Moment causes people to treat decisions as if they were independent of each other, and not part of a cohesive design.

Finally, I am sincerely amazed about the absolute emphasis the Agile community places on the process, focusing on project management instead of software engineering. Apparently having a Sprint retrospective to discuss how to improve the process is considered by many Agile practitioners to be much more important than conducting proper code an design reviews. In the words of Thiel:

“Arguing over process has become a way to endlessly defer making concrete plans.”

In recent years it became almost a consensus that startups should adopt the Lean Feedback Loop, based on evolution through experimentation and incremental improvements. However I think that this evolution requires a software architecture that supports extensibility and adaptability. In the absence of such an Adaptable Design, the only alternative are endless mutations in the form of changes and patches. To conclude with a final quote by Thiel:

“Darwinism may be a fine theory in other contexts, but in startups, intelligent design works best.”

What do you think? Please share your experience in the comments below.

Posted in Adaptable Design, Agile, Lean Development, Software Architecture, Software Evolution | Tagged , , , , | 12 Comments

Productivity Hack: Using Effectively Your Small Fractions Of Time

ClocksI’m constantly looking for new ways to be more productive and avoid wasting my time. One of my biggest challenges is how to use some small fractions of free time, something like less than 10 minutes.

I think that, for most people today, the natural tendency would be to use these few minutes to check their social networks: go see if there is any interesting post on Facebook, Twitter or LinkedIn. But in general this would be a way to waste our time.

How can we measure if our time was used effectively or if it was wasted? I think that a good way is to ask ourselves if we will remember what we did.

Let’s say we spent a few minutes scanning some dozens of posts on Facebook. What will we remember after that? Did we really learn something new?

Sometimes we go over our stream so fast that we don’t even pay attention to who is posting what. Depending on who are our friends and how many friends do we have, we will just see a collection of random posts.

Thus I decided to look for alternative ways to use these small fractions of time. Currently I’m focusing on book summaries. For the most popular books, such summaries are available both as videos or slides.

In the case of slides, I enjoy the collection of book quotes by GetNugget on SlideShare:

In the case of videos, I enjoy the channel FightMediocrity on YouTube:

These are only two examples. I’m sure that it’s possible to find many other ways to make good use of our few minutes of free time. And certainly there are better ways to use our time than constantly visiting Facebook or Twitter.

What about you? How do you use your small fractions of free time? Please share with us in the comments below.

Posted in Efficacy | Tagged | Leave a comment

Talk: Adaptable Design Up Front (slides + video)

Last week I was invited to talk at the IASA eSummit on Architecture for Agile Software Development. My talk tries to answer the question: “How much Design Up Front should be done in an Agile project?” I present my approach of Adaptable Design Up Front (ADUF), describing its rationale, applications in practice and comparison to other approaches such as Emergent Design.

Here is the video of the entire talk (in English):

Here are the slides of my presentation:

I recommend you check additional talks at IASA Content Repository.

Please share your comments below.

Posted in Adaptable Design, Agile, Software Architecture, Software Evolution | Tagged , , , | Leave a comment

Invention Is Not Enough For Innovation

wheel

The cartoon above depicts a common problem in hi-tech companies: Someone invents a new technology, but is not able to promote its adoption. I think the solution is that instead of just providing the “wheels”, we need to build a prototype of the entire vehicle and then show how it can transport real stones…

The main problem with just inventing the wheels is that it may be very difficult to demonstrate its business value. Very often a new technology is only useful as part of a more complex system, and until this technology has not been integrated in such a system its potential contribution is only hypothetical.

In contrast, the picture below displays the wrong and right ways to build a Minimum Viable Product (MVP). In both cases we have a form of incremental development. But in the wrong way the initial stages are such that the business value cannot be assessed. In the right way the business value is evident from the beginning, even if the initial versions are quite different from the final product.

MVP

Therefore, Invention is not enough for Innovation. For an invention to be adopted we must be able to prove its value. For an additional discussion on this topic, please see also my article about the Minimum Viable Product and Incremental Software Development.

What do you think? What has been your experience with Invention and Innovation? Please share with us in the comments below.

Posted in Lean Development | Tagged | 8 Comments

The Challenges and Joy of Public Speaking

I love giving talks to diverse types of audiences. I consider Public Speaking to be at the same time an extremely challenging and gratifying activity, that very often requires you to get out of your comfort zone. My first experience giving talks was when I taught an introductory BASIC programming course in my high-school in Rio de Janeiro. At that time I was 16 and had to teach other students who were a few years younger than me. Several years later, as a M.Sc. student at the Technion in Israel, I became a Teaching Assistant. It was very difficult for me, because I was still learning Hebrew at that time. Today, thanks to this blog, I am very frequently being invited to speak at diverse meet-ups and conferences.

Below: Giving a talk about finding Hi-tech jobs in Israel to a group of new immigrants at the University of Haifa.

Gvahim

The Challenges of Public Speaking

I think the most challenging aspect of Public Speaking is preparing my presentation. But this includes much more than simply writing the slides, adding some nice pictures and drawing charts and diagrams. I always do lots of rehearsals, planning what exactly I will say at each slide, and measuring precisely my time. I also always give a talk to myself facing a large mirror, at least once. I personally don’t make a special effort to prepare jokes in advance, but I always think about alternative ways to better engage my audience. Finally, I always imagine what kinds of questions people may ask, and think about good answers. My experience is that, when I am really well prepared, talking to the public becomes relatively easy, and I don’t feel any particular fear nor anxiety.

Below: Giving a talk about the Role of the Software Architect to a group of students at the Ruppin Academic Center.

Ruppin

The Joy of Public Speaking

I think the most gratifying aspect of Public Speaking is when you are able to really engage your audience. You can feel as you speak when people are paying attention and thinking about your message, when they start asking lots of interesting questions, when they nod their heads in agreement with your statements. But it is also great when someone in the audience asks a particular challenging question, which makes people in the audience turn their heads to see who is asking. This is very often an opportunity to provide a detailed answer including also additional aspects that were not mentioned in the original question, conquering the attention of the entire audience. Then you have transformed your public speaking into a Public Dialogue.

What is your experience with Public Speaking? Please share with us in the comments below.

Posted in Efficacy | Tagged | 3 Comments

The Problem with Velocity in Agile Software Development

direction_vs_speedIn my previous blog post about the “Real Danger of Quick-and-Dirty Programming“, I criticized some Agile practices such as measuring the Velocity and drawing Burndown charts. In this post I would like to extend and clarify this criticism, bringing also some very relevant references.

Here is the definition of Velocity from Wikipedia:

“Velocity is a capacity planning tool sometimes used in Agile software development. The velocity is calculated by counting the number of units of work completed in a certain interval, the length of which is determined at the start of the project. The main idea behind Velocity is to help teams estimate how much work they can complete in a given time period based on how quickly similar work was previously completed.”

In my opinion, the main problem with Velocity is when it becomes a goal by itself. When software development teams are focused on keeping their Velocity or increasing their Velocity, they will be ready to make sacrifices.

Having Velocity as a goal introduces trade-offs:

  • Should we make it faster or should we make it better?
  • Do we have enough time for Refactoring?
  • Why not accumulate some Technical Debt to increase our Velocity?

In contrast to the Agile term, here is the definition of the Physical concept of velocity, also from Wikipedia:

“Velocity is a physical vector quantity; both magnitude and direction are needed to define it. The scalar absolute value (magnitude) of velocity is called ‘speed’, a quantity that is measured in metres per second (m/s) in the SI (metric) system.”

Thus apparently what we call Velocity in Agile should actually be called Speed, because there is no clear definition of the dimensions of “Direction”. Of course any User Story that is implemented should satisfy some basic quality attributes such as Correctness. But what about other desirable quality attributes, both functional and non-functional, such as low coupling, high cohesion, efficiency and robustness? Who is measuring them? Who is tracking them? Unfortunately I’ve never heard of an Agile team drawing charts to track the average Cyclomatic Complexity of their code.

When the only thing you are really measuring is your Speed, you will attempt anything to “make progress” as fast as possible. Anything that causes you to move slower may be considered a burden. Including improving your code. Including Refactoring. Including avoiding Technical Debt.

To conclude this article, I bring below two opinions about Velocity from people that we should listen to.

Sacrificing Quality For Predictability by Joshua Kerievsky

Joshua Kerievsky is the CEO of Industrial Logic and author of the book “Refactoring to Patterns“. Below is an extract from his article “Stop Using Story Points“:

“Technical practices like test-driven development and refactoring are often the first things to be dropped when someone is trying to ‘make their estimate.’

We see this behavior emerge from the outset, starting in training: a team that completes all work at the end of a timebox (say, two hours) often does so by sacrificing quality.

For years, we addressed this common problem by helping students experience what it was like to deliver on commitments while also producing quality code.

We explained how great teams ultimately learned to maintain technical excellence while also sustaining a consistent velocity (the same number of story points completed per sprint).

Yet the fact is, such teams are rare and their consistency is often fleeting as their team size changes or other pressures take hold.

For too many years, I thought it was important to maintain a consistent velocity, since it was supposed to be helpful in planning and predicting when a product/system could be completed.

Yet I’ve seen too many teams sacrifice quality and face compounding technical debt solely to make or exceed their estimates and keep their burndown charts looking pretty.”

Velocity is Killing Agility by Jim Highsmith

Jim Highsmith is an executive consultant with ThoughtWorks and author of several books on Agile software development, including “Agile Software Development Ecosystems” and “Agile Project Management: Creating Innovative Products“. Here is an extract from his article “Velocity is Killing Agility!“:

“Over emphasis on velocity causes problems because of its wide used as a productivity measure. The proper use of velocity is as a calibration tool, a way to help do capacity-based planning, as Kent Beck describes in ‘Extreme Programming: Embrace Change‘. Productivity measures in general make little sense in knowledge work—but that’s fodder for another blog. Velocity is also a seductive measure because it’s easy to calculate. Even though story-points per iteration are calculated on the basis of releasable features, velocity at its core is about effort.

While Agile teams try to focus on delivering high value features, they get side-tracked by reporting on productivity. Time after time I hear about comments from managers or product owners, ‘Your velocity fell from 24 last iteration to 21 this time, what’s wrong? Let’s get it back up, or go even higher.’ In this scenario velocity has moved from a useful calibration tool (what is our capacity for the next iteration?) to a performance (productivity) measurement tool. This means that two things are short-changed: the quality of the customer experience (quantity of features over customer experience) and improving the delivery engine (technical quality).”

What do you think? Should we continue measuring our Velocity? Please share your opinions in the comments below.

Posted in Agile, Refactoring, Technical Debt | Tagged , , | 2 Comments

On the Real Danger of Quick-and-Dirty Programming

quote-if-10-years-from-now-when-you-are-doing-something-quick-and-dirty-you-suddenly-visualize-that-i-edsger-dijkstra-50997As in Dijkstra‘s quote above, when people criticize Quick-and-Dirty programming they are in general focusing on the negative impact in the system being developed. Software that is built following a Quick-and-Dirty approach will certainly have some serious deficiencies, which are also called Technical Debt.

However the Quick-and-Dirty approach has a different consequence that is as important as its violation of software design principles: It destroys the morale of the development team. Programmers should be proud of their work, but when they adopt Quick-and-Dirty solutions they are sacrificing their own professionalism.

In one of my first posts in this blog, I claimed that “Nothing is More Effective Than Enthusiasm“. The idea is that the best programmers are always enthusiastic about their work. But working Quick-and-Dirty is actually an enthusiasm-killer. Eventually the developers will become so disgusted with the system they are building that they will prefer to leave the project than trying to fix it.

Doing It Wrong

Besides the suffering caused by intentionally creating a bad product, programmers constantly working in Quick-and-Dirty mode will never have the opportunity to gain experience, improve their skills and actually become professional software developers. As in the quote by Frank Sonnenberg:

“Practice doesn’t make perfect if you’re doing it wrong.”

Worse yet, working Quick-and-Dirty may become a habit. In this case the poor and simplistic solutions may be chosen even when there is enough time to implement a proper design and avoid the creation of Technical Debt. Instead of being an emergency solution, Quick-and-Dirty may become the standard approach.

Unfortunately I think this phenomenon is very common in Agile projects: Programmers are motivated to adopt Quick-and-Dirty solutions to increase their own Velocity and display more progress in the Burn-Down charts. I discuss that in my article “The End of Agile: Death by Over-Simplification“.

In conclusion, if you are a software developer currently working on a project in which most of your work is Quick-and-Dirty, I seriously recommend you should consider finding another job, either in a different project in your company or in another company. Good luck!

What do you think? Do you agree? Please share your experience in the comments below.

Posted in Agile, Programming, Technical Debt | Tagged , , | 11 Comments

Watch-It-Next: A Contextual TV Recommendation System

yahoo_labsLast week my colleague Raz Nissim presented our joint work at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), which was held at Porto, Portugal. This was part of a Yahoo Labs project in which we investigated Recommender Systems for TV shows, under the leadership of Ronny Lempel and in collaboration with Michal Aharon, Eshcar Hillel and Amit Kagian.

Title: Watch-It-Next: A Contextual TV Recommendation System

Abstract:

As consumers of television are presented with a plethora of available programming, improving recommender systems in this domain is becoming increasingly important. Television sets, though, are often shared by multiple users whose tastes may greatly vary. Recommendation systems are challenged by this setting, since viewing data is typically collected and modeled per device, aggregating over its users and obscuring their individual tastes.

This paper tackles the challenge of TV recommendation, specifically aiming to provide recommendations for the next program to watch following the currently watched program the device. We present an empirical evaluation of several recommendation methods over large-scale, real-life TV viewership data. Our extensions of common state-of-the-art recommendation methods, exploiting the current watching context, demonstrate a significant improvement in recommendation quality.

You can download the article from the publisher’s site.

You can also watch here a video (in English) of a previous presentation of our work.

Here are the slides of Raz’s presentation:

Feel free to share your comments below. Thanks!

Posted in Data Mining, Recommender Systems, Research, Yahoo! | Tagged , , , | Leave a comment

On Wisdom and Happiness

I spend considerable time thinking about the meaning of Wisdom and Happiness:

  • What are the characteristics of a wise person?
  • What are the characteristics of a happy person?
  • Can wisdom and happiness be measured?

Of course I also think about the relationship between the two:

  • Are wise people happier?
  • Can we increase our happiness by increasing our wisdom?

In the Venn diagram below I try to capture one possible aspect of wisdom: The relationship between the “things we want” and the “things that are good for us”. The set W is the intersection of these two kinds of things. I believe that the number of items in W may be a measure of wisdom, meaning that a wise person is able to understand what is good for him/her.

Wisdom

Consequently, happiness would be the capacity to obtain these “things that we want and that are good for us”. In the Venn diagram below, the set H is the intersection between W and the “things we have”. I believe that the number of items in H may be a measure of happiness, meaning that a happy person has many of these things that he/she wants and that are good for him/her.

Happiness

Of course I’m not talking only about material possessions. I believe that the “things you have” should consist mostly of the personal goals you achieved and the dreams you realized.

So what is the the path to happiness? First you increase the number of things in W, by understanding which things are good for you and “wanting them”. Then you increase the number of things in H, by obtaining the things that you want and that are good for you.

In other words, before realizing your dreams, make sure you are dreaming the right dreams.

What do you think? Do you agree? Please share your comments below.

Posted in Efficacy | Tagged | 2 Comments

Resource Adaptive Software Systems

The International Association of Software Architects (IASA) in Israel organized a special event about Adaptive Software Systems. We invited Tom Mueck to give a talk about “Resource Adaptive Software Systems”.

Title: Resource Adaptive Software Systems

Abstract: DARPA issued an announcement about building Resource Adaptive Software Systems this past April. More specifically “to build survivable, long-lived complex software systems that are robust to changes in the resources provided by their operational environment”. The motivation behind this project has to do with the short shelf life of modern day software systems:

  • Orders of magnitude less than other engineering artifacts.
  • Strongly inversely correlated with the rate and magnitude of change.
  • When changes do occur they lead to high software maintenance costs and premature obsolescence of otherwise functionally sound systems.

All of these factors greatly impact an enterprise architecture and every architect wishes there would be better technology around.

This talk will introduce a new general computing abstraction named Resource-Oriented Computing (ROC) and discuss these areas of particular interest to architects:

  • Coupled, loosely coupled and decoupled systems
  • Resource-Oriented Architectures
  • Messaging
  • Risk surface / Security

Bio: Tom Mueck is the Director of Business Development at 1060 Research – which was spun out of Hewlett Packard Labs in Bristol. Tom has an MA in Asian Studies and an MBA from the University of Chicago. Previously he was a Political Risk and Financial Analyst, and Sales Manager at Hotsip AB (spun out of Ericsson).

These are the original slides of Tom’s presentation:

Here is the video of the talk (in English):

Feel free to share your comments below.

Posted in IASA Israel, Software Architecture | Tagged , | Leave a comment