On Anzeneering, Pride and the Definition of Done (DoD)

Posted on February 8, 2016 by Hayim Makabee

wedwardsdeming378714

The concept of Anzeneering was created by Joshua Kerievsky, CEO of Industrial Logic and author of the book “Refactoring to Patterns“. It is derived from the Japanese word “anzen” which means “safety”. According to Joshua:

“Anzeneers protect people by establishing anzen in everything from relationships to workspaces, codebases to processes, products to services.”

“Anzeneers protect: Software makers from poor working conditions, including hostile relationships, death marches, burnout, hazardous software (poorly designed, highly complex, deeply defective code, lacking even basic safety nets like automated builds or automated tests), insufficient testing infrastructure, poor lighting, uncomfortable seating, excessive work hours and insufficient exercise.”

I think one important aspect of Anzeneering is the protection of software developers against the production of low-quality code. Note that the emphasis here is not on the negative business consequences of having poor software systems. Anzeneering focuses on the negative impact on the developers themselves. When programmers are forced to produce bad code, they cannot be proud of their work, and will suffer because of that.

Frank Sonnenberg, award-winning author of the book “Managing With a Conscience“, addresses similar issues in his article “If You’re Not Proud, You’re Not Done“. According to Frank:

“If you don’t put your heart into your activities, if you hand in incomplete work as finished, if you don’t do your best every time you start something, then you’re doing yourself a tremendous disservice. The truth is, if you’re not proud of what you do, you’re not done.”

“If you don’t do your best, you’re only developing bad habits, damaging your reputation, and letting your team down.”

And this brings us to the concept of Definition of Done (DoD) in Agile software development.

Definition of Done (DoD)

In Agile software development, the Definition of Done (DoD) are the acceptance criteria that must be met for a new feature (user story) to be released. Here is the DoD definiton by the Agile Alliance:

“The team agrees on, and displays prominently somewhere in the team room, a list of criteria which must be met before a product increment ‘often a user story’ is considered ‘done’. Failure to meet these criteria at the end of a sprint normally implies that the work should not be counted toward that sprint’s velocity.”

A concrete example is provided by Mike Cohn, Agile consultant and author of the book “Succeeding with Agile: Software Development Using Scrum“, in his article “Multiple Levels of Done“:

“A typical DoD would be something similar to:

  • The code is well written. (That is, we’re happy with it and don’t feel like it immediately needs to be rewritten.)
  • The code is checked in. (Kind of an “of course” statement, but still worth calling out.)
  • The code was either pair programmed or peer reviewed.
  • The code comes with tests at all appropriate levels. (That is, unit, service and user interface.)
  • The feature the code implements has been documented in any end-user documentation such as manuals or help systems.”

In Mike’s example above it is not sufficient for the feature to be implemented, what is normally called “fully-functional”. The code may be complete, correct and still have lots of Technical Debt. Thus Mike says “the code is well written”.

Conclusion

Following the Anzeneerign approach, I would add to the DoD that the developer must be proud of his work. If a programmer is not proud of the code he wrote, he is not done, even if his code satisfies all the other requirements.

I want to end with a quote by Ed Yourdon, a pioneer in the field of Software Development who unfortunately left us last month at age 71. In his book “The Rise and Resurrection of the American Programmer“, Yourdon writes:

“If you think your management doesn’t know what it’s doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.”

Posted in Agile, Efficacy, Psychology of Programming | Tagged , , | 2 Comments

Beware the Lean Feedback Loop: Over-fitting to Early Adopters

Posted on January 24, 2016 by Hayim Makabee

the-lean-startup-bookToday many software startups follow the ideas introduced by Eric Ries in his best-selling book “The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses”. Ries is responsible for the popularization of the concept of the MVP: “The Minimum Viable Product is that version of a new product which allows a team to collect the maximum amount of validated learning with the least effort.” The MVP should have the core features which allow it to be deployed to some real customers in order to get initial feedback, but not more.

build-measure-learnThe MVP is used in the context of the “build-measure-learn” feedback loop. From the Lean Startup site: “The first step is figuring out the problem that needs to be solved and then developing a minimum viable product (MVP) to begin the process of learning as quickly as possible. Once the MVP is established, a startup can work on tuning the engine. This will involve measurement and learning and must include actionable metrics that can demonstrate cause and effect question.”

As Ries explains in one of his blog posts: “In other words, the Minimum Viable Product is a test of a specific set of hypotheses, with a goal of proving or disproving them as quickly as possible. One of the most important of these hypotheses is always: what will the customer care about? How will they define quality?”

This application of the build-measure-learn feedback loop is proposed by Eric Ries as the most effective way to learn about what the customers really want. There is a basic assumption that the first users of our product may provide us valuable data from which we will be able to extrapolate the needs of all other users. But there is another very important book that should make us question this assumption.

Crossing the Chasm

crossing the chasmGeoffrey A. Moore is the author of the best-selling book “Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers“, in which he describes the five main customer segments in a technology startup’s market: innovators, early adopters, early majority, late majority and laggards.

Innovators: “Pursue new technology products aggressively. They sometimes seek them out even before a formal marketing program has been launched.”

Early Adopters: “Buy into new product concepts very early in their life cycle… find it easy to imagine, understand, and appreciate the benefits of new technology.”

Early Majority: “Are driven by a strong sense of practicality… want to see well-established references before investing substantially.”

Late Majority: “Wait until something has become an established standard, and even so they want to see lots of support.”

Laggards: “Don’t want anything to do with new technology.”

These customer segments correspond to the stages in the Technology Adoption Life Cycle: the innovators are the first to buy a new product, followed by the early adopters, and so on. However, Moore claims that there is a chasm between the early adopters and the early majority, which is illustrated in the picture below.

chasm

According to Moore, there is a huge difference between the early adopters and the early majority:

Early adopters are buying some kind of change agent. “By being the firsts to implement this change in their industry, the early adopters expect to get a jump on the competition… They expect a radical discontinuity between the old ways and the new… They also are prepared to bear with the inevitably bugs and glitches that accompany any innovation just coming to market.”

In contrast, the early majority want a productivity improvement. “They want evolution, not revolution. They want technology to enhance, not overthrow, the established ways of doing business. And above all, they do not want to debug someone’s else product… They want it to work properly and to integrate appropriately with their existing technology base.”

Over-fitting to Early Adopters

In my opinion it is clear that if we adopt the build-measure-learn feedback loop while in the early adopters stage, we are incurring the risk of learning about user needs and preferences of a small segment which is very different from the main market, represented by the early and late majority segments.

Another way to say that is that we are using the build-measure-learn feedback loop in order to build a model to predict the behavior of our users. In this case we are incurring the risk of over-fitting this model to the particular user segment which is currently providing its feedback.

Definition of over-fitting: “A modelling error which occurs when a function is too closely fit to a limited set of data points. Over-fitting the model generally takes the form of making an overly complex model to explain idiosyncrasies in the data under study. In reality, the data being studied often has some degree of error or random noise within it. Thus attempting to make the model conform too closely to slightly inaccurate data can infect the model with substantial errors and reduce its predictive power.”

Note that the feedback provided by the early adopters has the two main problems that appear in the definition above:

  • It is a “limited set of data points”: the early adopters are a small and very specific segment, and not a random sample of our potential users.
  • It “has some degree of error or random noise within it” since the behavior of the early adopters is different from the majority of the users.

In Summary

We should be very careful when using the build-measure-learn feedback loop while in the early adopters stage. Possible ways to reduce the risks of over-fitting to this very specific user segment require understanding the differences between the early adopters and the majority of our potential users.

Posted in Agile, Lean Development | Tagged , | 3 Comments

Indefinite Optimism: the Problem with the Agile and Lean Mindsets

Posted on January 5, 2016 by Hayim Makabee

In the last decade we have seen a decrease of interest in Software Design and a sharp increase of interest in Agile methods. The first graph below compares the trends between Software Design and Agile. The second graph compares the trends between Design Patterns and Scrum. They are very similar. And in my personal opinion, they are very sad.

design vs agile

patterns vs scrum

I am a firm believer in the importance of Software Design. In particular, I strongly believe that Software Design is essential in the context of Agile and Lean software development. In a series of previous posts I defined my Adaptable Design Up Front approach that is focused exactly on supporting Agility: having a software architecture that embraces changes through adaptation.

Thus I am trying to understand this phenomenon that has caused so many software developers to lose their interest in Software Design and get so enthusiastic about Lean and Agile methods. I think one of the reasons is the current predominance of Indefinite Optimism.

Definite vs. Indefinite Optimism

zero to oneThe difference between Definite and Indefinite Optimism is described by Peter Thiel in his book “Zero To One, Notes on Startups, Or How to Build the Future“.

Definite Optimist: has concrete plans for the future and strongly believes that the future will be better than today.

“To a definite optimist, the future will be better than the present if he plans and works to make it better.”

Indefinite Optimist: has great expectations about the future but lacks any design or plan for how to make such a future a reality.

“To an indefinite optimist, the future will be better, but he doesn’t know how exactly, so he won’t make any specific plans.”

Indefinite Optimism, Agile and Lean

If we go back to the graphs above, in my opinion it is clear that they depict a transition in software development from definite to indefinite optimism. Agile and Lean methods are characterized by the lack of planning and design, by the emphasis on the process and by a tendency to postpone decisions.

For example, the popular YAGNI principle from eXtreme Programming instructs developers to avoid implementing things that they are not sure if they will need. In practice, programmers adopt the optimistic guess that they will be able to implement anything when it is really needed. Then, they not only avoid the implementation of new features, they also avoid preparing for these features. In other words, YAGNI causes people to avoid designing for extensibility.

In another example, the Last Responsible Moment principle from Lean Software Development instructs developers to postpone their decisions as much as possible. In practice, programmers adopt the optimistic guess that this Last Responsible Moment will always happen in the future. Then, they not only avoid making premature decisions, they also avoid thinking about the consequences of these decisions. In other words, the Last Responsible Moment causes people to treat decisions as if they were independent of each other, and not part of a cohesive design.

Finally, I am sincerely amazed about the absolute emphasis the Agile community places on the process, focusing on project management instead of software engineering. Apparently having a Sprint retrospective to discuss how to improve the process is considered by many Agile practitioners to be much more important than conducting proper code an design reviews. In the words of Thiel:

“Arguing over process has become a way to endlessly defer making concrete plans.”

In recent years it became almost a consensus that startups should adopt the Lean Feedback Loop, based on evolution through experimentation and incremental improvements. However I think that this evolution requires a software architecture that supports extensibility and adaptability. In the absence of such an Adaptable Design, the only alternative are endless mutations in the form of changes and patches. To conclude with a final quote by Thiel:

“Darwinism may be a fine theory in other contexts, but in startups, intelligent design works best.”

What do you think? Please share your experience in the comments below.

Posted in Adaptable Design, Agile, Lean Development, Software Architecture, Software Evolution | Tagged , , , , | 12 Comments

Productivity Hack: Using Effectively Your Small Fractions Of Time

Posted on December 31, 2015 by Hayim Makabee

ClocksI’m constantly looking for new ways to be more productive and avoid wasting my time. One of my biggest challenges is how to use some small fractions of free time, something like less than 10 minutes.

I think that, for most people today, the natural tendency would be to use these few minutes to check their social networks: go see if there is any interesting post on Facebook, Twitter or LinkedIn. But in general this would be a way to waste our time.

How can we measure if our time was used effectively or if it was wasted? I think that a good way is to ask ourselves if we will remember what we did.

Let’s say we spent a few minutes scanning some dozens of posts on Facebook. What will we remember after that? Did we really learn something new?

Sometimes we go over our stream so fast that we don’t even pay attention to who is posting what. Depending on who are our friends and how many friends do we have, we will just see a collection of random posts.

Thus I decided to look for alternative ways to use these small fractions of time. Currently I’m focusing on book summaries. For the most popular books, such summaries are available both as videos or slides.

In the case of slides, I enjoy the collection of book quotes by GetNugget on SlideShare:

In the case of videos, I enjoy the channel FightMediocrity on YouTube:

These are only two examples. I’m sure that it’s possible to find many other ways to make good use of our few minutes of free time. And certainly there are better ways to use our time than constantly visiting Facebook or Twitter.

What about you? How do you use your small fractions of free time? Please share with us in the comments below.

Posted in Efficacy | Tagged | Leave a comment

Talk: Adaptable Design Up Front (slides + video)

Posted on December 24, 2015 by Hayim Makabee

Last week I was invited to talk at the IASA eSummit on Architecture for Agile Software Development. My talk tries to answer the question: “How much Design Up Front should be done in an Agile project?” I present my approach of Adaptable Design Up Front (ADUF), describing its rationale, applications in practice and comparison to other approaches such as Emergent Design.

Here is the video of the entire talk (in English):

Here are the slides of my presentation:

I recommend you check additional talks at IASA Content Repository.

Please share your comments below.

Posted in Adaptable Design, Agile, Software Architecture, Software Evolution | Tagged , , , | Leave a comment

Invention Is Not Enough For Innovation

Posted on December 10, 2015 by Hayim Makabee

wheel

The cartoon above depicts a common problem in hi-tech companies: Someone invents a new technology, but is not able to promote its adoption. I think the solution is that instead of just providing the “wheels”, we need to build a prototype of the entire vehicle and then show how it can transport real stones…

The main problem with just inventing the wheels is that it may be very difficult to demonstrate its business value. Very often a new technology is only useful as part of a more complex system, and until this technology has not been integrated in such a system its potential contribution is only hypothetical.

In contrast, the picture below displays the wrong and right ways to build a Minimum Viable Product (MVP). In both cases we have a form of incremental development. But in the wrong way the initial stages are such that the business value cannot be assessed. In the right way the business value is evident from the beginning, even if the initial versions are quite different from the final product.

MVP

Therefore, Invention is not enough for Innovation. For an invention to be adopted we must be able to prove its value. For an additional discussion on this topic, please see also my article about the Minimum Viable Product and Incremental Software Development.

What do you think? What has been your experience with Invention and Innovation? Please share with us in the comments below.

Posted in Lean Development | Tagged | 8 Comments

The Challenges and Joy of Public Speaking

Posted on December 9, 2015 by Hayim Makabee

I love giving talks to diverse types of audiences. I consider Public Speaking to be at the same time an extremely challenging and gratifying activity, that very often requires you to get out of your comfort zone. My first experience giving talks was when I taught an introductory BASIC programming course in my high-school in Rio de Janeiro. At that time I was 16 and had to teach other students who were a few years younger than me. Several years later, as a M.Sc. student at the Technion in Israel, I became a Teaching Assistant. It was very difficult for me, because I was still learning Hebrew at that time. Today, thanks to this blog, I am very frequently being invited to speak at diverse meet-ups and conferences.

Below: Giving a talk about finding Hi-tech jobs in Israel to a group of new immigrants at the University of Haifa.

Gvahim

The Challenges of Public Speaking

I think the most challenging aspect of Public Speaking is preparing my presentation. But this includes much more than simply writing the slides, adding some nice pictures and drawing charts and diagrams. I always do lots of rehearsals, planning what exactly I will say at each slide, and measuring precisely my time. I also always give a talk to myself facing a large mirror, at least once. I personally don’t make a special effort to prepare jokes in advance, but I always think about alternative ways to better engage my audience. Finally, I always imagine what kinds of questions people may ask, and think about good answers. My experience is that, when I am really well prepared, talking to the public becomes relatively easy, and I don’t feel any particular fear nor anxiety.

Below: Giving a talk about the Role of the Software Architect to a group of students at the Ruppin Academic Center.

Ruppin

The Joy of Public Speaking

I think the most gratifying aspect of Public Speaking is when you are able to really engage your audience. You can feel as you speak when people are paying attention and thinking about your message, when they start asking lots of interesting questions, when they nod their heads in agreement with your statements. But it is also great when someone in the audience asks a particular challenging question, which makes people in the audience turn their heads to see who is asking. This is very often an opportunity to provide a detailed answer including also additional aspects that were not mentioned in the original question, conquering the attention of the entire audience. Then you have transformed your public speaking into a Public Dialogue.

What is your experience with Public Speaking? Please share with us in the comments below.

Posted in Efficacy | Tagged | 3 Comments

The Problem with Velocity in Agile Software Development

Posted on November 11, 2015 by Hayim Makabee

direction_vs_speedIn my previous blog post about the “Real Danger of Quick-and-Dirty Programming“, I criticized some Agile practices such as measuring the Velocity and drawing Burndown charts. In this post I would like to extend and clarify this criticism, bringing also some very relevant references.

Here is the definition of Velocity from Wikipedia:

“Velocity is a capacity planning tool sometimes used in Agile software development. The velocity is calculated by counting the number of units of work completed in a certain interval, the length of which is determined at the start of the project. The main idea behind Velocity is to help teams estimate how much work they can complete in a given time period based on how quickly similar work was previously completed.”

In my opinion, the main problem with Velocity is when it becomes a goal by itself. When software development teams are focused on keeping their Velocity or increasing their Velocity, they will be ready to make sacrifices.

Having Velocity as a goal introduces trade-offs:

  • Should we make it faster or should we make it better?
  • Do we have enough time for Refactoring?
  • Why not accumulate some Technical Debt to increase our Velocity?

In contrast to the Agile term, here is the definition of the Physical concept of velocity, also from Wikipedia:

“Velocity is a physical vector quantity; both magnitude and direction are needed to define it. The scalar absolute value (magnitude) of velocity is called ‘speed’, a quantity that is measured in metres per second (m/s) in the SI (metric) system.”

Thus apparently what we call Velocity in Agile should actually be called Speed, because there is no clear definition of the dimensions of “Direction”. Of course any User Story that is implemented should satisfy some basic quality attributes such as Correctness. But what about other desirable quality attributes, both functional and non-functional, such as low coupling, high cohesion, efficiency and robustness? Who is measuring them? Who is tracking them? Unfortunately I’ve never heard of an Agile team drawing charts to track the average Cyclomatic Complexity of their code.

When the only thing you are really measuring is your Speed, you will attempt anything to “make progress” as fast as possible. Anything that causes you to move slower may be considered a burden. Including improving your code. Including Refactoring. Including avoiding Technical Debt.

To conclude this article, I bring below two opinions about Velocity from people that we should listen to.

Sacrificing Quality For Predictability by Joshua Kerievsky

Joshua Kerievsky is the CEO of Industrial Logic and author of the book “Refactoring to Patterns“. Below is an extract from his article “Stop Using Story Points“:

“Technical practices like test-driven development and refactoring are often the first things to be dropped when someone is trying to ‘make their estimate.’

We see this behavior emerge from the outset, starting in training: a team that completes all work at the end of a timebox (say, two hours) often does so by sacrificing quality.

For years, we addressed this common problem by helping students experience what it was like to deliver on commitments while also producing quality code.

We explained how great teams ultimately learned to maintain technical excellence while also sustaining a consistent velocity (the same number of story points completed per sprint).

Yet the fact is, such teams are rare and their consistency is often fleeting as their team size changes or other pressures take hold.

For too many years, I thought it was important to maintain a consistent velocity, since it was supposed to be helpful in planning and predicting when a product/system could be completed.

Yet I’ve seen too many teams sacrifice quality and face compounding technical debt solely to make or exceed their estimates and keep their burndown charts looking pretty.”

Velocity is Killing Agility by Jim Highsmith

Jim Highsmith is an executive consultant with ThoughtWorks and author of several books on Agile software development, including “Agile Software Development Ecosystems” and “Agile Project Management: Creating Innovative Products“. Here is an extract from his article “Velocity is Killing Agility!“:

“Over emphasis on velocity causes problems because of its wide used as a productivity measure. The proper use of velocity is as a calibration tool, a way to help do capacity-based planning, as Kent Beck describes in ‘Extreme Programming: Embrace Change‘. Productivity measures in general make little sense in knowledge work—but that’s fodder for another blog. Velocity is also a seductive measure because it’s easy to calculate. Even though story-points per iteration are calculated on the basis of releasable features, velocity at its core is about effort.

While Agile teams try to focus on delivering high value features, they get side-tracked by reporting on productivity. Time after time I hear about comments from managers or product owners, ‘Your velocity fell from 24 last iteration to 21 this time, what’s wrong? Let’s get it back up, or go even higher.’ In this scenario velocity has moved from a useful calibration tool (what is our capacity for the next iteration?) to a performance (productivity) measurement tool. This means that two things are short-changed: the quality of the customer experience (quantity of features over customer experience) and improving the delivery engine (technical quality).”

What do you think? Should we continue measuring our Velocity? Please share your opinions in the comments below.

Posted in Agile, Refactoring, Technical Debt | Tagged , , | 2 Comments

On the Real Danger of Quick-and-Dirty Programming

Posted on October 22, 2015 by Hayim Makabee

quote-if-10-years-from-now-when-you-are-doing-something-quick-and-dirty-you-suddenly-visualize-that-i-edsger-dijkstra-50997As in Dijkstra‘s quote above, when people criticize Quick-and-Dirty programming they are in general focusing on the negative impact in the system being developed. Software that is built following a Quick-and-Dirty approach will certainly have some serious deficiencies, which are also called Technical Debt.

However the Quick-and-Dirty approach has a different consequence that is as important as its violation of software design principles: It destroys the morale of the development team. Programmers should be proud of their work, but when they adopt Quick-and-Dirty solutions they are sacrificing their own professionalism.

In one of my first posts in this blog, I claimed that “Nothing is More Effective Than Enthusiasm“. The idea is that the best programmers are always enthusiastic about their work. But working Quick-and-Dirty is actually an enthusiasm-killer. Eventually the developers will become so disgusted with the system they are building that they will prefer to leave the project than trying to fix it.

Doing It Wrong

Besides the suffering caused by intentionally creating a bad product, programmers constantly working in Quick-and-Dirty mode will never have the opportunity to gain experience, improve their skills and actually become professional software developers. As in the quote by Frank Sonnenberg:

“Practice doesn’t make perfect if you’re doing it wrong.”

Worse yet, working Quick-and-Dirty may become a habit. In this case the poor and simplistic solutions may be chosen even when there is enough time to implement a proper design and avoid the creation of Technical Debt. Instead of being an emergency solution, Quick-and-Dirty may become the standard approach.

Unfortunately I think this phenomenon is very common in Agile projects: Programmers are motivated to adopt Quick-and-Dirty solutions to increase their own Velocity and display more progress in the Burn-Down charts. I discuss that in my article “The End of Agile: Death by Over-Simplification“.

In conclusion, if you are a software developer currently working on a project in which most of your work is Quick-and-Dirty, I seriously recommend you should consider finding another job, either in a different project in your company or in another company. Good luck!

What do you think? Do you agree? Please share your experience in the comments below.

Posted in Agile, Programming, Technical Debt | Tagged , , | 11 Comments

Watch-It-Next: A Contextual TV Recommendation System

Posted on September 16, 2015 by Hayim Makabee

yahoo_labsLast week my colleague Raz Nissim presented our joint work at the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), which was held at Porto, Portugal. This was part of a Yahoo Labs project in which we investigated Recommender Systems for TV shows, under the leadership of Ronny Lempel and in collaboration with Michal Aharon, Eshcar Hillel and Amit Kagian.

Title: Watch-It-Next: A Contextual TV Recommendation System

Abstract:

As consumers of television are presented with a plethora of available programming, improving recommender systems in this domain is becoming increasingly important. Television sets, though, are often shared by multiple users whose tastes may greatly vary. Recommendation systems are challenged by this setting, since viewing data is typically collected and modeled per device, aggregating over its users and obscuring their individual tastes.

This paper tackles the challenge of TV recommendation, specifically aiming to provide recommendations for the next program to watch following the currently watched program the device. We present an empirical evaluation of several recommendation methods over large-scale, real-life TV viewership data. Our extensions of common state-of-the-art recommendation methods, exploiting the current watching context, demonstrate a significant improvement in recommendation quality.

You can download the article from the publisher’s site.

You can also watch here a video (in English) of a previous presentation of our work.

Here are the slides of Raz’s presentation:

Feel free to share your comments below. Thanks!

Posted in Data Mining, Recommender Systems, Research, Yahoo! | Tagged , , , | Leave a comment