The Wisdom of Agile Development

Posted on July 24, 2011 by Hayim Makabee

What is wisdom? This is a definition from the Ethics Resource Center:

“The ability to make good judgments based on what you have learned from your experience.”

According to this definition, a wise man is someone with vast knowledge and experience, a person who can teach others.

However, the Jewish sages had a different view of wisdom, as expressed in the ancient book “Ethics of the Fathers” written almost 2000 years ago:

“Who is wise? One who learns from every man.”

Thus, to become wise, a person needs to have contact with many others, to be exposed to their opinions, and to learn from their knowledge and experience.

I believe that Agile software development, among its many advantages, creates many such opportunities for sharing knowledge and learning from others. Let’s consider a few examples of situations in which this may occur:

Code Reviews

Agile methodologies give emphasis to the importance of Code Reviews. This is a great chance for each developer to receive some feedback on the quality of the code he is writing. My experience is that Code Reviews have the potential to significantly and quickly improve the programming skills of the members of the team. They create opportunities to discuss design alternatives and analyze the implications of the adopted algorithms and data structures. Of course there is no contradiction between Code Reviews and traditional development methodologies, but in practice Agile teams dedicate more time to this activity.

Pair Programming

Pair Programming is one of the most fundamental concepts of Extreme Programming (XP). The idea is that programmers should not work alone, but in pairs. Then, at the exact moment one programmer is typing the code the other programmer is reviewing it. It is like an immediate code review, with the advantage that the developer always has someone to discuss his decisions at the moment he is writing his code. Of course this is considerably more expensive than traditional development, but experience has proved that Pair Programming has a great positive impact on the quality of code.

Stand-Up Meetings

During the daily Stand-Up meeting in Scrum, people can share what they are doing. The meeting itself is not the appropriate place to discuss technical issues, but it strengthens the connections among the team members, removing communication barriers and creating new opportunities for the exchange of knowledge and experience. After Stand-Up meetings each developer understands what the others are doing, so that he knows from whom he should ask for help on a particular issue.

Retrospective Meetings

At the end of each Sprint, the team organizes a meeting to evaluate its own performance. This is a great opportunity to discuss what was done right as well as what were the mistakes, thus learning from the shared experience. This kind of feedback mechanism is essential in a self-management approach such as Scrum. Its main goal is to constantly improve the process, but another consequence is that it forms more effective team members.

In summary, Agile methods introduce many new opportunities to learn from others by giving emphasis to team work. According to the ancient Jewish sages, this is the key to real wisdom.

Posted in Agile, Jewish Sources | Tagged , | 1 Comment

Who is an Expert? – On Rules and Exceptions

Posted on July 18, 2011 by Hayim Makabee

It is relatively easy to find very young people who are brilliant programmers, who seem to know every detail about a particular language, platform or technology. Some of them will be able to implement a nice Android app in a few days, others can write cool Facebook games in a week.

These bright young programmers are frequently very quick learners as well. Give them a new programming language and they will grasp its syntax in a couple of hours. Present them a new API and they will very soon understand all the differences among similar methods and the roles of each parameter.

Now the question is: Do their knowledge and ability to learn transform them into expert software developers?

Whenever I meet these young programmers, I remember this quote:

“The young man knows the rules, but the old man knows the exceptions.” – Oliver Wendell Holmes

So my answer to the previous question is “No”. Knowing the rules of a programming language or platform can make you very effective when working in a particular environment. But to really be an expert you must know the exceptions. You must know when to violate the rules.

1st Example: Composition versus Inheritance

I once worked with a good Java programmer that followed the rule “favor composition over inheritance”. The problem is he was fanatic about it: In all situations he used only composition. In his programs you could not find a single class extending another class. If he wanted to reuse a class, he would just encapsulate it in a new class and forward most method calls. A rule with no exceptions allowed…

2nd Example: Const-Correctness

I once participated in the development of a very large C++ system. I noticed that almost none was using const methods, and as a consequence they could not pass const parameters, even if an object was not supposed to be changed by a function.  I was surprised; it seemed that most people were not aware of the importance of const-correctness.

But then I discovered another reason for this mystery: Most programmers didn’t know about the possibility of declaring a data member as mutable. Then, many classes that used synchronization mechanisms had not const methods, even when the state of the object was not being changed. Even the accessor methods could not be declared as const, because they were changing some lock.

The rule says that const methods should not change the values of data members. But the real intention is that they should not change the object state. Hence the exception to the rule is that Semaphores and Mutexes can be changed by any method, because they are not part of the object state. They should be declared as mutable.

Conclusion: It is relatively easy to learn the rules, but only experience can teach you the exceptions. If you know the rules and the exceptions, then you are an expert.

I will be happy to hear in your comments more examples of rules and exceptions. Please share your experience. Thanks!

Posted in Efficacy, OOD, OOP | Tagged , , | 4 Comments

Design Patterns and the Tower of Babel

Posted on July 11, 2011 by Hayim Makabee

This is the story of the Tower of Babel from Wikipedia:

According to the biblical account, a united humanity of the generations following the Great Flood, speaking a single language and migrating from the east, came to the land of Shinar, where they resolved to build a city with a tower “with its top in the heavens… lest we be scattered abroad upon the face of the Earth.” God came down to see what they did and said: “They are one people and have one language, and nothing will be withholden from them which they purpose to do.” So God said, “Come, let us go down and confound their speech.” And so God scattered them upon the face of the Earth, and confused their languages, and they left off building the city, which was called Babel “because God there confounded the language of all the Earth.” (Genesis 11:5-8).

There is a very clear message in the story of the Tower of Babel: When people speak a common language, they may succeed in very challenging projects. If, however, everyone speaks a different language, the project will surely fail.

The catalogue of Design Patterns in the Gang-of-Four book was a very important contribution to the software development industry worldwide. Many programmers improved their object-oriented design skills, learning from the examples collected in this work. But, in my opinion, the book’s main gift to us was the creation of a common language on a higher level of abstraction.

Before this book, software engineers tried to express their ideas using the basic concepts of Object-Oriented Programming (OOP): inheritance, composition, encapsulation, dynamic binding and abstract methods. There were very few names to capture a portion of the design, i.e. the relationships among several classes in a system. One of the few examples was Model-View-Controller (MVC).

Then, when the Design Patterns book became popular, as an immediate consequence a common vocabulary was created. Suddenly people were able to talk about Singletons, Visitors, Factories and Strategies. Today, when you mention any of these names, you expect other developers to understand exactly what you are talking about, without any further explanation.

We know that it is a characteristic of each profession to have its specific vocabulary, which allows work to be done more effectively. In this sense, I believe that the Design Patterns book was a big step for the maturity of software engineering as a profession.

Posted in Design Patterns, OOD | Tagged , | 2 Comments

The Facebook Killer: Lack of Reciprocity

Posted on July 3, 2011 by Hayim Makabee

Some people claim that Facebook will be replaced by some other social network such as Google+. I think that the real Facebook killer will be the Lack of Reciprocity:

If you are constantly posting about your personal life, you will eventually discover that your hundreds of “friends” know everything about you, but most of them don’t want to tell you about their lives.

This is my classification of Facebook users, according to the way they use it and the kind of information they publish: 

    • Naïve Users: Are very active users, publishing lots of personal information and family pictures. They will let you know everything about their lives: In which restaurants they eat, what clubs they frequent, where they are planning their vacations. They have many albums with lots of pictures, some of them very personal. They also frequently comment and like the posts of others.
    • The Jokers: They like to post statuses that they think are very funny and/or smart, such as: “How are turkeys called in Turkey?” They will never publish anything personal. If they upload a photo, it will probably be something bizarre they saw in the street. They don’t comment or like the statuses of others, but it is very important for them to have as many people as possible liking their own posts.
    • The Linkers: They provide lots of links to things they think are interesting and should be shared, like videos on YouTube, TED talks and news articles in digital journals. They also like to be the first to share some breaking news like “Michael Jackson is dead!” The Linkers never post any kind of personal information.
    • The Gamers: They spend most of their time playing games at Facebook, and their many posts are all notifications of the kind “Nancy has found a lost cow in his farm!” The Gamers don’t post personal information, but they will frequently invite you to play with them.
    • The Fans: They use Facebook very actively to follow everything their friends are publishing. They frequently comment and like other people’s posts, but never write about themselves. They make a true effort to be nice and friendly, but do not share about their own lives.
    • The Wishers: They only use Facebook to wish a “Happy Birthday” to their friends. They connect every day, check who has a birthday, and then post a congratulation message on his/her wall. If there are many such friends, they use “copy-and-paste”. The Wishers rarely write anything on their own wall.
    • The Scanners: They use Facebook to always keep up-to-date about what is happening to his many contacts. They are very interested in other people’s lives, so they read every status and see every picture. If there is not enough activity in the News Feed, the Scanners will start checking who the friends of their friends are. As the name suggests, they never write anything, so they cause you the impression they are not using Facebook. But they are, and a lot!

It is obvious from this classification that the only people publishing any personal information are Naïve Users. Moreover, people using Facebook to read their friends’ posts are interested mostly in the personal information provided by the Naïve Users. The Jokers’ jokes, Linkers’ links and Gamers’ notifications are different types of garbage that only pollute the News Feed. There is nothing really “social” about them, so they do not contribute much in a social network.

I believe that eventually the Naïve Users will understand that they are sharing too much personal information with too many people. They will discover that details of their lives became public domain, while the people consuming this information, their “friends”, do not really reciprocate. Then I doubt they will continue sharing that much.

At the moment that Facebook will lack the personal information provided by the Naïve Users, its social aspect will be lost, and this will greatly affect the motivation of all other types of users.

Technology can provide us new tools to communicate, but it cannot change the inherent defects in human nature.

Posted in Social Networks | Tagged | 2 Comments

The Agile Architect

Posted on June 28, 2011 by Hayim Makabee

The International Association of Software Architects (IASA) in Israel was honored to host a talk by Johanna Rothman last week (June/21/2011).

Johanna Rothman is a consultant, author and speaker with vast experience with project management.

She was visiting Israel and generously agreed to give us a presentation.

In this special meeting, Johanna Rothman talked about “The Agile Architect”.

These are some of the interesting issues that she addressed:

  • What is the role of the Software Architect in Scrum?
  • In Agile, should the architecture be done in advance or incrementally together with implementation?

Johanna was very kind to publish the slides of her presentation.

To complement this material, you can read an interview with Johanna Rothman about “The Role of the Software Architect in Agile Development”.

We at IASA Israel would like to thank Johanna for this very interesting talk.

We would also like to thank SAP Labs Israel for hosting the event and providing the accommodations and refreshments.

If you are interested in future events of IASA Israel, please join our LinkedIn group.

Posted in Agile, IASA Israel, Software Architecture | Tagged , , | Leave a comment

The Single Question Interview

Posted on June 9, 2011 by Hayim Makabee

The routine of a software developer normally turns around design, implementation and testing. But hopefully there will be times in which the team needs to grow, and then he will be asked to do something extraordinary: Interviews.

In the software industry, it is natural to ask technical people to conduct interviews: After all, the company needs to evaluate the professional skills of the candidate, sometimes in a very specific field or technology, and this surely cannot be done by a HR person.

But planning a successful interview is a real challenge. The interviewer must conclude in a short time if the candidate is suitable or not to the open position. Thus, the time of the interview cannot be wasted on questions that do not really contribute to a decision, or, worse yet, questions that may cause the interviewer to make the wrong decision.

A recent article at TechCrunch claims that the most important thing to consider in the recruiting process is the candidate’s achievements. The author recommends:

“Don’t interview anyone who hasn’t accomplished anything. Ever. Certificates and degrees are not accomplishments; I mean real-world projects with real-world users.”

Similarly, Joel Spolsky says that the ideal candidate must be Smart and Get Things Done. According to Joel:

“The only way you’re going to be able to tell if somebody Gets Things Done is to see if historically they have tended to get things done in the past.”

I agree with these views, and I always start my interviews with a question about the candidate’s previous professional experience. I ask the candidate to select one of the projects he has participated in the past, and then describe me its requirements, high-level design and some implementation details. If the project is too big or complex, I may ask the candidate to talk about a particular module, or about a particular aspect, such as scalability or robustness.

In any case, if the candidate really contributed to this project and knows what he is talking about, this first question is just the opening for many interesting ones:

  • What was the biggest challenge in this project?
  • Did this project have any conflicting requirements?
  • What were the trade-offs you had to consider in your design decisions?
  • What were the non-functional requirements and how they affected the design?
  • What were the design patterns that you used in this project?

Thus, by talking about the candidate’s previous projects, we can learn a lot about his real skills and experience. In my opinion, this is much more valuable than discussing generic questions about software engineering or asking the candidate to solve logic puzzles.

But we must pay attention not only to what the candidate says, we must observe how he talks about his experience:

  • Does the candidate talk with enthusiasm?
  • Is he proud of his previous work?

Ideally, the candidate should be so enthusiastic when talking about his contributions to past projects that he should forget that he is in an interview. He shouldn’t feel he is in a kind of exam, but that he is talking to a fellow developer, enjoying the opportunity to describe all the smart things that he did and that he is proud of.

As I said in a previous post, I believe there is nothing more effective than enthusiasm. So we want to hire someone who is smart and gets things done, but with enthusiasm. Achievements alone are not valuable if the candidate is not proud of his achievements.

Good luck hiring the best people; to find them you don’t need more than a single question.

Posted in Hiring | Tagged | 4 Comments

Divide-and-Conquer: Coping with Complexity

Posted on June 6, 2011 by Hayim Makabee

Sometimes a problem is simply too complex for us to solve. Our only chance seems to be breaking it into smaller parts that we know how to deal with.

This strategy of reducing the complexity of a problem by dividing it into simpler sub-problems is known as “Divide-and-Conquer”. In this case there are two assumptions that are normally true:

  • A problem can be divided in several parts, so that each part can be handled independently.
  • Handling each one of the smaller parts that compose a problem is less complex than handling the whole problem, and thus we can “conquer” it.

There are several algorithms that are based in this strategy. For example, the Quicksort algorithm works by recursively dividing the list of values to be sorted into two smaller lists, in which one sub-list has the smaller values and the other the bigger values.

Another example is the Motion Planning problem in robotics, in which the robot needs to find a path to cross a room avoiding several obstacles. It is possible to adopt a divide-and-conquer approach to solve this problem, so that each obstacle is treated independently: The robot first finds possible paths between each pair of obstacles, then merges these smaller steps to create a single plan to pass all obstacles.

Given a complex problem, these are the steps that must be followed to solve it using a divide-and-conquer strategy:

  1. Define a method to decompose the complex problem into smaller parts.
  2. For each part, find a way to solve it.
  3. Combine the solutions of the parts to obtain the solution of the original problem.

Some problems can be solved recursively, as in the example of Quicksort. In this case all sub-problems are solved in the same way in step 2, and it is relatively easy to combine their solutions in step 3.

In the case of a problem that cannot be decomposed recursively, each sub-problem in step 2 may require a different solution, and it may be more difficult to combine the solutions in step 3. This would be the case of Motion Planning if the robot needs to do other types of tasks besides only crossing obstacles. For example, in the middle of its path the robot may be required to recharge its batteries or to carry some object.

In any case, a divide-and-conquer strategy will provide a systematic way to cope with complexity, making difficult problems more tractable by our limited human capabilities.

Posted in OOD, OOP | Tagged , | 2 Comments

On Bubbles and the Value of Data

Posted on June 1, 2011 by Hayim Makabee

In a recent article at Newsweek, Daniel Lyons claims that:

“Data, by itself, is worthless. Facebook is betting it can create software algorithms to extract value from that data—essentially to perform a kind of techno-alchemy and turn zillions of meaningless bits into billions of dollars. Nobody knows if the company can really pull this off…”

I couldn’t disagree more. Today, creating software algorithms to extract value from data is a science, applied by all big companies on the Web.

For instance, the research field of Data Mining focuses on discovering hidden patterns in large data sets. When these patterns are found, what was previously raw data becomes very useful information and knowledge. One example would be an E-commerce company that sells a large diversity of items on the Web. Data mining tools can be used to predict future buying trends and forecast supply demands.

Another important research field is Recommender Systems, whose goal is to recommend items of interest to users based on preferences they have expressed, either explicitly or implicitly. Recommender systems increase both the user satisfaction and the revenue of the companies using it, because the user is exposed to the items that interest him most and thus has a higher probability of acquiring them. The best-known example is Amazon, which is very successful in recommending books to their customers based on their previous purchases.

In the case of Facebook, at this moment they have almost 700 million users, and continue to grow fast in new markets. For each one of these users they have very valuable data, including detailed user profiles and their connections. These data can be easily used to select precisely targeted advertisements, what as a consequence increases the price they can ask for each such advertisement.

We just recently saw the IPO of LinkedIn, and it is now valued at more than U$10bn. This value is not derived from paying users, because most services are free. The real value of LinkedIn comes from more than 100 million very detailed professional profiles, including precious personal information that no other company has. Again, this allows them to display advertisements that have a high probability of being clicked, and thus a high probability of generating revenue.

Facebook and LinkedIn are not bubbles. I and my friends at Yahoo! Labs are working hard devising new ways to extract value from data. We know how valuable data can be. None should underestimate it.

Posted in Data Mining, Recommender Systems | Tagged , | Leave a comment

Soft Skills vs. Software Skills

Posted on May 23, 2011 by Hayim Makabee

When building a team of software developers, it is very important to define the profile of the engineers we want to compose it: Do we want to give importance mostly to the software skills, or do we want also to give a big weight to the soft skills?

This is how Wikipedia defines soft skills:

“Soft skills is a sociological term relating to a person’s EQ (Emotional Intelligence Quotient), the cluster of personality traits, social graces, communication, language, personal habits, friendliness, and optimism that characterize relationships with other people. Soft skills complement hard skills (part of a person’s IQ), which are the occupational requirements of a job.”

It is clear that the ideal developer must have a combination of the two types of skills. We don’t want someone in our team who is a brilliant programmer but is never ready to share his knowledge with others. We also do not want someone who is very nice and pleasant, but writes low quality code and has low productivity.

I think it is natural that in the software industry, in general, companies will have some preference for hiring the best engineers, even if they are not the best team players. For example, this is what Eric Schmidt recently said in an interview about Google’s hiring process:

“You are going to have to deal with the odd people. Not every single one of these incredibly smart people is a team player… even if people don’t want them around, we still need them.”

In some cases, people with poor soft skills can be really prejudicial. My experience is that a problematic personality is normally the main reason for a software developer to be fired or transferred to a different project.

The amount of damage that a developer with poor soft skills can cause to the team depends on the development methodology being used:

  • If the developers are working following traditional methods, in which they do not have many dependencies on each other tasks, then it is easier to make the most benefit of that black-belt programmer that does not happen to have the best inter-personal skills.
  • If the team is using Agile methods such as Scrum, then we do expect that the developers will work closely together, and being a good team-player becomes essential. In a Scrum team, a programmer with poor soft skills may cause more damage than benefit.

At the bottom line, when considering a candidate for an engineering position, I believe there should be some essential requirements regarding the software skills, but I would never hire someone who clearly has poor soft skills.

After all, you can always teach a good engineer some new technology, but it can be very hard to improve someone’s soft skills…

Posted in Agile, Hiring | Tagged , | 3 Comments

Inheritance and the Power of Abstraction

Posted on April 11, 2011 by Hayim Makabee

Programming is an exercise in abstraction. If a program is intended to solve some problem, we must first be able to model this problem in an abstract way, in order to express it in a programming language. Only after the problem is modeled the solution can be built.

An algorithm can be seen as an abstract solution for a category of problems. For example, the same sorting algorithms can be used to sort anything: numbers, names or dates.

Thus, we can talk about levels of abstraction: In the first level, we have the problem of sorting a list of values, and we need to abstract this problem in order to write a solution (a program) for it. In a higher level of abstraction, we are able to understand that the solution is independent of the type of values being sorted, and then we can create a sorting algorithm. When we create an algorithm, we are generalizing the solution.

Design Patterns are another example of abstractions. In this case, experienced software developers were able to identify recurring designs in the systems they have built, express these designs in an abstract way, and classify them according to the scenarios in which they should be used. When a programmer decides to apply a pattern, he needs to adapt the abstract design to the entities in his specific system. When we use a design pattern, we are specializing an abstract design.

Therefore, the logic of abstraction can be used in both directions:

  • Generalization: Creating a more abstract solution by ignoring details.
  • Specialization: Creating a less abstract solution by adding details.

In object-oriented programming, abstraction is naturally expressed by inheritance. In a well-structured class hierarchy, each superclass is a generalization of its subclasses, and each subclass is a specialization of its superclass.

Sometimes we start with a set of concrete classes and then we understand that they have something in common, despite their differences. In this case we abstract their properties, preserving the shared ones and removing the specific ones, and the result is a superclass that generalizes all the previous classes.

In other situations we start with a single concrete class and then understand that there are several possible ways to extend its functionality. In this case we specialize the initial class by creating new concrete subclasses, each with its own properties and behavior, but all of them sharing the functionality defined in the superclass.

Without a mechanism such as inheritance we would be severely limited in our capacity to model a problem and its solution. Programming languages without inheritance lack expressive power.

Accordingly, programmers that don’t know how to make good use of inheritance are less effective. Invest your time in mastering the correct usage of abstraction, and you will get very concrete quality improvements!

Posted in OOD, OOP | Tagged , | 3 Comments