The Agile Architect

Posted on June 28, 2011 by Hayim Makabee

The International Association of Software Architects (IASA) in Israel was honored to host a talk by Johanna Rothman last week (June/21/2011).

Johanna Rothman is a consultant, author and speaker with vast experience with project management.

She was visiting Israel and generously agreed to give us a presentation.

In this special meeting, Johanna Rothman talked about “The Agile Architect”.

These are some of the interesting issues that she addressed:

  • What is the role of the Software Architect in Scrum?
  • In Agile, should the architecture be done in advance or incrementally together with implementation?

Johanna was very kind to publish the slides of her presentation.

To complement this material, you can read an interview with Johanna Rothman about “The Role of the Software Architect in Agile Development”.

We at IASA Israel would like to thank Johanna for this very interesting talk.

We would also like to thank SAP Labs Israel for hosting the event and providing the accommodations and refreshments.

If you are interested in future events of IASA Israel, please join our LinkedIn group.

Posted in Agile, IASA Israel, Software Architecture | Tagged , , | Leave a comment

The Single Question Interview

Posted on June 9, 2011 by Hayim Makabee

The routine of a software developer normally turns around design, implementation and testing. But hopefully there will be times in which the team needs to grow, and then he will be asked to do something extraordinary: Interviews.

In the software industry, it is natural to ask technical people to conduct interviews: After all, the company needs to evaluate the professional skills of the candidate, sometimes in a very specific field or technology, and this surely cannot be done by a HR person.

But planning a successful interview is a real challenge. The interviewer must conclude in a short time if the candidate is suitable or not to the open position. Thus, the time of the interview cannot be wasted on questions that do not really contribute to a decision, or, worse yet, questions that may cause the interviewer to make the wrong decision.

A recent article at TechCrunch claims that the most important thing to consider in the recruiting process is the candidate’s achievements. The author recommends:

“Don’t interview anyone who hasn’t accomplished anything. Ever. Certificates and degrees are not accomplishments; I mean real-world projects with real-world users.”

Similarly, Joel Spolsky says that the ideal candidate must be Smart and Get Things Done. According to Joel:

“The only way you’re going to be able to tell if somebody Gets Things Done is to see if historically they have tended to get things done in the past.”

I agree with these views, and I always start my interviews with a question about the candidate’s previous professional experience. I ask the candidate to select one of the projects he has participated in the past, and then describe me its requirements, high-level design and some implementation details. If the project is too big or complex, I may ask the candidate to talk about a particular module, or about a particular aspect, such as scalability or robustness.

In any case, if the candidate really contributed to this project and knows what he is talking about, this first question is just the opening for many interesting ones:

  • What was the biggest challenge in this project?
  • Did this project have any conflicting requirements?
  • What were the trade-offs you had to consider in your design decisions?
  • What were the non-functional requirements and how they affected the design?
  • What were the design patterns that you used in this project?

Thus, by talking about the candidate’s previous projects, we can learn a lot about his real skills and experience. In my opinion, this is much more valuable than discussing generic questions about software engineering or asking the candidate to solve logic puzzles.

But we must pay attention not only to what the candidate says, we must observe how he talks about his experience:

  • Does the candidate talk with enthusiasm?
  • Is he proud of his previous work?

Ideally, the candidate should be so enthusiastic when talking about his contributions to past projects that he should forget that he is in an interview. He shouldn’t feel he is in a kind of exam, but that he is talking to a fellow developer, enjoying the opportunity to describe all the smart things that he did and that he is proud of.

As I said in a previous post, I believe there is nothing more effective than enthusiasm. So we want to hire someone who is smart and gets things done, but with enthusiasm. Achievements alone are not valuable if the candidate is not proud of his achievements.

Good luck hiring the best people; to find them you don’t need more than a single question.

Posted in Hiring | Tagged | 4 Comments

Divide-and-Conquer: Coping with Complexity

Posted on June 6, 2011 by Hayim Makabee

Sometimes a problem is simply too complex for us to solve. Our only chance seems to be breaking it into smaller parts that we know how to deal with.

This strategy of reducing the complexity of a problem by dividing it into simpler sub-problems is known as “Divide-and-Conquer”. In this case there are two assumptions that are normally true:

  • A problem can be divided in several parts, so that each part can be handled independently.
  • Handling each one of the smaller parts that compose a problem is less complex than handling the whole problem, and thus we can “conquer” it.

There are several algorithms that are based in this strategy. For example, the Quicksort algorithm works by recursively dividing the list of values to be sorted into two smaller lists, in which one sub-list has the smaller values and the other the bigger values.

Another example is the Motion Planning problem in robotics, in which the robot needs to find a path to cross a room avoiding several obstacles. It is possible to adopt a divide-and-conquer approach to solve this problem, so that each obstacle is treated independently: The robot first finds possible paths between each pair of obstacles, then merges these smaller steps to create a single plan to pass all obstacles.

Given a complex problem, these are the steps that must be followed to solve it using a divide-and-conquer strategy:

  1. Define a method to decompose the complex problem into smaller parts.
  2. For each part, find a way to solve it.
  3. Combine the solutions of the parts to obtain the solution of the original problem.

Some problems can be solved recursively, as in the example of Quicksort. In this case all sub-problems are solved in the same way in step 2, and it is relatively easy to combine their solutions in step 3.

In the case of a problem that cannot be decomposed recursively, each sub-problem in step 2 may require a different solution, and it may be more difficult to combine the solutions in step 3. This would be the case of Motion Planning if the robot needs to do other types of tasks besides only crossing obstacles. For example, in the middle of its path the robot may be required to recharge its batteries or to carry some object.

In any case, a divide-and-conquer strategy will provide a systematic way to cope with complexity, making difficult problems more tractable by our limited human capabilities.

Posted in OOD, OOP | Tagged , | 2 Comments

On Bubbles and the Value of Data

Posted on June 1, 2011 by Hayim Makabee

In a recent article at Newsweek, Daniel Lyons claims that:

“Data, by itself, is worthless. Facebook is betting it can create software algorithms to extract value from that data—essentially to perform a kind of techno-alchemy and turn zillions of meaningless bits into billions of dollars. Nobody knows if the company can really pull this off…”

I couldn’t disagree more. Today, creating software algorithms to extract value from data is a science, applied by all big companies on the Web.

For instance, the research field of Data Mining focuses on discovering hidden patterns in large data sets. When these patterns are found, what was previously raw data becomes very useful information and knowledge. One example would be an E-commerce company that sells a large diversity of items on the Web. Data mining tools can be used to predict future buying trends and forecast supply demands.

Another important research field is Recommender Systems, whose goal is to recommend items of interest to users based on preferences they have expressed, either explicitly or implicitly. Recommender systems increase both the user satisfaction and the revenue of the companies using it, because the user is exposed to the items that interest him most and thus has a higher probability of acquiring them. The best-known example is Amazon, which is very successful in recommending books to their customers based on their previous purchases.

In the case of Facebook, at this moment they have almost 700 million users, and continue to grow fast in new markets. For each one of these users they have very valuable data, including detailed user profiles and their connections. These data can be easily used to select precisely targeted advertisements, what as a consequence increases the price they can ask for each such advertisement.

We just recently saw the IPO of LinkedIn, and it is now valued at more than U$10bn. This value is not derived from paying users, because most services are free. The real value of LinkedIn comes from more than 100 million very detailed professional profiles, including precious personal information that no other company has. Again, this allows them to display advertisements that have a high probability of being clicked, and thus a high probability of generating revenue.

Facebook and LinkedIn are not bubbles. I and my friends at Yahoo! Labs are working hard devising new ways to extract value from data. We know how valuable data can be. None should underestimate it.

Posted in Data Mining, Recommender Systems | Tagged , | Leave a comment

Soft Skills vs. Software Skills

Posted on May 23, 2011 by Hayim Makabee

When building a team of software developers, it is very important to define the profile of the engineers we want to compose it: Do we want to give importance mostly to the software skills, or do we want also to give a big weight to the soft skills?

This is how Wikipedia defines soft skills:

“Soft skills is a sociological term relating to a person’s EQ (Emotional Intelligence Quotient), the cluster of personality traits, social graces, communication, language, personal habits, friendliness, and optimism that characterize relationships with other people. Soft skills complement hard skills (part of a person’s IQ), which are the occupational requirements of a job.”

It is clear that the ideal developer must have a combination of the two types of skills. We don’t want someone in our team who is a brilliant programmer but is never ready to share his knowledge with others. We also do not want someone who is very nice and pleasant, but writes low quality code and has low productivity.

I think it is natural that in the software industry, in general, companies will have some preference for hiring the best engineers, even if they are not the best team players. For example, this is what Eric Schmidt recently said in an interview about Google’s hiring process:

“You are going to have to deal with the odd people. Not every single one of these incredibly smart people is a team player… even if people don’t want them around, we still need them.”

In some cases, people with poor soft skills can be really prejudicial. My experience is that a problematic personality is normally the main reason for a software developer to be fired or transferred to a different project.

The amount of damage that a developer with poor soft skills can cause to the team depends on the development methodology being used:

  • If the developers are working following traditional methods, in which they do not have many dependencies on each other tasks, then it is easier to make the most benefit of that black-belt programmer that does not happen to have the best inter-personal skills.
  • If the team is using Agile methods such as Scrum, then we do expect that the developers will work closely together, and being a good team-player becomes essential. In a Scrum team, a programmer with poor soft skills may cause more damage than benefit.

At the bottom line, when considering a candidate for an engineering position, I believe there should be some essential requirements regarding the software skills, but I would never hire someone who clearly has poor soft skills.

After all, you can always teach a good engineer some new technology, but it can be very hard to improve someone’s soft skills…

Posted in Agile, Hiring | Tagged , | 3 Comments

Inheritance and the Power of Abstraction

Posted on April 11, 2011 by Hayim Makabee

Programming is an exercise in abstraction. If a program is intended to solve some problem, we must first be able to model this problem in an abstract way, in order to express it in a programming language. Only after the problem is modeled the solution can be built.

An algorithm can be seen as an abstract solution for a category of problems. For example, the same sorting algorithms can be used to sort anything: numbers, names or dates.

Thus, we can talk about levels of abstraction: In the first level, we have the problem of sorting a list of values, and we need to abstract this problem in order to write a solution (a program) for it. In a higher level of abstraction, we are able to understand that the solution is independent of the type of values being sorted, and then we can create a sorting algorithm. When we create an algorithm, we are generalizing the solution.

Design Patterns are another example of abstractions. In this case, experienced software developers were able to identify recurring designs in the systems they have built, express these designs in an abstract way, and classify them according to the scenarios in which they should be used. When a programmer decides to apply a pattern, he needs to adapt the abstract design to the entities in his specific system. When we use a design pattern, we are specializing an abstract design.

Therefore, the logic of abstraction can be used in both directions:

  • Generalization: Creating a more abstract solution by ignoring details.
  • Specialization: Creating a less abstract solution by adding details.

In object-oriented programming, abstraction is naturally expressed by inheritance. In a well-structured class hierarchy, each superclass is a generalization of its subclasses, and each subclass is a specialization of its superclass.

Sometimes we start with a set of concrete classes and then we understand that they have something in common, despite their differences. In this case we abstract their properties, preserving the shared ones and removing the specific ones, and the result is a superclass that generalizes all the previous classes.

In other situations we start with a single concrete class and then understand that there are several possible ways to extend its functionality. In this case we specialize the initial class by creating new concrete subclasses, each with its own properties and behavior, but all of them sharing the functionality defined in the superclass.

Without a mechanism such as inheritance we would be severely limited in our capacity to model a problem and its solution. Programming languages without inheritance lack expressive power.

Accordingly, programmers that don’t know how to make good use of inheritance are less effective. Invest your time in mastering the correct usage of abstraction, and you will get very concrete quality improvements!

Posted in OOD, OOP | Tagged , | 3 Comments

Nothing is more effective than enthusiasm

Posted on March 28, 2011 by Hayim Makabee

We all know that some programmers are much more effective than others. In one study, it was shown that the best programmers are ten times more efficient than the worse programmers. This degree of difference has not been observed in any other kind of industry, and this makes effective software development a unique challenge.

Now the question is: Why are some programmers much better than others? Can this difference be explained in terms of intellectual capacity, aptitude or experience?

In fact, it was observed that some programmers are much more effective than others even if they are very similar on all other aspects, such as IQ, problem solving capacity, math and logic orientation, and years of experience as a professional software developer.

I believe that the main difference is the enthusiasm! Some people simply love programming, more than any other intellectual activity. If these people weren’t assigned any coding task, they would just use their imagination and invent some new program to write, probably as challenging as possible.

I think this is the origin of all open-source projects, which are very successful and sometimes go against the intuition that complex software development requires centralized management. Actually, these passionate programmers are not only able to produce high-quality code in their free time, they are also able to organize themselves in very effective development teams.

So, do you want to find a real programmer? It’s easy: They can’t stay too long without writing a line of code. It’s kind of an addiction to programming!

Posted in Efficacy, Hiring, Psychology of Programming | Tagged , , | 9 Comments

Effective Software Reuse

Posted on February 28, 2011 by Hayim Makabee

For many years Software Reuse has been seen as a solution to both reduce the costs of software development and improve the quality of the systems being developed. However, in practice, most companies today have the feeling that they are not enjoying the full potential of software reuse.

In general, the most successful type of reuse today is the one exploring third-party components, in the form of libraries and frameworks. It is hard to imagine that someone would implement an application to parse some standard file format like XML, or to implement some standard communication protocol, without first looking for libraries that already provide the required functionality.

In contrast, a more difficult type of reuse is the one of components developed inside an organization. This happens because very often the components being implemented as part of an application are not planned to be reusable.

The effective development of reusable components requires a serious investment, including:

  • The design of the component’s interface, that must be both rich and generic.
  • The implementation of the component’s functionality, that must be efficient and independent of the application.
  • The full testing of the component’s behavior, that must take in consideration a great diversity of contexts.
  • The detailed documentation of the component’s usage, that must guide its potential users providing all required details.
  • The indexing of the component in a shared repository where it can be easily found.

In my opinion, this can be achieved only by a team that is dedicated to the development of reusable components. This team should receive requests from the product teams in the organization, with the description of components they would like to have in their applications and that are considered good candidates to become reusable components. Ideally, the communication between these teams should include the participation of a software architect, who should be responsible for approving the requests.

The biggest challenges of the Reusable Components Team (RCT) are:

  • Understanding the requirements from different product teams so that the components developed are really reusable across different applications.
  • Creating new components according to the time schedules of the product teams, respecting their deadlines and avoiding any delays in their plans.
  • Being responsive fast enough  when the product teams ask for changes or extensions in existing reusable components.

Therefore, the product teams are the clients of the RCT. The product teams should treat the components developed by the RCT as-if they were third-party reusable components, and never duplicate the code or change it by themselves.

From my experience, if the RCT is not able to meet its challenges, being responsive enough and fast enough, the product teams will “take ownership” of the reusable components, creating their own versions and ruining the very idea of software reuse.

Good luck with your reuse plans, I will be glad to hear about your own experience.

Posted in OOD, Software Reuse | Tagged , | 6 Comments

Effective Remote Work

Posted on January 20, 2011 by Hayim Makabee

After reading the book “The 4-Hour Workweek” by Tim Ferriss, I was eager to test his concept of Mini-retirements:

“To distribute recovery periods and adventures throughout life on a regular basis and recognize that inactivity is not the goal. Doing that which excites you is.”

I work for Yahoo! Labs in Israel, and I discovered that Yahoo! also has offices in Brazil. Then I decided that in my next trip to Brazil I would combine my two-week vacations with a period of remote work, so that in the total I would be able to stay there more than a month and have many additional weekends to enjoy being abroad.

My main doubt was: “How effective will be my remote work?”

I’m still in Brazil, but after two weeks working remotely I think that overall the experience was very positive. Here are the main points:

Communicating with co-workers: I didn’t feel any disruption in my formal communication with other members of my team, especially because there is only a 4-hour difference between Brazilian summertime and Israeli time. We even didn’t have to change the schedule of our weekly meeting, that was done via audio-conference, in the early afternoon in Israel and the morning in Brazil. We continued to communicate normally through Email and IM, and the only difference was a slight increase in the volume and frequency of messages, but nothing disturbing.

Performing my tasks: We are now in the early stages of our project, and I’m doing basically investigation of technologies and programming tasks. At this stage there are few dependencies between my tasks and the ones being performed by my co-workers. Actually, the fact that I’m abroad has drastically reduced the number and frequency of interruptions of my work, so that I’m certainly more productive here. I intentionally don’t have my mobile phone activated here, and that’s a great change from being at home.

Access to resources: When I arrived in the Yahoo! offices in Brazil, I was immediately given a desk with a phone and network connection. I brought my laptop with me from Israel, so that in a few minutes I was already connected and able to continue my work. Since we store everything in the company network that is globally shared, I have access to all resources I need, including Wikis, documentation, source code repositories and bug-tracking tools. Thus, despite being very far from home I’m working seamlessly.

The only thing I really miss are the social interactions with my co-workers, like the jokes in the corridors and the heated discussions at lunchtime. But this is compensated by the wonderful opportunity to meet and make new friends with my colleagues at Yahoo! offices in Brazil.

Posted in Efficacy, Yahoo! | Tagged , | 1 Comment

Identifying Anti-Patterns

Posted on December 22, 2010 by Hayim Makabee

In my previous post, I’ve discussed the importance of identifying domain-specific Design Patterns in an organization. Equally important is the need to identify Anti-Patterns.

This is the definition of Anti-Patterns from Wikipedia:

“An anti-pattern is a pattern that may be commonly used but is ineffective and/or counterproductive in practice.”

The first interesting aspect in this definition is that anti-patterns are commonly used despite being ineffective or counterproductive. There may be two main reasons for this clear contradiction:

  • The programmers using the anti-pattern are inexperienced, so they are unaware of the fact that these patterns are ineffective or counterproductive.

  • The programmers using the anti-pattern have experience on a different kind of systems, which were less demanding or based on older architectural approaches. Thus, they are not aware that the patterns they have used in the past are no longer appropriate for the new category of systems under development.

The second interesting aspect in the definition is that an anti-pattern is ineffective or counterproductive, but it still implements correct functionality, so the use of an anti-pattern is not a bug. Most often the anti-pattern provides a poor implementation that fails to satisfy non-functional requirements. Here also there are two main cases:

  • Bad design: The usage of the anti-pattern causes the code to be hard to understand and hard to maintain.

  • Bad performance: The usage of the anti-pattern causes the code to be inefficient, not scalable or non-robust.

Normally, inexperienced programmers use anti-patterns that cause the system to have both bad design and bad performance. It is necessary to show them the problematic code and teach them better alternatives.

Experienced programmers, on the other hand, may use anti-patterns that implement very elegant designs, but that provide bad performance. This happens because these patterns were good enough for the systems they have implemented previously, but they are no longer able to satisfy the more demanding non-functional requirements of current systems, such as higher scalability and increased robustness. Another common problem is that patterns that were once appropriate for older architectural approaches are no longer suitable for systems with modern architectures.

In an organization, there are three main opportunities to identify anti-patterns: During design reviews, during code reviews and during refactoring of legacy code.

Design reviews are a great opportunity to find anti-patterns. A software architect should be present in each review, and be responsible for indicating if there is any problematic aspect in the design being proposed, either because it doesn’t fit the overall system architecture or because it doesn’t satisfy the non-functional requirements.

Code reviews are the right place to locate bad code, either because it is inefficient or because it is hard to understand and to maintain. They are an excellent opportunity to teach inexperienced programmers about better alternatives.

During the refactoring of legacy code it should be easy to locate anti-patterns, simply because the system requirements have evolved since the code was written. In this case, the person responsible for the refactoring should have a plan for converting each anti-pattern into appropriate code that satisfies the current requirements. Very often this involves adapting old code to fit the new architecture.

So good luck finding your anti-patterns, there will be only benefits!

Posted in Design Patterns, OOD | Tagged , | 3 Comments