Communication Problems in Software Projects

The cartoon above is the best illustration I know for the so usual communication problems in software projects. This difficulty to capture the customer needs and correctly translate them in the form of requirements, design and working code is one of the main motivations for the methodology of Agile software development.

As I discussed in a previous post, I believe in iterative methods and fast prototyping. I have been applying this approach in all my projects in the last years, and I’m very happy with the results. Among the benefits, I have been able to deliver more frequently and adapt more easily to changing requirements.

Thus, if you are frustrated with your current development lifecycle, you should know that there are alternatives, and try them. Then, perhaps this cartoon will become a thing of the past…

Posted in Agile, Requirements Specification | Tagged , | 2 Comments

How to find the best candidates without interviews

Interviewing candidates for a job is a very expensive process. If your intention is to hire only the best people, as it should be, you will probably have to interview dozen of candidates until you find one good match. And this of course is after the screening process, in which the potential candidates were already selected through careful analysis of their skills and experience, and perhaps other filtering mechanisms such as phone interviews.

What is the cost of each interview? There is the obvious cost of the interviewers’ time, including their preparation before the interview and the time spent writing their conclusions after it. Companies differ in their selection methods, but in general the entire hiring process will always include a series of interviews with people with rising levels of seniority. Thus, to find one good match, the people involved with the hiring process need to invest dozens of work hours, many of them dedicated to candidates that will be rejected. Clearly, we would like to make this process more effective, both to save money and to make it less frustrating for all the people involved, including the interviewers and the candidates.

There is an additional cost if the candidates are coming through job placement agencies. In this case, the agency will normally get one full month of salary for each candidate that is hired. Of course it would be much more cost-effective if we could attract good candidates without the help of these agencies. Better yet, it would be great if we could get the interest from people who are not actively looking for work. We want to hire the best people, but the best people normally already have good jobs and are happy with them. Unfortunately, the truth is that the pool of people actively looking for work is below average, either regarding their skills or experience.

To summarize, we would like a hiring process that:

  • Is extremely cost-effective.
  • Requires a minimum number of personal interviews.
  • Is able to attract good candidates without the help of placement agencies.
  • Is able to attract good candidates that are not actively looking for a new job.

This is exactly what was idealized by Hola, an Israeli Internet start-up. They have created a promotion in which software developers have to solve a programming challenge in order to win money prizes. A programmer submitting a “perfect” solution gets the top prize of 2000 Shequels (~500 US dollars). Their idea is viral since people can also win prizes if they forward the challenge to their friends, and if their friends solve it. If you are a programmer, you can try their challenge.

According to Ofer Vilenski, Hola’s CEO, several hundreds of Israeli programmers already tried to solve the challenge, but only a few were able to do it perfectly. These few ones that won the prize are considered by Hola to be top programmers, exactly the kind of people they would like to hire. The entire process so far cost to them only a few thousand Shequels, plus the time required to check the solutions, which they claim is very small for each solution. The prize winners are being invited for interviews, even if they were not actively looking for a job.

I’m personally very impressed with the efficacy of this model, which is also very original. But I’m not sure if I would consider adopting it in the future. As I said in a previous post, I believe in the importance of soft skills, but that cannot be measured in a programming challenge.

What do you think? I will be glad to hear your opinion about this idea. Please leave a comment below.

I also invite you to check my previous posts about the hiring process. And if you are a candidate looking for a job, I wish you good luck!

Posted in Hiring | Tagged | 4 Comments

An Interview Question for Fresh Graduates

Hire Me!It is extremely difficult to evaluate a candidate in a job interview, and even more difficult if this candidate is a fresh graduate. In this case we can talk about his grades, the courses he took and the projects he did during his studies. But we cannot learn from his previous experience. Of course we can focus on his soft skills or his problem-solving abilities, but we still need to predict his performance doing a job that requires specific technical skills.

In the case of software developers, we would like to predict if the candidate will be an effective programmer and a good team member. As I’ve written in a previous post, I believe that nothing is more effective than enthusiasm. Then, the question becomes: Will this candidate enjoy being a programmer? Will he be enthusiastic about his tasks?

I believe this question is extremely relevant today, because software development became a popular profession with very high salaries. In the past, people wanted to study Computer Sciences because they really were interested in working with computers; they were very technical-oriented and liked Math. But unfortunately today it is very common to meet students that chose Computer Sciences just because they know that it will be easy to find a good job with a great salary. And I say “unfortunately” because if these students do not really enjoy programming they will not be happy in their future jobs.

Thus, when interviewing the fresh graduate, we want to know: Is this guy of the first kind, the one that always loved computers and has a passion for programming? Or is he of the second type, the pragmatic that chose his faculty with the goal of finding a good job afterwards, but without any special attraction to algorithms and data structures?

Malcolm Gladwell, in his book “What the Dog Saw”, has a chapter entitled “Most Likely to Succeed” in which he discusses the challenge of hiring the best candidate. He mentions one corporate recruiter who used to ask the question “Give me a typical day”. I think this question is excellent, especially for fresh graduates, and I can imagine it in an interview:

Interviewer: “Please describe your typical day.”

Candidate: “I wake up at 7:00. I arrive at my first class at 8:00 and then I’ve classes until 12:00. I eat lunch with my friends, and then I spend the afternoon doing my home works or projects. Sometimes I need to go to the library or to a computers farm in the faculty. Around 18:00 I go either to the gym or to the swimming pool. When I come back home I have dinner and then I watch TV or surf the Web until I go to sleep around 11:30.”

Interviewer: “And when do you code?”

Candidate: “Well, I normally have projects in which I have to code, and some of my home works also require some programming.”

Interviewer: “Sure. But when do you write your own programs?”

If a candidate presents himself as a programmer, I expect him to have his own pet projects, besides home works or projects he had to do during his studies. Today there are many platforms that allow programmers to develop interesting ideas with relatively small effort: Web sites, Facebook games, iPhone or Android apps. These are examples of applications that can be implemented by a single person working during his free time. And of course there is also the option of contributing to a much greater project as part of an Open-Source effort.

If you are a fresh graduate, perhaps you got a nice diploma, but if you do not have any personal project I doubt you truly enjoy programming. And since I believe that you must really have fun coding to be a great programmer, I’m sorry, but if you do not have pet projects I would not hire you.

You may want to take a look in my other posts about the hiring process.

Feel free to leave a comment or question, and good luck!

Posted in Hiring | Tagged | 7 Comments

IASA Israel meeting about TRIZ

The first activity of the International Association of Software Architects in Israel in the year of 2012 was a meeting about TRIZ – The Theory of Inventive Problem Solving.

We had an introductory talk by Ido Lapidot, TRIZ expert at Intel Haifa, followed by a panel with other TRIZ specialists to discuss the applications of TRIZ in the industry.

Our invited speakers were:

Ido Lapidot: Directed Evolution and Systematic Innovation Expert (TRIZ) at Intel Strategic Planning Group.

Vladimir Petrov: TRIZ Master, President of the Israeli TRIZ Association and VP R&D at Innovation Technology.

Shadi Saba: Senior Research and Design Automation Team Manager at Intel Corporation.

This meeting was sponsored by SAP Labs Israel.

If you would like to be informed about our next meetings, please join IASA Israel group on LinkedIn.

You can watch below Ido Lapidot’s talk about TRIZ and its applications (in Hebrew):

Introduction to TRIZ: Theory & Practice from Michael Kalika on Vimeo.

Posted in IASA Israel | Tagged | Leave a comment

Separation of Concerns

The most important principle in Software Engineering is the Separation of Concerns (SoC): The idea that a software system must be decomposed into parts that overlap in functionality as little as possible. It is so central that it appears in many different forms in the evolution of all methodologies, programming languages and best practices.

Dijkstra mentions it in 1974: “separation of concerns … even if not perfectly possible is yet the only available technique for effective ordering of one’s thoughts”. Information Hiding, defined by Parnas in 1972, focuses on reducing the dependency between modules through the definition of clear interfaces. A further improvement was Abstract Data Types (ADT), by Liskov in 1974, which integrated data and functions in a single definition.

In the case of Object Oriented Programming (OOP), encapsulation and inheritance proved to be essential mechanisms to support new levels of modularity. Design-by-Contract, proposed by Meyer in 1986, provides guidelines of how to improve interfaces using pre-conditions and post-conditions. Finally, the separation of cross-cutting concerns is the most important motivation for the proponents of Aspect Oriented Programming (AOP).

Since the first software systems were implemented, it was understood that it was important for them to be modular. It is necessary to follow a methodology when decomposing a system into modules and this is generally done by focusing on the software quality metrics of coupling and cohesion, originally defined by Constantine:

Coupling: The degree of dependency between two modules. We always want low coupling.

Cohesion: The measure of how strongly-related is the set of functions performed by a module. We always want high cohesion.

All methodologies try to reduce coupling and increase cohesion. Information Hiding reduces coupling by isolating the details of the implementation of state. ADTs reduce coupling by defining clear and abstract interfaces. An ADT that specifies in a single definition the set of functions that can be executed on a type is certainly more cohesive than a global data structure that is modified by external functions.

OOP adds another step in the reduction of coupling with the enforcement of encapsulation and the introduction of dynamic binding and polymorphism. Inheritance allows us to increase cohesion by defining hierarchies based on generalization and specialization, in which we can separate the functionality that belongs to the superclass from its subclasses. AOP provides a solution for the problem of cross-cutting concerns, so that both the aspects and the affected methods may become more cohesive.

There are many benefits that software developers expect to obtain when making a system more modular, reducing coupling and increasing cohesion:

Maintainability: A measure of how easy it is to maintain the system. As a consequence of low coupling, there is a reduced probability that a change in one module will be propagated to other modules. As a consequence of high cohesion, there is an increased probability that a change in the system requirements will affect only a small number of modules.

Extensibility: A measure of how easily the system can be extended with new functionality. As a consequence of low coupling, it should be easier to introduce new modules, for example a new implementation for an existing interface. As a consequence of high cohesion, it should be easier to implement new modules without being concerned with aspects that are not directly related to their functionality.

Reusability: A measure of how easy it is to reuse a module in a different system. As a consequence of low coupling, it should be easier to reuse a module that was implemented in the past for a previous system, because that module should be less dependent on the rest of the system. Accordingly, it should be easier to reuse the modules of the current system in new future systems. As a consequence of high cohesion, the functionality provided by a module should be well-defined and complete, making it more useful as a reusable component.

As software developers, after we recognize the importance of SoC, we need to apply this principle in at least two ways: Understanding the power of our programming language tools and patterns, and learning how to evaluate and compare different designs in terms of coupling and cohesion.

Tools: For each mechanism in the programming languages we use, we should understand how it can be applied to reduce coupling and increase cohesion. For example: How encapsulation, dynamic binding, polymorphism and generic types can be used to separate concerns? Similarly, for each Design Pattern, we can analyze how it helps to make a system more modular.

Designs: When evaluating and comparing our own design alternatives, it is always useful to think in terms of coupling and cohesion. Sometimes a design seems to be more complex than others, but this may be a consequence of a better separation of concerns, with fewer dependencies between modules through the definition of additional layers. Another design may at first appear to have too many classes, but this may be an advantage if each class becomes more cohesive.

Producing effective software designs requires lots of experience, but principles such as the Separation of Concerns are essential to perform a great work. So reduce coupling, increase cohesion and good luck!

Posted in AOP, Design Patterns, OOD, OOP, Software Reuse | Tagged , , , , | 52 Comments

Events for Software Developers in Israel

Something very nice is happening in the Software Development Community in Israel: People are organizing meetings to share their knowledge, but not only with their colleagues in the workplace – these are meetings open to the public, including engineers from “competing” companies. I personally think that this strong sense of community and this desire to share knowledge and experience is a wonderful characteristic of the Israeli hi-tech industry, well described in the book “Start-up Nation“.

I am myself one of the co-founders of the International Association of Software Architects in Israel (IASA IL). We are organizing an event next week, focused on TRIZ – the Theory of Inventive Problem Solving. The meeting will include an introductory talk about TRIZ, followed by a panel to discuss its applications in the industry. Our speakers are Ido Lapidot and Shadi Saba, from Intel Haifa, and Vladimir Petrov, president of the Israeli TRIZ association. You can register in the event. You can also join our LinkedIn group.

Another very active group is the Software Craftsmanship in Israel (SCISR), leaded by Uri Lavi. They are having their 10th meeting this week, with the special participation of Corey Haines. The meetings organized by SCISR are extremely popular, and if you don’t register fast you will surely find yourself in the waiting list.

The ILTechTalks initiative, organized by Ori Lahav, has as its goal to “facilitate knowledge and experience exchange between Israeli tech professionals”. They have a rich list of very interesting talks, with highly qualified speakers that volunteer to present these subjects in your own company.

The Agile community in Israel also became very active in the last couple of years. They have several groups on LinkedIn, in which you can always find interesting blog posts. They are organizing a great two-day event next week, the Agile Practitioners 2012 Conference, with the participation of Gojko Adzic, David Evans and Corey Haines. There will also be talks by several of the leading members of the Agile movement in Israel.

Besides the opportunity to exchange knowledge and experience, these meetings are the best places to socialize and increase your network. In a recent article at the Fast Company, Kevin Purdy talks about the importance of the Third Place:

“The First Place is your home, and the Second Place is your office. You have assigned roles and tasks at each place, and you know nearly all the people in each. The Third Place is where you meet with people you don’t know that well, or maybe at all, and you exchange ideas, learn about other people, and enrich society and yourself.”

 So don’t miss these events! Hope to see you there!

Posted in Agile, IASA Israel, Israel | Tagged , , | 1 Comment

Learning from a Shipwreck

In the last week we followed the story of the Costa Concordia disaster, the cruise ship that partially sank after hitting a reef off the Italian coast. When I first heard about it, I could not understand how this was possible: We know that this kind of ships has a very well planned route, which should leave no risk for such a disaster. And we also know that with current technology it is very easy to provide alerts in the case a ship deviates from its planned route.

But then we learned that the captain had intentionally modified the ship’s route, apparently just to please some of his friends. And he certainly also ignored the alarms that followed this change in the route:

“Costa Cruises CEO, Pier Luigi Foschi, stated that the company’s ships have computer-programmed routes and alarms, both visual and sound, if the ship deviates by any reason from the stated route as stored in the computer and as controlled by the GPS, but that these alarms can be manually overridden.”

Finally, it was also reported that the evacuation of the passengers was delayed for no clear reason, which is another violation of the defined procedures. Actually, in the beginning the passengers were not even informed that there was a problem, probably to avoid panic.

This sad story makes me think about failures in software projects. As experienced software developers, we know that there are many risks involved when implementing a new system. We try to avoid failures by having:

  • A well-defined execution plan.
  • Mechanisms to measure progress with alerts for potential problems.
  • Recovery plans in the case something goes wrong.

However, what the Costa Concordia disaster teaches us is that these plans and alarms are only effective if the people executing them have discipline. A successful execution cannot be guaranteed by good plans alone; it depends on the ability and skills of the people implementing it. And for this reason, when choosing the right software developers for your team, you should always look for the best.

Posted in Efficacy, Hiring | Tagged , | 6 Comments

Hanukkah and Iterative Software Development

Hanukkah is a Jewish holiday that is celebrated during eight days. The tradition is to light candles every day, starting with one candle in the first day, then two candles in the second day, and continue adding one candle a day until there are eight candles in the last day. One possible meaning for this tradition is that we should always grow: The fact that a person is constantly growing and making progress becomes more important than the particular level he may be in a specific moment in his life.

Accordingly, in Iterative Software Development we understand that we do not have all the answers in the beginning of the project. We don’t even expect to have all the requirements clearly defined when we are assigned a new task. And we know that even if all the requirements were specified before we started the design, most probably some of them would change during the implementation. It is much easier to declare a code freeze than a “requirements freeze”.

Software as a Baby

In his classic book The Mythical Man-Month, Fred Brooks explains that software development is an incremental process, like pregnancy:

“The bearing of a child takes nine months, no matter how many women are assigned.”

The development of a software system cannot be reduced to the execution of a list of tasks completely known in advance, in which we can minimize the total implementation time by assigning more people to execute these tasks. Our understanding of the system will certainly change during the implementation, and thus these tasks cannot be really executed in parallel.

Now the question is: If we cannot know all requirements in advance, how can we progressively discover the requirements?

Software Prototyping

Some aspects of a software system cannot be completely defined before there is at least a partial implementation. For instance, the User Interface (UI): Without testing it, it is difficult to determine if a particular design of the UI will satisfy the customer needs.

The idea of Software Prototyping is to provide an incremental and progressive way to develop systems with partial functionality, allowing the developers to adapt to missing or changing requirements.

The process starts with the identification of the basic requirements, for which a prototype is built. Then the customers review this prototype and provide feedback on additions and changes. Using this feedback, the initial prototype is enhanced. These steps can be repeated several times: Each new version of the prototype is reviewed and can then be further improved.

Software Gardening

This incremental approach of software development is very different from other more traditional engineering disciplines. For example, we normally expect that the blueprint of a house should include all the details before we start building it.

In contrast, the authors of the book The Pragmatic Programmer say:

“Rather than construction, programming is more like gardening.”

In an interview, they claim that there are two main aspects that make software development so similar to gardening:

• You can never plan all the small details in advance: The plants in your garden will certainly grow.
• You need to invest a lot in maintenance: The plants may grow in unexpected and undesirable ways.

Going back to our candles metaphor, our role as software developers is to assure that the system is growing in the right direction, incrementally and progressively.

And of course, if you celebrate Hanukkah, then I wish you Chag Sameach!

Posted in Agile, Jewish Sources, Requirements Specification | Tagged , , | 7 Comments

TDD and the Gamification of Testing

The Test-Driven Development (TDD) methodology was invented to improve the way we test the code we write, with the final goal of increasing the quality of systems being implemented.  The basic idea is that the tests must be written before the code, and following the same requirements. In practice, adopting the TDD approach really caused an improvement in quality, even if in general the total development time also increased.

But in my opinion TDD is much more than a different way to test a system. I believe that it modifies completely the nature of the testing tasks. Actually, I think that this change is so profound that TDD was able to transform a hated activity in something that programmers can really enjoy.

In the past testing was something boring. Programmers didn’t like to spend much time testing their code; they preferred to delegate this task to the QA team. And it was considered normal if the testers in the QA team did find some bugs. After all, the software developers were too busy coding; they did not have time for all the repetitive actions related to testing. Most programmers felt that testing was an activity that did not require from them much creativity, neither used their special skills.

This reaction of programmers in relation to testing can be explained by concepts of Positive Psychology. According to the Flow theory, a person can only enjoy a task if there is a match between his abilities and the complexity of the task. As depicted in the graph on the right, when the person is skilled but the challenge is low the activity is considered boring. In contrast, when the person is skilled and the challenge is high, this can bring him to a state of arousal. In traditional testing methodologies there was a mismatch between the ability of software developers and the complexity of the testing task. The consequence is that programmers could not enjoy testing.

But TDD changes this picture because it transforms testing in just another type of programming task. When the software developers implement automated tests, they are actually writing code. Therefore, there is a match between their skills and the complexity of the testing task. Suddenly, testing becomes something that programmers can really enjoy doing.

This match between skills and complexity is in the root of any successful game. And TDD actually has some other characteristics that make it similar to games. This reminds me the concept of Gamification:

“Gamification is the use of game design techniques and mechanics to solve problems and engage audiences. Typically gamification applies to non-game applications and processes, in order to encourage people to adopt them. Gamification works by making technology more engaging, by encouraging users to engage in desired behaviors, by showing a path to mastery and autonomy, and by taking advantage of humans’ psychological predisposition to engage in gaming. The technique can encourage people to perform chores that they ordinarily consider boring.”

In TDD, using a framework such as xUnit makes the progress clearly measurable, like earning points. You start with a set of tests that are all “red” and gradually make them become “green”. It’s like a game: When all your tests are green, you win and this level of the game is over. Then you can move to a more advanced level, augmenting your challenge, either by adding new, more complex tests, or by performing some refactoring in the code that can potentially break some of the existing tests.

According to the book Gamification by Design: Implementing Game Mechanics in Web and Mobile Apps, another very important characteristic of games is their immediate feedback:  

“One of the most straightforward and important game mechanics, and one that is increasingly a cornerstone of the gamification movement, is feedback. Broadly defined, feedback is returning information to players and informing them of where they are at the present time, ideally against a continuum of progress.”

Every time you modify the code, you are actually taking the risk of breaking some tests. This generates the suspense of executing the tests after you made several changes in the code, to check the tests are still working. In practice, code refactoring can become an activity full of adrenaline. But the immediate feedback provided by automated tests allows you to take corrective actions in the case there were failures. At this stage, the coding-testing process can progress in fast iterations.

Sometimes you need to modify the tests because some requirement has changed. Of course in this case the right thing to do is to first change the tests and then the code. But I know that some programmers do exactly the opposite, and they first change the code and then check what tests were broken. This may be easier, because sometimes it is difficult to identify all the tests that are related to some specific requirement. Changing the code first is surely not the right thing to do, but it can be more fun.

If you still have any doubt that TDD is like a game, take a look in the TDGotchi application. I think it is a very original idea (but I must also say that I did not install it myself to check if it works).

So go on and have fun testing. It’s possible!

Posted in Agile, Gamification, Psychology of Programming, Refactoring, TDD | Tagged , , , , | 27 Comments

Yahoo! Cocktails: When the client is in the cloud

One of the reasons I’m proud to work at Yahoo! is that we have some very nice products that reach hundreds of millions of people worldwide. Another reason is that Yahoo! is behind the development of several advanced technologies and platforms, such as Hadoop. This is no coincidence, since sophisticated solutions are required to serve efficiently so many users and to handle effectively the huge quantities of data they consume and generate.

But the engineers inside the company are not the only ones to benefit from these innovations. Very often Yahoo! makes its technologies available to the broader development community. The latest such contribution is called Cocktails, a solution for the implementation of Web applications that addresses both the client and the server-side. Cocktails is based on a mix of standard Web technologies such as HTML5, Node.JS, CSS3 and JavaScript.

Yahoo! Cocktails is composed of Mojito, an environment-agnostic JavaScript web application framework, and Manhattan, a hosted platform (PaaS) for Mojito-based applications.

Mojito

Mojito is a JavaScript web application framework that can run both on the client and on the server-side. This allows developers to maintain the same codebase for the browser frontend and the server backend. During execution, if JavaScript is enabled in the device, the Mojito module will be deployed from the server runtime to the browser runtime. Otherwise, if JavaScript is not enabled, the same application will simply run on the server side.

Manhattan

Manhattan is a server-side JavaScript hosting environment for Mojito-based applications running on Yahoo!’s Cloud. It is a platform that provides essential capabilities such as security, scalability, high availability and monitoring. Manhattan is an extension of Node.JS that has a very simple interface for the deployment of Mojito-based applications. The fact that it runs on Yahoo!’s Cloud makes it possible for these applications to easily access services such as the Yahoo! Content Distribution Network.

Write once, run anywhere

One of the main promises of Cocktails is that the client code, based on JavaScript, can run on any connected device: Web browsers on the desktop, smartphones and tablets. The possibility of maintaining a single code base for all devices is a real advantage for developers. The benefits are in the form of reduced development time, easier maintenance and, in the bottom line, lower costs:

“Going forward, Mojito and Yahoo! Manhattan will allow you to develop ONE app and deploy it on several devices to lower cost, increase agility and keep a uniform design across platforms (web, mobile, tablet). With the power of Yahoo! Cocktails, you’ll be able to build that app – deployable on several platforms –in the same time it currently takes to build each app individually.”

When the client becomes part of the cloud

I think that the most innovative concept of Cocktails is that it transforms the client device in an additional machine running in the cloud. Since the client and server sides execute the same language, all code that runs in the client side may also run in the server side.

When the client starts, the server offloads its code to the device. If the client device is not able to run JavaScript, the client business logic simply runs on the server side. Moreover, if for some reason the client functionality becomes too heavy for execution on the device, it can migrate back to the server side and be executed there, improving the perceived performance.

In other words, the functionality associated with the client may run on the server, and this may be changed dynamically, following a decision based on performance issues. This in practice blurs the boundaries between client and server machines, and transforms the client device in an integral part of the cloud.

Present and Future of Cocktails

Cocktails is the technology behind one of Yahoo!’s most recent products: Livestand, a personalized magazine for tablets.

In the future, Yahoo! plans to make the Mojito framework available as open source and the Manhattan platform accessible to the community of developers of Mojito applications.

Check the video below for more information about Yahoo! Cocktails:

Enjoy your cocktails!

Posted in Cloud Computing, Software Architecture, Web Development, Yahoo! | Tagged , , , | Leave a comment