When Machines Learn: Predictive Analytics

CP 500In 1983, when I was 13, I got my first personal computer. It was a Brazilian CP-500 (in the picture on the right). I was one of the first kids in my class to have a computer at home, and so many of my friends wanted to come to my house to play with it. It was really a big attraction for boys my age at that time.

Once I invited a friend and started showing him all kinds of commands in BASIC. He was impressed by how the computer executed all my orders. Then he said:

“Ask the computer who is going to win the Brazilian soccer championship.”

I found that idea extremely funny. Trying not to laugh, I explained to him that computers were not able to make such predictions. At most they could execute some very simple instructions…

Well, 33 years have passed since then. And now I’m not a young boy programming in BASIC, I’m a data scientist applying machine learning algorithms to build predictive models. Yes, I’m developing systems that predict what will happen in the future.

More specifically, my goal at Pontis is to analyse the history of how subscribers reacted to marketing offers in the past, and then build models that predict how subscribers will react to marketing offers in the future. These models are built using very sophisticated machine learning algorithms.

Based on these predictions, the system automatically decides which marketing offer is most appropriate for each subscriber. This allows a new level of personalization, and accordingly increases the success rate of our marketing campaigns.

Thus my young friend was right after all. Computers may predict the future. Give them enough data and some smart algorithms, and they will learn how the world behaves.

Posted in Data Mining, Machine Learning, Programming | Tagged , , | 1 Comment

Copy-and-Paste Programming

I’m recently seeing lots of funny O’Reilly book covers being created by inspired programmers, but below is my favorite one. I like it because it depicts a big change in software development since the beginning of my career. In the past you could be an expert in some programming language and know very well all its standard libraries. But today, with the fast proliferation of new languages, libraries, frameworks and platforms, it became simply impossible to remember all the necessary details. As a consequence, programmers are always looking for coding examples, and copy-and-pasting them. What would we do without Stack Overflow?

copying_and_pasting

Posted in Programming | Tagged | Leave a comment

Agile and Wrong: The Problems with Emergent Design in Pictures

Many idealistic Agile practitioners propose the idea of Emergent Design:

“With emergent design, a development organization starts delivering functionality and lets the design emerge. Development will take a piece of functionality A and implement it using best practices and proper test coverage and then move on to delivering functionality B. Once B is built, or while it is being built, the organization will look at what A and B have in common and refactor out the commonality, allowing the design to emerge. This process continues as the organization continually delivers functionality. At the end of an agile release cycle, development is left with the smallest set of the design needed, as opposed to the design that could have been anticipated in advance.”

Based on my 20+ years of experience with Software Development, I’m convinced that Emergent Design does not work. Below I discuss its main problems, illustrated with some nice pictures.

1st Problem: Software Systems Change

emergent1Emergent Design may be enough to create an initial design that will be right until it is changed. If a software design is not planned to support change, it will deteriorate very fast. As illustrated on the picture on the right, it may be challenging to keep the design coherent even for very simple systems. Since software systems always change, a design that does not support change is simply wrong.

 

emergent62nd Problem: Software Systems Grow

Emergent Design may be enough to create an initial design that will be right until the system grows. If a software design is not planned to support growth, its complexity will increase very fast. As illustrated on the picture on the right, the initial design may become a disaster when one of the components grows. Since software systems always grow, a design that does not support growth is simply wrong.

 

emergent33rd Problem: Dependencies are Important

Emergent Design is based on ignoring the big picture and designing only for the lower-level tasks (user stories). This may work well when these tasks are completely independent from each other, but in most software systems the dependencies are very important. If individual tasks are designed and implemented in the wrong order, ignoring their dependencies, the result may be a total mess, as illustrated on the picture on the right.

 

emergent24th Problem: Coordination is Essential

Emergent Design is based on the individual work of software developers, ignoring the need of a higher-level shared architecture and team-level cooperation. Each programmer is responsible for his own low-level tasks, which he tries to implement with minimal coordination with other team members. As illustrated on the picture on the right, this lack of coordination may cause a total deadlock.

 

The Solution: Define Clear Interfaces

screwdriver-setThe only solution to all the problems above is to define very clear interfaces among the modules in a software system. Of course, this definition of the interfaces must be done much before the developers start the implementation of the individual components. At the component level the low-level design may emerge, but at the system level there must be a pre-defined shared architecture. I give more information about my proposed approach for software design in my posts about Adaptable Design Up Front.

What do you think? Please share your experience in the comments below.

Posted in Adaptable Design, Agile, Software Architecture, Software Evolution | Tagged , , , | 11 Comments

Employability and the Three Stages of a Successful Career Path

career_pathI have many young friends who ask for my personal advice on long-term career strategies. Of course it seems that today almost everyone dreams about becoming an entrepreneur and having his own start-up company. But as a Plan B, if you can’t become a millionaire, it is quite nice to have a good job. Let’s start with some definitions:

Employability = The ability to find a new job.

Successful Career Path = A career path in which your Employability is always increasing.

Most professionals are afraid that as they get older it may become difficult for them to find a new job. Indeed, this happens quite frequently. For this reason, after a certain age most people start looking for more stable workplaces. Their goal is to reduce the risk that they will be fired.

However I don’t believe that this must be always the case. I think that, with proper planning and investment, many professionals can actually increase their Employability as they get older. Below I will describe the three stages of such a Successful Career Path.

1st Stage: You actively look for a new job

When you are a fresh graduate or when you do not have much experience, you must actively look for a new job. The less experience you have, the less differences when compared to other candidates. Thus your employability at this stage depends mostly on your performance at job interviews. And to be invited to job interviews, you must have a good CV. At this stage you don’t even have many friends working for other companies who may recommend you.

2nd Stage: You are approached by head-hunters

At this stage you are approached by head-hunters because you have special experience and skills which are highly in demand and hard to find. It will not help you if your skills are not in demand, or if there are many other professionals with similar experience. There is a reason they are called head-hunters and not hands-hunters. In general, head-hunters will offer you jobs which are better than your current one.

How to move from the 1st to the 2nd stage:

  • Work for companies which have a reputation of hiring only highly-qualified professionals. For example, if you land a job at Microsoft or Google it’s just a question of time until you start being approached by head-hunters offering you better jobs in other, smaller companies.
  • Make sure you are acquiring experience and skills in fields which are highly in demand and that do not have enough supply. For example, today it would be a smart move to work on projects related to Big Data using technologies such as Spark.
  • Keep a rich LinkedIn profile containing detailed and relevant information so that it will be easy for the head-hunters to find you.

3rd Stage: You are invited to join a company

At this stage you are invited to join a company because of your unique reputation, because you have proved achievements besides your experience and skills. You may be contacted by a friend or acquaintance who is a C-level executive or a VP in another company. Or you may be invited to be a co-founder in a seed-stage start-up. In any case you will be offered a leadership position with greater authority and responsibility than your current job.

Note that when you are approached by a head-hunter, they want someone like you. The head-hunter will say: “You have the exact profile we are looking for.”

In contrast, when you are invited to join a company, they want you. The invitation will be: “We want you to come work with us.”

When you are approached by a head-hunter you still need to make formal interviews, there may be other potential candidates, and you may not be hired.

When you are invited to join a company you still need to talk to other managers or co-founders so they will know you, but in general you are currently the only candidate for this role, and you will be hired unless someone really dislikes you.

How to move from the 2nd to the 3rd stage:

  • Build your reputation by making your achievements visible. This may include being a public speaker at conferences and meet-ups, writing a book or a professional blog. You must become a recognized authority in your field.
  • Build your network by connecting to all the people you have been in any kind of professional relationship. This may include friends from College, co-workers, customers and new people you should regularly meet in conferences and meet-ups.
  • Achievements mean results. People must be aware of your contribution to the companies you’ve worked for. They must know that you had a central role in the creation of new products that were successful in the market and that drove significant revenues to your company.

The number of opportunities that appear to you will depend on both your reputation and the size of your network. Even if you have a very big network, most people will not remember you if you don’t make your achievements visible. On the other hand, it is not very useful to have a very strong reputation in a small network. For example, you may be considered the topmost expert inside your company, but this would not help you get job offers in other companies.

In Summary:

Our goal should be to develop strategies to have a Successful Career Path in which our Employability is always increasing.

  • In the 1st stage you need a good CV. The only goal of the CV is to be invited for an interview when you apply for a job.
  • In the 2nd stage you need a good LinkedIn profile. The goal of your profile is to make it easy for head-hunters to find you and offer you a relevant job.
  • In the 3rd stage you need your name to be a brand. You need to have an active presence both virtually and in real-life events, so that people will remember you.

What do you think? Please share your opinion in the comments below.

Posted in Efficacy, Hiring, Social Networks | Tagged , , | 6 Comments

Please Stop the Work-Life Balance Bullshit

work-life-balanceA friend of mine was really unhappy, so he decided to have a conversation with a personal coach.

He complained: “My work is totally meaningless. I’m tired of doing every day the same tasks that are neither interesting nor challenging. My work does not use my experience neither my skills.”

The coach asked: “So why don’t you look for a new job? Why do you continue working for this company if you feel so miserable?”

My friend said: “Because I don’t have to work hard. I don’t have to stay long hours at the office. And then I can have lots of time to be with my wife and children.”

The coach replied: “Is it really worth it? Your wife has lots of time with an unhappy husband. Your children have lots of time with a Daddy who feels miserable.”

My friend quit his job. He now works harder but is happier. He now has less time to be with his wife and children, but this is quality time.

Work-Life Balance

The topic of Work-Life balance has become very popular recently, with many articles trying to help people to find the proper division of time and energy between their Work and the rest of their Lives.

This is one definition of Balance from a dictionary: “a state in which different things occur in equal or proper amounts or have an equal or proper amount of importance”. In other words, balance applies to the equilibrium of quantities or to the equilibrium of priorities.

However, as illustrated in the short story above, the real problem is not one of quantities nor priorities. The problem is Quality. If we hate our job, the solution is not to work less. Actually, by working less, we are drastically reducing the chances that we may do anything meaningful. People who are really passionate about their careers have no problem of working hard.

This is best expressed in a quote by Confucius:
“Choose a job you love, and you will never have to work a day in your life.”

Life Quality

Another problem with the proponents of Work-Life Balance is the assumption that it’s easy to enjoy the time we are not working. As if all the time we are not in the office could be considered leisure time, in which we have fun with our family and friends, we practice sports and get distracted with our hobbies.

However the sad reality for lots of people is that their Lives outside of Work are not so much enjoyable than their time in the office. Ask people who have to take care of small children or old parents (or both) how easy are their Lives. Ask a couple which is having fights and considering divorce if the solution to their problems is to simply spend more time together, without improving their relationship.

So again the issue is not one of finding the Balance between Work and Life. The goal is improving the Quality of both our Work and our Life. And improving the quality of the time we spend with our family and friends is not necessarily easier than being happy with our job.

As a friend once said to me:
“I have fun at the office. My work starts when I arrive at home.”

In Summary

Our goal should not be the Balance between Work and Life. Our goal should be the constant improvement of the Quality of both our Work and our Life.

Think about it, your problem never is work-life balance:

  • If you hate your job and you hate your life – Your problem is not work-life balance.
  • If you hate your job but you love your life – You need to find a job you love.
  • If you love your job but you hate your life – You need some counselling.
  • If you love your job and you love your life – Congratulations! You don’t care about work-life balance.
Posted in Efficacy | Tagged | Leave a comment

On Anzeneering, Pride and the Definition of Done (DoD)

wedwardsdeming378714

The concept of Anzeneering was created by Joshua Kerievsky, CEO of Industrial Logic and author of the book “Refactoring to Patterns“. It is derived from the Japanese word “anzen” which means “safety”. According to Joshua:

“Anzeneers protect people by establishing anzen in everything from relationships to workspaces, codebases to processes, products to services.”

“Anzeneers protect: Software makers from poor working conditions, including hostile relationships, death marches, burnout, hazardous software (poorly designed, highly complex, deeply defective code, lacking even basic safety nets like automated builds or automated tests), insufficient testing infrastructure, poor lighting, uncomfortable seating, excessive work hours and insufficient exercise.”

I think one important aspect of Anzeneering is the protection of software developers against the production of low-quality code. Note that the emphasis here is not on the negative business consequences of having poor software systems. Anzeneering focuses on the negative impact on the developers themselves. When programmers are forced to produce bad code, they cannot be proud of their work, and will suffer because of that.

Frank Sonnenberg, award-winning author of the book “Managing With a Conscience“, addresses similar issues in his article “If You’re Not Proud, You’re Not Done“. According to Frank:

“If you don’t put your heart into your activities, if you hand in incomplete work as finished, if you don’t do your best every time you start something, then you’re doing yourself a tremendous disservice. The truth is, if you’re not proud of what you do, you’re not done.”

“If you don’t do your best, you’re only developing bad habits, damaging your reputation, and letting your team down.”

And this brings us to the concept of Definition of Done (DoD) in Agile software development.

Definition of Done (DoD)

In Agile software development, the Definition of Done (DoD) are the acceptance criteria that must be met for a new feature (user story) to be released. Here is the DoD definiton by the Agile Alliance:

“The team agrees on, and displays prominently somewhere in the team room, a list of criteria which must be met before a product increment ‘often a user story’ is considered ‘done’. Failure to meet these criteria at the end of a sprint normally implies that the work should not be counted toward that sprint’s velocity.”

A concrete example is provided by Mike Cohn, Agile consultant and author of the book “Succeeding with Agile: Software Development Using Scrum“, in his article “Multiple Levels of Done“:

“A typical DoD would be something similar to:

  • The code is well written. (That is, we’re happy with it and don’t feel like it immediately needs to be rewritten.)
  • The code is checked in. (Kind of an “of course” statement, but still worth calling out.)
  • The code was either pair programmed or peer reviewed.
  • The code comes with tests at all appropriate levels. (That is, unit, service and user interface.)
  • The feature the code implements has been documented in any end-user documentation such as manuals or help systems.”

In Mike’s example above it is not sufficient for the feature to be implemented, what is normally called “fully-functional”. The code may be complete, correct and still have lots of Technical Debt. Thus Mike says “the code is well written”.

Conclusion

Following the Anzeneerign approach, I would add to the DoD that the developer must be proud of his work. If a programmer is not proud of the code he wrote, he is not done, even if his code satisfies all the other requirements.

I want to end with a quote by Ed Yourdon, a pioneer in the field of Software Development who unfortunately left us last month at age 71. In his book “The Rise and Resurrection of the American Programmer“, Yourdon writes:

“If you think your management doesn’t know what it’s doing or that your organisation turns out low-quality software crap that embarrasses you, then leave.”

Posted in Agile, Efficacy, Psychology of Programming | Tagged , , | 2 Comments

Beware the Lean Feedback Loop: Over-fitting to Early Adopters

the-lean-startup-bookToday many software startups follow the ideas introduced by Eric Ries in his best-selling book “The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses”. Ries is responsible for the popularization of the concept of the MVP: “The Minimum Viable Product is that version of a new product which allows a team to collect the maximum amount of validated learning with the least effort.” The MVP should have the core features which allow it to be deployed to some real customers in order to get initial feedback, but not more.

build-measure-learnThe MVP is used in the context of the “build-measure-learn” feedback loop. From the Lean Startup site: “The first step is figuring out the problem that needs to be solved and then developing a minimum viable product (MVP) to begin the process of learning as quickly as possible. Once the MVP is established, a startup can work on tuning the engine. This will involve measurement and learning and must include actionable metrics that can demonstrate cause and effect question.”

As Ries explains in one of his blog posts: “In other words, the Minimum Viable Product is a test of a specific set of hypotheses, with a goal of proving or disproving them as quickly as possible. One of the most important of these hypotheses is always: what will the customer care about? How will they define quality?”

This application of the build-measure-learn feedback loop is proposed by Eric Ries as the most effective way to learn about what the customers really want. There is a basic assumption that the first users of our product may provide us valuable data from which we will be able to extrapolate the needs of all other users. But there is another very important book that should make us question this assumption.

Crossing the Chasm

crossing the chasmGeoffrey A. Moore is the author of the best-selling book “Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers“, in which he describes the five main customer segments in a technology startup’s market: innovators, early adopters, early majority, late majority and laggards.

Innovators: “Pursue new technology products aggressively. They sometimes seek them out even before a formal marketing program has been launched.”

Early Adopters: “Buy into new product concepts very early in their life cycle… find it easy to imagine, understand, and appreciate the benefits of new technology.”

Early Majority: “Are driven by a strong sense of practicality… want to see well-established references before investing substantially.”

Late Majority: “Wait until something has become an established standard, and even so they want to see lots of support.”

Laggards: “Don’t want anything to do with new technology.”

These customer segments correspond to the stages in the Technology Adoption Life Cycle: the innovators are the first to buy a new product, followed by the early adopters, and so on. However, Moore claims that there is a chasm between the early adopters and the early majority, which is illustrated in the picture below.

chasm

According to Moore, there is a huge difference between the early adopters and the early majority:

Early adopters are buying some kind of change agent. “By being the firsts to implement this change in their industry, the early adopters expect to get a jump on the competition… They expect a radical discontinuity between the old ways and the new… They also are prepared to bear with the inevitably bugs and glitches that accompany any innovation just coming to market.”

In contrast, the early majority want a productivity improvement. “They want evolution, not revolution. They want technology to enhance, not overthrow, the established ways of doing business. And above all, they do not want to debug someone’s else product… They want it to work properly and to integrate appropriately with their existing technology base.”

Over-fitting to Early Adopters

In my opinion it is clear that if we adopt the build-measure-learn feedback loop while in the early adopters stage, we are incurring the risk of learning about user needs and preferences of a small segment which is very different from the main market, represented by the early and late majority segments.

Another way to say that is that we are using the build-measure-learn feedback loop in order to build a model to predict the behavior of our users. In this case we are incurring the risk of over-fitting this model to the particular user segment which is currently providing its feedback.

Definition of over-fitting: “A modelling error which occurs when a function is too closely fit to a limited set of data points. Over-fitting the model generally takes the form of making an overly complex model to explain idiosyncrasies in the data under study. In reality, the data being studied often has some degree of error or random noise within it. Thus attempting to make the model conform too closely to slightly inaccurate data can infect the model with substantial errors and reduce its predictive power.”

Note that the feedback provided by the early adopters has the two main problems that appear in the definition above:

  • It is a “limited set of data points”: the early adopters are a small and very specific segment, and not a random sample of our potential users.
  • It “has some degree of error or random noise within it” since the behavior of the early adopters is different from the majority of the users.

In Summary

We should be very careful when using the build-measure-learn feedback loop while in the early adopters stage. Possible ways to reduce the risks of over-fitting to this very specific user segment require understanding the differences between the early adopters and the majority of our potential users.

Posted in Agile, Lean Development | Tagged , | 3 Comments