Data Science: Set Similarity Metrics

Posted on February 27, 2019 by Hayim Makabee

In this article I would like to present several metrics to calculate the similarity between sets of items. I’ve been analyzing diverse metrics as part of an investigation I’m doing to improve the targeting of digital advertising campaigns. These set similarity metrics are very useful to address the problem of audience expansion.

The basic idea is: we start with a set of users that have engaged with the ad, for example clickers. Then we try to find other similar sets of users that we can target. We have the expectation that, because of their similarity, the targeted users will also engage positively with the ad. These similar users become our expanded audience.

Jaccard Similarity

The Jaccard Similarity is defined as the size of the intersection divided by the size of the union of the sets.

Given two sets, A and B, the Jaccard Similarity is defined as:

The Jaccard Similarity ranges between zero and one.

Also called: Jaccard index, Intersection over Union

For more information: Jaccard Similarity

Sorensen Coefficient

The Sorensen Coefficient equals twice the number of elements common to both sets divided by the sum of the number of elements in each set.

Given two sets, X and Y, the Sorensen Coefficient is defined as:

The Sorensen Coefficient ranges between zero and one.

Also called: Sorensen–Dice index, Sorensen index, Dice’s coefficient

For more information: Sorensen Coefficient

Tversky Index

For sets X and Y, the Tversky Index is given by:

Note that  are parameters of the Tversky Index.

The Tversky Index ranges between zero and one.

The Tversky Index can be seen as a generalization of the Jaccard Similarity and the Sorensen Coefficient:

  • Setting  produces the Jaccard Similarity.
  • Setting  produces the Sorensen Coefficient.

Tversky measures with  are of special interest.

For more information: Tversky Index

Overlap Coefficient

The Overlap Coefficient is defined as the size of the intersection divided by the size of the smaller of the two sets.

For sets X and Y, the Overlap Coefficient is given by:

If set X is a subset of Y or the converse then the Overlap Coefficient is equal to 1.

Also called: Szymkiewicz–Simpson Coefficient

For more information: Overlap Coefficient

Posted in Data Science, Machine Learning | Tagged , | 5 Comments

My Workshop on Software Architecture for Agile Development

Posted on January 29, 2019 by Hayim Makabee

Yesterday I had the pleasure to present a full-day workshop about Software Architecture for Agile Development. The audience was composed of software architects, team leaders and senior software developers. This was the description of the workshop:

This workshop will be dedicated to the topic of Software Architecture in the context of Agile Development. We will answer the question: “How much Design Up Front should be done in an Agile project?” Hayim will present his approach of Adaptable Design Up Front (ADUF), describing its rationale, applications in practice and comparison to other approaches such as Emergent Design. He will explain why adaptability is essential for the development of complex software systems using Agile methods. We will also cover related approaches such as the Adaptive Object Model and Evolutionary Architecture. The concepts will be illustrated through practical software architectures such as micro-services, and examples of real software systems that were developed applying these principles. The workshop will also include exercises on the definition and evolution of the design of an interesting system.

What you will learn:

  • The Architectural challenges of modern software development
  • The need of Adaptability in the context of Agile development
  • The approach of Adaptable Design Up Front (ADUF)
  • The Adaptive Object Model (AOM)
  • The approach of Evolutionary Architectures

These are some nice pictures of the workshop participants doing an exercise:

 

Posted in Adaptable Design, Agile, Software Architecture, Software Evolution | Tagged , , , | Leave a comment

Doing difficult things for the first time

Posted on December 26, 2018 by Hayim Makabee

If I summarize the year of 2018 I can say that I tried to do many difficult things, and many of these things I have tried to do for the first time.

This is the reality for most entrepreneurs. When you are the founder of a startup company you have to do many things that you have never done before, and many of these things are difficult.

Actually, most things you are doing as an entrepreneur would have been difficult even for someone who has experience doing them:

  • Hiring the right people
  • Doing market research
  • Understanding your competition
  • Developing your go-to-market strategy
  • Defining your MVP
  • Writing a Business Plan
  • Talking to investors
  • Acquiring your first users
  • Contacting potential customers

Naturally, by doing these hard things for the first time, there will be many failures. You will make many mistakes. You will have to learn from these mistakes and adjust your plans very fast.

You will also discover what are the things that you like to do and what are the things that you don’t like to do. Then you will have to learn how to delegate tasks and trust your partners.

But you will also have some good surprises. You will experience success in activities that you have never done before. You will discover that you have strengths that you didn’t know before.

“Doing the difficult things that you’ve never done awakens the talents you never knew you had.” — Robin S. Sharma

So I wish that in 2019 I will continue to do difficult things, and I will continue to learn how to do them. Happy New Year!

Quotefancy-183740-3840x2160

Posted in Startups | Tagged | 4 Comments

Networking is Overrated: Focus on Your Reputation

Posted on September 28, 2018 by Hayim Makabee

Network of peopleI’ve observed in recent years that many professionals became obsessed with networking. These are the people who constantly participate in conferences, meetups and other kinds of networking events. They like to collect business cards and feel excited whenever they connect to other professionals on LinkedIn.

I must say that in the past I also was a networking enthusiast. However after investing lots of time and effort into growing my network, I finally understood that this approach was not producing the results that I expected. My goal was to have more opportunities, but instead I was losing my focus.

It is very easy to meet new people, exchange business cards and create a connection. However, it is much more difficult to keep in touch with all these people over time. The natural tendency is to simply stop having any kind of interaction with our new contacts until we completely forget about each other.

To confirm this point, we just need to do a simple exercise. Do you have a business cards collection? Please select one of them randomly and answer these questions:

  • Do you remember where did you meet this person? Do you remember in which circumstances did you meet?
  • Did someone introduce you to this person? Do you remember who introduced you?
  • Do you remember the face of this person? Would you be able to recognize him/her in the street?
  • Did you ever communicate with this person? Do you remember when was your last interaction?
  • If you would see the name of this person somewhere else, would you remember that you once met and exchanged business cards?

If you don’t keep a business cards collection, you could ask the same questions about your contacts on LinkedIn. Actually, I’ve witnessed more than once the following situation which is quite funny:

Two people meet on a networking event. After a few minutes of conversation, one of them says: “Let’s connect on LinkedIn! What is your name?” Then he opens his LinkedIn app and searches for the profile of his new contact… when he discovers: “Oh! We are already connected!”

If this has happened to you, you can be sure that you have been doing too much networking recently.

Focus on Your Reputation

online-reputation-managementIn general the goal of networking is to create new opportunities. These may be business opportunities, partnership opportunities or job opportunities. But in my opinion what really creates new opportunities is our reputation.

Having a good reputation means that:

  • People will remember us. They will remember us for many years since we had our last interaction.
  • People will recommend us. They will introduce us to their own contacts whenever they think we may contribute.
  • People will constantly offer us new opportunities. They will invite us when they have a job opening, or when they need a partner or an adviser.

In other words, if you have a good reputation your name will be present even if you are not physically present. People will mention you in the right context, at the right places and at the right times.

There are many good articles and books about how to build your reputation. It is certainly something that requires lots of time, effort and dedication. But it will bring you much better results than meeting random people in networking events.

What do you think? Do you agree? What has been your own experience? Please share with us in the comments below.

Posted in Efficacy, Social Networks | Tagged , | 1 Comment

Do we need more time or do we need more energy?

Posted on July 30, 2018 by Hayim Makabee

Busy people in general apply a simple formula: invest extra hours to get more work done. However I think there is a flawed assumption here: that the quantity of work accomplished grows linearly with the amount of time invested. Besides that, I also believe that in general there is a decrease in the quality of work done in these extra hours.

I’ve witnessed several times people forcing themselves to work additional hours when clearly they were not being productive anymore. I also observed tired people making terrible mistakes or making very bad decisions. The time required to fix these mistakes can actually eliminate any benefit obtained from working extra hours.

In my opinion people should focus on Energy instead of Time. We should ask ourselves: do I have the right energy level to produce this work? And if not, how can I obtain more energy, so that I will be able to produce the best possible work in my available time?

I don’t intend to prescribe here any formula on how people can get more energized. I think this is a very personal question, and each individual needs to find his/her own ways to obtain the energy they need to produce their work. I believe that following this approach people can be more productive while avoiding any need to work extra hours.

Energy Level

Posted in Efficacy | Tagged | Leave a comment

The Myth of Innovation and the First-Mover Advantage

Posted on June 29, 2018 by Hayim Makabee

Innovation_300x300Lots of people today dream about having their own startup company. These aspiring entrepreneurs are constantly looking for ideas, and in general they believe that a good startup idea must be very innovative, and better yet if they can be the first-mover in a new market. However if we analyze the most successful hi-tech companies today, we can observe that many of them were not so innovative, and most of them were not first-movers. A good example are Instant Messaging platforms.

WhatsApp is the leading Instant Messaging app today. It was created in 2009, and by the end of 2013 it already had 400 million active users. WhatsApp was acquired by Facebook in February 2014 for the amazing value of U$19 billion. By February 2018, WhatsApp already had a user base of over one and a half billion people globally. This is certainly a great success story! But can we say that WhatsApp was really innovative? Did it have any first-mover advantage?

Do you remember ICQ? ICQ was created by the Israeli company Mirabilis in 1996. That’s 22 years ago! It was acquired by AOL on June 1998 for U$400 million, which was the highest price ever paid to purchase an Israeli startup company. At its peak around 2001, ICQ had more than 100 million accounts registered. ICQ provided the same functionality that is provided by WhatsApp today, but of course ICQ ran on a desktop, and not on mobile devices.

So, can we say that WhatsApp’s innovation was providing Instant Messaging (IM) over mobile devices? Certainly not! Actually, I myself worked for two different Israeli startup companies that provided IM on mobile. The first was NomadIQ (latter acquired by OmniSky), and the second was Followap (which was acquired by Neustar).

NomadIQ was founded in Jerusalem in 1999. We developed IM and Location Based services for handheld devices such as the Palm and the PocketPC. These devices, which were also called Personal Digital Assistants (PDAs), were very similar in shape and size to modern smartphones. But they had to be connected to external modems in order to access the Internet. NomadIQ was acquired by OmniSky in January 2001 for U$40 million. Below you can watch a promotional video explaining our services at that time.

Followap was founded in Haifa in 1999. We developed IM services for the first mobile phones, which are also known as “feature phones”. At this time Nokia was the leading mobile phone manufacturer. The screen sizes were very small, and people had to write textual messages using the numeric keyboard. Even so, Followap services reached more than 200 million subscribers. Followap was acquired by Neustar in November 2006 for U$140 million. You can watch below a video explaining Followap’s services.

Now, after you have watched these videos, I ask again: Can we say that WhatsApp was really innovative? Did it have any first-mover advantage?

I think the answer to both questions is “no”. So perhaps what startup companies really need to be successful is the right timing. Companies such as NomadIQ and Followap had the right vision, but they were simply ahead of their time.

Posted in Israel, Startups | Tagged , | 1 Comment

Data Science vs. Software Engineering

Posted on May 30, 2018 by Hayim Makabee

Data ScienceThe work of a Software Engineer is to analyze a problem, think about a good solution, design it, implement it and then test it. Software Engineers do problem-solving. The work of the Software Engineer ends when he finishes implementing the solution for the problem.

The work of a Data Scientist is to make good predictions or classifications based on past behavior. The Data Scientist does not create solutions, he creates models that are able to generate such predictions or classifications. The Data Scientist then tries to optimize these models, so that the new models will generate better predictions or classifications than the previous models. The work of the Data Scientist never ends. In particular there is always more work when behavior is changing over time.

It may take lots of time until the Data Scientist is able to generate his/her first good model. The Data Scientist needs to analyze the data, clean the data, generate new features, select the best features, train models, try different Machine Learning algorithms, try different parameters for each Machine Learning algorithm and measure diverse offline metrics. Each one of these steps may require several weeks of work or even months.

Thus, the work of the Data Scientist should not have hard deadlines. It is very difficult for a Data Scientist to provide good estimates of when he/she may have a model that is good enough to be deployed in production.

Implementation vs. Experimentation

ExperimentSoftware Engineering teams are mostly busy implementing new features. Data Science teams are mostly busy running experiments.

A feature has a clear functionality being provided, and in general it is a solution to a problem. By the end of the implementation, the new feature becomes part of the system. The goal of implementing features is to add functionality to the system.

An experiment is a way to check a hypothesis. Depending on the results of the experiment, we may prove that the hypothesis is true or false. If we prove that the hypothesis is false, this does not mean that the experiment has failed. Our goal was to learn and we have learned. Based on what we have learned, we can make a new hypothesis and plan a new experiment.

It is possible that in most experiments we will prove that the hypothesis is false. This does not mean that we are wasting our time. It is natural that when facing new and complex problems we need to try many different approaches until we find one that works.

Data Science teams are like a laboratory. If we measure the progress of a Data Science team in the same way we measure the progress of the Software Engineering teams, we may reach the wrong conclusion that the Data Science team is not productive enough.

Posted in Data Science, Machine Learning, Software Engineering | Tagged , , | Leave a comment

Priorities and Maslow’s hierarchy of needs

Posted on April 30, 2018 by Hayim Makabee

Recently, when I was thinking about the priorities of a project I’m working on, I used as a source of inspiration the Maslow’s hierarchy of needs. This hierarchy is usually depicted as a pyramid, as you can see in the image below. According to this theory, human beings have some basic needs that must be addressed before anything else. I believe that this is true also in the case of a project: we must be able to identify which requirements are absolutely necessary, and which ones may wait until the basic needs were satisfied.

Maslow

Posted in Efficacy, Requirements Specification | Tagged , | 1 Comment

KashKlik Pitch at TheHive Demo Day

Posted on March 30, 2018 by Hayim Makabee

DemoDayLast week KashKlik presented at the Demo Day, the event that concludes TheHive accelerator by Gvahim. There were more than a hundred people in the audience, including a panel of investors that judged the startups that participated at TheHive together with us.

KashKlik Team

KashKlik team during the event (from left to right): Ariel, Jacob, Galina and Hayim.

Below is the video of my pitch, followed by questions from the panel (in English):

From TheHive’s site:

TheHive was created in 2011 as the entrepreneurial branch of Gvahim, a nonprofit organization aiming at providing new immigrants with the tools to succeed in the Israeli labour market. Alongside Gvahim’s career programs open to a wide range of sectors, and special channels dedicated to medical professions and software engineers, TheHive responds to a specific tendency among new immigrants, who are more prone to launch into entrepreneurial adventure than the average population.

The accelerator is a multinational entrepreneurial environment where Olim, returning citizens and Israeli entrepreneurs all operate in a shared working space, with a wide mix of talents and ideas, to create a truly international atmosphere.

The 5-month Start-up accelerator program includes:

  • A community of international entrepreneurs who learn from, motivate and support each other.
  • Participation in the Google Launchpad: a one-week bootcamp at the Google Campus in Tel Aviv.
  • Access to capital: Connections to top investment firms, VCs and angels.
  • Top-tier mentors.
  • Office space in a creative co-working space in its two centers: Tel Aviv University and Ashdod.
  • Training, networking and office hours with experts.

 

Posted in KashKlik, Startups | Tagged , | Leave a comment

Everything is an experiment: on decisions and retrospectives

Posted on February 23, 2018 by Hayim Makabee

DecisionsIn decision-making, we try as much as we can to impact future results based on our previous experience. But every decision is also based on the assumption that its positive consequences will be greater than any possible negative outcome. Therefore, we should treat decisions as experiments. We should always preserve the option of reconsidering our decisions in the future, when we will be able to measure their real impact.

When making decisions, we should also prepare the tools and mechanisms that will allow us to collect feedback and evaluate the obtained results. In the methodology of Agile software development there are several such tools, including Burndown charts and Velocity charts. These charts help us measure the progress in the delivery of new features, providing immediate feedback that indicates if the execution is according to our plans.

Another fundamental concept in Agile methodologies are the Retrospectives. They are a planned opportunity for teams to analyze their own performance and discuss if there is any need to change the current processes. Thus we never assume that we are already following the most effective process and nothing needs to be changed. An integral part of the Agile mindset is the focus on continuous improvement, always looking for opportunities to deliver better results.

For example, a software development team may decide to adopt Test-Driven Development (TDD). The expected results may include improved quality and better design. But this decision should be considered an experiment. After some period of time, the team should verify if indeed TDD is having the intended consequences, and if there is no drastic negative impact such as a big increase on the time required to develop and deploy new features.

By treating decisions as experiments, and by having the tools to measure their results, we can be sure we are making progress in the right direction.

Posted in Agile, Efficacy, Software Quality | Tagged , , | 1 Comment