Data Science vs. Software Engineering

Data ScienceThe work of a Software Engineer is to analyze a problem, think about a good solution, design it, implement it and then test it. Software Engineers do problem-solving. The work of the Software Engineer ends when he finishes implementing the solution for the problem.

The work of a Data Scientist is to make good predictions or classifications based on past behavior. The Data Scientist does not create solutions, he creates models that are able to generate such predictions or classifications. The Data Scientist then tries to optimize these models, so that the new models will generate better predictions or classifications than the previous models. The work of the Data Scientist never ends. In particular there is always more work when behavior is changing over time.

It may take lots of time until the Data Scientist is able to generate his/her first good model. The Data Scientist needs to analyze the data, clean the data, generate new features, select the best features, train models, try different Machine Learning algorithms, try different parameters for each Machine Learning algorithm and measure diverse offline metrics. Each one of these steps may require several weeks of work or even months.

Thus, the work of the Data Scientist should not have hard deadlines. It is very difficult for a Data Scientist to provide good estimates of when he/she may have a model that is good enough to be deployed in production.

Implementation vs. Experimentation

ExperimentSoftware Engineering teams are mostly busy implementing new features. Data Science teams are mostly busy running experiments.

A feature has a clear functionality being provided, and in general it is a solution to a problem. By the end of the implementation, the new feature becomes part of the system. The goal of implementing features is to add functionality to the system.

An experiment is a way to check a hypothesis. Depending on the results of the experiment, we may prove that the hypothesis is true or false. If we prove that the hypothesis is false, this does not mean that the experiment has failed. Our goal was to learn and we have learned. Based on what we have learned, we can make a new hypothesis and plan a new experiment.

It is possible that in most experiments we will prove that the hypothesis is false. This does not mean that we are wasting our time. It is natural that when facing new and complex problems we need to try many different approaches until we find one that works.

Data Science teams are like a laboratory. If we measure the progress of a Data Science team in the same way we measure the progress of the Software Engineering teams, we may reach the wrong conclusion that the Data Science team is not productive enough.

Posted in Data Science, Machine Learning, Software Engineering | Tagged , , | Leave a comment

Priorities and Maslow’s hierarchy of needs

Recently, when I was thinking about the priorities of a project I’m working on, I used as a source of inspiration the Maslow’s hierarchy of needs. This hierarchy is usually depicted as a pyramid, as you can see in the image below. According to this theory, human beings have some basic needs that must be addressed before anything else. I believe that this is true also in the case of a project: we must be able to identify which requirements are absolutely necessary, and which ones may wait until the basic needs were satisfied.

Maslow

Posted in Efficacy, Requirements Specification | Tagged , | Leave a comment

KashKlik Pitch at TheHive Demo Day

DemoDayLast week KashKlik presented at the Demo Day, the event that concludes TheHive accelerator by Gvahim. There were more than a hundred people in the audience, including a panel of investors that judged the startups that participated at TheHive together with us.

KashKlik Team

KashKlik team during the event (from left to right): Ariel, Jacob, Galina and Hayim.

Below is the video of my pitch, followed by questions from the panel (in English):

From TheHive’s site:

TheHive was created in 2011 as the entrepreneurial branch of Gvahim, a nonprofit organization aiming at providing new immigrants with the tools to succeed in the Israeli labour market. Alongside Gvahim’s career programs open to a wide range of sectors, and special channels dedicated to medical professions and software engineers, TheHive responds to a specific tendency among new immigrants, who are more prone to launch into entrepreneurial adventure than the average population.

The accelerator is a multinational entrepreneurial environment where Olim, returning citizens and Israeli entrepreneurs all operate in a shared working space, with a wide mix of talents and ideas, to create a truly international atmosphere.

The 5-month Start-up accelerator program includes:

  • A community of international entrepreneurs who learn from, motivate and support each other.
  • Participation in the Google Launchpad: a one-week bootcamp at the Google Campus in Tel Aviv.
  • Access to capital: Connections to top investment firms, VCs and angels.
  • Top-tier mentors.
  • Office space in a creative co-working space in its two centers: Tel Aviv University and Ashdod.
  • Training, networking and office hours with experts.

 

Posted in KashKlik, Startups | Tagged , | Leave a comment

Everything is an experiment: on decisions and retrospectives

DecisionsIn decision-making, we try as much as we can to impact future results based on our previous experience. But every decision is also based on the assumption that its positive consequences will be greater than any possible negative outcome. Therefore, we should treat decisions as experiments. We should always preserve the option of reconsidering our decisions in the future, when we will be able to measure their real impact.

When making decisions, we should also prepare the tools and mechanisms that will allow us to collect feedback and evaluate the obtained results. In the methodology of Agile software development there are several such tools, including Burndown charts and Velocity charts. These charts help us measure the progress in the delivery of new features, providing immediate feedback that indicates if the execution is according to our plans.

Another fundamental concept in Agile methodologies are the Retrospectives. They are a planned opportunity for teams to analyze their own performance and discuss if there is any need to change the current processes. Thus we never assume that we are already following the most effective process and nothing needs to be changed. An integral part of the Agile mindset is the focus on continuous improvement, always looking for opportunities to deliver better results.

For example, a software development team may decide to adopt Test-Driven Development (TDD). The expected results may include improved quality and better design. But this decision should be considered an experiment. After some period of time, the team should verify if indeed TDD is having the intended consequences, and if there is no drastic negative impact such as a big increase on the time required to develop and deploy new features.

By treating decisions as experiments, and by having the tools to measure their results, we can be sure we are making progress in the right direction.

Posted in Agile, Efficacy, Software Quality | Tagged , , | 1 Comment

To Increase Productivity, Avoid Multitasking

Most people look for ways to increase their productivity. Many of them believe that they will be more productive by handling several tasks simultaneously, what is known as “multitasking”. But this is actually a trap! By trying to perform many tasks in parallel we just lose our focus and reduce the quality of our work. To be more productive we should actually avoid multitasking!

Multitasking

Posted in Efficacy | Tagged | 1 Comment

Digital Nomads: The Upside of Being Always Connected and Working from Anywhere

Today lots of people complain about being always connected to work. A recent study shows that:

  • 45 percent of workers feel obligated to respond to emails after hours.
  • 42 percent of employees feel obligated to check in with work while on vacation.
  • 47 percent feel guilty if they don’t work when sick, either on site or from home.

This situation certainly has many disadvantages, including the increase in work-related stress and the reduction of leisure time people spend with their family and friends. However I think each person has also the opportunity to enjoy the advantages of being always connected and able to work from anywhere.

The extreme example are the Digital Nomads, people who travel while they continue to work with clients or employers. They are able to visit new places and learn about other cultures and lifestyles, as if they were tourists, while at the same time they continue working remotely an making money.

But it is also possible to be a Digital Nomad part of our time. In the same way that we can travel for a business-related conference and enjoy our free time for tourism, we can travel mostly for tourism and enjoy some of our time to do work, without feeling guilty.

It may be very nice also to work near our home instead of working from home. We can work from places that inspire us, such as a beach, a park, a restaurant with a nice view or even a beautiful library. Such changes in environment probably have a positive impact in our creativity.

So I suggest this as a new-year-resolution for 2018: instead of complaining about being always connected to work, let’s find new ways to feel better while working remotely.

Happy New Year!

20171219_134353

Posted in Efficacy | Tagged | Leave a comment

Temporary Solutions: Technical Debt in Pictures

Software developers are constantly told that they should avoid Technical Debt. But very frequently it is difficult to resist the temptation. Why not implement a good-enough temporary solution that satisfies all the functional requirements? The main problem is that very often this solution ignores completely the non-functional requirements. It is not efficient, nor robust, nor maintainable, nor scalable. I think this problem may be very nicely illustrated with the pictures below.

Car Dashboard1) Functional requirement:

The dashboard must have a knob to control the air-conditioning.

 

 

 

car window cleaner2) Functional requirement:

The back-window must have a wiper.

 

 

 

lock3) Functional requirement:

The door must have a lock.

 

 

 

mirror4) Functional requirement:

The car must have a side-mirror.

 

 

 

 

tire5) Functional requirement:

The car must have four tires.

Posted in Requirements Specification, Software Quality, Technical Debt | Tagged , , | 1 Comment