One of my favorite topics during my recent studies at ThePowerMBA was the Lean Startup. I read the Lean Startup book by Eric Ries several years ago, and since then I have been trying to follow its main principles. In my opinion his methodology is applicable not only to startup companies, but also to any project which is very innovative and involves a high degree of risk.

One of the fundamental ideas in the Lean Startup approach is that we must plan experiments to test our hypotheses. This is called the Build-Measure-Learn Feedback Loop. 

Build-Measure-Learn Feedback Loop

Quoting from the Lean Startup principles page:

“A core component of Lean Startup methodology is the build-measure-learn feedback loop. The first step is figuring out the problem that needs to be solved and then developing a minimum viable product (MVP) to begin the process of learning as quickly as possible. Once the MVP is established, a startup can work on tuning the engine. This will involve measurement and learning and must include actionable metrics that can demonstrate cause and effect question.”

In the quote above, I emphasized the concept “actionable metrics”. One of the main mistakes that entrepreneurs do when trying to apply the idea of the feedback loop is that instead of measuring actionable metrics they measure vanity metrics.

Actionable vs. Vanity Metrics

Here is a very nice definition of Vanity Metrics from the Tableau website:

“Vanity metrics are metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies. These metrics are exciting to point to if you want to appear to be improving, but they often aren’t actionable and aren’t related to anything you can control or repeat in a meaningful way.”

In contrast, Actionable Metrics are useful to make decisions and adapt our strategies, because they are related to things we can control and repeat.

Here is the definition from the Wikipedia:

“Actionable metrics can lead to informed business decisions and subsequent action.”

And here is the definition from Ash Maurya:

“An actionable metric is one that ties specific and repeatable actions to observed results.”

In the sections below we will analyze the main differences between vanity metrics and actionable metrics. I will illustrate these differences with relevant examples of both kinds of metric that are frequently used in startup companies.

Measuring Cause and Effect

In the build-measure-learn feedback loop our main goal is to learn. In the early-stages of a startup company we want to reach product-market fit.

Vanity Metrics are not good for learning because they are not necessarily a sign of a better product-market fit.

Actionable Metrics are good for learning because they are a clear sign of having stronger product-market fit.

For example, the number of followers on a social network or the number of visitors on a website are vanity metrics. A company may purchase followers to its page on Facebook. In the same way, a company may drastically increase the number of visitors in its website using paid campaigns on Google AdWords. However, these metrics are not indications of product-market fit. We can increase the values of these metrics by simply increasing our marketing spend.

In contrast, the conversion rate or the activation rate are actionable metrics. In an early-stage startup company, we can gradually learn how to increase the conversion rate. We can plan a series of experiments and test different hypotheses of how to improve the activation rate. If a company is able to increase these metrics, it is clearly getting closer to product-market fit.

What does the Metric Represent?

When using metrics, it’s very important to understand what we are measuring. In the context of the build-measure-learn feedback loop we want to focus on metrics that help us validate our product and our business model.

In general, Vanity Metrics are used to measure the size of the business.

On the other hand, Actionable Metrics are used to measure individual behaviour.

For example, the number of leads or the number of apps downloaded are vanity metrics. They certainly provide an indication of the size of our operation, but they do not help us validate our business model. We certainly want to increase the number of leads or have a higher number of apps downloaded, but this growth does not necessarily indicate product-market fit. Again, a company may increase its number of leads or have a higher number of apps downloaded just by increasing its marketing spend.

In contrast, the Cost per Acquisition and the Customer Life-Time Value (CLTV) are actionable metrics. An early stage startup company may try different strategies to reduce its cost per acquisition. In other words, this company may plan experiments to learn how to decrease its cost per acquisition. In the same way, a company may adopt several tactics to increase the CLTV. We may define a hypothesis of how to improve the CLTV and test it in practice. By working on the refinement of the Cost per Acquisition and the CLTV we are gradually validating our business model and getting closer to product-market fit.

Measuring Aggregates and Ratios

As we have observed above, in general the vanity metrics are used to measure the size of the business while the actionable metrics are used to measure individual behavior. This is reflected in the respective types of these metrics.

Vanity Metrics: In general, measure gross quantities.

Actionable Metrics: In general, measure ratios and unit economics.

Additional examples of vanity metrics are the  total number of registered users and the total revenues. These measures are important, but they are an aggregation. This kind of metrics alone are not useful to provide an indication of the “health” of our business. Actually, it is very common that a company may be increasing these totals at the same time that it is deteriorating other metrics. For example, an early stage startup company may increase the total number of registered users by increasing the user acquisition cost (which of course is undesirable), or it may increase its total revenues at the same time that it is decreasing the average CLTV.

Compare this to some additional examples of actionable metrics, such as the Repeat Rate and the Churn Rate, both of which are ratios and not aggregates. The Repeat Rate measures the ratio of users who are coming back to use our services or buy our products again. The Repeat Rate is an indication of customer loyalty and retention. The opposite is the Churn Rate, which measures the relative number of users who are leaving our product or platform. The Churn Rate is an indication of attrition and loss of users over time. Both the Repeat Rate and the Churn Rate have a clear impact on the CLTV, and as such are very valuable metrics.

When to Use the Metrics

Now that we have a clear understanding of vanity metrics and actionable metrics, and we have seen several examples for each kind of metric, it is time to ask when to use them.

In general, Vanity Metrics are important when the company wants to scale.

Actionable Metrics are important when the company is trying to reach product-market fit.

Thus, when a startup company reaches its growth stage, the vanity metrics may be useful to measure this growth. It makes sense for a company that is scaling up to measure the number of logins in its website, the number of apps downloaded and its total revenues. The values of these metrics over time provide an indication of the growth rate. In other words, these metrics together tell the history of growth. But we need to observe the actionable metrics in order to conclude if this growth is sustainable.

When a startup is still trying to reach its product-market fit, it must focus on actionable metrics such as the conversion rate, the user acquisition cost and the retention rate. Actually these metrics should be used to guide the company in its efforts to validate its product and validate its business model. If a company makes the mistake of being driven by vanity metrics in its early-stage, it may consume all its resources before it reaches product-market fit. In other words, an entrepreneur following only the vanity metrics will probably fail and also be surprised by this failure.

Please feel free to share in the comments below about your own experience with vanity metrics and actionable metrics. In my next posts I will continue to share interesting insights about the topics I’m learning at ThePowerMBA.

See below a summary of the differences between Vanity Metrics and Actionable Metrics (extracted from ThePowerMBA materials):