Compass is a powerful feature that can help you identify behaviors that are predictive of retention or conversion. These *inflection metrics* -- the moments when a user has reached a critical threshold in your product -- are instrumental in driving user growth. Facebook found early on that adding seven friends in the first ten days was the strongest signal of long-term retention. More recently, Netflix published an analysis on exactly how many episodes it takes per TV show to get hooked on the show.

Before progressing further, we suggest you review our documentation on Compass here. The rest of this article assumes that you have a general understanding of how the analysis works.^{1}

To find an inflection metric, you first need to determine your target cohort. For most teams, it is finding out how to better encourage new users to retain. In this example the Base Cohort is new users and the Target Cohort is retained users.^{2}

Because this is a common use case, it is the default setup for Compass: New Users as the Base Cohort and 2nd Week Retention (Days 8-14) as the Target Cohort. However, Compass is fully extensible to any set of base and target cohorts and therefore it is not restricted to this specific use case.

When finding an inflection metric it is also important to note that it is not absolute. It does not mean that a user will convert specifically at that point but instead is an indication of the behavior you want your organization (e.g. the product and marketing team) to encourage your users to have.

**Table of Contents**

- Where to Begin?
- Proportion Above Threshold
- True Positive Ratios - PPV and Sensitivity
- True Negative Ratios - NPV and Specificity
- Conclusion
- Footnotes

## Where to begin?

The best way to begin using Compass is to investigate your events that you believe could be good predictors of retention. This is especially important for teams that have implemented an onboarding process already -- part of the value of using Compass is confirming your hypotheses and/or discovering what does not drive retention.

Once you have selected an event to analyze, the next most logical question is, “What is a good correlation?” But before we answer that question, let’s explore how to interpret the Compass metrics.

## Proportion Above Threshold

The proportion above threshold tells you how many new users actually performed the event in their first *N* days. This matters because there needs to be a large enough sample of users meeting the threshold for Compass to understand how well it correlates with retention.

One way to change the proportion is by increasing the number of performance days in the window: we allow between one to seven days. More performance days gives users more time to reach the threshold, hence increasing the proportion. In addition, if you are investigating an event property, consider looking at the overall event as that may have a high enough proportion above the threshold.

Note that there is not a “perfect proportion above threshold." Too low and it is unlikely you can get new users to perform that event that many times; too high and you do not have any room for improvement.^{ 3}

This metric is important because it takes into account the balancing of finding your inflection metric: in the Facebook example, getting a new user to add one friend is not great because almost everyone does that. However, getting a new user to add 100 friends, while highly correlated with retention, is not feasible because such a low percentage of users actually do that.

## True Positive Ratios - PPV and Sensitivity

Assuming you have a reasonable proportion above the threshold, next you want to see how well reaching the event frequency positively correlates with retention. This is done by looking at the Positive Predictive Value (PPV) and Sensitivity.

PPV looks at the ratio of users who reached the event frequency and retained (True Positive) to all users who reached the event frequency (True Positive + False Positive).

Sensitivity looks at the ratio of users that retained and reached the event frequency (True Positive) to all users retained (True Positive + False Negative). You want both of them to be high.^{ 4}

### A Contingency Table

Event Frequency |
Retained |
Not Retained |
---|---|---|

≥ n Times | True Positive | False Positive |

< n Times | False Negative | True Negative |

### Example 1: High PPV, Low Sensitivity

For example, let’s say that the PPV is high but the Sensitivity is low -- what does that mean? It means that this event is a predictor of retention, but not many new users are reaching the threshold. So the event would be a great event to test and see if you can get more users to do it. It also means that there might be another inflection metric you haven’t looked at yet; something else is likely correlated with retention as people who aren’t reaching this frequency are still retaining.

Event Frequency |
Retained |
Not Retained |
---|---|---|

≥ 5 Times |
10 |
1 |

< 5 Times | 100 | 10 |

### Example 2: Low PPV, High Sensitivity

What if it is the other way around (low PPV, high Sensitivity)? In this example, the event frequency is capturing a lot of the retained users, but the overall retention for the product is likely very low. This is not a good candidate for an inflection metric because either the product’s retention is very low or a very high percentage of users meeting the event frequency are not retaining.

Event Frequency |
Retained |
Not Retained |
---|---|---|

≥ 5 Times | 10 | 100 |

< 5 Times | 1 | 10 |

## True Negative Ratios - NPV and Specificity

In addition to our inflection moment being a positive predictor, you also want a user NOT reaching the threshold to be a negative predictor of retention (e.g. churn). This is captured through the Negative Predictive Value (NPV) and Specificity.

NPV looks at the ratio of users who did* not* reach the event frequency and did *not* retain (True Negative) to all users who did *not* reach the event frequency (True Negative + False Negative).

Specificity looks at the ratio of users who did *not* reach the event frequency and did *not* retain (True Negative) to all users who did *not* retain (True Negative + False Positive). As in the examples above, you are hoping to maximize both of these values.^{6}

### Example 3: High NPV, Low Specificity

For example, let’s investigate an event frequency where NPV is high but Specificity is low. In this instance, one of two things is likely happening. Either the PPV is also low (as in Example 2) or the proportion above the threshold is so high that there will not be any improvement by encouraging this action (Example 3 below). Neither are great inflection metrics.

Event Frequency |
Retained |
Not Retained |
---|---|---|

≥ 5 Times | 1000 | 100 |

< 5 Times | 1 | 10 |

### Example 4: Low NPV, High Specificity

What if you have low NPV and high Specificity? Similarly, either the Sensitivity is also low (as in Example 1), or the retention is so high that there aren’t many users to convert (Example 4 below - a good problem to have!).

Event Frequency |
Retained |
Not Retained |
---|---|---|

≥ 5 Times | 1000 | 1 |

< 5 Times | 100 | 10 |

## Conclusion

As you might have guessed by now, we are basically running the Compass analysis to try to uncover event frequencies that maximize the upper left (True Positive) and bottom right (True Negative) quadrants of the contingency matrix (or if you are familiar with statistics, minimize the Type I and Type II errors).

These inflection metrics tend to balance all five of the detailed statistics we have described to accomplish this, and depending on the type of product, a good correlation will be in the range of 0.2-0.4 depending on the number of performance days (1-7) for the event.^{5}

It is important to note that Compass exposes *correlations* from your data -- hypotheses that you can now test by making changes to your product and/or lifecycle marketing. The only way to prove a causal relationship is to run an A/B or split test to isolate those changes. You can read more about how you can analyze A/B test results on Amplitude.

We hope that this primer has been helpful as you start to explore your own product and uncover possible inflection metrics to drive your company goals. Our hope is that you find your ‘A-ha’ moment with Compass soon. If you have any questions, do not hesitate to reach out to your Success Manager.

## Footnotes

1. For ease of reading, we maintain the new user/retention use case but new user can be replaced with any base cohort, and retained user can be replaced with any target cohort.

2. A Base Cohort is an initial set of users you are analyzing (e.g. new users or logged in users). A Target Cohort is a set of users that have successfully completed a targeted action (e.g. retention, conversion).

3. There are some extreme cases where a low proportion above the threshold can still result in a high correlation. For example, if a web application has high traffic but forces login for all new users.

4. More reading on true positives, false positives, and other values in a contingency matrix can be found here.

5. In addition, it is also important to check that the sample size is sufficient to draw conclusions regarding the analysis. There is not a magic number for this as it depends on your overall user volume, but you can see the effect of sample size by clicking on the blue +- number (the 95% confidence interval) next to the correlation. Sample size can be increased by changing the date range -- you can use up to 90 days of data.

6. There is an edge case where a high NPV and high Specificity can lead to a strong correlation but not be a great inflection metric. This occurs when a very high proportion of users fall into the True Negative bucket and the proportion above the threshold is very low. Take for example a website where a very small proportion login, but login blocks every other event from occurring. In this instance, most events will have a high correlation with retention because most users do not do any event. To prevent this, change the base cohort to better reflect an actual user (e.g. someone who logs in).