Compass helps find thresholds of user behavior that predict long-term retention and ultimately lead to growth for your application.
Table of Contents
- Setting Up the Report
- "A-Ha Moment"
- Reading the Report
- Why Is Correlation a Good Metric?
- Statistical Significance
- Why Not Use the Diagnostic Odds Ratio or Other Similar Metrics?
- Add to Dashboard
- Advanced Usage
- Best Practices Article
- Video Walkthrough
Note: Compass is only available on the Enterprise plan.
An "A-Ha Moment" is the point in time at which the user decides to either consciously or subconsciously become an active user of your application. Compass quantifies these moments by looking at which events new users performed during their first days of using the application and whether or not those users become retained users down the line.
The most famous example of an "A-Ha Moment" is Facebook's "7 Friends in 10 Days". In this instance, Facebook found that new users who added at least seven friends in their first ten days on the site were almost certain to be retained in the long term. On the other hand, users who have failed to add at least seven friends in their first ten days were almost guaranteed to churn.
Facebook's metric is important for at least three reasons:
- The focus on friends emphasizes that new users should be encouraged to add friends rather than take other actions such as adding photos, writing posts, or filling out their profile.
- The threshold of adding at least seven friends in the first ten days is very predictive of retention (see below for a more detailed discussion of what predictive means). This predictive power makes it a good metric to measure success by.
- The fact that this metric can be measured within the first ten days of use puts it close to the top of the user retention funnel. This early measurement allows Facebook to iterate quickly and be data driven without sacrificing speed.
Setting Up the Report
Compass can be found on the "Create Chart" page under the "Retain Your Users" section. To begin, select a time range of new users to analyze. Make sure that this time range is sufficiently long to increase the size of your sample and far enough in the past to ensure that all users in your sample have had an opportunity to be retained. For example, if you are looking at '[Amplitude] 2nd Week Retention' as your target cohort, then you should make sure that the new users in your base cohort have been new for longer than two weeks.
Next, you can use the left module to choose what group of users you want as your base cohort. You can select between the default cohort of '[Amplitude'] New Users' and any other behavioral cohort you have created.
Finally, you can use the right module to configure how many days after first use to look at for each user. By default, you can look at the target cohorts of '2nd Week Retention', '3rd Week Retention', '4th Week Retention', and '2nd Month Retention'. However, you can also use the dropdown to select a target cohort that you have created in the Behavioral Cohorts section. The Compass report is essentially looking at how users in your starting cohort end up in your target cohort. For example, if you select '2nd Week Retention' as your target cohort, this means that Compass will look at whether a user came back between 8 and 14 days after first use (if the user was new on September 1st then they will count as retained if they came back anytime from September 9th-15th).
Here are the definitions for the out-of-the-box '[Amplitude]' prefixed cohorts:
- [Amplitude] 2nd Week Retention: Users will belong in this cohort if they were new during the selected time range in the date picker and triggered an active event two weeks after they were new.
- [Amplitude] 3rd Week Retention: Users will belong in this cohort if they were new during the selected time range in the date picker and triggered an active event three weeks after they were new.
- [Amplitude] 4th Week Retention: Users will belong in this cohort if they were new during the selected time range in the date picker and triggered an active event four weeks after they were new.
- [Amplitude] 2nd Month Retention: Users will belong in this cohort if they were new during the selected time range in the date picker and triggered an active event two months after they were new.
Like all other charts in Amplitude, you can read the chart control panel like a sentence. The default setup shows you for your new users in that time range, how does performing any event predict they will be 2nd week retained.
Choosing a Specific Event
If you are unsure of which events to look at, you can generate the default report of '[Amplitude] Any Event' to get an idea. However, if you do have a specific event in mind, you can choose to select which event to analyze by using the dropdown in the left module and within how many days of first use you want to look at for new users. For example, let's say we wanted to look more specifically at the correlation between the event 'AddFriend' if users perform this event within the first seven days of joining our app. Additionally, just like in other parts of the Amplitude platform, you can see a more granular view by using a "+where" clause. The following setup would allow you to analyze the event 'AddFriend' for your iOS users in a Compass report.
Group By Property
You can also group by a particular property to generate a report that calculates how different property values affect the correlation between your base cohort and your target cohort. For example, the following chart shows the correlations of the event 'PlaySong' by 'Type' property values between the base cohort ('[Amplitude] New Users') and the target cohort ('2nd Week Retention'). Note that you can also search for a particular property value if there is a specific one you wanted to analyze.
The output of Compass shows how a new user performing an event correlates with that user being retained. More precisely, it shows how a new user performing an Event X for n number of times is correlated with that user being retained. For example, if performing Event X at least four times in the first five days is highly correlated with retention, then "Performing Event X at least four times in the first five days" is a good candidate for an "A-Ha Moment".
As someone with a lot of domain-specific knowledge about your app, you should use the following as a starting point but not ignore unique aspects of your app and the space you are in when interpreting this analysis. If you do not already have an event or hypothesis in mind, you can create the default Compass report where the event field is '[Amplitude] Any Event' as a good first step. This report gives you a quick summary of what the top correlated events are for a base cohort of users to convert to a target cohort.
Below is an example of how the summary report would look like. If you click on a specific cell, you will see a popup of more detailed information regarding the event you selected. Additionally, if you click on the light blue numerical day column labels you can sort the report based on ascending or descending correlation. The summary report is useful for looking at your data from an overall view, e.g. looking for events that should have been at the top but were not.
Reading the Report
Let's look at how performing the event 'AddFriend' within the first seven days of use for new users correlates with second-week retention and walk through the different aspects of the generated reports.
The left side of the report shows you the correlation of various frequencies of that event. By default, the report will show you the frequency with the highest correlation. You can see that users who performed 'AddFriend' >= 2 times had the highest correlation score. You can click on the different buckets to view each individual report for different frequencies of that particular event. In addition, you can also click on the blue text above the bar chart to show the chart in a different metric. Note that as the threshold goes up due to small sample sizes, the correlation will also decrease so keep in mind the sample sizes your Compass report is showing.
On the right side of the report, you can view the correlation score for this event at this particular frequency and your target cohort. While it is hard to generalize across all apps, even correlations near 0.2 can be considered when looking at smaller numbers of initial days for each user. Thus, we have generalized correlations into four categories:
- Highly Predictive: correlation >= 0.4
- Moderately Predictive: 0.4 > correlation >= 0.3
- Slightly Predictive: 0.3 > correlation >= 0.2
- Not Predictive: correlation <= 0.2
You can create a cohort out of these specific users (users who performed 'AddFriend' >= 2 times) by using the "Create Cohort" button below the correlation score, and you can compare their retention to new user retention by clicking the "Click to Compute Comparison" button.
If you click "Show" next to the "Correlation Table" section, you can see a detailed contingency table that shows the count of users in your base cohort who constitute your True Positives, False Positives, False Negatives, and True Negatives.
If you click "Show" next to the "Detailed Statistics" section, you can see the Correlation, Positive Predictive Value, Negative Predictive Value, Sensitivity, Specificity, and Proportion Above Threshold. You can read more about these statistics here.
Additionally, you can export the Compass report as a CSV file by using the "Export" button.
Correlation is a measure (ranging from -1 to 1) of how two statistical variables relate to each other. In Compass, the variables for each user are whether or not the user performed at least the threshold number of events and whether or not the user was retained. You may have heard of different variations and definitions of correlation, including Matthews correlation, Pearson correlation, phi coefficient, and R-value. In this case, all of these definitions are equivalent because Compass looks at pairs of binary random variables.
Remember, correlation is not causation so hypotheses generated by Compass still must be tested and verified in the real world. Here are some more technical intuitive definitions of correlation:
- Correlation of X and Y is the covariance of X and Y divided by the geometric mean of their variances.
- If X is modeled as an affine function of Y and Y is modeled as an affine function of X, each with minimal root mean squared error, then the correlation of X and Y is the geometric mean of the predictive coefficients of these two functions.
Why Is Correlation a Good Metric?
As mentioned above, part of finding an "A-Ha Moment" is internal marketing and motivation for your team. However, this subsection will instead consider the product side and how to find the metric that will most influence your product growth.
When looking for a good threshold of user activity, it is important that most users above the threshold go on to be retained and most users below the threshold are not retained. This is essentially finding a threshold with a good Positive Predictive Value (PPV) and a good Negative Predictive Value (NPV). Check out our blog post for some tips on determining which events lead to growth.
However, another important consideration is how easy it is to move users across the threshold. A threshold with a very strong PPV and NPV but that is very difficult to move users across will not be helpful to your goal of growing your app. Intuitively, if barely any of your users have crossed the threshold or almost all of your users have already crossed it, then it will probably be difficult to increase the fraction of users above the threshold. Note: You may have domain-specific knowledge telling you otherwise, but in the absence of such knowledge, this is a good assumption.
Luckily, correlation takes into account the PPV, the NPV, and the proportion above the threshold. So, if the PPV is higher, the NPV is higher, or the fraction of users above the threshold is closer to 50%, then the correlation will also be higher. Likewise, if the PPV is lower, the NPV is lower, or the fraction of users above the threshold is further from 50%, then the correlation will be lower. Note: Technically, the above statements are not always true for negative correlations, but you will generally not be looking at negative correlations when using Compass.
Compass allows you to toggle on and off the 95% confidence interval of the correlation. Click on the blue numerical text on the right-hand side of the table to display the interval on the left-hand side bar chart.
Why Not Use the Diagnostic Odds Ratio or Other Similar Metrics?
The diagnostic odds ratio is commonly used in medicine to determine if a treatment is effective or not. However, the medical community mainly cares about statistical significance and can ask patients to move across a treatment threshold. In your case, you might know that using your app at least 50 times in the first day is perfectly predictive of retention and statistically significant, but that does not help you because almost no users do it and it is not realistic to push many more users toward that threshold.
Add to Dashboard
Once you have found an interesting Compass report, you can save the report and also add it to a dashboard to monitor over time.
The darker colors in the added Compass report correspond to the correlation of each cell.
Using Custom Behavioral Cohorts
Some users may want to do more advanced analyses that involve specific behavioral cohorts of new users for definitions of "good users" that are different from retention. In this case, it is possible to create custom cohorts and insert them in place of the base and target cohorts.
Please note that before any analysis is run, the base cohort will be filtered down to new users in the selected time range. Then, the target cohort will be filtered down to users who are also in the base cohort. In the dropdown below we can see that there are several other options for our target cohort beyond the default ones prefixed by '[Amplitude]'.
Best Practices Article
Watch the video here.