Dig deeper into experimentation data with Experiment Results

This article will help you:

• Extend the analytic power of A/B tests you create in Amplitude Experiment

Previously, Amplitude customers had to use Amplitude Experiment’s feature flagging system if they wanted to take advantage of experimentation features built into the platform. With Experiment Results, Amplitude Analytics customers who have invested in a non-Amplitude feature flagging platform, whether third party or homegrown, can now take advantage of Amplitude’s planning, tracking, and analysis tools for Experiment—while still using the A/B tracking data generated by their own feature flagging platform.

Before you begin

Prior to using Experiment Results, you’ll need to ensure you’ve instrumented the metric events that are relevant to your experiment. Without them, you’ll be unable to create the success metrics and goals that Experiment Results needs to compare each variant in its analysis.

You will also want to make sure you’ve instrumented the necessary exposure events, which represent the delivery of a variant to a user participating in the experiment. See the Amplitude developer documentation on exposure tracking for more detailed information.

There is also an Experiment Results FAQ article that could provide guidance as you use this chart for the first time. 

Analyze an A/B test using Experiment Results

To create an A/B test and see the results, follow these steps:

1. Click Create New > Analysis > Experiment Results.
2. In the Primary Metric module, click + Add Metric or + Define single-use metric to begin setting up your primary metric. A third option, Import, allows you to import your experiment settings as JSON. 
3. If adding a single-use metric, use the drop-down menu to specify the metric type in the Define Metric fly-out panel:

• • • • • Unique conversions
        • Event totals
        • Sum of property value
        • Average of property value
        • Funnel conversion
        • Formula
        • Retention

The first four are available for individual event metric analyses, while funnel conversion allows you to define a multi-step journey that must be completed in order for conversion to be counted. The Formula metric allows you to define a formula centered around a selected event or events.

The last option, Retention, allows you to measure the percentage of users who return to perform the selected event on a specific day (Return on nth day) after being exposed to the experiment. By default, the Retention metric does not support CUPED, exposure attribution settings, nor calendar day windows. Instead, the metric will calculate exposure attribution settings using any exposure and the nth day value based on 24-hour window increments. See this FAQ help center article for more information on how the Retention metric is calculated. 

NOTE: Any of the above metrics can be used as a custom metric during the design phase in Amplitude Experiment. 

4. Next, specify the event that will be used for this metric. You can also filter the event using a + where clause. When you’re finished, click Done. 
   

   

   Optionally, click + Add Metric or + Define single-use metric in the Secondary Metrics module to add a second, subordinate metric to the analysis. You can add multiple secondary metrics as necessary.
   

5. Click + Add Event in the Exposure module to define your experiment’s exposure event. The exposure event is the event users must trigger to become part of the experiment.
6. In the Variants performed by module, add your variants. All experiments require at least one variant, which is known as the control. Add a variant by clicking + Add Experiment Variant.
   
   Choose the properties and values that will define your variant and click Apply.
   

7. Click + Add Experiment Variant to add more variants as necessary to reflect the experiment setup in your feature flagging system. 

Amplitude will calculate your statistical results on the fly and display them in the results. The results also allow you to modify your experiment's statistical settings, such as from the default Sequential test to a T-test. 

Interpret your results

While the specifics may vary depending on the metric types you’re using, you’ll see four charts depicting your results:

• Confidence interval of absolute performance over time: This chart is for sequential testing only. It can help you identify when the experiment reaches statistical significance; which occurs when the confidence interval no longer includes zero. 
• Cumulative exposure: This chart details the number of users who are exposed to your experiment over time. The x-axis displays the first date of a user's exposure, and the y-axis displays a cumulative, running total of users exposed to the experiment.
• Performance by variant: The title of this chart will be the metric you're focused on. The chart will either show the number of users who did each step of a funnel, or the means of each variant if the metric is not a funnel.
• Mean over time (cumulative or non-cumulative): This chart is like the Conversion Over Time graph in a funnel analysis except that it works for non-conversion metrics. The x-axis will show the date the user was first exposed, and the y-axis will show the mean of the selected metric for each variant. The chart gives the option for either a cumulative or non-cumulative view. The cumulative view can help smooth out noise and make interpretation easier.

These charts are also helpful when learning from your end-to-end experiment, and can be found in the Analysis panel of Amplitude Experiment. 

NOTE: By default, the primary metric is selected in experiment results. You can choose a different metric in the Analysis module. Click on the metric's name in the metric table to see its results.