A/B Tests

Running A/B tests—what we call experiments—is a powerful way to learn what works. But knowing how to interpret your results is just as important as running the test itself. Here's how to make sense of the data, know when you have a "winner," and understand what it really means to be statistically significant.

Start With the Basics

In your A/B Tests analytics page, you’ll see performance broken down by variant—A, B, C, etc.—including metrics like:

• Conversion rate (CVR)

• Revenue generated

Look for meaningful differences between variants. If one clearly outperforms the others across multiple metrics, it may be your winner.

What Is Statistical Significance?

Statistical significance helps you determine if a result is real, or just a fluke due to random chance.

In plain terms:
A result is statistically significant if it’s unlikely that the difference in performance happened by accident. This usually depends on how large the difference is and how many people received each variant.

Key points:

• The more recipients in your test, the more confident you can be in the result

• Small performance differences are often not significant if the sample size is small

• You generally want at least a few hundred recipients per variant for meaningful analysis

We display results for each experiment, and we highlight results with statistical significance to help indicate winners—especially useful if the differences are subtle.

Pro Tip: Test One Variable at a Time

To isolate what's driving results, keep your experiments simple:

• Change only the call-to-action, or

• Change only the offer, or

• Change only the image

If you change multiple things at once, it’s hard to know what caused the outcome.

A/B testing is a long-term habit, not a one-time tool. The more you test, the more you’ll learn—and the better your campaigns will perform. Use your results to guide future strategy, and don’t be discouraged by “inconclusive” outcomes. Every test teaches you something.

Ready to set up your first experiment? 🧪 Read our guide to A/B testing.

A/B Test Reporting

Check out this video for an overview of how A/B test reporting works and how it differs from our conventional reporting methods.