How do I use the A/B test calculator?
Enter the number of visitors and conversions for each variant. Variant A is your original version (the control). Variant B is the change you are testing. Choose a confidence level. The calculator shows both conversion rates, the uplift, and whether the difference is statistically significant.
What does statistically significant mean?
It means the difference between A and B is unlikely to be down to chance. At a 95% confidence level, a result is significant if the p-value is below 0.05. In other words, if there were no real difference, you would see a gap this large less than 5% of the time.
What is a p-value?
The p-value is the chance of seeing a difference at least this large if A and B actually perform the same. A small p-value is evidence of a real difference. It does not tell you how big or how important the difference is.
How does the calculator work out the result?
It uses a two-tailed, two-proportion z-test. It pools both variants to estimate the standard error, then compares the gap between the two conversion rates to that error. This is the standard test for conversion rates.
How many visitors do I need?
Enough to detect the smallest change you care about. Small differences need many thousands of visitors per variant. Use the "Plan a test" tab to work out the number before you start. Do not stop early because the result looks good. Stopping early raises the chance of a false positive.
How does the "Plan a test" tab work?
Enter your current conversion rate and the smallest relative uplift you want to detect. For example, a 4% baseline and a 10% uplift means you want to detect a move from 4.0% to 4.4%. Choose a confidence level and a power. The calculator shows how many visitors each variant needs. If you enter your daily visitors, it also estimates how many days the test will take.
What is statistical power?
Power is the chance that the test finds a real difference of the size you set. A power of 80% means that, if the uplift is real, you would detect it 8 times out of 10. Higher power needs more visitors.
Is my data uploaded anywhere?
No. Every calculation runs in your browser. The numbers you enter never leave your device.
Can I work out ad costs and returns too?
Yes. Use the ad metrics calculator to work out CTR, CPA, ROAS and ROI for each variant.