How do I use the A/B test calculator?
Enter the number of visitors and conversions for each variant. Variant A is your original version (the control). Variant B is the change you are testing. Choose a confidence level. The calculator shows both conversion rates, the uplift, and whether the difference is statistically significant.
What does statistically significant mean?
It means the difference between A and B is unlikely to be down to chance. At a 95% confidence level, a result is significant if the p-value is below 0.05. In other words, if there were no real difference, you would see a gap this large less than 5% of the time.
What is a p-value?
The p-value is the chance of seeing a difference at least this large if A and B actually perform the same. A small p-value is evidence of a real difference. It does not tell you how big or how important the difference is.
How does the calculator work out the result?
It uses a two-tailed, two-proportion z-test. It pools both variants to estimate the standard error, then compares the gap between the two conversion rates to that error. This is the standard test for conversion rates.
How many visitors do I need?
Enough to detect the smallest change you care about. Small differences need many thousands of visitors per variant. Decide your sample size before you start the test, and do not stop early because the result looks good. Stopping early raises the chance of a false positive.
Is my data uploaded anywhere?
No. Every calculation runs in your browser. The numbers you enter never leave your device.
Can I work out ad costs and returns too?
Yes. Use the ad metrics calculator to work out CTR, CPA, ROAS and ROI for each variant.