A/B test calculator
Statistical significance for an A/B test, with support for more than one variant, a confidence interval on the lift, and a sample size planner.
Test result
Best result
-
Enter your test data
| Group | Rate | Abs. diff | Rel. lift | 95% CI on diff | z | p-value | Result |
|---|
How it works
An A/B test significance calculator determines whether the difference in conversion rates between a control and one or more variants is statistically real and not just random chance, using a two-proportion z-test. The sample size planner works out how many visitors a test needs before you run it.
- Enter your data
Visitors and conversions for the control, then each variant you tested.
- Pick a confidence level
95% is the common default. Add a second or third variant if you tested more than one.
- Read the p-value
Below your significance threshold means the difference is unlikely to be chance.
The two-proportion z-test
z = (rate(variant) − rate(control)) / √(pooled × (1−pooled) × (1/n(control) + 1/n(variant)))
The pooled rate is total conversions divided by total visitors across both groups. The p-value comes from the standard normal distribution: a two-tailed test checks for a difference in either direction, a one-tailed test checks only for an improvement. A p-value below your significance threshold, 0.05 at 95% confidence, means the result is unlikely to be random chance.
Sample size, before you run the test
n = (z(confidence) + z(power))² × (p1(1−p1) + p2(1−p2)) / (p2 − p1)²
p1 is the baseline rate and p2 the rate you want to detect. Planning ahead of time avoids peeking bias, stopping a test early because it happens to look significant, which inflates false positives. A smaller minimum detectable effect or a higher confidence level both require a larger sample.
Before you call a winner
Run a test for at least one full business cycle, typically one to two weeks, so it captures day-of-week variation, even if significance appears sooner. Each group generally needs at least 100 conversions for a reliable result. A statistically significant result is not a guarantee the lift holds after launch, only that it is unlikely to be random noise in this data.
Frequently asked questions
What does statistical significance mean in an A/B test?
Statistical significance means the observed difference in conversion rates between control and a variant is unlikely to have happened by random chance. At 95% confidence, a p-value below 0.05 means there is only a 5% probability the result is a fluke. It does not guarantee the lift will hold in production, only that there is enough statistical evidence to act on the result with reasonable confidence.
How many conversions do I need for A/B test significance?
There is no single number: it depends on the baseline conversion rate and the size of the lift you want to detect. As a general rule, each group needs at least 100 conversions, and ideally several hundred, for a reliable result. Use the Plan sample size mode: enter the baseline rate and the minimum detectable effect to get the exact visitor count needed per group before starting the test.
What is a p-value in an A/B test?
The p-value is the probability that the observed difference, or a larger one, would occur if there were actually no real difference between groups. A p-value of 0.03 means there is only a 3% chance the result is due to random variation. Most teams use a threshold of p below 0.05, 95% confidence, before calling a winner, though a high-stakes decision may call for p below 0.01, 99% confidence.
Is statistical significance the same as sample size?
No. Statistical significance is an outcome, the p-value, that depends on the actual data collected. Sample size is a planning input that controls how sensitive a test is to detecting a given lift. Running a test to significance without pre-planning sample size can lead to peeking bias, stopping a test early because results happen to look significant by chance, which inflates the false positive rate.
Can I compare more than one variant against control at once?
Yes. Add up to four variants and each is tested against the control with its own z-test, confidence interval and p-value, so you can see every result side by side. Testing several variants at once does increase the chance that one looks significant purely by luck, so treat a borderline result among many variants with extra caution and consider retesting it alone.