Skip to content

A/B test calculator

Statistical significance for an A/B test, with support for more than one variant, a confidence interval on the lift, and a sample size planner.

Control
Variants
Test settings

Tests only whether a variant is better than control, not whether it differs in either direction. Two-tailed is the safer default.

Test result

Best result

-

Enter your test data

GroupRateAbs. diffRel. lift95% CI on diffzp-valueResult

How it works

An A/B test significance calculator determines whether the difference in conversion rates between a control and one or more variants is statistically real and not just random chance, using a two-proportion z-test. The sample size planner works out how many visitors a test needs before you run it.

  1. Enter your data

    Visitors and conversions for the control, then each variant you tested.

  2. Pick a confidence level

    95% is the common default. Add a second or third variant if you tested more than one.

  3. Read the p-value

    Below your significance threshold means the difference is unlikely to be chance.

The two-proportion z-test

z = (rate(variant) − rate(control)) / √(pooled × (1−pooled) × (1/n(control) + 1/n(variant)))

The pooled rate is total conversions divided by total visitors across both groups. The p-value comes from the standard normal distribution: a two-tailed test checks for a difference in either direction, a one-tailed test checks only for an improvement. A p-value below your significance threshold, 0.05 at 95% confidence, means the result is unlikely to be random chance.

Sample size, before you run the test

n = (z(confidence) + z(power))² × (p1(1−p1) + p2(1−p2)) / (p2 − p1)²

p1 is the baseline rate and p2 the rate you want to detect. Planning ahead of time avoids peeking bias, stopping a test early because it happens to look significant, which inflates false positives. A smaller minimum detectable effect or a higher confidence level both require a larger sample.

Before you call a winner

Run a test for at least one full business cycle, typically one to two weeks, so it captures day-of-week variation, even if significance appears sooner. Each group generally needs at least 100 conversions for a reliable result. A statistically significant result is not a guarantee the lift holds after launch, only that it is unlikely to be random noise in this data.

Frequently asked questions

What does statistical significance mean in an A/B test?

Statistical significance means the observed difference in conversion rates between control and a variant is unlikely to have happened by random chance. At 95% confidence, a p-value below 0.05 means there is only a 5% probability the result is a fluke. It does not guarantee the lift will hold in production, only that there is enough statistical evidence to act on the result with reasonable confidence.

How many conversions do I need for A/B test significance?

There is no single number: it depends on the baseline conversion rate and the size of the lift you want to detect. As a general rule, each group needs at least 100 conversions, and ideally several hundred, for a reliable result. Use the Plan sample size mode: enter the baseline rate and the minimum detectable effect to get the exact visitor count needed per group before starting the test.

What is a p-value in an A/B test?

The p-value is the probability that the observed difference, or a larger one, would occur if there were actually no real difference between groups. A p-value of 0.03 means there is only a 3% chance the result is due to random variation. Most teams use a threshold of p below 0.05, 95% confidence, before calling a winner, though a high-stakes decision may call for p below 0.01, 99% confidence.

Is statistical significance the same as sample size?

No. Statistical significance is an outcome, the p-value, that depends on the actual data collected. Sample size is a planning input that controls how sensitive a test is to detecting a given lift. Running a test to significance without pre-planning sample size can lead to peeking bias, stopping a test early because results happen to look significant by chance, which inflates the false positive rate.

Can I compare more than one variant against control at once?

Yes. Add up to four variants and each is tested against the control with its own z-test, confidence interval and p-value, so you can see every result side by side. Testing several variants at once does increase the chance that one looks significant purely by luck, so treat a borderline result among many variants with extra caution and consider retesting it alone.