A/B Test Significance Calculator
Analyze results or plan your next experiment
Test Data
Variant A (Control)
Variant B (Test)
Results
Required Sample Size
About A/B Test Significance
This calculator uses a two-proportion z-test to determine if the difference between two conversion rates is statistically significant. A p-value below 0.05 (95% confidence) indicates that the observed difference is unlikely due to random chance. For reliable results, ensure each variant has at least 100 visitors and the test has run for a full business cycle.
Frequently Asked Questions
What does statistical significance mean in an A/B test?
Statistical significance means the observed difference in conversion rates between Variant A and Variant B is unlikely to have happened by random chance. At 95% confidence (p-value below 0.05), there is only a 5% probability the result is a fluke. It does not guarantee the lift will hold in production — it just provides enough statistical evidence to act on the result with reasonable confidence.
How many conversions do I need for A/B test significance?
There is no universal number — it depends on your baseline conversion rate and the size of the lift you want to detect. As a general rule, each variant needs at least 100 conversions, and ideally several hundred, to produce reliable results. Use the sample size planner: enter your baseline rate and minimum detectable effect to get the exact visitor count needed per variant before starting your test.
What is p-value in an A/B test?
The p-value is the probability that the observed difference (or a larger one) would occur if there were actually no real difference between variants. A p-value of 0.03 means there is only a 3% chance the result is due to random variation. Most marketers use a threshold of p < 0.05 (95% confidence) before calling a winner, though high-stakes decisions may warrant p < 0.01 (99% confidence).
Is statistical significance the same as sample size?
No — this is a critical misconception. Statistical significance is an outcome (the p-value) that depends on your actual data. Sample size is a planning input that controls how sensitive your test is to detecting a given lift. Running a test to significance without pre-planning sample size can lead to peeking bias: stopping the test early when results happen to look significant by chance, inflating false positive rates.
How long should I run an A/B test?
Run your test for at least one full business cycle — typically one to two weeks — to capture day-of-week variation in user behavior, regardless of whether significance is reached earlier. Calculate the required sample size upfront and do not stop the test early just because the significance badge appears. Stopping early is one of the most common A/B testing mistakes and routinely produces false winners.