A/B Test Significance Checker
Enter visitors and conversions for each variant. Instantly see conversion rates, uplift, z-score, p-value — and whether the difference is statistically significant at 95% confidence.
What statistical significance means
"Significant" does not mean "big" or "important" — it means the difference between your variants is unlikely to be random noise. A significance test starts from the assumption that A and B are really identical (the null hypothesis) and asks: if that were true, how surprising would our observed gap be? If the gap would happen less than 5% of the time by pure chance, we call the result significant at 95% confidence and act as if B genuinely differs from A.
The formula
This calculator uses the two-proportion z-test, the standard way to compare conversion rates between two groups:
Here p1 and p2 are the two conversion rates, and p is the pooled rate — total conversions divided by total visitors. The denominator is the standard error: how much the difference would wobble under pure chance. Dividing the observed gap by that wobble gives the z-score, which the standard normal distribution converts into a two-tailed p-value.
A worked example
Suppose variant A converts 100 out of 1,000 visitors (p1 = 10%) and variant B converts 120 out of 1,000 (p2 = 12%). The pooled rate is 220 ÷ 2,000 = 11%, so the standard error is √(0.11 × 0.89 × (1/1000 + 1/1000)) ≈ 0.014. The z-score is (0.12 − 0.10) ÷ 0.014 ≈ 1.43, giving a two-tailed p-value of about 0.15. That is above 0.05, so the result is not significant — despite the tempting 20% uplift, this data could easily be noise. With ten times the traffic (1,000/10,000 vs 1,200/10,000), the same rates give z ≈ 4.52 and p < 0.0001: highly significant.
Common mistakes
- Peeking early. Checking the test daily and stopping the moment it turns green inflates false positives — the 5% error rate assumes you looked once, at a pre-planned sample size.
- Tiny samples. With only a handful of conversions, one lucky click moves the percentage wildly. This calculator warns you when either variant has fewer than 100 visitors.
- Confusing significance with importance. A huge sample can make a 0.1% uplift "significant". Always pair the p-value with the actual uplift and ask whether the gain is worth shipping.
Frequently asked questions
How many visitors do I need for a valid A/B test?
There is no fixed number — it depends on your baseline conversion rate and the uplift you want to detect. Small uplifts on low conversion rates need thousands of visitors per variant; big uplifts need far fewer. As a rule of thumb, aim for at least a few hundred conversions per variant before trusting the result, and always decide your target sample size before the test starts rather than stopping when you like the numbers.
What does the p-value actually mean?
The p-value is the probability of seeing a difference at least this large if the variants were actually identical and the difference were pure noise. A p-value of 0.15 means there is a 15% chance randomness alone could have produced what you see. It is not the probability that B is better, and it is not the probability the result is wrong — it is a measure of how surprising the data would be under the "no real difference" assumption.
Why is 95% confidence the standard?
It is a convention that balances two errors: calling a dud a winner (false positive) and missing a real winner (false negative). 95% confidence means accepting a 5% false-positive rate. A stricter level like 99% cuts false positives but needs much larger samples and makes you slower to ship. For most marketing tests, 95% is the pragmatic middle ground.
Significant but tiny uplift — ship it?
Statistical significance is not the same as practical importance. With enough traffic, even a 0.1% uplift can be significant — but shipping it may not be worth the engineering time, maintenance cost or added complexity. Judge tiny significant wins on business impact: revenue per visitor, implementation cost, and whether the change makes the product harder to maintain.
What if my variants have unequal traffic?
The two-proportion z-test handles unequal sample sizes naturally — each variant is weighted by its own sample size in the pooled standard error, so a 50/50 split is not required. What does matter is that traffic was split randomly and the imbalance was stable throughout the test; a split that changes mid-test (for example, ramping B from 10% to 50%) can introduce time-based bias the test cannot see.