Free tool
A/B test significance and sample size calculator
The short answer
Use the sample size calculator before the test to see how many visitors each version needs. Use the significance calculator after the test has reached that number, not before. Both run in your browser, and the page address keeps your numbers so a result can be shared.
01Before the test: sample size
Sample size
20 means 3.0% to 3.6%
Per version
13,914
Total (A + B)
27,828
Days at your traffic
56 (8 wk)
Detecting 3% to 3.60%. Two-sided test, 50/50 split. Round up to whole weeks so every weekday is covered.
02After the test: is the difference real?
Significance
Rate A
3.00%
Rate B
3.80%
Relative lift
26.7%
p-value
0.0273
Significant at 95%. A gap this large would appear by chance about 2.7% of the time if the versions performed the same.
95% confidence interval for the difference (B minus A): 0.09 to 1.51 percentage points. Two-sided two-proportion z-test.
How to read the result
- p-value: how often a gap at least this large would appear by chance if the versions were identical. Below 0.05 is the usual bar.
- Confidence interval: the plausible range for the true difference. If it spans zero, you cannot rule out no effect.
- Relative lift: B's rate compared with A's. A big lift on a small sample is usually noise.
Checking repeatedly and stopping the moment the p-value dips below 0.05 inflates false wins. Decide the sample size first and check once.
The method is a two-sided two-proportion z-test with a pooled standard error, and the sample size uses the standard normal approximation. Both are reliable when each version has at least a few dozen conversions.
More on running tests on modest traffic: A/B testing for small business.