Web and product analytics
A/B testing for small business: traffic and significance
The short answer
Decide the one metric and the smallest change worth detecting before you start, use a sample size calculator to see how many visitors each version needs, run the test for full weeks until you reach that number, and only then check significance. Sites with a few hundred conversions a month can only detect large differences.
Key takeaways
- Set the sample size before you start and check significance once, at the end.
- At a 3% conversion rate, detecting a 20% lift needs about 14,000 visitors per version.
- If you cannot reach the sample in a couple of months, test something bigger or just ship.
On this page 5 sections
Traffic is the constraint
With a 3% conversion rate, detecting a lift to 3.6% (a 20% relative improvement) at 95% confidence and 80% power needs roughly 14,000 visitors per version. Detecting 3% to 4.5% needs about 2,500 per version. Use the calculator with your own numbers.
Rules that prevent false wins
- Set the sample size in advance and do not stop early because one version is ahead.
- Run whole weeks, so weekday and weekend visitors are both included.
- Test one change, or accept that you will not know which part worked.
- Do not slice results into many segments looking for a winner; some will look significant by chance.
When not to test
If you cannot reach the sample size in a couple of months, make the change you believe in and watch the trend, or test something bigger: a new offer, a new page, a price change. Fix obvious problems (a broken form on mobile) without testing.
Testing beyond the website
The same arithmetic applies to email subject lines, ad copy and survey results such as a change in NPS between two groups.
Questions
- What does 95% significance mean?
- If there were no real difference between the versions, a gap this large would appear less than 5% of the time by chance. It does not mean there is a 95% chance the winner is better.
Next step
Try the a/b test significance and sample size calculator
Related guides
Marketing attribution models explained
First-click, last-click, linear, time-decay, position-based and data-driven attribution, explained with one worked example and how to pick one.
How to measure NPS (Net Promoter Score)
How to ask the NPS question, calculate the score, choose when and where to survey, and turn the follow-up comments into decisions.