What Is an A/B Test Sample Size Calculator?
An A/B test shows two versions of a page, ad or email to different visitors and compares how many convert. The hard part is knowing when to stop. Stop too early and a lucky streak looks like a winner. This calculator tells you how many visitors each version needs before the difference you see is likely to be real.
It uses the standard sample size formula for comparing two conversion rates with a two sided test. You choose the significance level, which controls false winners, and the power, which controls how often a real improvement is caught.
How to Use the Calculator
- Enter your current conversion rate from GA4 or your testing tool, for the exact page and goal you will test.
- Set the smallest lift worth finding. A 15% relative lift on a 3% rate means detecting a move to 3.45%.
- Keep 95% significance and 80% power unless you have a reason to change them. These are the common defaults.
- Add your daily visitors and the share of traffic in the test to see how many days the test needs.
- Check the table to see how much faster a test finishes when you aim for a bigger lift.
What Each Setting Means
| Setting | What it controls | Typical value |
|---|---|---|
| Baseline conversion rate | How often visitors convert today | From your reports |
| Minimum detectable effect | The smallest change the test can reliably find | 10% to 20% relative |
| Significance level | How sure you want to be that a winner is not luck | 95% |
| Statistical power | The chance of catching a real improvement of that size | 80% |
| Number of versions | Each extra version needs its own full sample | 2 |
The formula is n = (zα/2 × √(2p̄(1 − p̄)) + zβ × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₂ − p₁)², where p₁ is the current rate, p₂ the rate you want to detect, and p̄ their average.
Tips for Trustworthy A/B Tests
- Fix the sample size first. Checking results every day and stopping when one version leads is the most common way to get a false winner.
- Run whole weeks. People behave differently on weekends, so cover at least one full week, ideally two.
- Test bold changes on small sites. Low traffic pages can only detect big lifts in a sensible time.
- Track one main goal. Pick the conversion you care about before you start and judge the test on that.
Keep Improving
Find ideas worth testing in our guide to conversion rate optimization, apply the same thinking to ads with PPC ad copy testing, and make sure your goal is tracked correctly using GA4 events vs conversions. Testing ad clickthrough rates instead? The CTR calculator checks whether one ad really beats another, and you can track each version cleanly with the UTM builder.
Frequently Asked Questions
1How many visitors do I need for an A/B test?
It depends on your conversion rate and the lift you want to detect. A page converting at 3% needs about 24,000 visitors per version to detect a 15% relative lift at 95% significance and 80% power. Bigger lifts and higher conversion rates need far fewer.
2What is a minimum detectable effect?
It is the smallest improvement your test can reliably find. If the real lift is smaller than this, the test will usually end without a clear winner. Smaller effects need much larger samples.
3Can I stop the test once it reaches significance?
Not safely. Significance moves up and down as data comes in, so stopping at the first good reading inflates false winners. Decide the sample size up front and let the test reach it.
4Is my data stored?
No. The calculator runs entirely in your browser and nothing you enter is sent anywhere.