The sample size of an A/B test is the number of people each variant needs to receive so that a difference of the size that matters shows up with the confidence and power set before the test. The calculator uses the classic two-proportion formula and turns the total into full weeks of traffic, so you know whether the experiment fits the deadline before you start.
How to use
- Enter the current conversion of the metric the test wants to move.
- Set the minimum effect that would justify adopting the variant, relative or in percentage points.
- Choose confidence, power, test type and number of variants (the defaults are 95%, 80%, two-tailed and two).
- Add the visitors per day entering the experiment to see the duration in full weeks.
- Use the sensitivity table to agree the minimum effect and the deadline with the team before you start.
What each field asks for
| Field | What to enter |
|---|---|
| Current conversion | Today's rate for the metric the test wants to move. |
| Minimum detectable effect | The smallest change that would justify adopting the variant. The smaller it is, the larger the sample. |
| Effect type | Relative: 10% on 4% gives 4.4%. Absolute: 0.5 points on 4% gives 4.5%. |
| Confidence level | Protects against a false winner. Raising it increases the sample. |
| Statistical power | The chance of detecting the effect if it really exists. |
| Test type | Two-tailed detects improvement and decline; one-tailed only looks one way. |
| Variants (including control) | 2 for an A/B test; 3 for A/B/C, and so on. |
| Visitors per day (optional) | People entering the experiment per day, across all variants. |
Example
A fictional online shop converts 4% of visitors and wants to test a new product page. The team will only adopt the change if it lifts conversion by at least 10% relative, to 4.4%. With 95% confidence, 80% power and a two-tailed test, each variant needs 39,475 visitors, 78,950 in total. With 6,000 visitors per day in the experiment, that is 14 days: two full weeks. Detecting a 5% effect would take 8 weeks.
When to use
- Before launching an A/B test, to know whether it finishes within an acceptable time.
- To decide whether a test is worth running or traffic is too small for the expected effect.
- To agree with the team on the smallest effect that would change the decision.
- To explain to anyone asking for a quick result why the test needs more weeks.
Frequently asked questions
What is the minimum detectable effect?
It is the smallest difference between control and variant that the test can detect with the chosen power. Set it by business value: the smallest gain that would pay for the change. The smaller the effect, the larger the sample, and it grows with the square of the reduction.
What is statistical power?
It is the chance that the test detects an effect that really exists. With 80% power, a real effect of the chosen size goes unnoticed 20% of the time. Raising it to 90% reduces that risk and increases the sample.
Why not stop the test as soon as it is significant?
The calculation assumes you read the result once, at the end. Checking every day and stopping at the first significant number multiplies the chance of a false winner. Fix the duration in advance and stick to it.
Why is the duration rounded up to full weeks?
Behaviour changes between weekdays and weekends. A test that covers only part of the week measures a different audience from the real one. Full weeks balance out that effect.
What if traffic is low?
Test bigger changes that produce bigger effects, or choose a more frequent metric closer to the action being tested. If the test still runs past 8 weeks, prefer qualitative methods or a prototype test.
How does the calculation work with more than two variants?
Each variant is compared with the control, and the significance of each comparison is divided by the number of comparisons to keep the overall error at the chosen level. That is why each extra variant increases both the sample per group and the total.
Is this calculator Bayesian or frequentist?
Frequentist: it uses the normal approximation for two proportions, the same formula as the classic statistical references. Bayesian approaches answer a different question (the probability that the variant is better) and also need a sufficient sample to be stable.