Statistical significance is the criterion that separates a real difference between variants from one produced by sampling chance. In an A/B test it is measured by the p-value: the chance of observing a difference as large as the one seen if the variants were the same. The calculator runs the two-proportion test, shows the difference interval and checks whether the planned sample was reached.
How to use
- Write the control in the first row and the variants in the following rows (up to four in total).
- Enter visitors and conversions for each one, counted over the same period.
- Use the same confidence and test type you set before the experiment.
- Enter the planned sample per variant to find out whether the test stopped too early.
- Read the verdict, the difference interval and the warnings before deciding.
What each field asks for
| Field | What to enter |
|---|---|
| Variant | E.g. current checkout |
| Visitors | People who received the variant. |
| Conversions | People who converted, at most equal to visitors. |
| Confidence level | The same one you set before the test. |
| Test type | Two-tailed detects improvement and decline; one-tailed only looks at improvement. |
| Planned sample per variant (optional) | The number worked out before the test. Used to warn you if the test stopped early. |
Example
A fictional online shop tested showing shipping costs on the product page. The current checkout had 20,000 visitors and 800 purchases (4%); the variant had 20,050 visitors and 905 purchases (4.51%). The relative change is +12.8%, with a p-value of 0.0109: significant at 95% confidence. The difference interval runs from +0.12 to +0.91 percentage points, the traffic split is balanced and both variants passed the 18,000 visitors planned.
When to use
- At the end of an A/B test, to decide whether to adopt, iterate on or drop the variant.
- To review a result someone presented as a win and check whether it holds up.
- When the test has three or four variants and the error of multiple comparisons must be controlled.
- To check whether the traffic split between variants came out as planned.
Frequently asked questions
What is the p-value?
It is the probability of seeing a difference equal to or larger than the one observed if the variants were the same. It is not the probability that the variant is better. Below the threshold set before the test (0.05 for 95% confidence), the result is called significant.
Does significant mean important?
No. With a very large sample, tiny differences become significant. Compare the difference interval with the minimum effect that would justify the change: only that tells you whether the gain is worth it.
Can I stop the test as soon as it is significant?
It is not advisable. The p-value assumes a single reading at the end of the planned sample. Checking every day and stopping at the first good result greatly increases the chance of a false winner, which is why the calculator warns you when the planned sample was not reached.
What does the difference interval mean?
It is the range of values where the real effect probably lies, at the same confidence level as the test. If the interval includes zero, the test did not separate the variants. If it is all positive but very wide, the gain exists with little precision about its size.
One-tailed or two-tailed test?
Two-tailed detects improvement and decline and is the standard. One-tailed needs a smaller sample but only looks at improvement: a real decline shows up as "no difference". Choose before the test, never after seeing the data.
How does it work with more than two variants?
The calculator runs an overall chi-square test and compares each variant with the control. The significance of each comparison is divided by the number of comparisons, so that the overall error stays at the chosen level.
What is the traffic split warning?
If the planned split was equal and visitors ended up too unbalanced to be chance, something failed in the assignment: a redirect, a filter, a logging error. In that case the result is not reliable until the cause is found.
The result was not significant. What now?
A lack of significance does not prove that the variants are the same. If the planned sample was reached, the real effect is probably smaller than the minimum effect you chose: record the learning and decide whether a bolder change is worth testing.