An experiment plan is the document that fixes, before the test, everything that could later be adjusted to favour a result: the hypothesis, the primary metric, the guardrail metrics, the unit of randomisation, the sample, the duration and the rule that decides between adopting, iterating or stopping. Writing the plan first protects the team from reading the data to suit itself.
How to use
- Write the hypothesis, or import the one you built in the hypothesis generator.
- Set one primary metric and the guardrail metrics that must not get worse.
- Choose the unit of randomisation, the audience and the variants.
- Enter the current value, minimum effect and traffic to work out sample and duration.
- Tick the threats to validity already handled and agree the decision rule with whoever decides.
What each field asks for
| Field | What to enter |
|---|---|
| Hypothesis | The bet the experiment will test: change, audience and outcome. |
| Primary metric | Just one: it decides the result. |
| Guardrail metrics | What must not get worse while the primary metric improves. |
| Unit of randomisation | What is assigned to each variant. It must match the metric. |
| Experiment audience | Who enters the test and who is left out. |
| Variants | One per line, starting with the control. Up to four. |
| Current value of the metric (optional) | As a percentage. With the minimum effect, it works out the sample. |
| Minimum effect (optional) | In % relative: the smallest improvement that would justify the change. |
| Visitors per day (optional) | People entering the experiment per day, across all variants. |
| Confidence level | Protects against a false winner. |
| Statistical power | The chance of detecting the effect if it exists. |
| Maximum time in weeks (optional) | How long the business is willing to wait for the answer. |
| Threats already handled | Tick what the plan already covers. The rest becomes an open item. |
Example
A fictional online shop plans to test estimated shipping on the product page. The primary metric is basket-to-purchase conversion (4% today), with average order value and return rate as guardrails, randomisation by user and two variants. To detect a 15% improvement (from 4% to 4.6%), the plan needs 17,943 people per variant, 35,886 in total: with 3,000 visitors per day, two full weeks, within the four-week deadline. Two threats remain open: unusual dates and contamination between groups.
When to use
- Before asking engineering to build the variants of an A/B test.
- To agree with leadership and partner teams what counts as success before seeing the data.
- When a previous test sparked debate about what the result meant.
- To record the experiment design alongside the decision taken at the end.
Frequently asked questions
What goes into an experiment plan?
Hypothesis, primary metric, guardrail metrics, unit of randomisation, audience, variants, sample, duration, decision rule and threats to validity. All of it set before the test starts.
What are guardrail metrics?
They are the metrics that must not get worse while the primary one improves, such as average order value, returns or support tickets. They stop you declaring victory on a change that shifts the loss somewhere else.
How do I choose the unit of randomisation?
Randomise what matches the metric: if the metric is per person, randomise by user; if it is per customer company, by account. Randomising by session when the metric is per person lets the same person see both variants and mixes the effects.
Why set the decision rule in advance?
Once you see the data, there is always a reading that favours the preferred option. With the rule written first (adopt, iterate or stop), the decision follows the agreed criterion rather than the mood of the day.
What are the main threats to validity?
Faulty instrumentation, an unbalanced traffic split, stopping the test early, novelty effects, unusual dates, contamination between groups and bot traffic. The checklist shows which ones the plan already covers and which remain open.
What if the test does not fit the deadline?
Increase the minimum effect, put more traffic into the experiment or choose a more sensitive metric closer to the change. If none of that is enough, an A/B test is not the right method: prefer a prototype or qualitative research.