Product Experimentation

Sample questions

Two questions from this practice test, in the same style as the certification exam. Answer before opening the explanation.

1. Question 1

   A ride-hailing app tests a change to its pricing algorithm. The experiment randomises passengers: half see the new price and half the old one, in the same city and over the same period. The same drivers serve both groups. The result shows a 6 per cent rise in completed rides in the test group.

   1. Accept the result, because the randomisation was correctly done by passenger
   2. Distrust the result, because both groups compete for the same supply of drivers
   3. Repeat the test with a larger sample to confirm the 6 per cent rise observed
   4. Accept the result and monitor driver satisfaction over the following period
   Show answer and explanation

   Correct answer: B. Distrust the result, because both groups compete for the same supply of drivers

   Randomisation guarantees the two groups are comparable at the start, and that is not enough when they compete for the same limited resource. Here the passengers in the test group, by paying a different price, change the availability of drivers the control group finds. What is measured stops being the effect of the change and becomes the effect plus whatever one group took from the other. If the change merely redistributes rides between the two groups, the test shows a gain and the whole operation gains nothing once the change is applied to everyone. Accepting the result is attractive because randomising by passenger looks impeccable and intuition equates correct randomisation with a valid result. Enlarging the sample is attractive because more data always seems better, and here it only measures with greater precision a number that is already wrong, which is the worst possible combination: high confidence in a biased estimate. Monitoring driver satisfaction addresses a consequence and not the cause. The condition an experiment requires is that what happens to one unit does not affect the outcome of the others. When that condition falls, the answer lies in changing the unit of randomisation, for example alternating the configuration by region or by time window.

2. Question 2

   A team tests a new sign-up flow and randomises by session. A returning user may see the old flow on one visit and the new one on another. The declared primary metric is sign-up completion rate per user, and about 40 per cent of users make more than one visit before completing.

   1. Keep randomising by session, because it produces a larger sample in the same time
   2. Keep randomising by session and analyse the result by session, not by user
   3. Keep the randomisation and exclude from the analysis users who saw both versions
   4. Randomise by user, because the metric and the experience belong to the user
   Show answer and explanation

   Correct answer: D. Randomise by user, because the metric and the experience belong to the user

   The unit of randomisation has to follow the unit where the effect happens and where the decision will be taken. Signing up is a process a person goes through across several visits, and the decision to launch will be about the user’s experience, not the experience of an isolated session. By randomising per session, four in ten people go through both flows, which mixes the treatments inside the same unit of analysis and compresses the difference between groups: the test starts underestimating both the gain and the loss. Keeping it by session for sample size is attractive because more units appear to raise the power of the test, and what is gained in count is lost in purity of contrast. Analysing by session seems to fix things by aligning analysis with randomisation, and it remains invalid, because the session of a user who has already seen the other flow carries its effect. Excluding those who saw both versions is the most dangerous path: the exclusion happens after the draw and depends on user behaviour, which breaks the comparability randomisation had created and reintroduces selection on a criterion correlated with the outcome. The practical rule is to randomise at the broadest unit where the experience is consistent.