Certified Product Experimentation Specialist

Who this is for

• You design and interpret experiments

  And you have noticed the hard part is not running the test: it is deciding the unit of randomisation and the metric before seeing any number.

• You answer for the decision to launch or drop

  The conclusion leaves your report and becomes product. When a result is inconclusive, you are the one explaining how that differs from "it does not work".

• You have watched a test be read the wrong way

  Stopping at the first significant reading, a finding in one segment out of eighteen, a comparison with last month. And you want the argument to refuse each one.

What you take with you

• A permanent verification code

  Anyone confirms your credential at pmcert.org/verify, without having to ask you for anything.

• Approval that does not expire

  A credential you earned does not lapse, asks for no annual renewal and charges no subscription.

• LinkedIn and a public profile

  Add it to your licenses and certifications section, or share the link to your profile.

• A certificate with your score by domain

  In PDF, showing how you performed in each area assessed.

See what the questions are like

Two questions from the free open assessment. The certification exam uses different ones in the same format: a work situation and a decision.

1. A ride-hailing app tests a change to its pricing algorithm. The experiment randomises passengers: half see the new price and half the old one, in the same city and over the same period. The same drivers serve both groups. The result shows a 6 per cent rise in completed rides in the test group.

   1. AAccept the result, because the randomisation was correctly done by passenger
   2. BDistrust the result, because both groups compete for the same supply of drivers
   3. CRepeat the test with a larger sample to confirm the 6 per cent rise observed
   4. DAccept the result and monitor driver satisfaction over the following period
   Show answer and explanation

   Answer: B

   Randomisation guarantees the two groups are comparable at the start, and that is not enough when they compete for the same limited resource. Here the passengers in the test group, by paying a different price, change the availability of drivers the control group finds. What is measured stops being the effect of the change and becomes the effect plus whatever one group took from the other. If the change merely redistributes rides between the two groups, the test shows a gain and the whole operation gains nothing once the change is applied to everyone. Accepting the result is attractive because randomising by passenger looks impeccable and intuition equates correct randomisation with a valid result. Enlarging the sample is attractive because more data always seems better, and here it only measures with greater precision a number that is already wrong, which is the worst possible combination: high confidence in a biased estimate. Monitoring driver satisfaction addresses a consequence and not the cause. The condition an experiment requires is that what happens to one unit does not affect the outcome of the others. When that condition falls, the answer lies in changing the unit of randomisation, for example alternating the configuration by region or by time window.

2. A team tests a new sign-up flow and randomises by session. A returning user may see the old flow on one visit and the new one on another. The declared primary metric is sign-up completion rate per user, and about 40 per cent of users make more than one visit before completing.

   1. AKeep randomising by session, because it produces a larger sample in the same time
   2. BKeep randomising by session and analyse the result by session, not by user
   3. CKeep the randomisation and exclude from the analysis users who saw both versions
   4. DRandomise by user, because the metric and the experience belong to the user
   Show answer and explanation

   Answer: D

   The unit of randomisation has to follow the unit where the effect happens and where the decision will be taken. Signing up is a process a person goes through across several visits, and the decision to launch will be about the user’s experience, not the experience of an isolated session. By randomising per session, four in ten people go through both flows, which mixes the treatments inside the same unit of analysis and compresses the difference between groups: the test starts underestimating both the gain and the loss. Keeping it by session for sample size is attractive because more units appear to raise the power of the test, and what is gained in count is lost in purity of contrast. Analysing by session seems to fix things by aligning analysis with randomisation, and it remains invalid, because the session of a user who has already seen the other flow carries its effect. Excluding those who saw both versions is the most dangerous path: the exclusion happens after the draw and depends on user behaviour, which breaks the comparability randomisation had created and reintroduces selection on a criterion correlated with the outcome. The practical rule is to randomise at the broadest unit where the experience is consistent.

• A/B test sample size calculatorHow many people each variant needs to see, and for how many weeks, for the test to detect the effect that matters.Open
• A/B test significance calculatorVisitors and conversions for each variant become a p-value, a difference interval and a plain-language verdict.Open
• Experiment plannerHypothesis, metrics, sample, duration and decision rule in a single document, agreed before you start.Open
• Hypothesis generatorFrom observation to a sentence a test can confirm or knock down, with metric, number and timeframe.Open