Cimentar Futuro Foundation

Learning Objectives

Design a valid A/B test with a clear hypothesis
Avoid common statistical pitfalls like stopping a test too early
Interpret results honestly, including inconclusive ones

An A/B test randomly splits users into two groups, one seeing the current version (control) and one seeing a proposed change (variant), to measure the change's actual effect on a chosen metric rather than relying on opinion. A valid test starts with a clear, falsifiable hypothesis ("adding a progress bar will increase checkout completion by at least 5%") and needs a sufficient sample size and duration to reach statistical significance; stopping a test early just because results look favorable is one of the most common ways teams fool themselves, since early results are often noise that regresses toward the true effect over time. You will also practice interpreting inconclusive results honestly rather than cherry-picking a secondary metric that happened to move favorably.

Module Quiz

1. Stopping an A/B test early because early results look favorable is risky because:

It guarantees a false negativeEarly results are always completely accurateIt is illegal to stop a test earlyEarly results are often noisy and can regress toward the true effect with more data

2. In an A/B test, the 'control' group refers to:

The group seeing the new, proposed versionThe group that continues seeing the current, unchanged versionThe engineering team running the testThe group that is excluded from all results

3. If a test's primary metric shows no significant change but a secondary metric moved favorably by chance, a PM should:

Report honestly that the primary hypothesis was not supportedHide the test results entirelyReport the secondary metric as the headline resultImmediately roll out the change to all users regardless

Module 9: A/B Testing & Experimentation