Skip to interview prep
Apexed

7. How would you design an A/B test?

MediumStatistics & experiments

Technical
Show model answer

Define one primary metric and guardrail metrics, the hypothesis, and the minimum effect worth detecting. Calculate the sample size for your significance level and power, randomise at the right unit (user, not page view), and fix the duration in advance (cover full weekly cycles).

Check sample ratio mismatch, avoid peeking and stopping early, then analyse with confidence intervals and consider novelty effects and segments.

What interviewers listen for

  • Primary metric and guardrails
  • Sample size and power
  • Randomisation unit
  • No peeking
How would you design an A/B test — Data & analytics interview question — Apexed