63. An experiment assigned to split traffic equally reports a persistent 60/40 exposure split. What should the PM do before interpreting conversion?
MediumMetrics & analytics
Case
Show model answer
Treat the mismatch as a reason to investigate assignment, eligibility, exposure logging and data processing. Check whether the treatment itself changes which users appear in the analysis; filtering after assignment can distort the comparison.
Involve the experimentation owner and preserve diagnostic data. Do not explain away the split or select a favorable result until the population and measurement issue is understood.
What interviewers listen for
- Check assignment and exposure integrity
- Inspect treatment-dependent filtering
- Resolve validity before interpreting lift