65. A team checks a fixed-horizon experiment daily and stops the first time the result looks significant. What is the concern?
MediumMetrics & analytics
Technical
Show model answer
Repeatedly choosing when to stop based on favorable results changes the error properties of a fixed-horizon test. A nominal significance threshold does not retain its original interpretation under arbitrary repeated looks.
Agree the decision rule and analysis approach in advance with the experimentation team. Use a suitable sequential method when continuous decisions are necessary, and retain guardrail monitoring for customer harm throughout.
What interviewers listen for
- Predefine the stopping rule
- Use an appropriate sequential method
- Continue harm monitoring