Skip to interview prep
Apexed

65. A team checks a fixed-horizon experiment daily and stops the first time the result looks significant. What is the concern?

MediumMetrics & analytics

Technical
Show model answer

Repeatedly choosing when to stop based on favorable results changes the error properties of a fixed-horizon test. A nominal significance threshold does not retain its original interpretation under arbitrary repeated looks.

Agree the decision rule and analysis approach in advance with the experimentation team. Use a suitable sequential method when continuous decisions are necessary, and retain guardrail monitoring for customer harm throughout.

What interviewers listen for

  • Predefine the stopping rule
  • Use an appropriate sequential method
  • Continue harm monitoring
A team checks a fixed-horizon experiment daily and stops the first time the result looks significant. What is the concern — Product management interview questio