60. A team checks a campaign experiment hourly and stops as soon as the result looks favorable. What is the concern?
MediumAnalytics & budget
Case
Show model answer
Repeated unplanned looks and selective stopping can change the error properties of the analysis. The most favorable moment may reflect random variation rather than a stable improvement, even if the dashboard displays a familiar significance threshold.
Agree a suitable fixed or sequential design before launch. Use predefined stopping and safety rules, disclose deviations, and do not present a selected snapshot as the originally planned procedure.
What interviewers listen for
- Recognize selection from repeated looks
- Use an appropriate planned design
- Disclose stopping rules and deviations