47. An experiment measures twenty outcomes and only one has a small p-value. How would you interpret that result?
MediumStatistics & experiments
Technical
Show model answer
Check which outcome and decision rule were specified before the experiment. Searching many outcomes increases the chance of finding an apparently unusual result by chance, even when no treatment effect exists.
Use a multiplicity approach suited to the decision, or label the finding exploratory and confirm it independently. Report effect sizes, intervals and the full set of tested outcomes instead of presenting the selected result as a prespecified success.
What interviewers listen for
- Distinguish planned from selected outcomes
- Account for multiple comparisons
- Confirm exploratory findings independently