26. A researcher tests many outcomes and highlights only the smallest p-value. What is the problem?
HardStatistics & inference
Technical
Show model answer
The selection process increases the opportunity to find an apparently unusual result by chance, while hiding the full set of analyses. A single reported p-value does not describe that search.
Distinguish planned and exploratory outcomes, disclose the testing family and use an appropriate multiplicity strategy where needed. Confirm important discoveries with suitable independent evidence instead of presenting the most favourable comparison as the only test conducted.
What interviewers listen for
- Account for the analysis search
- Disclose outcome selection
- Plan adjustment or confirmation