74. An experiment has no statistically significant increase in serious errors. Can the team claim the feature is equally safe?
HardMetrics & analytics
Technical
Show model answer
Failure to detect a difference does not establish equivalence, especially when serious errors are rare and the experiment has limited information. Review uncertainty, exposure and the largest plausible deterioration compatible with the evidence.
Agree an acceptable risk margin and an appropriate evaluation approach with the responsible experts. Combine relevant testing and monitoring, and state what remains uncertain instead of translating a nonsignificant result into proof of safety.
What interviewers listen for
- Distinguish absence of evidence from equivalence
- Examine exposure and uncertainty
- Define an acceptable risk margin