Skip to interview prep
Apexed

66. A very large experiment detects a tiny improvement with a small p-value. Does that make the feature worth shipping?

EasyMetrics & analytics

Technical
Show model answer

Statistical evidence against a null hypothesis does not establish that the improvement is valuable enough to justify the change. Examine the effect size, uncertainty, affected population and relevant user or business outcome.

Compare expected benefit with implementation, maintenance and opportunity costs, plus guardrail effects. A small gain may matter at scale, but explain that case explicitly rather than treating significance as a product decision.

What interviewers listen for

  • Interpret effect size and uncertainty
  • Estimate value at realistic scale
  • Compare full costs and guardrails
A very large experiment detects a tiny improvement with a small p-value. Does that make the feature worth shipping — Product management interview question — Ape