Skip to interview prep
Apexed

65. A recommender improves an offline ranking metric. Why might it still fail to improve the live product?

MediumMachine learning basics

Technical
Show model answer

Offline labels reflect what an earlier system chose to show, and the metric may reward relevance without capturing novelty, diversity, latency or user value. Better ranking on that dataset need not translate into useful behavioral change.

Check how the evaluation data was collected and which candidate items were eligible. Use live validation with guardrails when feasible, and inspect user groups rather than assuming the offline improvement proves causal product benefit.

What interviewers listen for

  • Inspect exposure and label-selection bias
  • Align ranking metrics with user value
  • Validate live impact with guardrails
A recommender improves an offline ranking metric. Why might it still fail to improve the live product — Data & analytics interview question — Apexed