Skip to interview prep
Apexed

38. A delivery marketplace changes dispatch rules. Why might a standard user-level A/B test give a misleading result?

HardStatistics & experiments

Technical
Show model answer

Treatment and control may compete for the same drivers, so changing dispatch for one group alters the other group’s experience. Capacity, geography and time also connect outcomes, violating an assumption that assignments act independently.

Consider geographic clusters or time-based switchbacks with suitable transition periods, depending on operational spillovers. Analyze at the assignment unit and account for seasonality and carryover. Explain remaining contamination and validate customer, driver and efficiency guardrails before recommending wider use.

What interviewers listen for

  • Shared-resource interference
  • Appropriate assignment unit
  • Carryover and seasonality
  • Multiple-side guardrails
A delivery marketplace changes dispatch rules. Why might a standard user-level A/B test give a misleading result — Data & analytics interview question — Apexed