38. A delivery marketplace changes dispatch rules. Why might a standard user-level A/B test give a misleading result?
HardStatistics & experiments
Technical
Show model answer
Treatment and control may compete for the same drivers, so changing dispatch for one group alters the other group’s experience. Capacity, geography and time also connect outcomes, violating an assumption that assignments act independently.
Consider geographic clusters or time-based switchbacks with suitable transition periods, depending on operational spillovers. Analyze at the assignment unit and account for seasonality and carryover. Explain remaining contamination and validate customer, driver and efficiency guardrails before recommending wider use.
What interviewers listen for
- Shared-resource interference
- Appropriate assignment unit
- Carryover and seasonality
- Multiple-side guardrails