Skip to interview prep
Apexed

54. Some users assigned a new feature never use it. Why might comparing actual users with nonusers mislead?

MediumStatistics & experiments

Technical
Show model answer

Use is a choice that may reflect motivation, need or access. Comparing users with nonusers therefore discards the protection of randomized assignment and can confound the feature's effect with pre-existing differences.

An intention-to-treat comparison estimates the effect of offering or assigning the feature, including incomplete uptake. Define that estimand clearly. Estimating an effect among actual users requires additional assumptions or a suitable design, not simply filtering to engaged users.

What interviewers listen for

  • Preserve the randomized comparison
  • Define the effect of assignment
  • Do not treat uptake as random
Some users assigned a new feature never use it. Why might comparing actual users with nonusers mislead — Data & analytics interview question — Apexed