54. Some users assigned a new feature never use it. Why might comparing actual users with nonusers mislead?
MediumStatistics & experiments
Technical
Show model answer
Use is a choice that may reflect motivation, need or access. Comparing users with nonusers therefore discards the protection of randomized assignment and can confound the feature's effect with pre-existing differences.
An intention-to-treat comparison estimates the effect of offering or assigning the feature, including incomplete uptake. Define that estimand clearly. Estimating an effect among actual users requires additional assumptions or a suitable design, not simply filtering to engaged users.
What interviewers listen for
- Preserve the randomized comparison
- Define the effect of assignment
- Do not treat uptake as random