47. An AI feature improves its average score but makes more mistakes on a small high-impact workflow. Would you ship it?
MediumAI products
Case
Show model answer
Inspect the affected workflow, error severity, evaluation coverage and uncertainty before deciding. An average improvement can conceal a regression that makes the product unsuitable for an important use.
Consider limiting the release, retaining the earlier behavior for that workflow or delaying until a mitigation is validated. Establish explicit acceptance criteria with the responsible domain owner and monitor the affected group after release.
What interviewers listen for
- Inspect high-impact slices
- Consider a bounded release
- Set explicit regression criteria