58. The same AI agent passes a task once and fails it on another run. How should that affect your evaluation plan?
HardAI products
Technical
Show model answer
Recognize that a single successful run is weak evidence of reliability when behavior varies. Use repeated trials on a defined set of representative tasks, recording versions, relevant settings and consequential failure types.
Report the success distribution and uncertainty alongside cost and completion time. Avoid tuning to a small visible set, and inspect trajectories when outcome scores alone do not explain a failure.
What interviewers listen for
- Use repeated versioned trials
- Report reliability and uncertainty
- Inspect consequential failure trajectories