Skip to interview prep
Apexed

58. The same AI agent passes a task once and fails it on another run. How should that affect your evaluation plan?

HardAI products

Technical
Show model answer

Recognize that a single successful run is weak evidence of reliability when behavior varies. Use repeated trials on a defined set of representative tasks, recording versions, relevant settings and consequential failure types.

Report the success distribution and uncertainty alongside cost and completion time. Avoid tuning to a small visible set, and inspect trajectories when outcome scores alone do not explain a failure.

What interviewers listen for

  • Use repeated versioned trials
  • Report reliability and uncertainty
  • Inspect consequential failure trajectories
The same AI agent passes a task once and fails it on another run. How should that affect your evaluation plan — Product management interview question — Apexed