Skip to interview prep
Apexed

77. Two reviewers disagree on whether an answer is correct. How would you improve the evaluation before adding more labels?

MediumStatistics & experiments

Technical
Show model answer

Inspect disagreements to determine whether the task, reference answer or scoring rule is ambiguous. Create explicit criteria and examples for borderline cases, and use independent labeling before discussion where practical.

Measure agreement using a method suited to the label structure and inspect per-category disagreements. Adjudicate important cases and revise the rubric transparently; forcing a majority label without understanding the disagreement can conceal a poorly defined evaluation target.

What interviewers listen for

  • Investigate rubric ambiguity
  • Preserve independent initial judgments
  • Adjudicate and document important disagreements
Two reviewers disagree on whether an answer is correct. How would you improve the evaluation before adding more labels — Data & analytics interview question — A