77. Two reviewers disagree on whether an answer is correct. How would you improve the evaluation before adding more labels?
MediumStatistics & experiments
Technical
Show model answer
Inspect disagreements to determine whether the task, reference answer or scoring rule is ambiguous. Create explicit criteria and examples for borderline cases, and use independent labeling before discussion where practical.
Measure agreement using a method suited to the label structure and inspect per-category disagreements. Adjudicate important cases and revise the rubric transparently; forcing a majority label without understanding the disagreement can conceal a poorly defined evaluation target.
What interviewers listen for
- Investigate rubric ambiguity
- Preserve independent initial judgments
- Adjudicate and document important disagreements