71. Before using an LLM to grade thousands of answers, how would you assess the grader itself?
HardMachine learning basics
Technical
Show model answer
Define a clear rubric and compare the grader with independently reviewed examples, including ambiguous answers and consequential errors. Examine disagreements, consistency and sensitivity to presentation rather than relying on a few agreeable demonstrations.
Keep a separate validation set for the grader and review important error categories manually. Record its prompt and model version, and revalidate changes; automated scores inherit the grader's limitations and do not become ground truth through scale.
What interviewers listen for
- Validate against independent judgments
- Inspect consequential disagreements
- Version and revalidate the grader