100. A model scores highly on synthetic evaluation data generated by a similar model. What would you require before using that score as deployment evidence?
HardBusiness cases
Case
Show model answer
Inspect whether the synthetic tasks cover the real population and whether shared generation patterns make them unusually easy for the evaluated system. Check label validity, diversity and possible contamination rather than assuming synthetic scale creates representativeness.
Validate on independently reviewed real or realistically sourced cases that are authorized for evaluation. Keep synthetic cases for useful controlled tests, but report their origin and limits separately from evidence about actual operating conditions.
What interviewers listen for
- Inspect generation and label artifacts
- Check realistic coverage independently
- Separate controlled tests from deployment evidence