37. Why can randomly splitting rows give an overoptimistic predictive result when several rows come from the same specimen or site?
HardData & reproducibility
Technical
Show model answer
Related rows can share information, allowing the test set to resemble observations already seen during training. The evaluation may then measure recognition of specimens or sites rather than generalisation to genuinely new units.
Split or validate at the level relevant to the intended use, preserving temporal or group boundaries where appropriate. Fit preprocessing only on training information and explain which future conditions the evaluation does and does not represent.
What interviewers listen for
- Identify shared information across rows
- Validate at the intended deployment unit
- Keep test information out of fitting