A colleague tunes a churn model by trying 40 hyperparameter settings, checking accuracy on the 20% test set after each one, and reporting the best test accuracy she found: 84.1%.
What is wrong with the reported 84.1%?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
Every time you pick the setting with the best test score, you are fitting a choice to the test set. After 40 looks, the winner is partly the one that got lucky on those particular customers, so its score overstates future performance. Tune on a validation split (or cross-validation) and touch the test set once, at the end.
Selection is training in slow motion. The test set stayed out of .fit() but it drove the decision, and that is enough to bias the estimate.
More trials make it worse: the more candidates you compare on the same test set, the luckier the winner is likely to be.
Size reduces noise, not the bias from repeatedly choosing against the same data. A 50/50 split also throws away half the training data.