A model predicts whether a support ticket will be escalated. The dataset has 80,000 tickets from 6,000 customers; many customers have dozens of tickets, and tickets from the same customer look alike. A 5-fold KFold cross-validation reports AUC 0.91.
Why might 0.91 be optimistic, and which splitter fixes it?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
The unit the model will face in production is a new customer, but plain KFold lets the model see the same customer's other tickets during training — a form of leakage across rows. GroupKFold keeps every customer entirely inside one fold, so the validation score reflects generalization to customers the model has never seen. Stratification is a separate concern and can be combined (StratifiedGroupKFold).
More folds reduce variance of the estimate, not its bias. The leak across customers is the same at 5 or 10 folds.
Stratifying balances the label across folds; it does nothing about the same customer appearing on both sides of the split.
KFold assumes rows are exchangeable. Repeated customers, time order, and hierarchies all violate that assumption.