A team forecasts daily store sales with a gradient-boosting model using lag and calendar features. They evaluate with a random 80/20 train–test split and report MAPE of 4%. In production the first month's MAPE is 14%.
What is the most likely cause of the gap?
Why did you pick that answer? Two or three sentences. The act of articulating it is what builds the judgment — not the click that follows.
With a random split, a test day in March is surrounded by training days from the same week, and its lag features are built from actual sales the model also trained on. The model is interpolating, not forecasting. A time-ordered split — train through February, test on March — mimics production, where the future is unknown. Expect the honest number to be closer to 14% than to 4%, and tune against that.
Changing the scoreboard would not close a gap caused by the model seeing the future during evaluation.
The problem is not model strength — the offline score was too good, not too bad. More trees would overfit the leaked information further.
Some degradation is normal; a 3.5× gap is a validation bug. The evaluation must replicate the production setting.