← ISOM 835 · Session 2
Interactive · When Did You Know That?

The Leakage Time Machine

A model is only as honest as the moment it pretends to live in. Drag the prediction moment along the timeline and watch three things change: which columns exist yet, how good the model looks, and whether there is still a decision left to make. Every number below comes from a real gradient-boosting model trained on the real data with exactly the features that exist at that moment — nothing is simulated.

Model quality by moment · ROC-AUC
At this moment
—
ROC-AUC on held-out data—
Caught in the top 10%—
Precision of that top slice—
Features the model can see—

The one question that catches every leak

Would this value exist, in the database, at the moment the decision is made? Not "is it correlated with the target" — the leaks are the most correlated columns. Not "does it feel like cheating" — duration feels like an innocent column about the call. The question is about time: a value written after the decision cannot inform it, and a value written after the outcome is the outcome wearing a costume.

Two habits that follow. (1) Write the prediction moment at the top of every notebook — "prediction is made at booking / before the call / on the 1st of the month" — and audit every column against it, which is exactly Homework 1, question 4. (2) When a score jumps by ten points after you add one column, do not celebrate; ask what time it is.

Session 2 builds this on Olist and Telco · the bank data is UCI Bank Marketing (its own documentation warns that duration "should be discarded if the intention is to have a realistic predictive model") · the hotel data is the course file, 83,573 bookings.