Reference · print it
The Course on One Page
Every template, formula, and checklist from the thirteen sessions. Keep it open during homework; bring it to your project presentation. Each block names the session that teaches it.
01 // FRAME IT · SESSION 1
Four lines before the keyboard
Unit
One row = one what? A customer, an order, an hour, a house.
Target
What exactly is predicted — a label (yes/no), a number, a count?
Horizon
How far ahead, and at what moment is the prediction made?
Decision
What changes if the prediction is right? Who owns that decision today?
Then name the baseline: majority class, the mean, or same-period-last-time. Every model must beat it.
02 // LEAKAGE TEST · SESSIONS 2, 3, 9, 11
Seven questions before you trust a score
- Would I know this value at the moment I need the prediction?
- Is it recorded after the outcome (call duration, reservation status, follow-up visits)?
- Is it downstream of a decision the model will influence (offer sent, line reduced)?
- Is it aggregated over a window that includes the future (lifetime average)?
- Did any fitted transformer (scaler, imputer, encoder, selector) see the test rows?
- Does the split match production — new customers (GroupKFold), later dates (TimeSeriesSplit)?
- Did the score jump to near-perfect from one column? That is a leak, not a discovery.
03 // THE TEMPLATE · SESSION 3
The pipeline every project starts from
Split first. Everything that learns from data lives inside the Pipeline. Swap the last step; nothing else changes.
python
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=835)
num = X.select_dtypes(include='number').columns
cat = X.columns.difference(num)
prep = ColumnTransformer([
('num', Pipeline([('imp', SimpleImputer(strategy='median')), ('sc', StandardScaler())]), num),
('cat', OneHotEncoder(handle_unknown='ignore'), cat),
])
model = Pipeline([('prep', prep), ('clf', ANY_ESTIMATOR)])
cv = StratifiedKFold(5, shuffle=True, random_state=835)
print(cross_val_score(model, X_tr, y_tr, cv=cv, scoring='roc_auc').mean()) # tune here
model.fit(X_tr, y_tr) # then touch the test set once 04 // METRIC MENU · SESSIONS 4, 5, 11
Which number for whom
| Task | Metric | Reads as | Report to |
|---|---|---|---|
| Regression | MAE | typical miss, in units | operations, sales |
| Regression | RMSE | typical miss, big misses count extra | risk, finance |
| Regression | MAPE | typical miss as a % | executives — never near zero |
| Regression | R² | share of variance explained vs. the mean | analysts |
| Classification | ROC-AUC | ranking quality; P(positive scores above negative) | model comparison, balanced classes |
| Classification | PR-AUC | precision across recall levels | rare positives (fraud, default) |
| Classification | Precision / Recall | "of those flagged…" / "of those who actually…" | at the chosen threshold |
| Classification | Brier / calibration curve | does 0.8 mean 80%? | before any dollar calculation |
| Forecasting | MAE vs. seasonal naive | gain over "same hour last week" | always report the baseline |
05 // THE CONFUSION MATRIX · SESSION 4
Four cells, four business meanings
| predicted: yes | predicted: no | |
| actually yes | TP · caught | FN · missed (the expensive cell) |
| actually no | FP · false alarm (wasted action) | TN · left alone |
Precision = TP / (TP + FP) — of those we flagged, how many were right?
Recall = TP / (TP + FN) — of those who actually were, how many did we catch?
Accuracy = (TP + TN) / N — averages both mistakes; hides which one you make.
F1 = harmonic mean of precision and recall — assumes both mistakes cost the same. They never do.
06 // FROM SCORES TO DECISIONS · SESSION 5
The threshold is a business decision
Break-even probability
p* = cost of action ÷ (cost of action + value of a catch)
A $50 offer and a $450 save → p* = 0.10. Act on anyone above it. With a contact budget, sort by score and take the top k instead (gain chart). Check calibration before the probabilities enter any dollar arithmetic; a churn score is not an uplift score.
python
from sklearn.model_selection import TunedThresholdClassifierCV
from sklearn.metrics import make_scorer, confusion_matrix
def profit(y_true, y_pred): # write the business scorer in dollars
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
return VALUE_OF_CATCH * tp - COST_OF_ACTION * (tp + fp)
tuned = TunedThresholdClassifierCV(model, scoring=make_scorer(profit), cv=5).fit(X_tr, y_tr)
print(tuned.best_threshold_) # break-even ≈ cost / (cost + value) 07 // VALIDATION CHOOSER · SESSION 9
Match the splitter to production
Rows are independent, label is rareStratifiedKFold
The same customer / agent / patient appears many timesGroupKFold (or StratifiedGroupKFold)
Time matters — tomorrow is predicted from yesterdayTimeSeriesSplit; test period after training
Small dataset, honest final number needednested CV (tune inside, evaluate outside)
Hyperparameter search budgetRandomizedSearchCV → Optuna; tune 2–3 knobs
08 // MODEL CHOOSER · SESSIONS 4–12
The 2026 default stack
Need to explain every coefficientLogistic / linear regression (+ ridge or lasso)S4 →Tabular business data, any sizeHistGradientBoosting / LightGBM / XGBoost, early-stoppedS8 →Tabular, < 50k rows, no time to tuneTabICL or TabPFN (foundation model) as instant baselineS8 →Want a strong no-tuning ensemble fastRandomForest (300 trees, min_samples_leaf=5)S7 →Text columnTF-IDF → logistic; or sentence embeddings → any modelS12 →Time seriesseasonal naive → StatsForecast → lags + GBM → ChronosS11 →No labelsk-means on scaled RFM; DBSCAN for shape; IsolationForest for anomaliesS10 →
Knobs that matter for boosting: learning rate × number of trees (with early stopping), then leaves / depth, then regularization. Trees need no scaling; distance and gradient models do.
09 // TRUST · SESSIONS 9, 13
The four questions an executive will ask
Why did it say that?
SHAP waterfall (one case) and beeswarm (the book); partial dependence for the shape.
How sure is it?
Calibration curve first; conformal prediction (MAPIE) for 90% sets or intervals that keep their promise.
Is it fair?
False-negative and false-positive rates by group. Equal accuracy can hide unequal harm.
What happens after launch?
Persist the pipeline + threshold; monitor input drift weekly; retrain trigger; model card with an owner.
10 // THE MEMO · PROJECT
Five sections, one page each at most
1
The question
2
The data & the model
3
The honest evaluation
4
The decision & the money
5
The trust story
Full brief, rubric, and deadlines on the Project page.