Cross-validation that doesn't lie, hyperparameter search with Optuna, SHAP for one customer and the whole book, conformal intervals that keep their promise, fairness slices, and the one-page model card
By the end of tonight you'll have both the concept and the working code — every objective maps to a cell you run yourself.
StratifiedKFold for rare labels, GroupKFold when the same customer appears many times, TimeSeriesSplit when tomorrow must be predicted from yesterday. The wrong splitter is the most common way a great offline score becomes a bad live one.
RandomizedSearchCV, HalvingRandomSearchCV, and a ten-line Optuna study — tuned on training folds, judged once on a test set you never touched.
TreeExplainer on the boosted hotel model: a waterfall for one booking, a beeswarm for the whole book, and the sentence you say to an executive about each.
Conformal prediction with MAPIE turns any model into one with 90% intervals — a promise you can keep. Then slice every metric by group, and write the model card that says what the model is for and where it fails.
A business question, a dataset, a model — and the judgment to evaluate it.
Three true stories: a support-ticket model that memorized customers (plain KFold on repeated customers), a demand forecast that saw the future (random split on time series), and a competition team that tuned on the public leaderboard. All three had the same disease — the validation didn't match production.
Grid search is exhaustive and slow; random search finds a near-best point in a tenth of the budget; successive halving spends the budget on promising configurations; Optuna learns from each trial. A ten-line Optuna objective on the hotel GBM, with the CV inside and the test set outside.
Shapley values split a prediction fairly among the features. TreeExplainer computes them exactly for boosted trees in milliseconds. A waterfall plot for booking #4,117: base rate 37% → non-refundable deposit +28 points → 200-day lead time +9 → repeat guest −6 → 68% cancellation probability.
Stretch, coffee, project teams meet.
The beeswarm: every booking, every feature, colored by value — the global picture in one plot. Mean |SHAP| as the honest importance ranking. Then PartialDependenceDisplay with ICE lines: the shape of lead time's effect, and where individual bookings deviate from the average.
A point forecast is a bet. Conformal prediction uses a calibration set's residuals to wrap any model in intervals with guaranteed coverage — 90% means 90%, no distributional assumptions. MAPIE's SplitConformalRegressor on Ames prices, SplitConformalClassifier on hotel cancellations (prediction sets: {cancel}, {stay}, or {both} when the model is unsure).
Equal accuracy can hide unequal harm: compute false-negative and false-positive rates by group (customer type, market segment, country) and look for gaps. Then the one-page model card — intended use, data, metrics by slice, known failure modes, owner. Milestone 3 of the final project is assigned in full — tonight is where your tuned model and error analysis come from.
Why one split is a coin flip and five splits are a measurement.
Notebooks, readings, explorers, and the stories behind the methods.
GroupKFold and TimeSeriesSplit demos, an Optuna study on the hotel GBM, SHAP waterfall and beeswarm, partial dependence with ICE, MAPIE conformal intervals and prediction sets, fairness slices, and a model-card template.
Open in ColabSet a coverage target, watch the calibration residuals size the band, then draw new cases a hundred times and check the promise. Underfit the model, shrink the calibration set, or shift the data to see what breaks it.
Open the explorerK-fold, animated: watch the folds rotate and the estimate stabilize.
Open the explainerThe self-contained tutorial that made conformal prediction mainstream: the split-conformal recipe, coverage guarantees, and extensions to classification, time series, and distribution shift.
Read the paperAn explorable on why equal accuracy can hide unequal error rates, and the trade-offs between fairness definitions. Do this before the fairness-slicing lab.
Open the explainerA fairness criterion you can drag: equalize true- and false-positive rates across groups and see what it costs.
Open the explainerThe reference for SHAP, partial dependence, ICE, permutation importance, and counterfactuals — with the honest caveats for each.
Read onlineMicrosoft's open-source toolkit for assessing and mitigating fairness issues: MetricFrame for sliced metrics is the one-liner behind tonight's fairness section.
Open the docsEvery splitter with a picture: KFold, StratifiedKFold, GroupKFold, StratifiedGroupKFold, TimeSeriesSplit. The visualization section alone is worth the visit.
Open the guideThe canonical walkthrough from the library's authors: what Shapley values are, why they are the fair attribution, and how to read every plot type we use tonight.
Read the introductionscikit-learn-contrib's conformal prediction library. The 1.x API (SplitConformalRegressor, CrossConformalRegressor, SplitConformalClassifier) is what we use; the theory pages are short and readable.
Open the docsDefine an objective, call study.optimize, read study.best_params. The tutorial's first example is the ten lines we write in class.
Open the sitePermutation importance, partial dependence, and SHAP values in four short lessons with exercises. The fastest path to fluency before Milestone 3.
Start the courseZuckerberg San Francisco General replaced a vendor readmission model with an in-house gradient-boosted tree that includes social-determinants data — and reports lower readmissions and narrower equity gaps. Explanation, slicing, and monitoring in a real hospital.
Read the storyThe paper that defined the one-page model card: intended use, factors, metrics, evaluation data, ethical considerations, caveats. Your final project includes one.
Read the paperk-fold, stratified, leave-one-out, shuffle-split, and grouped cross-validation, then grid search with nested CV. The slow version of tonight's first hour.
Open in ColabOpen the notebook and build it with us.
Take the early-stopped hotel GBM from Session 8. Explain ten bookings with SHAP, wrap the model in 90% conformal prediction sets, and check whether the error rates differ between city and resort guests.
shap.TreeExplainer(gbm) → shap_values on the test set. Waterfall for one booking, beeswarm for all. Write one sentence for each.
Split the training data into fit and calibration halves. MAPIE SplitConformalClassifier with confidence_level=0.9. Count how many test bookings get a two-label set — those are the ones to route to a human.
False-negative rate and false-positive rate at your chosen threshold, by hotel type and by customer_type. Any gap above 5 points gets a sentence in the model card.
Fill the model-card template: intended use, training data, metrics overall and by slice, known failure modes, owner, review date.
import shap, numpy as np, pandas as pd
from mapie.classification import SplitConformalClassifier
from sklearn.model_selection import train_test_split
# gbm, X_tr, X_te, y_tr, y_te: the early-stopped HistGradientBoosting model and split from Session 8
explainer = shap.TreeExplainer(gbm)
sv = explainer(X_te.iloc[:2000])
shap.plots.waterfall(sv[0]) # one booking: base rate → each feature's push → prediction
shap.plots.beeswarm(sv, max_display=12) # the whole book
# Conformal prediction sets: fit on one half, calibrate on the other, 90% coverage
X_fit, X_cal, y_fit, y_cal = train_test_split(X_tr, y_tr, test_size=0.3, stratify=y_tr, random_state=835)
cp = SplitConformalClassifier(estimator=gbm, confidence_level=0.9, prefit=True)
cp.conformalize(X_cal, y_cal)
y_pred, y_sets = cp.predict_set(X_te)
n_labels = y_sets[:, :, 0].sum(axis=1)
print(f'coverage: {np.mean(y_sets[np.arange(len(y_te)), y_te.values, 0]):.3f} '
f'singletons: {np.mean(n_labels == 1):.1%} ambiguous {{both}}: {np.mean(n_labels == 2):.1%}')
# Fairness slice: error rates by hotel type at threshold 0.5
pred = gbm.predict(X_te)
for grp, idx in X_te.groupby('hotel', observed=True).groups.items():
yt, yp = y_te.loc[idx], pd.Series(pred, index=X_te.index).loc[idx]
fnr = ((yt == 1) & (yp == 0)).sum() / (yt == 1).sum(); fpr = ((yt == 0) & (yp == 1)).sum() / (yt == 0).sum()
print(f'{grp:13s} FNR {fnr:.3f} FPR {fpr:.3f}') Three questions an executive will ask, three tools. Why did it say that? — the SHAP waterfall. How sure is it? — the conformal set; the bookings labeled {both} are exactly the ones a human should look at. Is it fair? — the sliced error rates. A model that can answer all three is one you can defend in front of a regulator, a customer, or a board. The model card is where the answers live.
Match the splitter to production: GroupKFold when units repeat, TimeSeriesSplit when time matters, stratify when labels are rare. Tune on the training folds; touch the test set once.
SHAP answers 'why this prediction?' locally and 'what drives the model?' globally. It attributes the model's output; it does not tell you what happens if you change the input.
Conformal prediction turns any model into one with guaranteed coverage, and sliced error rates turn 'is it fair?' into a table. Together with a model card, that is what 'trustworthy' means in practice.
Complete these before next Monday — they are the practice that makes the next session land.
Exam-style questions for the ideas in this session — Trust & Structure. Every wrong answer names the misconception behind it.
Learning without labels: RFM segmentation with k-means, choosing k, hierarchical clustering and DBSCAN, PCA and UMAP for seeing high-dimensional data, and Isolation Forest for finding the transactions that don't belong.