A model produces a probability; a business produces a decision. Expected value, profit curves, tuned thresholds, calibration, lift and gain — and the launch of the Midterm Model Competition
By the end of tonight you'll have both the concept and the working code — every objective maps to a cell you run yourself.
A retention offer costs $50; a lost customer costs $1,500. Put those numbers in a cost matrix and the 'right' threshold falls out of the arithmetic — and it is nowhere near 0.5.
Plot profit against every threshold, find the peak, and use scikit-learn's TunedThresholdClassifierCV to choose it by cross-validation with a business scorer — the same model, thousands of dollars better.
AUC says the ranking is good; a calibration curve says whether 0.8 means 80%. Boosted trees and forests often need CalibratedClassifierCV before their scores can enter a budget calculation.
When you can only contact 10%, the question is 'who are the best 700?', not 'who is above 0.5?'. Lift and gain charts answer it, and the top-k view is how marketing actually works.
A business question, a dataset, a model — and the judgment to evaluate it.
Three confusion matrices and the best odds-ratio sentences from your HW2 credit submissions. Then back to Session 1's Telco model. predict() flagged 322 customers; 201 of them really churn. It missed 173. The 0.5 cutoff assumes both mistakes cost the same. They don't: an unneeded offer costs $50, a missed churner costs a $1,500 customer.
Sweep the threshold from 0 to 1, compute the confusion matrix at each, multiply by the cost matrix, plot total profit. The peak is the threshold. On Telco the profit-maximizing threshold sits near 0.2 and adds tens of thousands of dollars over 0.5 on the test set alone.
scikit-learn (≥ 1.5) wraps any classifier and tunes the cutoff by cross-validation against a scorer you write — including one that returns dollars. Write the business scorer with make_scorer, wrap the pipeline, and let CV pick the threshold that pays.
Stretch, coffee, finalize competition teams.
Bin customers by predicted probability and plot the actual churn rate per bin. Logistic regression tends to sit on the diagonal; random forests and boosting push scores toward the middle or the extremes. CalibratedClassifierCV (isotonic or sigmoid) repairs it — and it matters because every dollar in the profit curve was computed from those probabilities.
Sort customers by score, contact the top 10%: the gain chart shows what share of churners you reach, the lift chart shows how many times better than random. Then the honest caveat: a churn model finds people likely to leave, not people an offer will persuade. Uplift modeling (treatment vs. control) is the next course's problem — but you should know it exists.
Teams of 2–3. A hidden test set of hotel bookings; predict which reservations will be canceled. Kaggle community competition, public leaderboard on half the test set, private on the other half. Leaderboard closes Mon Nov 9, 4:30 PM; reveal and three-minute 'what worked' talks that evening in Session 8.
Notebooks, readings, explorers, and the stories behind the methods.
The cost matrix, the profit curve, TunedThresholdClassifierCV with a dollar scorer, calibration curves and CalibratedClassifierCV, lift and gain charts, and top-k targeting under a budget — all on Telco.
Open in ColabTwo overlapping score distributions, a draggable threshold, and a live profit curve. Set the offer cost, the value of a save, and a contact budget; watch the profit-optimal cutoff drift away from 0.5 and from the accuracy-optimal one.
Open the explorerLoads the Kaggle files, builds the Session 3 pipeline, cross-validates (≈ 0.89 AUC), and writes submission.csv. It reproduces the logistic baseline on the leaderboard exactly; everything after that is yours.
Open in ColabOptional companion notebook: a simulated randomized pilot, a churn model vs. a T-learner uplift model, the uplift curve, and why targeting the riskiest customers sends offers to lost causes. Pure scikit-learn, with an optional CausalML cell.
Open in ColabAn explorable from Google's People + AI Research team: what it means for a model's 70% to mean 70%, and how calibration is measured and fixed.
Open the explainerGoogle's 2024-rebuilt crash course on choosing a classification threshold — short, with an interactive exercise.
Read the lessonThe canonical figure: logistic regression on the diagonal, random forest and naive Bayes bowed off it. Runnable example.
Open the exampleIf the advanced uplift notebook hooked you: a scikit-learn-style uplift library with tutorials, and the 25-million-row Criteo dataset (ailab.criteo.com) for a real project.
Open the docsTunedThresholdClassifierCV and FixedThresholdClassifier, with the cost-sensitive learning example that mirrors tonight's lab.
Open the guideCalibration curves, CalibratedClassifierCV, isotonic vs. sigmoid, and the famous figure showing which model families are naturally calibrated and which are not.
Open the guideStripe reports its Radar models cut fraud by roughly a third for the average business. Every one of those decisions is a threshold on a probability, tuned against the cost of a blocked good customer versus a fraudulent charge — tonight's arithmetic at planetary scale.
Read about RadarWhy a churn score is not a targeting score: Wayfair explains the four customer types (persuadables, sure things, lost causes, sleeping dogs) and why you should only pay to reach the first.
Read the explainerThe invitation link is posted on Canvas. Join with your suffolk.edu Kaggle account, form a team of 2–3, download train.csv and test.csv from the Data tab, and read the rules — the leak columns are gone, but agent, company, and country deserve thought. Five submissions per day; leaderboard closes Mon Nov 9, 4:30 PM.
Join via Canvas linkThe expected-value framework, profit curves, lift and cumulative response charts — the chapters this session is built on. Not free, but the single best business treatment of tonight's ideas.
See the bookOpen the notebook and build it with us.
Take the Session 1 Telco logistic regression. Price the mistakes — a $50 offer, a saved customer worth $450 in expectation — and let scikit-learn find the threshold that maximizes profit on cross-validated folds. Then compare the confusion matrix and the money at 0.5 versus the tuned cutoff.
profit(y_true, y_pred) = 450 × TP − 50 × (TP + FP). Wrap it with make_scorer so cross-validation can use it.
TunedThresholdClassifierCV(pipeline, scoring=profit_scorer, cv=5). Fit on the training set; read best_threshold_.
Confusion matrix and total profit at 0.5 and at the tuned threshold. Note how many more churners you catch and what it costs in extra offers.
CalibrationDisplay.from_estimator on the test set. If the curve bows away from the diagonal, wrap the model in CalibratedClassifierCV and redo step 2.
import pandas as pd
from sklearn.model_selection import train_test_split, TunedThresholdClassifierCV
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import make_scorer, confusion_matrix
url = 'https://raw.githubusercontent.com/harslan/isom-835/master/public/data/telco_churn.csv'
df = pd.read_csv(url)
X = pd.get_dummies(df[['tenure', 'MonthlyCharges', 'Contract', 'InternetService', 'PaymentMethod']], drop_first=True)
y = (df['Churn'] == 'Yes').astype(int)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=835)
OFFER, SAVE = 50, 450 # $50 per offer; a caught churner is worth $450 in expectation
def profit(y_true, y_pred):
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
return SAVE * tp - OFFER * (tp + fp)
profit_scorer = make_scorer(profit)
base = LogisticRegression(max_iter=1000)
tuned = TunedThresholdClassifierCV(base, scoring=profit_scorer, cv=5).fit(X_tr, y_tr)
print(f'Tuned threshold: {tuned.best_threshold_:.2f}')
for name, model in [('threshold 0.50', base.fit(X_tr, y_tr)), ('tuned', tuned)]:
pred = model.predict(X_te)
print(f'{name:15s} profit on test set: ${profit(y_te, pred):,} matrix: {confusion_matrix(y_te, pred).tolist()}') The model did not change. Not one coefficient. What changed is that we finally wrote down what each mistake costs, and the arithmetic moved the cutoff from 0.5 to roughly 0.1. On 1,409 test customers that is worth tens of thousands of dollars; scaled to the full customer base it is the difference between a churn model that pays for itself and one that does not. The threshold is a business decision, and it belongs in the memo, not buried in a default argument.
A model produces a score; a business produces a decision. Write down the cost of each mistake, and the break-even threshold is p* = cost of a false alarm ÷ (cost of a false alarm + value of a catch).
AUC is ranking; calibration is meaning. Check the calibration curve before a probability enters any dollar calculation, and use CalibratedClassifierCV on a held-out fold when trees or boosting bow away from the diagonal.
Under a budget the question is 'who are the best k?' — gain and lift charts answer it. And a churn score is not an uplift score: it finds who will leave, not who an offer will persuade.
Complete these before next Monday — they are the practice that makes the next session land.
Exam-style questions for the ideas in this session — Classification & Decisions. Every wrong answer names the misconception behind it.
From yes/no to how much: linear regression on the Ames housing data, coefficients in dollars, log targets, ridge and lasso, and the residual plot that tells you what your model is missing.