Logistic regression as odds, kNN and Naive Bayes as contrasts, the confusion matrix in depth, precision vs. recall, ROC vs. PR — on 30,000 credit-card customers, 22% of whom default
By the end of tonight you'll have both the concept and the working code — every objective maps to a cell you run yourself.
The sigmoid turns a linear score into a probability; each coefficient is a log-odds change. e^0.69 ≈ 2 means 'doubles the odds' — the sentence a credit officer can repeat.
Logistic regression draws a line, kNN votes among neighbors, Naive Bayes multiplies likelihoods. Same data, three decision boundaries, three sets of strengths.
Four cells, four business meanings. Precision, recall, F1, specificity — computed by hand once, so you never again trust a single accuracy number.
ROC for balanced problems and ranking quality; precision–recall when the positive class is rare and false alarms cost money. Fraud at 0.2% makes the difference brutal.
A business question, a dataset, a model — and the judgment to evaluate it.
Three sign-up lists and three confusion matrices from your HW1 Telco submissions, then a new dataset: 30,000 credit-card customers in Taiwan, six months of bill and payment history, and a label for whether they defaulted the following month. 22.1% did.
Why linear regression fails on a 0/1 target, the sigmoid that fixes it, and coefficients as log-odds. Payment status last month has the biggest coefficient — and exponentiating it tells you exactly how much a late payment multiplies the odds of default.
kNN: no training, just voting among the k nearest customers — and a decision boundary that follows the data everywhere. Naive Bayes: multiply likelihoods, fast and surprisingly good on text. Decision-boundary plots on two features make the three models' personalities visible.
Stretch, coffee, pick a competition teammate — the midterm launches next week.
True positive: a defaulter we flagged. False negative: a defaulter we approved — the expensive cell. False positive: a good customer we declined — the cell that loses revenue and trust. Precision, recall, F1, specificity computed by hand, then classification_report.
The ROC curve sweeps every threshold and plots recall against false-positive rate; AUC is the probability a random defaulter scores above a random good customer. Then the PR curve — and why at 22% default ROC is fine, but at 0.2% fraud only PR tells the truth. class_weight='balanced' as the first fix for imbalance; why SMOTE is usually unnecessary.
We now have a probability for every customer. We have not decided what to do with it. predict() used 0.5 — who chose that? Next session: costs, thresholds, calibration, and profit curves. HW2 launches tonight.
The sigmoid, log-odds, and why we fit a line in a different space.
The best sixteen minutes you can spend before Session 5.
Notebooks, readings, explorers, and the stories behind the methods.
Logistic regression with odds ratios, kNN and Naive Bayes with decision-boundary plots, the confusion matrix by hand, classification_report, ROC and PR curves, and class_weight='balanced'.
Open in ColabNext week's explorer, available now: drag the threshold across the two score distributions and watch precision, recall, and the confusion matrix move. Ignore the profit panel until Session 5 — or don't.
Open the explorerThe sigmoid, log-odds, and the decision boundary, interactively. Pairs with tonight's odds-ratio reading.
Open the explainerSlide the threshold and watch the point walk along the ROC curve. The cleanest explanation of what AUC is (and is not) on the web.
Open the explainerPrecision, recall, F1 and the confusion matrix with a live example — the vocabulary of Session 4 in one scroll.
Open the explainerLogistic regression, why not linear regression, multiple logistic regression, and generative classifiers (LDA, QDA, Naive Bayes). Sections 4.1–4.4 are tonight; 4.5 compares them.
Read Chapter 4 (free PDF)Every metric from tonight with its formula: precision, recall, F1, ROC-AUC, average precision, and the confusion matrix. The section on multiclass averaging will save you later.
Open the guideYeh & Lien's 2009 dataset from a Taiwanese bank: 30,000 clients, April–September 2005. The documentation explains the payment-status codes (−1 = paid duly, 1 = one month late …) that make the strongest features.
Read the documentationLenders using ML underwriting report approving materially more applicants at constant default rates — Upstart has claimed ~43% more approvals at the same loss rate. Vendor numbers, so read them as claims; the mechanism (better ranking than a scorecard) is exactly tonight's AUC.
Read the claimsSearch the index for 'Logistic Regression' and 'ROC and AUC'. The ROC video in particular is the best 16 minutes you can spend before next week. Also embedded on this page.
Watch on YouTubeProf. Arslan's earlier notebook on Gaussian and multinomial Naive Bayes, including the text-classification example. Pairs with the SVM refresher if you want a fourth boundary shape. Hosted in the course repo — opens in Colab with one click.
Open in ColabThe book chapter on evaluation: precision, recall, F1, ROC and AUC, with the imbalanced-class discussion. A slower second pass over tonight.
Open in ColabOpen the notebook and build it with us.
30,000 credit-card customers, 22% default rate. Fit logistic regression, kNN, and Gaussian Naive Bayes in the same pipeline, compare ROC-AUC and PR-AUC, and then read the logistic model's confusion matrix cell by cell in the bank's language.
Unit: one cardholder. Target: default next month. Horizon: one month. Decision: reduce the credit line / call / do nothing. Baseline: 77.9% accuracy by predicting 'no default' for everyone.
Same preprocessing, swap the last step: LogisticRegression, KNeighborsClassifier(k=25), GaussianNB. Record ROC-AUC and average precision for each.
np.exp(coef_) for the logistic model. Find the feature with the largest odds ratio and say it in a sentence a credit officer would understand.
Confusion matrix at the default threshold. Name each cell in business terms. Compute precision and recall by hand, then confirm with classification_report.
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import roc_auc_score, average_precision_score, confusion_matrix
credit = fetch_openml(data_id=42477, as_frame=True)
X, y = credit.data, (credit.target == '1').astype(int) # 1 = defaulted next month
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=835)
models = {
'logistic': LogisticRegression(max_iter=2000),
'knn-25': KNeighborsClassifier(n_neighbors=25),
'naive bayes': GaussianNB(),
}
for name, clf in models.items():
pipe = make_pipeline(StandardScaler(), clf).fit(X_tr, y_tr)
p = pipe.predict_proba(X_te)[:, 1]
print(f'{name:12s} ROC-AUC {roc_auc_score(y_te, p):.3f} PR-AUC {average_precision_score(y_te, p):.3f}')
logit = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)).fit(X_tr, y_tr)
print(confusion_matrix(y_te, logit.predict(X_te))) # rows: actual, cols: predicted The three models rank customers similarly (AUC within a few points) — but look at the confusion matrix: at the default threshold, logistic regression flags only a fraction of the defaulters. That is not a modeling failure. It is a decision nobody made: predict() silently assumes a missed default costs the same as a wrongly declined customer. Next week we price the two mistakes and let the price choose the threshold.
Logistic regression is linear in log-odds: exponentiate a coefficient to get an odds ratio, and say it in words — 'a late payment last month roughly doubles the odds of default.'
The confusion matrix has four cells with four business meanings. Precision asks 'of those we flagged…', recall asks 'of those who actually…'. Accuracy averages over both mistakes and hides which one you are making.
ROC-AUC measures ranking; PR-AUC measures usefulness on rare positives. At 22% default either works; at 0.2% fraud only precision–recall tells the truth. And predict()'s 0.5 threshold is a decision you have not yet made.
Complete these before next Monday — they are the practice that makes the next session land.
Exam-style questions for the ideas in this session — Classification & Decisions. Every wrong answer names the misconception behind it.
The expected-value framework, profit curves, TunedThresholdClassifierCV, calibration plots, lift and gain charts, and targeting under a budget. Plus: the Midterm Model Competition launches.