What is a house worth? Linear regression as the interpretable baseline, coefficients in dollars, log targets, ridge and lasso, and the residual plot that tells you what the model is missing
By the end of tonight you'll have both the concept and the working code — every objective maps to a cell you run yourself.
Coefficients are slopes in business units: $112 per square foot, $9,800 per garage bay. You'll learn to state them, to standardize them before comparing them, and to stop reading them as causes.
RMSE punishes big misses, MAE reports the typical miss, MAPE explodes near zero, R² tells you how much of the variance you explained. Which one you report depends on who is asking.
Eighty features, one thousand houses: the recipe for overfitting. Penalized regression shrinks coefficients — and lasso sets most of them to exactly zero, which is feature selection for free.
Predicted vs. actual, residuals vs. fitted, and the fan shape that says 'take the log of the target.' The plots that separate a model that works from one that merely runs.
A business question, a dataset, a model — and the judgment to evaluate it.
Five sessions of yes/no; tonight a number. Classification predicts a label; regression predicts a number — a price, a demand, a delivery time, a lifetime value. Same lifecycle, different target, different metrics. The Ames housing data: 1,460 sales, 79 features, a median price of $163,000.
Price against living area. The least-squares line, the intercept, the slope, and what 'holding everything else fixed' means once we add a second feature. The whole of linear regression in one picture.
Pipeline from Session 3, target = log(price), LinearRegression at the end. Coefficients now live on the log scale — a coefficient of 0.10 means roughly +10% price. Standardize to compare importance; never compare a per-square-foot slope to a per-garage-bay slope raw.
Stretch, coffee, argue about what a garage is worth.
Add polynomial terms and watch train R² climb while test R² collapses — the Overfitting Explorer, now with real houses. Then RidgeCV, LassoCV, ElasticNetCV: one alpha, chosen by cross-validation, and the coefficient path that shows lasso switching features off one by one.
Residuals vs. fitted: a fan means heteroscedasticity, a curve means a missing nonlinearity, a cluster of big misses means a segment the model doesn't understand. Then the metric menu — RMSE, MAE, MAPE, R² — and a one-line rule for which to report to whom.
The regression template: pipeline → log target → regularized linear model → residual plots → metrics in business units. HW3 Part 2 is the Ames competition in miniature.
Why shrinking coefficients toward zero makes predictions better on new data.
And why the L1 penalty switches features off entirely.
Notebooks, readings, explorers, and the stories behind the methods.
One-feature line, full pipeline with log target, RidgeCV / LassoCV / ElasticNetCV with coefficient paths, residual diagnostics, and every metric on the menu in dollars.
Open in ColabTonight's polynomial demo as a living picture: drag the degree, watch test error turn back up, then add data and watch the wiggles get tamed.
Open the explorerLeast squares as a picture: drag points, watch the line and the residuals move, and see what R² actually measures.
Open the explainerDrag the data and watch the squared errors — literally drawn as squares — shrink as the line finds its place.
Open the explainerThe definitive free chapter on simple and multiple regression, plus the ridge and lasso section. Skip the math you don't need; read every figure caption.
Read the chapters (free PDF)The Ames data as a permanent Kaggle competition. Submit your HW3 predictions and see where a well-built linear model lands on the leaderboard — a warm-up for the midterm competition.
Open the competitionOLS, Ridge, Lasso, ElasticNet, and their CV variants, with the cost functions written out. The place to check what alpha actually multiplies.
Open the guideIn 2021 Zillow shut its home-flipping unit after its pricing models overpaid at scale, writing down more than $500 million. The residual plot nobody looked at — and a reminder that a point prediction without an error bar is a bet.
Read the storySearch the index for 'Ridge', 'Lasso', and 'Elastic Net' — three short videos that make the penalty geometry obvious. Also embedded on this page.
Watch on YouTubeProf. Arslan's earlier notebook: simple regression, polynomial basis functions, and the ridge/lasso intuition on a small example. The slow version of tonight's middle hour. Hosted in the course repo — opens in Colab with one click.
Open in ColabOpen the notebook and build it with us.
1,460 house sales in Ames, Iowa, with 79 features. Build a leak-proof pipeline that predicts log(price), let LassoCV choose the penalty, and report the error in dollars a buyer would understand.
Unit: one house. Target: sale price (modeled as log). Horizon: at listing. Decision: list price / offer price. Baseline: the mean price, RMSE ≈ $74K.
Session 3's ColumnTransformer, unchanged. Log-transform the target with np.log1p before fitting.
LassoCV(cv=5) at the end of the pipeline. Inspect how many coefficients survived and which features they are.
Predict on the test set, np.expm1 the predictions, then RMSE and MAE in dollars. Plot residuals vs. fitted and describe the shape in one sentence.
import numpy as np
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LassoCV
from sklearn.metrics import mean_absolute_error, root_mean_squared_error
ames = fetch_openml('house_prices', as_frame=True)
X, y = ames.data.drop(columns=['Id']), ames.target
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=835)
# prep = the Session 3 ColumnTransformer (impute+scale numeric, one-hot categorical)
model = Pipeline([('prep', prep), ('lasso', LassoCV(cv=5, random_state=835))])
model.fit(X_tr, np.log1p(y_tr))
pred = np.expm1(model.predict(X_te))
print(f'Baseline RMSE (predict the mean): ${root_mean_squared_error(y_te, np.full(len(y_te), y_tr.mean())):,.0f}')
print(f'Lasso RMSE: ${root_mean_squared_error(y_te, pred):,.0f} MAE: ${mean_absolute_error(y_te, pred):,.0f}')
coef = model.named_steps['lasso'].coef_
print(f'alpha chosen: {model.named_steps["lasso"].alpha_:.4f}, non-zero coefficients: {(coef != 0).sum()} of {len(coef)}') Look at which features lasso kept. Overall quality, living area, neighborhood dummies, year built — the list reads like a realtor's checklist, which is the point: a regularized linear model is the most explainable strong model you will ever ship. Then look at the residual plot: the biggest misses are the most expensive houses. The model knows less about a $450K house because it has seen fewer of them — and that is exactly the segment where a mistake costs the most.
A linear coefficient is a slope in business units, holding other features fixed — not a cause, and not comparable across features until you standardize.
Ridge shrinks, lasso selects. Choose alpha by cross-validation, scale features first, and remember that a lasso model with 25 surviving features is one a client can read.
Metrics are for audiences: MAE for operations, RMSE when big misses hurt, R² for fit, MAPE never near zero. And always look at the residual plot — the fan shape is the log transform asking to be applied.
Complete these before next Monday — they are the practice that makes the next session land.
Exam-style questions for the ideas in this session — Regression & Regularization. Every wrong answer names the misconception behind it.
How a tree splits, why it overfits, and how bagging hundreds of decorrelated trees fixes it. Out-of-bag scores, permutation importance, and the first model that needs no scaling at all.