Fit the residuals, again and again: HistGradientBoosting, LightGBM, XGBoost, early stopping — and the tabular foundation models that, since 2025, beat tuned boosting on small data with zero tuning. Plus: the competition reveal
By the end of tonight you'll have both the concept and the working code — every objective maps to a cell you run yourself.
Start with a weak model, fit a small tree to its residuals, add it with a small learning rate, repeat. Bias falls step by step — and so, eventually, does generalization unless you stop early.
scikit-learn's HistGradientBoosting with native categorical support, LightGBM, and XGBoost — same idea, different defaults. Learning rate, number of trees, depth, and regularization as the four knobs that matter.
A validation set, a patience parameter, and the iteration where validation loss bottomed out. The single most important habit in boosting.
Tree ensembles beat neural networks on tabular data (Grinsztajn 2022). Tabular foundation models — TabPFN-2.5, TabICLv2 — now match or beat tuned boosting on small and medium tables. When to use which, and what licensing means for production.
A business question, a dataset, a model — and the judgment to evaluate it.
The private leaderboard, unveiled. Each team gets three minutes: what you tried, what worked, what leaked. The pattern is always the same — clean pipelines beat clever models, and early-stopped boosting sits at the top.
Bagging averages independent deep trees to cut variance. Boosting does the opposite: a sequence of shallow trees, each fit to what the ensemble so far got wrong, each added with a small learning rate. Bias falls with every tree. The picture: a regression curve improving one stump at a time.
scikit-learn's HistGradientBoostingClassifier: histogram binning, native missing values, categorical_features='from_dtype'. LightGBM: the same idea, leaf-wise growth, blazing fast. XGBoost 3.x: the Kaggle veteran with the deepest regularization options. Same pipeline, three last steps.
Stretch, coffee, congratulate the winners.
Plot training and validation log-loss against the number of trees. Training keeps falling; validation bottoms out and rises. early_stopping=True with a validation fraction and n_iter_no_change picks the bottom automatically. Then a quick tour of the regularizers: max_leaf_nodes, min_samples_leaf, l2_regularization, subsample.
2022: Grinsztajn et al. show tree ensembles beat deep learning on tabular data. 2025: TabPFN-2.5 — a transformer pretrained on millions of synthetic tables — wins against default XGBoost on nearly every dataset under 10k rows, with zero tuning, in seconds. 2026: TabICLv2 from Inria does it open-source at up to 500k rows. We run TabICL on a 5,000-row slice and compare.
The 2026 default for tabular business data: a clean pipeline, early-stopped gradient boosting, and a tabular foundation model as the sanity check. Next week we learn to trust it: tuning, SHAP, conformal intervals, fairness.
Fit the residuals, scale by a learning rate, repeat. Watch before the notebook.
The Kaggle veteran, tree by tree.
Notebooks, readings, explorers, and the stories behind the methods.
Residual-fitting animation, HistGradientBoosting vs. LightGBM vs. XGBoost on hotel bookings, early-stopping curves, the four knobs, and an optional TabICL cell for the foundation-model comparison.
Open in ColabWatch residuals shrink tree by tree. Drag the learning rate and tree depth, step through the ensemble, and see validation loss turn back up when you keep going — early stopping as a picture.
Open the explorerThe best long-form explanation of boosting on the web: residual fitting with pictures, then the gradient-descent view, by the authors of the fast.ai ML course.
Read the essayChange the learning rate, depth, and number of trees on a 2-D classification problem and watch the decision surface sharpen — a second angle on our Boosting Explorer.
Open the explainerWhy very large models can generalize after they overfit — the phenomenon behind foundation models, explained visually. Frontier reading for the TabPFN discussion.
Open the explainerCategorical support, missing values, early stopping, and monotonic constraints — the section that makes HistGradientBoosting the right first boosting model in 2026.
Open the guideThe benchmark paper that settled the argument for three years: on 45 tabular datasets, tuned tree ensembles beat neural networks, and the paper explains why (uninformative features, irregular functions, rotation invariance).
Read the abstractUp to 50k rows and 2k features; reports a 100% win rate against default XGBoost on classification datasets up to 10k rows and parity with AutoGluon's four-hour ensembles. The paper that made 'zero-shot tabular' a serious option.
Read the paperpip install tabicl. scikit-learn-compatible, scales to ~500k rows, fully open source, and competitive with tuned gradient boosting on most of TabArena. The one we run in class.
Open the repoA continuously updated leaderboard of GBDTs, neural nets, foundation models, and AutoML on curated tabular datasets. Check it before choosing a model — and read the licensing column.
Open the leaderboardThe clearest short derivation of gradient boosting with trees, from the library's own docs. Read it once and the objective function stops being scary.
Read the tutorialSearch the index for 'Gradient Boost'. Part 1 (regression main ideas) and Part 3 (classification) are the ones to watch before Milestone 3 of your project. Also embedded on this page.
Watch on YouTubeOpen the notebook and build it with us.
Fit HistGradientBoostingClassifier with early stopping on the hotel data, plot the validation curve, read the iteration it stopped at, and compare to last week's forest. Then, on a 5,000-row slice, run TabICL with no tuning and see where it lands.
Session 7's leak-free hotel features. HistGradientBoostingClassifier(learning_rate=0.05, max_iter=2000, early_stopping=True, validation_fraction=0.15, n_iter_no_change=50).
model.n_iter_ tells you how many trees were kept. Plot model.validation_score_ to see the curve bottom out.
Test AUC vs. the random forest from Session 7 and vs. logistic regression from Session 3's pipeline. Same split, same metric.
pip install tabicl; TabICLClassifier().fit(X_small, y_small) on 5,000 rows. No tuning. Compare AUC with the boosted model trained on the same 5,000 rows.
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score
url = 'https://raw.githubusercontent.com/harslan/isom-835/master/public/data/hotel_bookings.csv'
df = pd.read_csv(url)
leaks = ['reservation_status', 'reservation_status_date', 'assigned_room_type']
X = df.drop(columns=['is_canceled'] + leaks)
for c in X.select_dtypes(object): X[c] = X[c].astype('category') # native categorical support
y = df['is_canceled']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=835)
gbm = HistGradientBoostingClassifier(
learning_rate=0.05, max_iter=2000, max_leaf_nodes=31,
early_stopping=True, validation_fraction=0.15, n_iter_no_change=50,
categorical_features='from_dtype', random_state=835)
gbm.fit(X_tr, y_tr)
print(f'trees kept: {gbm.n_iter_} test AUC: {roc_auc_score(y_te, gbm.predict_proba(X_te)[:, 1]):.3f}')
# Optional (Colab): the open-source tabular foundation model, zero tuning, small slice
# !pip install -q tabicl
# from tabicl import TabICLClassifier
# Xs, ys = X_tr.sample(5000, random_state=835), y_tr.loc[X_tr.sample(5000, random_state=835).index]
# print('TabICL AUC:', roc_auc_score(y_te, TabICLClassifier().fit(Xs, ys).predict_proba(X_te)[:, 1])) Early stopping is doing the work the Overfitting Explorer made you do by hand. And the foundation-model cell is the 2026 twist: on 5,000 rows a pretrained transformer with zero hyperparameters will typically match the boosted model you just tuned — in seconds. On 95,000 rows the GBDT pulls ahead again and runs in production without a license question. The right answer is now 'both': GBDT as the workhorse, a TFM as the fast baseline and the sanity check.
Boosting fits the residuals sequentially with shallow trees and a small learning rate; it reduces bias where bagging reduces variance. Lower rate + more trees + early stopping is the recipe.
Early stopping on a validation set is the one habit that separates a boosting model that generalizes from one that memorized — and every major library has it built in.
On tabular business data tree ensembles still beat neural networks; since 2025 tabular foundation models (TabPFN, TabICL) match tuned boosting on small and medium tables with zero tuning. Use a GBDT for scale, latency, and licensing; use a TFM as the instant baseline.
Complete these before next Monday — they are the practice that makes the next session land.
Exam-style questions for the ideas in this session — Trees, Forests & Boosting. Every wrong answer names the misconception behind it.
Cross-validation strategies that don't lie (grouped, time-ordered), Optuna, SHAP for one customer and for the whole book, partial dependence, conformal prediction with MAPIE, fairness slices, and the one-page model card.