Interactive · Fit the Residuals

The Gradient Boosting Explorer

Weekly demand for a product, seen through noisy weeks. Boosting starts with a flat guess, fits a tiny tree to what it got wrong, adds it scaled by the learning rate, and repeats. Step through the ensemble and watch the fit sharpen — then watch the held-out error turn back up when you keep going. That turning point is where early stopping stops.

The ensemble so far — and the residuals it will fit next

Training weeksHeld-out weeksEnsemble predictionNext tree (fit to residuals)True curve

0 trees — the ensemble is a flat line at the mean

Error vs. number of trees

Train RMSETest RMSE◆ early-stopping point (test minimum)

Learning rate 0.30

Tree depth 2

Noise 0.18

Training weeks 60

Show the true curve Show the next tree's contribution

Scorecard

Trees in the ensemble0

Train RMSE—

Test RMSE—

Best test RMSE so far—

Early-stopping point—

The algorithm: F₀ = mean(y). For t = 1…T: fit a small regression tree hₜ to the residuals y − Fₜ₋₁ on the training weeks, then Fₜ = Fₜ₋₁ + η · hₜ. A small learning rate η needs more trees but lands smoother; deeper trees fit faster and overfit faster. Early stopping picks the t where the held-out error was lowest — scikit-learn, LightGBM and XGBoost all do this for you with a validation set.