Learning without labels: RFM customer segmentation with k-means, choosing k honestly, hierarchical clustering and DBSCAN, PCA and UMAP for seeing high-dimensional data, and Isolation Forest for the transactions that don't belong
By the end of tonight you'll have both the concept and the working code — every objective maps to a cell you run yourself.
Recency, Frequency, Monetary from a million retail transactions, scaled and clustered — then named: Champions, Loyal, At Risk, Hibernating. A segmentation a marketing team can act on Monday.
The elbow on inertia, the silhouette score, and the truth: k is a business decision informed by the data, not decided by it.
PCA as the linear projection that keeps the most variance; UMAP as the nonlinear map that keeps neighborhoods. What each is for and what each distorts.
Isolation Forest scores every point by how easy it is to isolate. Fraud, sensor faults, data-entry errors — anomaly detection when labels are rare or absent.
A business question, a dataset, a model — and the judgment to evaluate it.
The best baselines and metrics from Milestone 2, then a shift: nine sessions with a y column, and now none. Unsupervised learning finds structure — groups, directions, outliers — and the business supplies the meaning. Meet the data: Online Retail II, 1,067,371 transactions from a UK gift retailer, 2009–2011.
Pick k centroids, assign every customer to the nearest, move each centroid to the mean of its members, repeat until nothing moves. The k-Means Explorer runs it one step at a time. Then the two things that go wrong: bad initialization (k-means++ fixes it) and unscaled features (StandardScaler and a log on the dollar column fix that).
Inertia falls with every extra k; the elbow is where it stops falling fast. Silhouette measures how much closer each point is to its own cluster than the next one. On the retail data both point to 4–5 — and the marketing team can staff campaigns for four. That settles it.
Stretch, coffee, name your favorite cluster.
Agglomerative clustering builds a dendrogram you can cut at any height — no k up front. DBSCAN finds dense regions of any shape and labels the rest noise. Gaussian mixtures give soft memberships: a customer can be 70% Loyal and 30% At Risk. Same RFM data, three different stories.
PCA rotates the data to the directions of greatest variance; the first two components are the best flat picture of a 20-column table and the explained-variance curve tells you how much you lost. UMAP bends space to keep neighbors together — better pictures, less trustworthy distances. Both make a cluster plot possible.
Random trees isolate outliers in a few splits and normal points in many; the average path length is the anomaly score. On a simulated card-transaction stream with 1% fraud, Isolation Forest puts roughly a third of the fraud into the top 5% of scores with no labels at all — a starting point for the review queue, not a finished detector. Then the project: if yours has no label, tonight’s segmentation notebook is your template; if it has a clock, next week’s forecasting notebook is.
Assign, update, repeat — and why you run it more than once.
Directions of greatest variance, explained with a picture.
Notebooks, readings, explorers, and the stories behind the methods.
RFM from Online Retail II, k-means with elbow and silhouette, cluster profiles and names, agglomerative and DBSCAN, PCA loadings and UMAP, and Isolation Forest on a synthetic fraud stream.
Open in ColabCustomers on a recency × frequency plane. Step the algorithm one assignment at a time, re-initialize, change k, and watch the elbow chart and the segment table update — including the case where k-means confidently finds clusters that aren't there.
Open the explorer1,067,371 transactions from a UK-based online gift retailer, Dec 2009 – Dec 2011, with customer IDs, quantities, prices, and countries. Cancellations are the invoices starting with 'C'.
Open the dataset pagePlace centroids by hand and step the algorithm on several datasets, including the ones where k-means fails. The original explorer our K-Means Explorer is modeled on.
Open the explainerDensity-based clustering step by step: watch core points, border points, and noise get labeled on smiley faces and moons.
Open the explainerRotate a 2-D and a 3-D cloud and watch the principal components find the directions of greatest variance.
Open the explainerPCA (12.2), k-means and hierarchical clustering (12.4), with the honest section on how unstable clusters can be. Read 12.2 and 12.4.
Read Chapter 12 (free PDF)The comparison table at the top — which algorithm for which shape, scale, and use case — is the single most useful page in the clustering literature.
Open the guideIsolation Forest, Local Outlier Factor, One-Class SVM, and the elliptic envelope compared on the same toy data. Read it before you pick a method for a real fraud problem.
Open the guideIndustry reporting on Mastercard's real-time fraud scoring: higher detection and materially fewer false declines, each decision in under 50 milliseconds. Anomaly scores plus supervised models plus a threshold the business chose — tonight and Session 6 combined.
Read the reportSearch the index for 'K-means' and 'PCA main ideas'. Eight and five minutes respectively. Also embedded on this page.
Watch on YouTubeProf. Arslan's earlier notebook: PCA on digits and faces, choosing the number of components, and PCA as noise filtering. The picture-heavy version of tonight's PCA segment. Hosted in the course repo — opens in Colab with one click.
Open in ColabOpen the notebook and build it with us.
Build RFM features for every customer in Online Retail II, scale them properly, choose k with the elbow and silhouette, profile the clusters, and give each one a name and a marketing action.
Drop cancellations (invoice starts with 'C') and rows without a customer ID. Group by customer: Recency = days since last invoice, Frequency = number of invoices, Monetary = total spend.
log1p on Frequency and Monetary (right-skewed), then StandardScaler on all three. Look at the histograms before and after.
KMeans(n_init=10) for k = 2…10; plot inertia and silhouette_score. Pick k where both agree and marketing can act.
groupby cluster: mean R, F, M in original units, plus size. Name each (Champions, Loyal, At Risk, Hibernating…) and write one action per segment.
import pandas as pd, numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
raw = pd.read_excel('online_retail_II.xlsx', sheet_name=None) # UCI zip → xlsx, two sheets
df = pd.concat(raw.values()).dropna(subset=['Customer ID'])
df = df[~df['Invoice'].astype(str).str.startswith('C') & (df['Quantity'] > 0)]
df['Amount'] = df['Quantity'] * df['Price']
snapshot = df['InvoiceDate'].max() + pd.Timedelta(days=1)
rfm = df.groupby('Customer ID').agg(
Recency=('InvoiceDate', lambda d: (snapshot - d.max()).days),
Frequency=('Invoice', 'nunique'),
Monetary=('Amount', 'sum'))
Z = StandardScaler().fit_transform(np.column_stack([rfm['Recency'], np.log1p(rfm['Frequency']), np.log1p(rfm['Monetary'])]))
for k in range(2, 9):
labels = KMeans(n_clusters=k, n_init=10, random_state=835).fit_predict(Z)
print(f'k={k}: silhouette {silhouette_score(Z, labels, sample_size=3000, random_state=835):.3f}')
km = KMeans(n_clusters=4, n_init=10, random_state=835).fit(Z)
rfm['segment'] = km.labels_
print(rfm.groupby('segment').agg(n=('Recency', 'size'), Recency=('Recency', 'mean'), Frequency=('Frequency', 'mean'), Monetary=('Monetary', 'mean')).round(1)) Look at the profile table and the names write themselves: a small cluster with low recency, high frequency, high spend — Champions; a large cluster that bought once, long ago — Hibernating. Now the question that makes this a business deliverable rather than a plot: what do you do differently for each? Champions get early access; At Risk get a win-back offer; Hibernating get nothing (the offer costs more than they are worth). A segmentation is only as good as the four sentences that follow it.
k-means finds compact clusters of similar size and finds them whether or not they exist. Scale (and log) first, use k-means++, and use the elbow and silhouette to inform k — then let the business decide.
PCA keeps variance; UMAP keeps neighborhoods. Cluster on the scaled features, plot on the embedding, and read PCA loadings to name the axes.
Isolation Forest scores by how easy a point is to isolate — anomaly detection with no labels. The alert rate is a business choice: how many cases can the fraud team review tomorrow?
Complete these before next Monday — they are the practice that makes the next session land.
Exam-style questions for the ideas in this session — Trust & Structure. Every wrong answer names the misconception behind it.
Why random splits lie on time series, lag and rolling features, gradient boosting as a forecaster, classical baselines with StatsForecast, and zero-shot forecasting with Chronos-2 and TimesFM.