Chapter 4: Clustering
From Supervised to Unsupervised Learning
In the first three chapters of this course you built a predictive machine: you gathered features \(X_1, X_2, \ldots, X_p\), attached a label \(Y\), and trained a model that maps inputs to outputs. That is supervised learning — the algorithm learns under the supervision of known answers.
This chapter marks a conceptual turning point. There is no \(Y\). No house-price target, no return to predict, no label telling you whether a customer will churn. Instead, the question becomes: are there natural groups hiding in the data? This is the domain of unsupervised learning, and clustering is its most widely deployed tool.
A Brief History
The mathematics of clustering has surprisingly deep roots. Euclidean distance — the straight-line metric that underlies almost every clustering algorithm — traces directly to ancient Greek geometry and Pythagoras’s theorem. But the computational story begins in earnest in the mid-20th century.
K-Means was invented by Stuart Lloyd at Bell Labs in 1957. Lloyd was working on signal quantization — how to compress a continuous signal into a small number of discrete levels while minimizing distortion. His iterative assign-and-update algorithm turned out to be a universal tool for partitioning any numerical data. The method stayed internal to Bell Labs for two decades before being published in 1982. In the meantime, James MacQueen rediscovered and named the method “K-Means” in 1967, and that name stuck. Today K-Means is arguably the most widely used clustering algorithm in industry, valued for its speed and interpretability.
Hierarchical clustering has a parallel lineage. Ward’s method — which minimizes within-cluster variance at each merge step and remains the recommended default — was published by Joe Ward in 1963. Stephen Johnson formalized the full agglomerative framework in 1967, the same year MacQueen named K-Means. The dendrogram visualization, which makes hierarchical clustering’s output so readable, was already standard in biological taxonomy long before statisticians adopted it.
Why Clustering Matters in Business
The business case for clustering is straightforward: heterogeneous populations require heterogeneous strategies. The phrase “the average customer” is usually meaningless — averages erase the variation that matters. A retailer whose customers range from bargain-hunters to luxury enthusiasts should not send the same promotion to everyone.
Customer segmentation is the canonical application. By grouping customers into clusters with similar behavioral and demographic profiles, a company can tailor its messaging, pricing, product assortment, and channel strategy to each segment. The impact can be dramatic: segment-specific campaigns routinely outperform generic mass marketing by 20–50% on conversion rates.
The technique extends far beyond marketing. Netflix groups its 200+ million subscribers into approximately 2,000 “taste clusters” based on viewing patterns; each cluster gets a different thumbnail, title, and recommendation sequence. Spotify’s Discover Weekly playlist — one of the company’s most-loved features — is built on clustering listeners by listening history and then surfacing tracks that similar listeners enjoyed. Banks use clustering for credit-risk segmentation: Basel III explicitly encourages banks to segment their retail portfolios into homogeneous risk groups for capital allocation. Hospitals cluster patient phenotypes to identify subgroups that respond differently to the same treatment, enabling personalized medicine at scale. HR departments cluster employees by performance and engagement metrics to identify flight-risk segments before they resign.
What You Will Learn — and Why It Matters for Your Career
This chapter covers two foundational clustering methods:
- K-Means — the workhorse of industry clustering; fast, interpretable, scalable
- Hierarchical Clustering — slower but richer; produces a dendrogram that reveals the full merging structure without requiring you to specify the number of groups in advance
Along the way you will master distance measures, standardization, the elbow method for choosing \(K\), and the art of interpreting cluster profiles. You will apply both methods to two illustrative datasets: NBA per-game statistics (can the algorithm recover player archetypes from box-score numbers alone?) and mall customer data (can we segment shoppers into actionable groups?).
Topic 4 carries 10% of the course grade — lighter than regression. But in industry, clustering is among the most frequently deployed analytics tools, precisely because it requires no labels and produces outputs that non-technical stakeholders can immediately understand and act on. The ability to segment customers, products, markets, or employees into coherent groups is a core data science skill that appears in virtually every analytics job description.
A conceptual shift from Topics 1–3: we are moving from supervised to unsupervised learning. No more \(Y\) variable — we discover structure in data without labels.
Foundations
Why Clustering?
Having stepped from supervised into unsupervised learning, the next question is the most practical one a manager will ever ask of an analyst: why bother? The answer is that almost every business decision implicitly assumes a notion of similarity. Pricing tiers assume that customers can be grouped by willingness to pay. Inventory strategies assume that products can be grouped by demand profile. Risk capital allocation assumes that loans can be grouped by default behaviour. Clustering is the formal, data-driven way to construct those groups instead of guessing at them. It is the moment in an analytics project when the data stops being a table of rows and starts being a map of communities.
The Business Motivation
Clustering is everywhere in modern business. Netflix segments its viewers into taste groups so that the recommendation engine can offer a different homepage to a documentary devotee than to a teenage thriller fan. Spotify constructs playlist groups from listening behaviour to power its Discover Weekly feature. Banks segment credit-risk customers to set interest rates, allocate marketing budget, and meet regulatory capital requirements. Retailers group products by purchasing patterns to design store layouts and assortment strategies.
The visual intuition behind all of these applications is the same. Imagine a scatter plot of customers measured on two features — perhaps annual income and spending score — and three clearly separated point clouds sitting in different regions of that two-dimensional space. Each point cloud is a cluster. The job of a clustering algorithm is to find these point clouds without being told in advance that they exist, and without any label distinguishing one cloud from another.
In supervised learning (regression, Topics 1–3) we had labels \(Y\). In clustering we discover natural groups with no labels — this is unsupervised learning!
The Clustering Workflow
Every clustering project follows the same five-step pipeline: collect the data, standardise the features, choose the number of clusters, run the algorithm, and interpret the result. The pipeline looks deceptively simple, but each step has its own pitfalls, and three of them deserve special attention before we proceed. Standardisation is critical because variables measured on different scales — dollars and years, for example — will distort the distance calculations on which every clustering algorithm depends. Choosing the number of clusters relies on diagnostic tools such as the elbow plot for K-Means or the dendrogram for hierarchical clustering, supplemented by domain judgment. Finally, the interpretation step is where business value lives. Naming the segments, profiling their members, and translating each cluster into a concrete action is what turns a numerical partition into a strategy.
Clustering finds natural groups in data without a target variable. This chapter develops two methods: K-Means and hierarchical clustering.
Clustering in Everyday Life
The intuition behind clustering is so natural that you already practise it every day, often without noticing. When you sort songs into a workout playlist, a chill playlist, and a study playlist, you are clustering: songs in the same playlist sound similar to one another in ways that matter for the listening context. When you sort laundry into whites, darks, and colours, you are clustering by a single feature — colour — without anyone handing you a labelled training set; the groups simply suggest themselves once you look at the clothes. When you walk through a supermarket and find dairy products in one aisle, produce in another, and snacks in a third, you are seeing the result of a clustering decision made by a store designer who grouped products that customers tend to buy together.
In each of these examples you look at features — tempo and energy for songs, colour for laundry, product category for groceries — and you group items that are similar on those features. The three laundry baskets, lined up in a corner of the room, are a good mental image: no one prelabels which shirt belongs in which basket. You discover the groups yourself by inspecting the items.
Clustering is the same idea applied to data: the computer looks at numerical features and groups data points that are close to each other. The key question is how to measure closeness.
Distance and Similarity
Measuring Similarity
Background
Every clustering algorithm rests on a single foundational question: how do we define “similar”? Before any algorithm can group points, it needs a ruler — a distance measure that assigns a number to every pair of observations. Smaller number means more similar; larger number means more different. The choice of ruler profoundly shapes the clusters you obtain.
Euclidean distance is the oldest and most natural metric. It is precisely Pythagoras’s theorem generalized to \(p\) dimensions: the distance between two points is the square root of the sum of squared coordinate differences. For two customers differing only in age, Euclidean distance reduces to the absolute age difference. For two customers differing in age, income, and spending score simultaneously, it combines all three differences into a single number via the Pythagorean formula.
Manhattan distance (also called the L1 norm or taxicab distance) was formalized by Hermann Minkowski in the early 20th century as part of a broader family of \(L_p\) norms. The name “taxicab” reflects the geometry of a grid city: a taxi can only travel along streets, not diagonally, so the distance between two intersections is the sum of the horizontal and vertical displacements, not the straight-line flight distance. Manhattan distance is less sensitive to large individual differences on one feature — it does not square the deviations — making it more robust to outliers in high-dimensional settings.
Correlation distance (\(1 - \text{corr}(A,B)\)) is conceptually different: it measures whether two observations have the same shape or profile across features, regardless of their absolute magnitude. Two players whose per-game stats rise and fall in the same pattern across categories will have a correlation distance of zero even if one player puts up roughly twice the raw totals. This is particularly valuable when you care about pattern similarity rather than absolute level.
Standardization is not itself a distance measure but a pre-processing step that makes distance measures meaningful. Without it, features measured in dollars (income = 40,000–140,000) will numerically dominate features measured in single digits (spending score = 1–100), simply because the scale is larger. The algorithm will effectively ignore the small-scale features, even if they are economically important. Standardizing every feature to mean zero and unit variance — the z-score transform — gives each feature an equal vote in the distance computation.
The single biggest clustering mistake in practice is skipping standardization. If you cluster Age (20–70) and Annual Income ($15,000–$140,000) without standardizing, the income variable dominates the distance calculation by a factor of roughly 1,000. The algorithm will produce clusters that are almost entirely determined by income, as if age does not exist. Always standardize first.
Euclidean Distance
For two points \(A=(a_1,\dots,a_p)\) and \(B=(b_1,\dots,b_p)\) the Euclidean distance is defined as
\[ d(A,B) = \sqrt{(a_1-b_1)^2 + (a_2-b_2)^2 + \cdots + (a_p-b_p)^2}. \]
Consider two customers described by three features: age, annual income in thousands of dollars, and a spending score on a 1-to-100 scale. Customer A is 28 years old, earns 62 thousand dollars per year, and has a spending score of 68. Customer B is 47 years old, earns 77 thousand dollars per year, and has a spending score of 53. Substituting these values into the formula gives
\[ d = \sqrt{(47-28)^2 + (77-62)^2 + (68-53)^2} = \sqrt{361+225+225} = \sqrt{811} \approx 28.5. \]
Work through the four arithmetic steps in your head — subtract, square, sum, square-root — and predict the distance to one decimal place before revealing.
Interpretation: The distance of roughly 28.5 is a raw number measured in the mixed units of the original features — part years of age, part thousands of dollars, part spending-score points. By itself, 28.5 tells you nothing useful until you compare it to the distances between other pairs of customers. If most customer pairs have distances between 5 and 15, then 28.5 is a relatively large gap and these two customers are quite different. If most pairs have distances between 50 and 200, then 28.5 means they are actually quite similar. This is precisely why Euclidean distance is most informative after standardisation: once all features are on a common z-score scale, a distance of 1.0 always means roughly one standard deviation apart across all features, giving you a calibrated ruler.
Pitfall: Do not interpret raw Euclidean distances as meaningful numbers without first understanding the scale of your features. A distance of 28.5 mixes years, thousands of dollars, and spending-score points, so the dimensions with the largest numerical spread dominate the calculation. Standardise first.
Computing Euclidean Distance by Hand
It is worth slowing down and computing a Euclidean distance one step at a time, because the same four operations underlie every distance computation in this chapter. Suppose Customer A has age 25, income 40 thousand dollars, and a spending score of 70, while Customer B has age 30, income 55 thousand dollars, and a spending score of 50. The first step is to subtract each feature of B from A, which yields differences of \(-5\), \(-15\), and \(20\). The second step is to square each difference, giving \(25\), \(225\), and \(400\). The third step is to add these squared differences, producing \(25 + 225 + 400 = 650\). The fourth and final step is to take the square root, so \(d(A,B) = \sqrt{650} \approx 25.5\).
Notice that the spending score (difference of 20) and the income (difference of 15) contribute far more to the total than the age (difference of 5). Features with larger numerical spreads dominate the distance, which is precisely why standardisation is needed.
Other Distance Measures
Euclidean distance is the default, but it is not the only ruler available. Three alternatives appear frequently in practice. Manhattan distance, also called the L1 norm, sums the absolute differences across features rather than squaring them, which makes it more robust to outliers and a natural fit for grid-like data. Correlation distance, defined as one minus the correlation coefficient between two observation vectors, measures whether two points have the same shape across features regardless of magnitude — useful when patterns matter more than levels. Euclidean distance itself remains the default choice for continuous features in low to moderate dimensions.
| Distance | Formula | When to use |
|---|---|---|
| Euclidean | \(\sqrt{\sum (a_i - b_i)^2}\) | Default; continuous features |
| Manhattan | \(\sum \lvert a_i - b_i \rvert\) | Robust to outliers; grid-like data |
| Correlation | \(1 - \text{corr}(A,B)\) | When shape matters, not magnitude |
Different distance metrics can produce very different clusters. Always think carefully about what similarity means for the business problem at hand before fixing the metric.
Why Standardize?
The picture below is the clearest motivation. The raw data has two real clusters separated top vs bottom — a horizontal cut would correctly divide them. But the \(x\)-axis is much wider than the \(y\)-axis, so a small horizontal gap of \(\sim 0.4\) contributes far more to Euclidean distance than the larger vertical gap of \(\sim 0.5\). K-Means follows the dominant axis and cuts the data left vs right with a vertical boundary — exactly the wrong split. After standardizing each feature to the same scale, the vertical gap finally counts as much as the horizontal one, and K-Means recovers the correct top/bottom split.
The problem is that without standardisation the axis with larger numbers dominates K-Means; the solution is the z-score transformation \(z_i = (x_i - \bar{x})/s_x\), which centres each feature on zero and rescales it to unit standard deviation. After this transformation every feature gets equal weight in the distance computation, regardless of whether it was originally measured in dollars, years, or score points.
Interpretation: After StandardScaler, the mean of each column is 0.00 and the standard deviation of each column is 1.00 up to numerical precision. A customer aged 47 sits at approximately +1.3 standard deviations above the mean age, while a customer earning $42,000 sits at approximately −1.3 standard deviations below the mean income. When Euclidean distance is now computed between two customers, each feature contributes in proportion to how unusual the difference is rather than to the raw units. A five-year age difference and a roughly $12,000 income difference each contribute about the same amount to the total distance, which is a far more defensible operational definition of similarity.
StandardScaler is the default choice for clustering. An alternative is MinMaxScaler, which maps each feature to the range \([0, 1]\) instead of standardizing to zero mean and unit variance. Min-max scaling is sensitive to outliers (one extreme value compresses everything else into a small range), so z-score standardization is generally preferred for clustering unless you have a specific reason to prefer bounded features — for example, when feeding data to a neural network with bounded activation functions.
Always standardize before clustering so every feature gets an equal vote.
K-Means Clustering
K-Means Clustering
Background: The K-Means Algorithm
K-Means is deceptively simple. The entire algorithm reduces to two alternating steps repeated until nothing changes: assign each point to its nearest centroid, then update each centroid to the mean of its members. This assign-update cycle is guaranteed to terminate — the objective function (WCSS, or Within-Cluster Sum of Squares) strictly decreases at each step and is bounded below by zero, so the algorithm must eventually converge.
The convergence guarantee comes with an important caveat: K-Means converges to a local minimum, not necessarily the global one. The final clustering depends on which \(K\) points were chosen as initial centroids. Two runs of the algorithm on the same data with different random seeds can produce different final clusters, each locally optimal but potentially far from each other in quality. This is why scikit-learn’s default is n_init=10: run the algorithm 10 times with different random initializations and keep the best result (the one with the lowest WCSS). For important analyses, increase n_init to 20 or 50.
A more sophisticated initialization strategy called K-Means++ (Arthur and Vassilvitskii, 2007) chooses initial centroids that are spread far apart from each other, dramatically reducing the probability of landing in a poor local minimum. Scikit-learn uses K-Means++ initialization by default (init='k-means++'), which is why you rarely need to increase n_init beyond 10 in practice.
Why does K-Means always converge? Each iteration can only change the cluster assignment of a point if a different centroid is now closer. Every such reassignment reduces WCSS (or leaves it unchanged). Since there are only finitely many possible partitions of \(n\) points into \(K\) groups, the algorithm cannot cycle and must stop. The formal proof uses the fact that WCSS is a Lyapunov function for the algorithm — a non-negative quantity that never increases — which is a standard tool in optimization theory.
The connection to Gaussian Mixture Models: K-Means is actually a special (hard-assignment, equal-variance) case of the Expectation-Maximization (EM) algorithm for fitting Gaussian Mixture Models (GMMs). Where K-Means assigns each point to exactly one cluster (hard assignment), a GMM assigns each point a probability of belonging to each cluster (soft assignment). If you need soft boundaries between clusters — for example, customers who are ambiguously between two segments — GMMs are the natural extension.
For most business applications with \(n < 100{,}000\) observations, K-Means with n_init=10 and K-Means++ initialization (sklearn’s default) is robust and fast. For very large datasets (\(n > 1{,}000{,}000\)), Mini-Batch K-Means (sklearn.cluster.MiniBatchKMeans) processes random subsets at each iteration, achieving comparable quality at a fraction of the computational cost.
The K-Means Algorithm
The K-Means algorithm proceeds in five compact steps. First, choose the number of clusters \(K\) and initialise \(K\) centroids at randomly selected positions. Second, assign each data point to its nearest centroid using Euclidean distance. Third, recompute each centroid as the arithmetic mean of the points currently assigned to it. Fourth, reassign every point to its nearest updated centroid. Fifth, repeat the assignment and recomputation steps until no point changes cluster between iterations. The algorithm alternates between an assignment phase and an update phase, and it is guaranteed to converge — though only to a local optimum, which is why running the algorithm multiple times with different initialisations is standard practice.
The Centroid
A centroid is simply the average position of all points in a cluster — its centre of gravity. A useful physical analogy is balancing a cardboard cutout of the cluster on your fingertip; the balance point is the centroid. Numerically, the centroid is the vector of column means across all points assigned to that cluster. If a cluster contains three customers with (age, income) coordinates \((20, 30)\), \((30, 50)\), and \((40, 40)\), then the centroid is \(((20+30+40)/3,\; (30+50+40)/3) = (30, 40)\). K-Means is named for this operation: the algorithm repeatedly moves each centroid to the mean of its members so that each centroid increasingly represents a typical member of its cluster.
The centroid is the mean of all points in the cluster. K-Means works by repeatedly moving centroids to better represent their members.
Step 1: Initialise Centroids
The first step is to choose a value for \(K\) — say \(K=3\) — and pick three random points as the initial centroids. At this stage every data point is unassigned; the centroids are simply placeholders, deliberately located somewhere in the feature space so that the assignment step has something to work with. The choice of initial positions can substantially affect where the algorithm eventually converges, which is why scikit-learn runs the entire algorithm multiple times (controlled by the n_init parameter) and keeps the best result.
Different random starting positions can lead to different final clusters. The n_init=10 argument tells scikit-learn to run the algorithm ten times and keep the run with the lowest within-cluster sum of squares.
Step 2: Assign Points to Centroids
The assignment rule is simple. For each data point, compute the Euclidean distance to every centroid and assign the point to the closest one. Suppose the three current centroids sit at locations \(\mu_1\), \(\mu_2\), and \(\mu_3\), and consider a single point \(P=(1.0, 4.0)\). If the distances work out to \(d(P,\mu_1) = 1.0\), \(d(P,\mu_2) = 2.8\), and \(d(P,\mu_3) = 1.6\), then \(P\) joins the cluster associated with \(\mu_1\) because \(\mu_1\) is the nearest centroid. After every point has been assigned in the same way, each point now wears the colour of its cluster.
Step 3: Recompute Centroids
Once every point has a cluster label, the update rule is to recompute each centroid as the mean of its current members. Suppose the cluster anchored by \(\mu_1\) now contains five points at \((0.5,1.5)\), \((1.0,2.0)\), \((0.8,1.2)\), \((1.5,1.8)\), and \((1.2,1.0)\). Averaging coordinate by coordinate gives a new centroid at \(\mu_1 = (1.0, 1.5)\). If the centroid was initialised at \((1.0, 3.0)\), it has moved a substantial distance toward its actual members. This step is why the algorithm is called K-Means: each centroid is, by construction, the mean of its members.
Step 4: Iterate Until Convergence
Once the centroids have moved, the new positions imply that some points may now be closer to a different centroid than the one they were previously assigned to. The algorithm therefore repeats the assignment and update steps: reassign every point to its nearest new centroid, then recompute the centroids again, and continue iterating until no point changes cluster between successive rounds. In practice K-Means typically converges in ten to twenty iterations. The full loop alternates between assign and update phases until the partition stabilises and the centroids cease moving.
K-Means in scikit-learn
The scikit-learn implementation reduces the entire workflow to a few lines of Python. The pattern is to instantiate the estimator with the desired hyperparameters, fit it on a standardised feature matrix, and then read off the cluster labels, centroid coordinates, and within-cluster sum of squares. The three arguments worth remembering are n_clusters, which fixes the number of clusters \(K\); random_state, which controls the seed used for centroid initialisation and makes the run reproducible; and n_init, which sets how many independent initialisations the algorithm tries before keeping the best one.
Interpretation: The output shows cluster sizes (approximately 50 members each, since we simulated three equal groups) and a WCSS value. The cluster sizes tell you whether K-Means found balanced groups or highly uneven ones — very uneven sizes (e.g., one cluster with 140 members and two with 5 each) are a warning sign that \(K\) may be wrong, or that the data has outliers pulling centroids. The WCSS value on its own is not informative; it only becomes useful when compared across different values of \(K\) in the Elbow Plot.
Pitfall: comparing cluster labels across runs. The cluster numbers K-Means assigns (0, 1, 2, …) are arbitrary. If you run K-Means twice on the same data, “Cluster 0” in run 1 might correspond to “Cluster 2” in run 2. Never compare cluster labels across different runs without first aligning them. This also means you cannot interpret cluster numbers as an ordering (Cluster 1 is not “better” than Cluster 2 in any sense).
Pitfall: outliers dragging centroids. A single extreme customer with income $2,000,000 will pull the centroid of whichever cluster it joins far away from the other members, producing a large but sparse cluster. Always check for outliers before clustering and consider whether to remove or Winsorize them.
Choosing K with the Elbow Plot
The natural diagnostic for choosing \(K\) is the within-cluster sum of squares, written
\[\text{WCSS} = \sum_{k=1}^{K} \sum_{x_i \in C_k} \| x_i - \mu_k \|^2.\]
WCSS always decreases as \(K\) increases: with more centroids, every point is closer to its nearest centre. The useful question is therefore not which \(K\) minimises WCSS — the answer would always be \(K=N\) — but at which value of \(K\) the rate of improvement begins to flatten out. The elbow plot answers this graphically by plotting WCSS against \(K\) and identifying the bend in the curve, the point where additional clusters yield only marginal gains.
Interpretation: The elbow plot shows WCSS on the vertical axis and \(K\) on the horizontal axis. As \(K\) increases from 1 to 10, WCSS always decreases — with more centroids, every point is closer to its nearest center. The useful information is in the rate of decrease. From \(K=1\) to \(K=2\), WCSS drops steeply because splitting one large cluster into two captures a real structure in the data. From \(K=2\) to \(K=3\), another big drop. From \(K=3\) to \(K=4\), the drop is much smaller. The “elbow” at \(K=3\) is the point of diminishing returns — adding a fourth cluster only marginally reduces within-cluster variance, which is not worth the added complexity.
When the elbow is ambiguous — the curve looks smooth with no sharp bend — consider using the silhouette score (higher is better; measures how much tighter a point is to its own cluster than to its nearest neighbor cluster) or the gap statistic (compares WCSS to a null reference distribution) as supplementary criteria. In practice, domain knowledge often resolves ambiguity: if you have a marketing budget for four campaigns, \(K=4\) is the natural choice even if the elbow is at \(K=3\).
Pitfall: picking \(K\) to match aesthetic preference. A common mistake is to pick \(K=5\) because “five segments feels like a round number” or \(K=3\) because “three is the classic Good/Better/Best framework.” Always ground your \(K\) choice in the elbow plot (or silhouette score) first, then verify the chosen \(K\) makes business sense. The algorithm and the business judgment should converge on the same answer.
WCSS: What Does It Actually Measure?
WCSS stands for Within-Cluster Sum of Squares, and the calculation is straightforward in plain language: for each cluster, measure how far every point is from its cluster’s centroid, square each of those distances so that bigger gaps count disproportionately more, and then add everything up across every cluster. Lower WCSS means tighter clusters and therefore a better numerical fit. Geometrically, WCSS measures how spread out the points are within their assigned groups; if every point sat exactly on its centroid, WCSS would be zero — a perfect but unrealistic outcome. A useful mental picture is a centroid drawn as a star with dashed lines connecting it to each member point; WCSS is simply the sum of the squared lengths of those dashed lines, summed across all clusters.
The Elbow Plot shows WCSS for different values of \(K\). As you add more clusters, WCSS always goes down because more centroids means shorter distances. The elbow is the point at which the improvement slows dramatically — beyond it, adding more clusters is not worth the added complexity.
Exercise: K-Means on Customer Data
You have a dataset of 200 mall customers measured on age, annual income, and spending score. Begin by standardising the three features with StandardScaler so that they share a common scale. Next, build an elbow plot for \(K = 1, 2, \ldots, 10\) and identify the bend in the curve. Then fit K-Means with your chosen \(K\), add the resulting cluster labels back to the DataFrame, and produce a scatter plot of income against spending score, colouring the points by cluster. A worked walkthrough appears in Section 5 of the companion notebook.
K-Means: Pitfalls and Limitations
K-Means has four important weaknesses that every practitioner should keep in mind. First, the algorithm is sensitive to its initial centroid positions: different random starts can converge to different final clusters, which is why setting n_init=10 and running the algorithm ten times is a sensible default. Second, it is sensitive to outliers, because a single extreme observation can drag the centroid of whichever cluster it joins away from the true centre of mass. Third, K-Means implicitly assumes that clusters are roughly spherical and of comparable variance; it struggles with elongated or crescent-shaped groups, where two interlocking shapes will be cut by a straight boundary that does not respect their geometry. Fourth, the value of \(K\) must be chosen in advance, and although the elbow plot helps, the final decision remains a judgment call.
Always visualise the resulting clusters and check whether they make business sense before acting on them.
A Worked Example: Netflix Viewer Segmentation
Netflix runs over 1,300 customer segments internally to drive its show recommendations, promotional emails, and even artwork choices for individual titles. Clustering is the bridge between raw view-time logs and these actionable groups: every subscriber is a vector of viewing behaviours, and unsupervised learning is what turns that vector cloud into a handful of marketable personas. The example below uses 20 simulated subscriber profiles to make the mechanics concrete. Each row records weekly viewing hours and the share of those hours spent on four content categories, and the goal is to recover three natural groups that a marketing team could plausibly target.
Reading the output, one cluster shows roughly 20 hours per week with about two-thirds of view time on action — the action household. A second cluster watches around 13 hours weekly with more than 80 per cent of that time on Korean drama — the K-drama fan. The third cluster watches fewer hours but devotes the majority to kids’ content — the family with young children. Real Netflix segments are built from hundreds of features, including time-of-day, device, churn risk, and price sensitivity, but the principle is identical: standardise, fit, inspect the centroids, then label what the algorithm found.
Hierarchical Clustering
Hierarchical Clustering
Background: Hierarchical Clustering
K-Means is a greedy partitioner: you tell it \(K\), it divides the data into \(K\) groups and stops. If you want to see what the data looks like with \(K=3\) and \(K=5\) and \(K=7\), you must run it three separate times. Hierarchical clustering takes a different philosophy: run the algorithm once and obtain a complete tree of all possible partitions, from \(N\) singletons (every point in its own cluster) to 1 giant group (everything in one cluster). After fitting, you choose \(K\) by cutting the tree at any height.
The agglomerative (bottom-up) approach starts with \(N\) clusters and merges them one pair at a time. At each step, the algorithm finds the two closest clusters and merges them. This merge is recorded as a node in the dendrogram at a height equal to the distance between the two clusters. After \(N-1\) merges, all points are in one cluster and the tree is complete.
The divisive (top-down) approach works in the opposite direction: start with all points in one cluster and recursively split. Divisive clustering is less commonly used in practice — it is computationally expensive and the splits are harder to define — so this chapter focuses on agglomerative methods.
Linkage methods define how you measure the distance between two clusters (not two points). This choice matters enormously:
- Single linkage uses the minimum distance between any two points across the two clusters. It is fast to compute but produces the “chaining” problem: a cluster can grow by absorbing one point at a time, creating long, stringy chains rather than compact groups.
- Complete linkage uses the maximum distance. It produces compact, roughly equal-sized clusters but is sensitive to outliers (one outlier in a cluster inflates the inter-cluster distance).
- Average linkage uses the mean of all pairwise distances — a reasonable compromise.
- Ward’s method (Joe Ward, 1963) asks: if we merge these two clusters, how much does total WCSS increase? It merges the pair that produces the smallest WCSS increase. Ward’s is the default for most business applications because it produces the most balanced, well-separated clusters and corresponds to the same objective function as K-Means.
The dendrogram is the visual output of hierarchical clustering. Each leaf at the bottom represents one data point. Moving up the tree, you see merges happening at increasing heights. The height of a merge reflects how dissimilar the two clusters were when they joined. Tall vertical lines before a merge mean the two groups were far apart — a clear separation. Short lines mean the groups were similar and the split at that level may not be meaningful.
Hierarchical clustering scales as \(O(n^2 \log n)\) in time and \(O(n^2)\) in memory (it must store the full distance matrix). For datasets with \(n > 10{,}000\) observations, K-Means is typically more practical. Hierarchical clustering is the preferred choice for \(n < 1{,}000\) when you want to explore the full cluster structure without committing to a specific \(K\) in advance — common in exploratory research, financial analysis, and genomics.
Hierarchical Clustering: Bottom Up
The agglomerative version of hierarchical clustering can be pictured as a sequence of three stages. At the start there are \(N\) clusters, with each individual point — A, B, C, D, and so on — sitting alone in its own cluster. The algorithm then identifies the two points that are closest to each other; suppose those are A and B, which now merge into a single cluster \(\{A, B\}\). In the next round the algorithm again finds the closest pair of clusters — perhaps the pair \(\{A, B\}\) and \(C\) — and merges them, producing \(\{A, B, C\}\) while D remains separate. The procedure continues until only one cluster remains.
Stated formally, the agglomerative algorithm has four steps: start with each data point as its own cluster, compute the distances between every pair of clusters, merge the two closest clusters, and repeat the distance computation and merge step until a single cluster contains all observations.
Unlike K-Means, hierarchical clustering does not require you to pre-specify \(K\). The choice of \(K\) is made afterward by cutting the dendrogram horizontally at the desired height.
How to Measure Cluster Distance: Linkage Methods
Once clusters contain more than one point, the algorithm needs a rule for measuring the distance between two whole clusters rather than between two individual points. This rule is the linkage method, and four choices appear in standard software. Single linkage uses the minimum distance between any two points across the two clusters, which is fast but prone to long chaining effects. Complete linkage uses the maximum distance and tends to produce compact, roughly equal-sized clusters. Average linkage uses the mean of all pairwise distances between the two clusters and represents a compromise between the two extremes. Ward linkage measures the increase in total within-cluster variance that would result from a merge and consistently produces the most balanced, well-separated clusters.
| Linkage | Distance between clusters | Characteristics |
|---|---|---|
| Single | Min distance between any two points | Can create long chaining clusters |
| Complete | Max distance between any two points | Tends to produce compact clusters |
| Average | Mean distance between all point pairs | Compromise between single and complete |
| Ward | Increase in total within-cluster variance | Most balanced, compact clusters |
Ward linkage tends to give the most balanced, well-separated clusters and is the recommended default for most business applications.
Computing Average Linkage: A Worked Example
Suppose the goal is to merge cluster \(\{A, B\}\) with cluster \(\{C, D\}\) using average linkage. Average linkage is defined as the mean of all pairwise distances between points in the two clusters. With two points in each cluster there are \(2 \times 2 = 4\) pairs to consider. If the four distances are \(d(A,C)=5\), \(d(A,D)=9\), \(d(B,C)=4\), and \(d(B,D)=7\), then the average linkage distance is \((5+9+4+7)/4 = 6.25\). The two alternative summaries make a useful contrast: single linkage would report the minimum of these four numbers, \(4\), while complete linkage would report the maximum, \(9\). Average linkage at \(6.25\) sits in between, which is exactly the compromise property the method is designed to deliver.
For the four distances \([5, 9, 4, 7]\), predict the three linkage values (single, complete, average) before revealing.
Hierarchical Clustering: Merge Example with Five Points
Consider five points labelled A through E with the Euclidean distance matrix below.
| A | B | C | D | E | |
|---|---|---|---|---|---|
| A | 0 | 2 | 6 | 10 | 9 |
| B | 2 | 0 | 5 | 9 | 8 |
| C | 6 | 5 | 0 | 4 | 5 |
| D | 10 | 9 | 4 | 0 | 3 |
| E | 9 | 8 | 5 | 3 | 0 |
Using single linkage — which always merges the pair separated by the smallest inter-cluster distance — the algorithm proceeds in four steps. First, A and B merge at distance 2, the smallest entry in the matrix. Second, D and E merge at distance 3, the next smallest. Third, C merges with the cluster \(\{D, E\}\) at distance 4, because the minimum of the C-to-D and C-to-E distances is 4. Fourth, the two remaining clusters \(\{A, B\}\) and \(\{C, D, E\}\) merge at distance 5. Drawn as a dendrogram, the four merges appear as horizontal bars at heights 2, 3, 4, and 5, recording both the order of merges and the dissimilarity at which each merge occurred.
The Dendrogram: Reading the Tree
A dendrogram is read from the bottom up. The leaves at the bottom of the tree are individual data points. Moving upward, each horizontal bar marks a merge, and the height of that bar represents the distance at which the two clusters were joined: tall bars indicate that the merged clusters were far apart, while short bars indicate they were similar. In a typical example, A and B might merge at height 1, D and E at height 1.5, the cluster \(\{A, B\}\) might join C at height 3, and finally \(\{A, B, C\}\) might merge with \(\{D, E\}\) at height 5.5. To extract a flat clustering with \(K\) groups, draw a horizontal cut line across the dendrogram and count the branches it crosses; a cut at height 2.5 in this example crosses three branches, yielding the partition \(\{A, B\}\), \(\{C\}\), \(\{D, E\}\).
Tall vertical lines before a merge mean those clusters were well separated. Short lines mean the merged groups were similar to begin with.
Dendrogram in Python
scipy gives you the dendrogram visualization; sklearn gives you the cluster labels directly.
Interpretation: The dendrogram height where two branches merge tells you how different those groups were when they joined. In this simulated dataset (three well-separated groups), you should see two early merges at low heights — points within the same true cluster joining each other — followed by a large gap before the inter-group merges at much higher heights. The correct \(K\) is the number of branches below the largest gap. If the tallest vertical lines you can draw without crossing a horizontal merge line spans three branches, then \(K=3\). The red cut line shows where you slice the tree horizontally to obtain your chosen number of clusters.
The AgglomerativeClustering output (cluster labels 0, 1, 2 …) is directly analogous to K-Means labels — you can add them to a DataFrame and use groupby to profile each cluster exactly as in Section 2.
Pitfall: choosing the wrong linkage for your data’s shape. Single linkage is prone to chaining — it will happily merge two compact groups through a thin bridge of outlier points. Ward linkage assumes roughly spherical, equal-variance clusters. For data with elongated clusters or very unequal cluster sizes, average linkage or complete linkage may be more appropriate. Visualize the dendrogram and the resulting scatter plot to check that the clusters look sensible.
Exercise: Hierarchical vs. K-Means
Using the same customer dataset, build a dendrogram with Ward linkage and cut it to produce five clusters. Then compare those assignments to the labels produced by K-Means at \(K=5\). The interesting question is the level of agreement: how many customers receive the same cluster from both methods, and how many disagree? The function scipy.cluster.hierarchy.fcluster(Z, t=5, criterion='maxclust') returns the flat clustering at \(K=5\).
K-Means vs. Hierarchical: When to Use Which?
The choice between the two methods can be reduced to two practical questions. The first question is whether \(K\) is known in advance: if not, hierarchical clustering is preferable because the dendrogram lets you explore the full merge structure and choose \(K\) afterward. The second question is the size of the dataset: for large datasets with more than a few thousand rows, K-Means scales better, while hierarchical clustering becomes slow and memory-intensive. The practical decision rules follow naturally. K-Means is fast and simple and works well when clusters are roughly round and evenly sized. Hierarchical clustering reveals the full merging structure, which is valuable when you do not yet know how many clusters the data contains. When in doubt, run both methods and compare them; agreement between the two algorithms provides additional confidence that the clusters reflect real structure rather than algorithmic artefact.
The bottom line is that no single method is universally correct. Run both, compare the partitions, and let the structure of the data guide the final choice.
Worked Examples
A Worked Example: Clustering NBA Players by Style
NBA front offices have used clustering to identify “player archetypes” — the pure shooter, the slashing wing, the playmaking big, the defensive specialist — since the early 2010s, when richer box-score and tracking data became widely available. Teams use these archetypes for lineup construction, scouting reports, and even contract negotiations: a “playmaking big” is paid differently from a “rim-running big” even when the two have similar point totals. The remarkable fact is that the same algorithm you have just learnt can recover these archetypes from per-game statistics alone, with no need for a coach’s labels or scout’s notes.
The NBA Player Dataset
The dataset contains 24 simulated NBA player season averages described by seven per-game statistics: points, assists, rebounds, three-point attempts, blocks, steals, and usage rate (the percentage of team possessions a player uses while on the floor). By construction the players come from three latent archetypes — pure scorers in the mould of a high-volume Curry-type guard, playmaker-bigs in the mould of a Jokić-type frontcourt hub, and defensive wings in the mould of a Holiday-type two-way guard — but those labels are withheld from the clustering algorithm. The question is whether Ward’s hierarchical clustering can recover the three archetypes from the per-game numbers alone.
A few example rows give a sense of what the algorithm sees:
| Player | points | assists | rebounds | three_pt_attempts | block_per_game | steal_per_game | usage_rate_pct |
|---|---|---|---|---|---|---|---|
| P01 (scorer) | 28.4 | 5.1 | 4.2 | 11.0 | 0.3 | 1.0 | 32.1 |
| P11 (playmaker-big) | 22.6 | 8.3 | 11.4 | 1.8 | 1.6 | 0.9 | 28.4 |
| P19 (defensive wing) | 14.7 | 4.8 | 4.9 | 4.2 | 0.5 | 1.9 | 21.0 |
Can the algorithm recover the three player archetypes from per-game stats alone, without ever seeing the archetype labels?
Building the Dendrogram
The code below builds and visualises a Ward dendrogram for the NBA player data. The first pyodide cell renders the illustrative dendrogram used in the previous discussion; the second cell simulates the 24-player dataset, fits Ward linkage, plots the dendrogram with the cut line, and prints the cross-tabulation between the algorithm’s three clusters and the known archetype labels.
The code uses Ward linkage, which minimises the within-cluster variance and is the recommended default for most business applications. The red dashed line marks the height at which the dendrogram is cut to produce \(K=3\) clusters, and the cross-tabulation in the final line compares the algorithm’s clusters against the known archetype labels — a form of external validation that tests whether per-game statistics alone can recover playing style.
Interpretation: The cross-tabulation (pd.crosstab) is the key output. A perfect recovery would show each cluster mapping exactly to one archetype. In practice, you expect near-perfect recovery for the most distinctive archetype (the playmaker-big, with its uniquely high rebound and assist totals) and some overlap between the two backcourt-oriented archetypes (scorers and defensive wings). “Misclassified” players are not errors — they are the most interesting data points. A defensive wing who clusters with scorers has a stat line more characteristic of a high-usage shot creator; that is a player worth a closer look from coaching and scouting staff.
The dendrogram also reveals within-archetype structure. Among the ten scorers, some merge early into tight sub-clusters of two or three players who share almost identical three-point and usage profiles, while others are more isolated. These sub-clusters may correspond to fine-grained styles — pure spot-up shooters versus high-usage isolation scorers — that the algorithm discovered without any labels.
Pitfall: clustering on irrelevant features. The NBA analysis uses seven per-game statistics that are known to be informative about playing style. If you added irrelevant features — say, jersey number, or a player’s height measured on a single day — you would introduce noise that distorts distances and degrades cluster quality. Feature selection matters as much in clustering as in supervised learning. Include only features that you believe carry information about the natural groupings you are trying to discover.
Pitfall: assuming clusters are “correct”. The K=3 clustering of NBA players is useful — it tells us the playmaker-big is statistically distinct from the two backcourt archetypes — but it is not “true” in any deep sense. A K=4 solution might split the scorers into spot-up shooters and high-usage shot creators, which could be equally valid. Clustering is a tool for description and exploration, not for establishing ground truth. The analyst’s judgment about which solution is most actionable is part of the method.
Reading the Dendrogram
Reading the NBA dendrogram requires moving from the bottom up. Players who merge at low height are very similar to one another — for example, two scorers with comparable three-point attempts and usage rates will join each other early. The red dashed line cuts the tree at the height that produces three clusters, the value of \(K\) chosen to match the number of known archetypes. Traced from bottom to top, the labels reveal which players fall together: the ten scorers tend to cluster among themselves, the defensive wings likewise, and the playmaker-bigs typically form their own branch that joins the rest only at a high merge height. That late merge is informative on its own: it tells you that the playmaker-big looks statistically very different from the two guard-oriented archetypes.
At the cut height that produces three clusters, the players group cleanly into the three archetypes. The cross-tab between the algorithm’s cluster assignments and the withheld ground-truth labels shows high accuracy, with the largest off-diagonal entries appearing where the two backcourt styles blur into each other — exactly where a human scout would also hesitate. The broader takeaway is that hierarchical clustering recovered meaningful playing-style groups from per-game statistics alone, supporting the long-standing claim from NBA analytics departments that players who play alike tend to look alike in the box score.
A Customer Segmentation Example
Background: The Marketing Science of Segmentation
Customer segmentation is the oldest application of clustering in business. Long before the term “data science” existed, marketers used demographic surveys and purchase records to classify customers into groups and tailor campaigns accordingly. What has changed is scale and precision: modern companies have behavioral data on millions of customers, and clustering algorithms can extract segments that are far more nuanced than the traditional demographic buckets (age group, income bracket, gender).
The foundational insight is the 80/20 rule (Pareto principle): in most retail businesses, roughly 20% of customers generate 80% of revenue. Knowing which 20% — and more importantly, what they look like so you can find more like them — is enormously valuable. Clustering is one of the primary tools for answering this question.
But the value of segmentation extends far beyond identifying the top customers. Each segment has its own price sensitivity, preferred communication channel, motivational triggers, and lifetime value trajectory. A “Budget Shopper” who visits frequently but spends little per visit may have enormous lifetime value if you can convert them to a higher-spending segment with the right loyalty program. An “Impulsive Buyer” with a high spending score but modest income may be vulnerable to economic downturns — a credit risk for BNPL (Buy Now Pay Later) programs.
The three-feature mall customer dataset (Age, Annual Income, Spending Score) is deliberately simplified for teaching purposes. In practice, a retail bank’s customer segmentation model might include hundreds of behavioral features: transaction frequency, average transaction size, product categories purchased, days since last purchase, response rate to past campaigns, customer service contact history, digital engagement metrics. The mechanics are identical — standardize, cluster, profile — but the interpretation requires deeper domain expertise.
Telecom customer churn segmentation is a canonical industry application. A telecom company clusters customers by usage patterns, contract tenure, complaint history, and plan type. One segment — typically younger, month-to-month contract, heavy data user — has 3–4× the churn rate of the overall population. By identifying this segment proactively, the retention team can offer targeted upgrades or discounts before the customer decides to leave, converting a reactive “save” into a proactive retention. Studies suggest proactive retention of high-churn segments costs 5–10× less than acquiring an equivalent new customer.
Mall Customer Segmentation: Business Context
The scenario is a shopping mall that wants to understand its customer base well enough to target marketing campaigns precisely. Each of the 200 customers is described by three variables: age, ranging from 18 to 70; annual income, ranging from $15,000 to $140,000; and a spending score on a 1-to-100 scale that reflects in-store purchase behaviour. The goal is to segment the customers into meaningful groups that can each receive a distinct marketing strategy.
A useful preview of the result is a two-dimensional map of income against spending score: five blobs occupy different corners of the plane. High-income, high-spending VIPs sit in the upper right. High-income but low-spending careful spenders sit in the lower right. Low-income but high-spending impulsive buyers sit in the upper left. Low-income, low-spending budget shoppers sit in the lower left. A fifth, more diffuse blob of middle-of-the-pack average customers sits near the centre. Each region calls for a different campaign.
Mall Customers: Elbow Plot
The elbow at \(K=5\) suggests five natural customer segments.
Interpretation: The steep drop from \(K=1\) to \(K=5\) and then the much flatter curve beyond \(K=5\) confirms that five is the natural number of segments in this data. From \(K=5\) to \(K=6\), WCSS drops by only a small amount relative to earlier steps — the sixth cluster is subdividing an already well-defined group, not capturing a new natural grouping. In a real dataset the elbow might be less sharp. When the elbow plot is ambiguous between \(K=4\) and \(K=5\), a useful sanity check is to run both and ask: do the two extra segments in \(K=5\) have meaningfully different profiles and require different marketing strategies? If yes, use 5. If the two extra segments look essentially the same, use 4.
Mall Customers: K-Means with K=5
The plot shows five clearly separated groups in (Income, Spending) space — one in each “corner” plus one in the middle.
Interpretation: The scatter plot immediately reveals the business-relevant structure: the five segments occupy distinct regions of (Income, Spending Score) space. This geometric separation is what makes the clustering actionable — each segment occupies a different “quadrant” of the customer behavior space. Notice that the cluster in the upper-right corner (high income, high spending) is the smallest by count but highest in value per customer. The cluster in the lower-left (low income, low spending) is probably the largest by count but lowest in value. The “average” cluster in the middle has the most variance — it is the catch-all for customers who do not fit neatly into any corner.
Pitfall: forgetting that clustering is for description, not prediction. The cluster labels assigned by K-Means do not predict whether a new customer will be a “VIP” — they describe the current customer base. To predict segment membership for new customers, you need a supervised classifier (e.g., logistic regression or decision tree) trained on the cluster labels as \(Y\). Clustering produces the labels; supervised learning uses them.
Mall Customers: Naming the Segments
| Cluster | Name | Income | Score | Marketing Strategy |
|---|---|---|---|---|
| 1 | VIP Clients | High | High | Loyalty programs, exclusive events |
| 3 | Impulsive Buyers | Low | High | Affordable premium, BNPL |
| 0 | Average Customers | Mid | Mid | Newsletters, general promos |
| 2 | Careful Spenders | High | Low | Personalized outreach, rewards |
| 4 | Budget Shoppers | Low | Low | Discount bundles, value packs |
Clustering transforms raw data into actionable business segments. Each cluster gets a tailored strategy — that is the real value of unsupervised learning.
Profiling Clusters with groupby
How do we know what each cluster “looks like”? Use groupby to compute the mean of each feature per cluster — the same pandas skill from Topic 2!
Example output (numbers vary slightly by seed):
| Cluster | Avg Age | Avg Income ($k) | Avg Spending Score |
|---|---|---|---|
| 0 | 42.7 | 55.3 | 49.5 |
| 1 | 32.7 | 86.5 | 82.1 |
| 2 | 41.1 | 88.2 | 17.1 |
| 3 | 25.3 | 25.7 | 79.4 |
| 4 | 45.2 | 26.3 | 20.9 |
The cluster profile table is how you name your segments. Cluster 1 has high income and high spending \(\Rightarrow\) “VIP.” Cluster 4 has low income and low spending \(\Rightarrow\) “Budget.” The numbers tell the story!
Interpretation: The profile table is the bridge from algorithm output to business action. Reading across each row, you reconstruct a “persona” for each cluster. Cluster 1 (high income \(\approx \$87k\), high spending score \(\approx 82\), relatively young \(\approx 33\)) is the VIP segment: affluent, engaged, and likely already loyal. The marketing strategy is retention: premium loyalty rewards, early access to new products, exclusive events. Cluster 4 (low income \(\approx \$26k\), low spending score \(\approx 21\), older \(\approx 45\)) is the Budget segment: value-sensitive, infrequent high-value purchases. The strategy is conversion: discount bundles, value-pack promotions, savings messaging.
The centroid coordinates (the cluster means) are the mathematical definition of the “average member” of each segment. They are the most useful single summary of a cluster for business communication: instead of describing a distribution, you describe a representative persona.
Naming segments is as much art as science. Common industry naming conventions: Transactors / Revolvers / Dormant (banking); Champions / Loyal / At-Risk / Lost (RFM marketing); Prime / Standard / Subprime (credit); Early Adopters / Mainstream / Laggards (technology adoption). The names should be memorable, non-pejorative (avoid calling any segment “bad”), and actionable — the name itself should suggest the strategy.
As an extension, run mall.groupby('Cluster').describe() to obtain not just the mean but also the minimum, maximum, and standard deviation of every feature within each cluster. Which cluster shows the most internal variation, and what does that tell you about the heterogeneity of the segment?
Hierarchical vs. K-Means agreement. In the exercise from Section 3, you compared hierarchical clustering (Ward, K=5) to K-Means (K=5) on the same customer data. When both methods agree on the assignment of most customers, you have strong evidence that the clusters are real structure in the data, not artifacts of one algorithm. When they disagree on many customers, the cluster boundary in that region is ambiguous — worth investigating whether those customers belong to a distinct segment or simply live in an overlapping zone between two segments.
Chapter Wrap-up
Chapter Summary
What You Have Learned
This chapter introduced the two foundational algorithms of unsupervised learning — K-Means and hierarchical clustering — and demonstrated their application to two real-world business problems. Here is what you should take away.
The unsupervised learning landscape. Clustering is the most widely deployed unsupervised method, but it sits within a broader family:
- K-Means and K-Medoids: partition-based; fast; require \(K\) in advance
- Hierarchical (agglomerative): tree-based; no \(K\) required upfront; produces dendrogram
- DBSCAN (Density-Based Spatial Clustering): finds clusters of arbitrary shape; handles noise/outliers explicitly; particularly useful for geographic data
- Gaussian Mixture Models (GMM): probabilistic, soft-assignment version of K-Means; useful when cluster boundaries are fuzzy
- Spectral Clustering: uses the eigenstructure of the similarity matrix; handles non-convex clusters; computationally expensive for large \(n\)
For most business applications, K-Means or Ward hierarchical clustering will get you 90% of the way there. Start simple.
The 5-step clustering workflow:
- Collect — gather relevant features that you believe encode group membership
- Standardize — apply
StandardScalerso every feature has equal weight in distance calculations - Choose \(K\) — use the Elbow Plot for K-Means; use the dendrogram height gap for hierarchical
- Cluster — fit the algorithm; for K-Means, use
n_init=10and a fixedrandom_state - Interpret — use
groupbyto profile clusters; name segments; connect to business actions
Step 5 is where most of the value is created — and where most of the effort should go. The algorithm does the math; you provide the judgment.
Decision flowchart: which method?
- \(n > 10{,}000\) and speed matters → K-Means
- \(n < 1{,}000\) and you want to explore the full structure → Hierarchical (Ward)
- You do not know \(K\) and want to discover it → Hierarchical (cut the dendrogram)
- You want soft cluster memberships → Gaussian Mixture Model
- Clusters may be non-spherical or of very different densities → DBSCAN
- In doubt → run both K-Means and hierarchical; compare results
What’s next. Clustering operates in the original feature space. When you have many features (say, 50 financial ratios), the distance calculations become noisy — in high dimensions, all points tend to be roughly equidistant from each other (the “curse of dimensionality”). The solution is dimensionality reduction: project the data into a lower-dimensional space that preserves the essential variation. Principal Component Analysis (PCA) is the linear version — it finds the directions of maximum variance and projects the data onto the first \(k\) principal components. t-SNE and UMAP are nonlinear alternatives that preserve local neighborhood structure, commonly used for visualizing high-dimensional clusters. These topics are the natural next step after mastering the algorithms in this chapter.
Further reading. The canonical references for the material in this chapter are:
- The Elements of Statistical Learning (Hastie, Tibshirani, and Friedman, 2009) — Chapter 14 covers unsupervised methods rigorously, including K-Means, hierarchical clustering, and principal components. Freely available at https://web.stanford.edu/~hastie/ElemStatLearn/
- An Introduction to Statistical Learning (James, Witten, Hastie, and Tibshirani, 2021) — Chapter 12 covers the same material at a more accessible level, with R code examples. Freely available at https://www.statlearning.com/
Final reflection. Every algorithm in this chapter is a mathematical object: it optimizes a criterion, converges to a fixed point, produces a partition. But the hard work — deciding which features to cluster on, choosing \(K\), naming the segments, and designing the marketing strategy for each — requires human judgment, domain knowledge, and creativity that no algorithm possesses. The machine finds the pattern; you decide what it means and what to do about it. That is the fundamental reason why data science skills amplify rather than replace business judgment.
Clustering is where mathematics meets business judgment. The algorithm finds the groups; you name them and act on them. Neither step works without the other.
Supervised vs. Unsupervised Learning
| Supervised | Unsupervised | |
|---|---|---|
| (Topics 1–3) | (Topic 4) | |
| Goal | Predict \(Y\) | Find groups |
| Labels? | Yes (need \(Y\)) | No (no target) |
| Methods | Regression, classification | Clustering, PCA |
| Evaluation | \(R^2\), MSE, accuracy | Inertia, silhouette, domain expertise |
| Example | Predict house price | Segment customers |
K-Means vs. Hierarchical Clustering
| K-Means | Hierarchical | |
|---|---|---|
| Choose \(K\)? | Before running | After (cut dendrogram) |
| Speed | Fast (large datasets) | Slower (\(O(n^2)\) memory) |
| Visualization | Scatter plot | Dendrogram |
| Shape | Spherical clusters | Any shape |
| Reproducibility | Depends on init | Deterministic |
| Best for | Large \(n\), clear groups | Small \(n\), exploring structure |
Use K-Means for speed and simplicity. Use Hierarchical for richer exploration and when you are unsure about \(K\). In practice, try both and compare!
Interactive Explorer: How K Changes the Clusters
K-Means requires you to commit to a number of clusters before it has even seen the data. The most direct way to build intuition for this choice is to watch the algorithm partition the same scatter of points under different settings of \(K\). The cell below generates three Gaussian blobs and then asks K-Means to recover them; edit the variables at the top to vary \(K\) from the true value of three to deliberate over- and under-fits, and read the resulting inertia (within-cluster sum of squares) to see how the objective falls as \(K\) grows.
As you raise \(K\), inertia falls monotonically — every additional centroid shortens at least one within-cluster distance — but the visual quality of the clusters improves only up to \(K=3\), the true number of generating blobs. Beyond that point K-Means begins to split a single natural group across two centroids, producing partitions that lower the objective without reflecting any underlying structure. This is precisely the phenomenon the elbow plot is designed to flag.
Debug Yourself: Forgetting to Standardise
Of every mistake a new practitioner makes with K-Means, omitting the standardisation step is the most common and the most damaging. The cell below clusters two simulated features — income in dollars and age in years — without scaling them, and a hidden assertion will refuse to pass until the bug is fixed. Inspect the printed cluster centres and ask yourself: do the two clusters differ in age, or only in income?
The bug is that K-Means uses Euclidean distance, and the income axis has a standard deviation of roughly 15,000 while age has a standard deviation of roughly 10 — a ratio of about 1,500 to 1. Income therefore dominates every distance calculation, and both cluster centres end up sitting at essentially the same age while differing only in income. The fix is to scale the features to a common range before fitting: insert from sklearn.preprocessing import StandardScaler; Xs = StandardScaler().fit_transform(X) and replace km.fit(X) with km.fit(Xs). Once the features share a scale, age contributes to the partition and the cluster centres separate along both axes.
Working with an AI Copilot
A modern analyst will be tempted to paste cluster centroids into an AI chat window and ask “what should I call these segments?” The AI will reply confidently with names like “tech-savvy millennials” or “value-driven empty nesters” — and almost none of those labels will be supported by the numbers in front of it. AI is fast at naming clusters but slow at validating whether the names make business sense. Three rules keep the copilot useful rather than misleading. First, before asking the AI for a label, confirm that the cluster means differ in ways that are both statistically detectable and practically meaningful — a two-dollar gap in average spending is not a segment. Second, ask the AI to state which features drove its proposed name; if it cannot point to specific feature values, the label is decorative rather than evidence-based. Third, the AI cannot tell you whether your \(K\) was right — that judgment requires the elbow plot, the silhouette score, and a domain expert who recognises whether the resulting partition matches how the business actually operates.
Decision Memo — How Many Customer Segments Should Marketing Target?
A clustering analysis is rarely the final deliverable. The decision it informs — how the marketing team will spend its next quarter of creative budget — is. The memo below shows the form in which a clustering result is normally communicated to a non-technical executive: a single recommendation, a small number of quantitative justifications, the assumptions that could overturn the recommendation, and a concrete next step that converts the analysis into a test.
To: VP Marketing, Netflix APAC From:
, customer analytics intern Subject: Re-target promotions to 3 — not 8 — viewer segments Date: 2026-05-15 Recommendation: Collapse the current 8-segment marketing model to 3 high-leverage segments.
Evidence: - K-Means with K=3 captures 68% of variance (within-cluster SS / total SS = 0.32). - Adding the 4th cluster only reduces WCSS by 4 additional percentage points — the elbow is at K=3. - The 3 segments map cleanly onto distinct content preferences: action, K-drama, family.
Caveats: - Features were standardised before clustering — without it, hours-per-week would dominate. - “Cluster identity” is a marketing label, not a customer attribute; people change clusters over time.
Next step: Run A/B test of cluster-specific creative on the next promotion vs. the current 8-segment baseline; measure click-through and conversion lift.
In 2006 Netflix launched a public competition with a one-million-dollar prize for any team that could improve its in-house recommendation algorithm by ten per cent. Three years later, a team called BellKor’s Pragmatic Chaos crossed the threshold using a 107-model ensemble that blended matrix factorisation, restricted Boltzmann machines, and a long list of regression refinements. Netflix paid the prize and then quietly never deployed the winning algorithm in production. Two reasons stand behind that decision. First, the engineering cost of running 107 models in serving infrastructure was prohibitive — the winning ensemble was roughly one hundred times slower than the model Netflix was already running, and it barely fit inside the latency budget that the product team required for a responsive interface. Second, the dataset had drifted under the contest’s feet: during the three-year competition Netflix migrated its business from mailing physical DVDs to streaming, and viewer behaviour on streaming was different enough that the model’s measured gains evaporated when scored against the new distribution. The lesson generalises directly to clustering. An analysis that wins on a clean static snapshot can fail in production once latency, infrastructure cost, and population drift enter the picture. Always ask whether the data on which the model was trained will still describe the population by the time the model runs.
Review Cards — Chapter 4
The cards below condense the chapter into seven concept-and-answer pairs. Click each card to reveal the answer, and revisit them on a spaced schedule before the exam.
Because K-Means uses Euclidean distance, and any feature with a larger scale (income in dollars vs. age in years) dominates the distance computation purely because of its units. Standardising to mean 0 / std 1 gives every feature an equal vote.
Assign each point to its nearest centroid, move every centroid to the mean of its assigned points, and repeat until no point changes cluster.
A centroid is the coordinate-wise mean of the points currently assigned to a cluster — its centre of gravity. K-Means is named because every iteration recomputes each cluster’s centroid as the mean of its \(K\) groups.
It plots the within-cluster sum of squares (WCSS) on the vertical axis against \(K\) on the horizontal axis. WCSS always falls as \(K\) rises, so you pick the \(K\) at the kink in the curve — the point where additional clusters yield only marginal reductions in WCSS.
K-Means is partitional and requires you to commit to \(K\) before fitting. Hierarchical clustering builds a full merge tree (dendrogram) once and lets you read off any \(K\) afterwards by cutting the tree at the desired height.
Ward merges the pair whose union produces the smallest increase in total within-cluster variance — equivalently, the smallest increase in WCSS. This is the same objective that K-Means minimises, which is why Ward and K-Means tend to agree.
Because the dendrogram records every merge from \(N\) singletons up to one cluster, a horizontal cut at any height crosses some number of branches and instantly defines a flat clustering at that \(K\). One fit, every possible \(K\).