Choosing K (elbow method, silhouette score)

The Elbow Method

The elbow method finds the K where adding more clusters stops giving you much benefit.

How It Works: Inertia as Your Guide

WHY this formula?
Each term $\|\mathbf{x} - \boldsymbol{\mu}_i\|^2$ is the squared distance from a point to its cluster center. Summing over all points in all clusters gives total "compactness". Lower inertia = tighter clusters.

Derivation from First Principles:

Goal: Minimize variance within each cluster
- Variance measures spread. For cluster $C_i$ with centroid $\boldsymbol{\mu}_i$ : $\text{Var}(C_i) = \frac{1}{|C_i|}\sum_{\mathbf{x} \in C_i} \|\mathbf{x} - \boldsymbol{\mu}_i\|^2$
- Total variance across all clusters: multiply by $|C_i|$ and sum → that's inertia!
Why squared distance?
- Penalizes outliers heavily (point 2× farther contributes 4× to inertia)
- Yields a closed-form centroid: minimizing $\sum \|\mathbf{x} - \boldsymbol{\mu}\|^2$ over $\boldsymbol{\mu}$ gives $\boldsymbol{\mu} = \text{mean}$ of the points. This is why K-means updates centroids to the mean—it's a coordinate-descent (Lloyd's algorithm), alternating point-assignment and mean-recomputation, not gradient descent.
- Matches Euclidean geometry assumption
The Elbow:
- As K increases, inertia always decreases (more clusters = points closer to some centroid)
- At K = N (number of points), inertia = 0 (each point is its own cluster)
- But somewhere between K=1 and K=N, the rate of decrease sharply slows—that's the elbow!

Figure — Choosing K (elbow method, silhouette score)

HOW to Apply:

Run K-means for = 1, 2, 3, .., K_max
Record inertia for each K
Plot K (x-axis) vs Inertia (y-axis)
Find the "elbow"—where the curve bends sharply

Worked example Worked Example: Customer Segmentation

You have 200 customers with 2 features (spending, frequency). Run K-means:

K	Inertia
1	15000
2	8000
3	4500
4	3800
5	3400
Why this step? From K=3 to K=4, inertia drops by 700. From K=4 to K=5, only 400. The marginal benefit shrinks dramatically after K=4.

Interpretation: K=4 captures major customer groups (e.g., low/high spending × occasional/frequent). K=5 starts splitting natural groups unnecessarily.

Silhouette Score

The silhouette score measures how well each point fits its cluster compared to neighboring clusters.

Derivation: Cohesion vs Separation

For a single point $\mathbf{x}_i$ in cluster $C_I$ :

Cohesion (how close to own cluster):
$a(i) = \frac{1}{|C_I| - 1} \sum_{\mathbf{x}_j \in C_I, j \neq i} d(\mathbf{x}_i, \mathbf{x}_j)$ Average distance to all other points in the same cluster.
WHY $|C_I| - 1$ in the denominator? There are $|C_I|$ points in the cluster, but we exclude $\mathbf{x}_i$ itself, leaving $|C_I| - 1$ neighbors to average over.
Separation (how far from nearest other cluster):
$b(i) = \min_{J \neq I} \frac{1}{|C_J|} \sum_{\mathbf{x}_j \in C_J} d(\mathbf{x}_i, \mathbf{x}_j)$ Average distance to points in the nearest neighboring cluster $C_J$ .
WHY minimum? We care about the closest alternative—if $\mathbf{x}_i$ is far from all other clusters, it's well-placed.
Silhouette coefficient:
$s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$

Derivation logic:

If $\mathbf{x}_i$ is well-clustered: $a(i)$ small (close to own cluster), $b(i)$ large (far from others) → $s(i) \approx 1$
If poorly clustered: $a(i) \approx b(i)$ → $s(i) \approx 0$
If misclassified: $a(i) > b(i)$ → $s(i) < 0$

WHY divide by max? Normalizes to $[-1, 1]$ range regardless of distance scale.

Worked example Worked Example: Iris Dataset (2D Projection)

3 flower species, 50 samples each. Run K-means with K=3: Point A (in Species 1 cluster):

$a(A) = 0.5$ (avg distance to other Species 1 points)
$b(A) = 1.8$ (avg distance to nearest other cluster, Species 2)
$s(A) = \frac{1.8 - 0.5}{\max(0.5, 1.8)} = \frac{1.3}{1.8} \approx 0.72$ ✓ (well-clustered)

Point B (in Species 2, near Species 3boundary):

$a(B) = 0.9$
$b(B) = 1.0$
$s(B) = \frac{1.0 - 0.9}{1.0} = 0.1$ ⚠ (ambiguous)

Overall score: Average all 150 points →0.58 (decent but not perfect).

Why this step? Point B's low score flags that Species 2 and 3 overlap—maybe they're one cluster, or K-means isn't the right algorithm (try DBSCAN for non-spherical clusters).

Using Silhouette to Choose K

Compute silhouette score for K = 2, 3, .., K_max
Pick K with highest average silhouette
Bonus: Plot silhouette for each point (silhouette diagram)—clusters with many negative scores are poorly defined

Combining Elbow and Silhouette

Method	What It Optimizes	Strength	Weakness
Elbow	Inertia drop rate	Simple, fast	Subjective (where's the elbow?)
Silhouette	Cohesion vs separation	Quantitative score	Biased toward fewer clusters

Best practice workflow:

Plot elbow curve → identify 2-3 candidate K values
Compute silhouette for those candidates
Check silhouette diagram (per-point scores)—look for balanced cluster sizes, few negatives
Validate with domain experts or downstream task

Recall Explain to a 12-Year-Old

Imagine you're sorting your toys into boxes. You could use1 giant box (everything together) or 100 tiny boxes (one per toy). Neither is useful!

Elbow method: Make a graph of "how mesy the boxes are" vs. "number of boxes." At first, adding boxes helps a LOT (mesy drops fast). But then it barely helps. The spot where it stops helping much? That's the elbow—your sweet spot.

Silhouette score: For each toy, check: "Am I way closer to toys in my box than toys in other boxes?" If yes, you get a high score. If you're kinda in-between boxes, low score. Average everyone's score—higher = better sorting.

Use both! Elbow finds where to stop, silhouette checks if your sorting actually makes sense.

Connections

K-Means Clustering: These methods only work for algorithms that need K upfront
DBSCAN: Density-based clustering doesn't require choosing K (finds it automatically)
Hierarchical Clustering: Use dendrograms instead of elbow/silhouette
Gap Statistic: Advanced alternative—compares inertia to null distribution
Hyperparameter Tuning: Choosing K is like choosing learning rate—requires validation
Bias-Variance Tradeoff: Too few clusters = high bias, too many = high variance (overfitting noise)

Flashcards

#flashcards/ai-ml

What does inertia measure in K-means? :: Within-cluster sum of squared distances—how tightly points cluster around centroids. Lower = tighter clusters.

Elbow method: What is the "elbow"?

The K value where inertia stops decreasing rapidly. After the elbow, adding clusters gives diminishing returns.

Why does inertia always decrease as K increases?

More clusters means points are closer to some centroid. At K=N (each point is its own cluster), inertia = 0.

Why does K-means use the mean as the centroid update?

Because minimizing sum of squared distances has a closed-form solution: the minimizer is the mean. This makes Lloyd's algorithm a coordinate-descent, not gradient descent.

Silhouette coefficient formula point i :: $s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$ where $a(i)$ = avg distance to own cluster, $b(i)$ = avg distance to nearest other cluster.

In the cohesion term a(i), why is the denominator |C_I| - 1?

The cluster has |C_I| points, but we exclude the point itself, leaving |C_I| - 1 neighbors to average distances over.

Silhouette score ranges and interpretation

Range: [-1, 1]. 1 = perfect (tight, separated), 0 = overlapping clusters, -1 = misclassified points.

Why does silhouette often favor K=2?

Splitting data in half maximizes separation. With more clusters, some inevitably overlap, lowering the score. Domain knowledge must override this bias.

What does a negative silhouette coefficient mean for a point?

The point is closer to a neighboring cluster than its own—likely misclassified.

Elbow method weakness

The elbow can be ambiguous or absent in mesy real-world data (no sharp bend in the curve).

Why divide silhouette by max(a, b)?

Normalizes the score to [-1, 1] regardless of distance scale, making it comparable across datasets.

Best practice for choosing K :: Use elbow + silhouette + domain knowledge together. No single method is perfect; they vote on the best K.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Jab ap K-means clustering karte ho, toh sabse pehla sawal ata hai—kitne clusters chahiye? Yeh number randomly nahi choose kar sakte, kyunki galat K se ya toh sab dataek pile mein mix ho jaayega, ya phir bahut zyada chhote-chhote groups ban jaayenge jo kaam ke nahi. Elbow method aur silhouette score do famous techniques hain jo is problem ko solve karti hain.

Elbow method ka logic simple hai—ap different values of K try karte ho (jaise 2, 3, 4 clusters) aur dekhte ho ki har K pe "inertia" (matlab points apne cluster center se kitna door hain) kaise change hoti hai. Initially jab K badhate ho, inertia fast drop hoti hai, lekin ek point ke bad improvement bahut slow ho jata hai. Us turning point ko "elbow" kehte hain, aur wahi optimal K hota hai. Ek important baat—K-means centroid ko har baar mean pe update karta hai, kyunki squared distance minimize karne ka closed-form solution hi mean hota hai. Ye

Choosing K (elbow method, silhouette score)

The Elbow Method

How It Works: Inertia as Your Guide

Silhouette Score

Derivation: Cohesion vs Separation

Using Silhouette to Choose K

Combining Elbow and Silhouette

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Unsupervised Learning

Connections