Distance metrics (Euclidean, Manhattan, cosine)

What is a distance metric? (First principles)

The Minkowski family (derive them ALL from one formula)

Formula Derivation of Euclidean (

$p=2$ ) WHY: the shortest straight line "as the crow flies." Comes directly from the Pythagorean theorem. In 2D, the hypotenuse of the right triangle with legs $|a_1-b_1|$ and $|a_2-b_2|$ is $\sqrt{(a_1-b_1)^2+(a_2-b_2)^2}$ . Generalize to $n$ dimensions. Set $p=2$ : $d_2(\mathbf{a},\mathbf{b}) = \left(\sum_i |a_i-b_i|^2\right)^{1/2} = \sqrt{\sum_{i=1}^{n}(a_i-b_i)^2}$ Since squares are already non-negative, $|a_i-b_i|^2 = (a_i-b_i)^2$ . This is Euclidean (L2).

Cosine similarity & distance (a DIFFERENT idea: angle, not length)

Formula Derivation from the dot product

Start: the geometric definition of the dot product: $\mathbf{a}\cdot\mathbf{b} = \|\mathbf{a}\|\,\|\mathbf{b}\|\cos\theta$ WHY: this is provable from the law of cosines applied to the triangle formed by $\mathbf{a}$ , $\mathbf{b}$ , $\mathbf{a}-\mathbf{b}$ . Solve for $\cos\theta$ : $\text{cosine similarity} = \cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|} = \frac{\sum_i a_i b_i}{\sqrt{\sum_i a_i^2}\,\sqrt{\sum_i b_i^2}}$ Range: $[-1, 1]$ (or $[0,1]$ for non-negative data). Then: $\text{cosine distance} = 1 - \cos\theta$ HOW to read it: $\cos\theta=1$ → same direction → distance $0$ . $\cos\theta=0$ → perpendicular → distance $1$ . $\cos\theta=-1$ → opposite → distance $2$ .

Figure — Distance metrics (Euclidean, Manhattan, cosine)

Worked Examples

Worked example Example 1 — all three on 2D points

$\mathbf{a}=(1,2)$ , $\mathbf{b}=(4,6)$ .

Manhattan: $|1-4|+|2-6| = 3+4 = 7$ . Why this step? L1 just sums the axis-wise moves — no squaring.

Euclidean: $\sqrt{(1-4)^2+(2-6)^2} = \sqrt{9+16} = \sqrt{25} = 5$ . Why this step? Squaring turns the two legs into a Pythagorean hypotenuse; the diagonal (5) is shorter than the grid walk (7) — always true.

Cosine: $\dfrac{1\cdot4 + 2\cdot6}{\sqrt{1+4}\,\sqrt{16+36}} = \dfrac{4+12}{\sqrt5\,\sqrt{52}} = \dfrac{16}{\sqrt{260}} \approx 0.992$ . distance $= 1-0.992 = 0.008$ . Why this step? Both points lie almost on the same ray from the origin, so their angle is tiny even though Euclidean says 5 apart. Different questions, different answers!

Common Mistakes (Steel-manned)

Flashcards

What single parent formula generates Euclidean, Manhattan, and Chebyshev?

Minkowski distance

d_p=(\sum_i|a_i-b_i|^p)^{1/p}

, with

p=2,1,\infty

respectively.

Write the Euclidean distance formula.

\sqrt{\sum_i (a_i-b_i)^2}

— from the Pythagorean theorem.

Write the Manhattan distance formula.

\sum_i |a_i-b_i|

— sum of absolute axis-wise differences (grid walk).

Derive cosine similarity from the dot product.

\mathbf{a}\cdot\mathbf{b}=\|\mathbf{a}\|\|\mathbf{b}\|\cos\theta \Rightarrow \cos\theta=\frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\|\mathbf{b}\|}

How do you turn cosine similarity into a distance?

distance

=1-\cos\theta

, ranging

0

(same direction) to

2

(opposite).

Which metric ignores vector magnitude and why is that useful?

Cosine — it measures angle/orientation only; ideal for text/TF-IDF where doc length shouldn't matter.

Which metric axiom does cosine distance violate?

The triangle inequality (so it's a dissimilarity, not a true metric).

Why must you scale features before Euclidean/Manhattan kNN?

Large-unit features dominate the sum, distorting "closeness"; standardize with

z=(x-\mu)/\sigma

For the same two points, which is larger: Euclidean or Manhattan?

Manhattan

\ge

Euclidean (grid walk is never shorter than the straight diagonal).

What does Minkowski

p\to\infty

give?

Chebyshev distance

\max_i|a_i-b_i|

Recall Feynman: explain to a 12-year-old

Imagine two houses on a map. Euclidean distance is how far a bird flies straight between them. Manhattan distance is how far a taxi drives on the streets — it can't fly, so it goes across then up, which is longer. Cosine is totally different: forget "how far," ask "are they in the same direction from the town center?" Two friends both walking east are going the "same way" even if one is 1 km out and the other 10 km out. A computer decides which houses are "neighbors" using one of these rulers — and picking the wrong ruler picks the wrong friends.

Connections

k-Nearest Neighbors — the metric decides who counts as a neighbor.
Feature Scaling & Standardization — mandatory before L1/L2 distances.
Curse of Dimensionality — why Euclidean degrades in high dimensions.
TF-IDF & Text Vectorization — where cosine shines.
K-Means Clustering — uses (squared) Euclidean in its objective.
RBF / Gaussian Kernel — $e^{-\gamma d^2}$ builds on Euclidean distance.
Pythagorean Theorem — the root of L2.
Dot Product & Law of Cosines — the root of cosine similarity.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, kNN jaise "instance-based" models kuch train nahi karte — bas saara data yaad rakhte hain, aur prediction ke time poochte hain "sabse paas wale points kaun se hain?". Lekin "paas" ka matlab tabhi banta hai jab hum distance metric define karein — yani ek ruler. Alag ruler → alag neighbors → alag answer. Isliye metric hi model ka asli inductive bias hai.

Teen main rulers hain. Euclidean (L2) = seedhi line, Pythagoras se aata hai: $\sqrt{\sum(a_i-b_i)^2}$ . Socho ek chidiya udd ke seedha jaati hai. Manhattan (L1) = grid pe taxi ki tarah, sirf axis ke along chalti hai, diagonal cut nahi kar sakti: $\sum|a_i-b_i|$ — isiliye ye hamesha Euclidean se bada ya barabar hota hai. Dono ek hi parent Minkowski formula ke bacche hain, bas $p=2$ ya $p=1$ daalo.

Cosine bilkul alag soch hai — ye "kitni door" nahi, "kis direction mein" poochta hai. Dot product se derive: $\cos\theta = \frac{a\cdot b}{\|a\|\|b\|}$ , aur distance $= 1-\cos\theta$ . Text data (TF-IDF) mein bahut kaam ka, kyunki document ki length (magnitude) matter nahi karti, sirf topic ki direction. Jaise "cat cat cat" aur "cat cat cat cat cat" same direction mein hain — cosine bolega bilkul same.

Do bade practical points yaad rakhna: (1) Euclidean/Manhattan use karne se pehle features ko scale/standardize karo, warna bade units wala feature (jaise income) sab kuch hijack kar lega. (2) Cosine similarity ko distance mein convert karna mat bhoolo ( $1-\cos\theta$ ), warna kNN ulte neighbors chun lega. Bas yeh dhyan rakha toh distance metrics easy hain.

Test yourself — Tree-Based & Instance Methods

Connections