Vector norms (L1, L2, L-infinity, Lp)

What is a norm? (first principles)

WHY these three axioms? They are the minimum needed for "size" to behave sensibly:

Axiom 1 says only the zero vector has zero length.
Axiom 2 says doubling a vector doubles its length (scaling is linear).
Axiom 3 says a detour is never shorter than the direct path.

The Lp family

HOW to read it: raise each coordinate's magnitude to power $p$ , sum, take the $p$ -th root. The root "undoes" the power so that homogeneity holds.

Derivation of the L∞ limit (from scratch). Let $M=\max_i|x_i|$ . Factor it out: $\|x\|_p=\left(\sum_i |x_i|^p\right)^{1/p}=M\left(\sum_i \left(\tfrac{|x_i|}{M}\right)^p\right)^{1/p}.$ Every ratio $\tfrac{|x_i|}{M}\le 1$ . Say $k$ coordinates equal the max (ratio $=1$ ), the rest are $<1$ . As $p\to\infty$ , terms with ratio $<1$ vanish, so the sum $\to k$ . Then $\|x\|_p \to M\cdot k^{1/p} \to M\cdot 1 = M,$ since $k^{1/p}\to 1$ for any fixed $k$ . Hence $\|x\|_\infty=\max_i|x_i|$ . ∎

Figure — Vector norms (L1, L2, L-infinity, Lp)

Worked examples

Let $x=(3,-4)$ throughout.

Common mistakes (Steel-manned)

Feynman check

Recall Explain to a 12-year-old

Imagine you're at home and your friend lives 3 blocks east and 4 blocks north.

If you walk on streets you go 3 + 4 = 7 blocks. That's the L1 distance.
If you could fly in a straight line it's 5 blocks (the diagonal). That's L2.
If you only care about the longest single stretch, that's 4 (the north part). That's L∞. Same two friends, three answers — because "how far" depends on how you're allowed to move.

Flashcards

What three axioms define a norm?

Non-negativity (

\|x\|=0\iff x=0

), absolute homogeneity (

\|\alpha x\|=|\alpha|\|x\|

), triangle inequality (

\|x+y\|\le\|x\|+\|y\|

General formula for the Lp norm?

\|x\|_p=\left(\sum_i |x_i|^p\right)^{1/p}

for

p\ge 1

L1 norm of

(3,-4)

|3|+|-4|=7

L2 norm of

(3,-4)

\sqrt{9+16}=5

L∞ norm of

(3,-4)

\max(3,4)=4

Why does L1 induce sparsity but L2 doesn't?

L1's unit ball is a diamond with corners on the axes, so optimal solutions land on axes → coordinates become exactly 0; L2's ball is smooth (a circle) with no corners.

General inequality among norms of the same vector?

\|x\|_\infty \le \|x\|_2 \le \|x\|_1

; larger

p

gives a smaller-or-equal norm.

Why is L0 not a true norm?

It counts nonzeros but violates homogeneity:

\|2x\|_0=\|x\|_0\ne 2\|x\|_0

Derive

\|x\|_\infty=\max_i|x_i|

Factor

M=\max|x_i|

\|x\|_p=M(\sum (|x_i|/M)^p)^{1/p}

; ratios

<1

vanish as

p\to\infty

, leaving

M\cdot k^{1/p}\to M

What norm relates to the dot product?

L2, since

\|x\|_2=\sqrt{x^\top x}

For which

p

is Lp a valid norm?

p\ge 1

(below that the triangle inequality fails).

Connections

Dot product and inner products — L2 is $\sqrt{x^\top x}$ .
Regularization L1 vs L2 (Lasso vs Ridge) — direct application of norm geometry.
Distance metrics and k-NN — Manhattan vs Euclidean choice.
Convexity and unit balls — why $p\ge1$ guarantees a convex ball.
Gradient clipping — clip by L2 norm threshold.
Matrix norms — generalization from vectors to matrices.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, norm ka matlab simple hai: "yeh vector kitna bada hai?" Lekin "bada" ka matlab depend karta hai ki aap distance kaise naapte ho. L1 (Manhattan) matlab city ke blocks mein chalke jaana — saare coordinates ke absolute values add kar do. L2 (Euclidean) matlab seedha diagonal udd ke jaana — Pythagoras wala $\sqrt{x_1^2+x_2^2}$ . L∞ matlab sirf sabse bada single coordinate uthao, baaki ignore. Example $(3,-4)$ : L1 = 7, L2 = 5, L∞ = 4. Notice karo — jitna bada $p$ , utni chhoti norm.

General formula ek hi hai: $\|x\|_p=(\sum|x_i|^p)^{1/p}$ , aur $p\to\infty$ karne se woh $\max$ ban jaata hai (note mein full derivation hai). Yaad rakho — power lagao, sum karo, root lagao; root isliye taaki units theek rahein.

ML mein yeh kyun important hai? Lasso = L1, Ridge = L2. L1 ka unit ball ek diamond hota hai jiske corners axes pe baithe hote hain — isliye solution axis pe land karta hai aur kuch weights bilkul zero ho jaate hain (sparsity!). L2 ka ball smooth circle hai, koi corner nahi, isliye weights chhote hote hain par exactly zero nahi. k-NN mein bhi aap decide karte ho Manhattan ya Euclidean distance — woh bhi yahi norm choice hai.

Ek common galti: L0 ko "norm" mat samjho — woh sirf non-zero count karta hai aur homogeneity todta hai, isliye woh pseudo-norm hai. Aur $p<1$ pe triangle inequality fail ho jaati hai, toh valid norm ke liye hamesha $p\ge 1$ chahiye.

What is a norm? (first principles)

The Lp family

Deriving the special cases

Worked examples

Common mistakes (Steel-manned)

Feynman check

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Test yourself — Linear Algebra Essentials

Connections