Dot product and its geometric meaning (projection, angle)

WHAT is the dot product?

WHY do we care (the AI-ML reason)? Cosine similarity, gradient descent (dot of gradient with a direction), attention scores (query·key), and projections onto principal components are all dot products. Master this and half of ML geometry unlocks.

WHY the two definitions agree — a derivation from scratch

We derive the geometric form using the Law of Cosines. Consider the triangle formed by $\mathbf{a}$ , $\mathbf{b}$ , and the vector $\mathbf{a}-\mathbf{b}$ (the side connecting their tips).

Step 1 — Law of cosines. The side opposite angle $\theta$ has length: $\|\mathbf{a}-\mathbf{b}\|^2 = \|\mathbf{a}\|^2 + \|\mathbf{b}\|^2 - 2\|\mathbf{a}\|\|\mathbf{b}\|\cos\theta$ Why this step? This is pure geometry — the generalization of Pythagoras when the angle isn't $90°$ .

Step 2 — Expand the left side algebraically. Using $\|\mathbf{v}\|^2 = \mathbf{v}\cdot\mathbf{v}$ : $\|\mathbf{a}-\mathbf{b}\|^2 = (\mathbf{a}-\mathbf{b})\cdot(\mathbf{a}-\mathbf{b}) = \mathbf{a}\cdot\mathbf{a} - 2\,\mathbf{a}\cdot\mathbf{b} + \mathbf{b}\cdot\mathbf{b}$ Why this step? The dot product distributes just like multiplication, so we can FOIL it out. $= \|\mathbf{a}\|^2 - 2\,\mathbf{a}\cdot\mathbf{b} + \|\mathbf{b}\|^2$

Step 3 — Equate the two expressions and cancel $\|\mathbf{a}\|^2+\|\mathbf{b}\|^2$ : $-2\,\mathbf{a}\cdot\mathbf{b} = -2\|\mathbf{a}\|\|\mathbf{b}\|\cos\theta$ $\boxed{\;\mathbf{a}\cdot\mathbf{b} = \|\mathbf{a}\|\|\mathbf{b}\|\cos\theta\;}$ Why this step? Dividing by $-2$ isolates the result. The algebraic sum $\sum a_ib_i$ and the geometric $\|\mathbf a\|\|\mathbf b\|\cos\theta$ are provably identical.

Angle and cosine similarity

Solve the boxed formula for the angle: $\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|\,\|\mathbf{b}\|}$

Projection — "how much of $\mathbf a$ lies along $\mathbf b$ "

Derivation. The scalar projection is the length of $\mathbf a$ in the direction of $\mathbf b$ : $\text{comp}_{\mathbf b}\mathbf a = \|\mathbf a\|\cos\theta$ Why? Basic trig: adjacent side = hypotenuse $\times \cos$ . Now substitute $\cos\theta = \dfrac{\mathbf a\cdot\mathbf b}{\|\mathbf a\|\|\mathbf b\|}$ : $\text{comp}_{\mathbf b}\mathbf a = \|\mathbf a\|\cdot\frac{\mathbf a\cdot\mathbf b}{\|\mathbf a\|\|\mathbf b\|} = \frac{\mathbf a\cdot\mathbf b}{\|\mathbf b\|}$

The vector projection points along the unit vector $\hat{\mathbf b} = \mathbf b/\|\mathbf b\|$ : $\text{proj}_{\mathbf b}\mathbf a = \left(\frac{\mathbf a\cdot\mathbf b}{\|\mathbf b\|}\right)\frac{\mathbf b}{\|\mathbf b\|} = \frac{\mathbf a\cdot\mathbf b}{\mathbf b\cdot\mathbf b}\,\mathbf b$ Why $\mathbf b\cdot\mathbf b$ ? Because $\|\mathbf b\|^2 = \mathbf b\cdot\mathbf b$ — cleaner and avoids square roots.

Figure — Dot product and its geometric meaning (projection, angle)

Worked examples

Common mistakes (steel-manned)

Active recall

Recall Predict before revealing (Forecast-then-Verify)

What is $(1,0,0)\cdot(0,1,0)$ and what does it tell you?
Rewrite $\|\mathbf v\|^2$ using a dot product.
Why do we divide by $\mathbf b\cdot\mathbf b$ in the vector projection?

Answers: 1) $0$ → orthogonal axes. 2) $\|\mathbf v\|^2=\mathbf v\cdot\mathbf v$ . 3) Because $\|\mathbf b\|^2=\mathbf b\cdot\mathbf b$ , and we need to scale $\mathbf b$ down to unit length twice (once for direction, once inside comp).

Recall Feynman: explain to a 12-year-old

Imagine two people pushing a shopping cart. If they push in the same direction, the cart moves fast — the dot product is big. If one pushes sideways to the other, that sideways push doesn't help at all — dot product is zero. If they push against each other, the dot product goes negative. The dot product just measures how much teamwork is in the same direction, times how hard each pushes.

Flashcards

What are the two equivalent definitions of the dot product?

Algebraic

\sum a_ib_i

and geometric

\|\mathbf a\|\|\mathbf b\|\cos\theta

What does a dot product of zero mean geometrically?

The vectors are orthogonal (perpendicular,

\theta=90°

Formula for the angle between two vectors?

\cos\theta=\dfrac{\mathbf a\cdot\mathbf b}{\|\mathbf a\|\|\mathbf b\|}

Vector projection of

\mathbf a

onto

\mathbf b

\text{proj}_{\mathbf b}\mathbf a=\dfrac{\mathbf a\cdot\mathbf b}{\mathbf b\cdot\mathbf b}\mathbf b

Scalar projection (component) of

\mathbf a

along

\mathbf b

\dfrac{\mathbf a\cdot\mathbf b}{\|\mathbf b\|}

Why use cosine similarity instead of raw dot product in ML?

It removes magnitude, comparing only direction; range

[-1,1]

What does a negative dot product indicate?

The angle is obtuse (

>90°

); vectors point partly away from each other.

Express

\|\mathbf v\|^2

as a dot product.

\mathbf v\cdot\mathbf v

Which law is used to prove the two dot-product definitions are equal?

The Law of Cosines.

\text{proj}_{\mathbf b}\mathbf a = \text{proj}_{\mathbf a}\mathbf b

No — the target vector's length is in the denominator, so they differ.

Connections

Vector norms and length — $\|\mathbf v\|=\sqrt{\mathbf v\cdot\mathbf v}$ comes straight from the dot product.
Cosine similarity and embeddings — direct ML application.
Orthogonality and orthonormal bases — dot product $=0$ is the definition.
Matrix multiplication as dot products — each entry of $AB$ is a row·column dot product.
Gradient descent — directional derivative is $\nabla f\cdot\mathbf d$ .
Attention mechanism — query·key scores are dot products.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, dot product basically batata hai ki do vectors kitna "same direction" mein point kar rahe hain. Do definitions hain jo actually same number dete hain: ek algebraic ( $a_1b_1+a_2b_2+\dots$ , yaani matching components multiply karke add karo), aur ek geometric ( $\|a\|\|b\|\cos\theta$ ). Jab dono aligned hote hain toh cosine 1 hota hai aur value badi positive; jab perpendicular hote hain toh $\cos 90°=0$ , isliye dot product zero — yeh orthogonality ka test hai. Aur agar opposite direction mein hain toh negative.

Angle nikalne ke liye formula simple hai: $\cos\theta = \frac{a\cdot b}{\|a\|\|b\|}$ . ML mein isko cosine similarity bolte hain — yeh magnitude ko ignore karke sirf direction compare karta hai. Isliye do documents ya embeddings alag-alag length ke ho sakte hain, par agar direction same hai toh similarity 1 aayegi. Yahi reason hai ki hum normalize karte hain.

Projection ka intuition "shadow" wala hai: agar tum $\mathbf b$ ki line par upar se light maaro, toh $\mathbf a$ ka jo shadow banega, woh projection hai. Formula: $\text{proj}_b a = \frac{a\cdot b}{b\cdot b}\,b$ . Denominator mein $b\cdot b$ isliye kyunki woh $\|b\|^2$ hai, aur do baar length se divide karna padta hai (ek direction ke liye, ek component ke liye).

Do galtiyan avoid karo: (1) bada dot product = zyada similar — galat, kyunki usme magnitude ghusi hoti hai, isliye cosine use karo. (2) Negative dot product ka matlab perpendicular nahi, balki obtuse angle (90° se zyada) hota hai. Perpendicular ka matlab exactly zero. Yeh concept matrix multiplication, attention scores, aur gradient descent — sab jagah kaam aata hai, isliye achhe se pakad lo.

Test yourself — Linear Algebra Essentials

Connections