Scalars, vectors, matrices, and tensors definitions

WHY do we care? (80/20)

WHAT are they? (definitions by rank)

The key organizing concept is rank (a.k.a. order or number of axes): how many indices you need to address one element.

Figure — Scalars, vectors, matrices, and tensors definitions

HOW do we count elements? (derive it from scratch)

Step 1 — Scalar. How many numbers? Just one. $N_0 = 1$ Why this step? Rank 0 means no index ranges over anything.

Step 2 — Vector of length $n$ . Index $i$ runs $1 \to n$ . $N_1 = n$ Why this step? One index, $n$ choices.

Step 3 — Matrix $m \times n$ . For each of the $m$ rows, there are $n$ columns. $N_2 = m \cdot n$ Why this step? Independent choices multiply (fundamental counting principle).

Step 4 — Tensor $d_1 \times \dots \times d_k$ . Generalize: multiply every axis length. $\boxed{N_k = \prod_{r=1}^{k} d_r}$ Why this step? Each index contributes an independent multiplicative factor — exactly like nested loops.

Transpose: a rank-2 operation

Why: it reflects the grid across the main diagonal. A vector $\mathbf{v}\in\mathbb{R}^n$ (a column) transposes to a row $\mathbf{v}^\top \in \mathbb{R}^{1\times n}$ . A scalar is its own transpose ( $a^\top = a$ ).

Worked Examples

Common Mistakes (Steel-manned)

Flashcards

How many indices address one element of a rank-

k

tensor?

Exactly

k

A scalar is a tensor of what rank?

Rank 0.

Shape of a vector in

\mathbb{R}^n

vs a matrix in

\mathbb{R}^{m\times n}

Vector:

(n,)

, rank 1. Matrix:

(m,n)

, rank 2.

Total number of elements in a

d_1\times\dots\times d_k

tensor?

\prod_{r=1}^k d_r

What does

(A^\top)_{ij}

equal?

A_{ji}

(swap the two indices).

Shape of a batch of 64 RGB

28\times28

images?

(64,28,28,3)

, a rank-4 tensor.

Why is a

1\times n

matrix NOT the same as an

n

-vector?

Different rank (2 vs 1) and shape; they broadcast/multiply differently.

"Tensor rank/order" vs "matrix rank" — difference?

Order = number of axes; matrix rank = number of linearly independent rows/columns.

Memory of a float32 tensor with

N

elements?

4N

bytes.

Transpose of a scalar?

The scalar itself (

a^\top=a

Recall Feynman: explain to a 12-year-old

Imagine boxes of numbers. A scalar is one number on a sticky note. A vector is a single row of lockers — say which locker (one number) and you get its value. A matrix is a whole wall of lockers — you say the row and the column (two numbers). A tensor is a giant building with many walls, floors, and buildings — you keep giving more numbers until you've pointed at exactly one locker. The number of things you have to say to find one value is the "rank". Multiply all the sizes together and you know how many lockers (numbers) there are in total.

Connections

Matrix multiplication — needs matching inner dimensions, built on these shapes.
Broadcasting rules — how scalars/lower-rank tensors combine with higher-rank ones.
Transpose and symmetry
Tensor reshaping and flattening — rearranging the $\prod d_r$ elements.
Dot product and matrix-vector product
Rank of a matrix (linear independence) — the other meaning of "rank".

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ye chaar cheezein sirf numbers ke containers hain, aur inme fark sirf itna hai ki ek single number tak pahunchne ke liye tumhe kitne index batane padte hain. Scalar matlab akela number — 0 index (jaise loss ka value 3.7). Vector matlab ek line me numbers ki list — 1 index (jaise ek ghar ke features: bedrooms, area, price). Matrix matlab grid, rows aur columns — 2 index (row aur column dono batao). Aur tensor matlab isi cheez ka general version, jitne chaaho utne index — jaise ek batch of images me (image number, row, column, color) = 4 index.

Sabse important 20% jo 80% kaam karwa deta hai: rank = number of axes (indices). Jaise hi tum ye samajh gaye, shapes samajh aa jaayenge, matrix multiplication ke rules logical lagenge, aur memory ka hisaab bhi. Total elements nikalne ke liye bas saare axis ki lengths multiply kar do: $\prod d_r$ . Nested loops ki tarah — har index ek multiply factor deta hai.

Ek common galti: "rank" word do jagah use hota hai. Yahan tensor ka rank matlab kitne axes hain. Lekin "matrix rank" ek alag cheez hai (independent rows/columns). Confuse mat hona — safe rehne ke liye axes wale ko "order" bol do. Doosri galti: $1\times n$ matrix aur $n$ -vector ek jaise dikhte hain par rank alag hai (2 vs 1), aur code me broadcasting alag behave karta hai — yahi silent bug ka top reason hai.

ML me ye foundation hai: har data — chahe loss ho, ek example ho, weights ho, ya images ka batch — inhi me se ek hai. Toh index count wala mental model pakka kar lo, aage sab easy lagega.

Test yourself — Linear Algebra Essentials

Connections