Foundations — PyTorch tensors and operations

Before you can read a single line of the parent note, you must be able to see what a shape like (32, 3, 224, 224) means, what dim=0 points at, and what a @ symbol actually computes. This page builds every one of those from nothing. We assume you know how to add and multiply numbers — nothing else.

1. A number, a list, a grid — the ladder of dimensions

Let us start with the smallest possible object and climb.

The picture below is the whole idea. Look at how each object is built from copies of the one before it.

WHY the topic needs this. Neural-network data is naturally shaped. A colour image is a grid of pixels (2 directions) times 3 colour channels (a 3rd direction). A batch of many images adds a 4th direction. The parent note's (32, 3, 224, 224) is exactly this: 32 images, 3 channels, 224 rows of pixels, 224 columns. A tensor is the only data structure that can hold all four directions at once.

2. The word "dimension" (and the trap inside it)

The word dimension gets used two different ways, and mixing them up is the #1 beginner mistake.

WHY numbering matters. When the parent note writes torch.sum(x, dim=0), it means "collapse the down-the-rows direction — add each column's entries together." Read the figure: the amber arrow is dim=0, the cyan arrow is dim=1. If you cannot instantly point to the arrow a dim= number names, no reduction operation will make sense.

3. Shape — the address system of a tensor

Think of the shape as a building address. To reach one number inside the box you give one index per axis — like [floor][room][shelf]. Every valid index list points at exactly one number.

WHY it is a product and not a sum. Picture the matrix in figure 2: it has 2 rows and 3 columns, and the numbers sit at every crossing of a row with a column. Each of the 2 rows meets each of the 3 columns, giving $2\times 3=6$ crossings. Add a third axis of size 4 and every one of those 6 crossings is copied 4 times: $6\times 4=24$. Each new axis multiplies the count — that is precisely what $\prod$ records.

4. dtype — what kind of number sits in each slot

WHY uniform. A tensor stores its numbers in one solid block of memory so the GPU can chew through them at full speed (see GPU acceleration). Mixed types would break that even, predictable layout. This "one flavour, tightly packed" rule is what makes tensors fast, and it is inherited directly from NumPy arrays.

5. The three operation families you must recognise

Every operation in the parent note is one of three shapes of behaviour. Learn to classify an operation and its effect on shape becomes obvious.

WHY three and only three. These three cover the entire forward pass of a network: a layer multiplies by weights (family 3), adds a bias and applies an activation (family 1), and the loss averages over the batch (family 2). Master these three pictures and the parent note reads like a story.

6. The dot product — the atom of matrix multiplication

Before matrix multiplication makes sense you need its single building block.

WHY this exact recipe, and not something else? A neural neuron's job is: "weigh each input by how much I care about it, then total the evidence." Multiplying input $u_k$ by weight $v_k$ = weighing; summing = totalling the evidence. That is literally $\mathbf{u}\cdot \mathbf{v}$. This is why the dot product, and not (say) adding or comparing, is the atom of every layer — it is the mathematical shape of "weighted evidence."

Now matrix multiplication is just a whole grid of dot products: entry $C[i,j]$ is the dot product of row $i$ of the first matrix with column $j$ of the second.

$C[i,j]={\sum }_{k}A[i,k]B[k,j]$

WHY the inner sizes must match. A dot product needs two vectors of equal length. Row $i$ of $A$ has as many entries as $A$ has columns; column $j$ of $B$ has as many entries as $B$ has rows. For the dot product to exist, columns of A must equal rows of B — that is the famous rule $(2,\underline{3})\times (\underline{3},4)\to (2,4)$: the underlined inner sizes must agree and then vanish, leaving the two outer sizes as the result's shape.

7. Sharing memory vs. copying — the invisible gotcha

WHY it exists. Copying millions of numbers is slow. So torch.from_numpy(...) and .view(...) hand you a new label on the old block instead of a fresh copy — instant and free. The price: an accidental edit leaks across. When you truly want an independent copy, use .clone(). Keep this in the back of your mind; it explains the parent note's .clone() warnings and the view vs reshape distinction (a view insists on sharing; reshape copies if it must).

Prerequisite map

This map feeds directly into backpropagation, building models, batch normalization and convolutional layers — all of which are just choreographed sequences of the three operation families above, built on neural-network fundamentals.

Equipment checklist

Cover the right side and see if you can answer each before revealing.