PyTorch tensors and operations: Visual walkthrough

We are deepening the parent PyTorch Tensors and Operations note. Everything below earns its symbols before using them.

Step 1 — One neuron is a weighted sum

WHAT. Start with the smallest possible piece: a single output neuron looking at a single sample. The sample is a list of numbers $x_{0}, x_{1}, x_{2}$ (the features — say three measurements of one flower). The neuron has one weight per feature, $w_{0}, w_{1}, w_{2}$ , and one extra number $b$ called the bias.

WHY. A neuron's whole job is to ask "how much does each feature matter, and what's my baseline?" A weight is the "how much it matters" knob; the bias is the "baseline" knob. The simplest way to combine "importance × value" over all features is to multiply each pair and add them up — a weighted sum. Nothing fancier is needed yet.

PICTURE. In the figure, each input dot (black) connects to the output dot (red) by an arrow carrying its weight. The neuron adds them all, then adds $b$ .

Read it left to right: multiply each feature by its weight, add the three products, then add the baseline $b$ . The output $y$ is one number.

Step 2 — The weighted sum is a dot product

WHAT. Collect the features into a row $x = [x_{0}, x_{1}, x_{2}]$ and the weights into a column $w = [w_{0}, w_{1}, w_{2}]^{⊤}$ (the $⊤$ , "transpose", just means "stand it up vertically"). The sum-of-products from Step 1 has a name: the dot product $x \cdot w$ .

WHY this tool? We want one operation that means "line the two lists up, multiply matching entries, add the results." That is precisely the definition of the dot product — so instead of writing three separate terms, we write one symbol. This is not a new idea, it is Step 1 folded into shorthand. The reason we care: this shorthand is what scales to thousands of features without the formula getting longer.

PICTURE. The row slides across the standing-up column; each matched pair (same color position) multiplies, and the products drop into a single sum.

Here $k$ is just a counter that walks through the feature positions $0, 1, 2$ . The $\sum$ ("sigma") symbol means "add up everything to my right as $k$ runs through those values." Same answer as Step 1 — new clothes.

Step 3 — Many neurons at once become a matrix

WHAT. One neuron gives one number. A layer has many neurons — say $2$ of them — each with its own weight column. Stack those columns side by side and you get a rectangular grid of numbers: a matrix $W$ . Its shape is $(in, out) = (3, 2)$ : three rows (one per input feature), two columns (one per output neuron).

WHY. Each output neuron reuses the same input $x$ but with different weights. Rather than run the dot product twice by hand, we lay the two weight columns next to each other and let a single operation — matrix multiplication — do all the dot products in one shot. Matrix multiplication is defined as "dot product of each row on the left against each column on the right", which is exactly the pattern we need.

PICTURE. The row $x$ (black) meets each column of $W$ (the red column is neuron 1's weights). Column 0 makes output $y_{0}$ , column 1 makes output $y_{1}$ .

The new index $j$ picks the output neuron ( $j = 0$ or $1$ ); $k$ still walks the features. $W_{k j}$ is the weight from feature $k$ into neuron $j$ . Notice the shapes: the inner numbers $(1, \underline{3}) (\underline{3}, 2)$ must match — that shared $3$ is what gets summed over and then vanishes, leaving the outer numbers $(1, 2)$ .

Step 4 — A whole batch: stack samples into rows

WHAT. Real training feeds many samples at once. Stack $32$ samples, each a row of $3$ features, into a matrix $X$ of shape $(32, 3)$ . The layer formula does not change — we just multiply the whole stack by $W$ .

WHY. GPUs are fastest when they do the same operation on lots of data at once (see GPU acceleration). Processing 32 samples in one matrix multiply is far faster than 32 separate ones. Because matrix multiplication treats each row of the left matrix independently, row $i$ of the output depends only on row $i$ of the input — the samples never mix. That independence is exactly why batching is safe.

PICTURE. Each black row of $X$ slides across all of $W$ and lands as a matching row of the red output $Y$ . Row 5 in → row 5 out.

Now $i$ is the sample index ( $0 \dots 31$ ), $j$ the neuron index, $k$ the feature index being summed away. The inner dimension $3$ matches and disappears; the output is $(32, 2)$ — one row per sample, one column per neuron. This is the parent note's $y = xW + b$ , now fully unpacked.

Step 5 — The bias must broadcast (edge case: shape mismatch)

WHAT. The output of the multiply is $(32, 2)$ , but the bias $b$ is only $(2,)$ — one baseline per neuron. Adding a $(2,)$ vector to a $(32, 2)$ matrix looks illegal at first: the shapes don't match. Broadcasting fixes this by copying the same bias row down all 32 rows.

WHY. Every sample should get the same baseline for a given neuron — neuron $0$ 's bias is $b_{0}$ whether it's sample 3 or sample 30. Copying $b$ into every row expresses exactly that. PyTorch's broadcasting rules (from the parent note) do this automatically: prepend a $1$ to make $b$ into shape $(1, 2)$ , then stretch that lone dimension from $1$ up to $32$ .

PICTURE. The single red bias row is stamped onto every row of the grid — same values, 32 times.

The one-picture summary

Every arrow you saw compresses into one diagram: features → dot products → matrix multiply over a batch → broadcast the bias → predictions. Shapes are annotated at every stage so you can trace a number from input to output.

Recall Feynman retelling — say it back in plain words

A neuron takes a list of measurements and asks "how much does each one matter?" It multiplies each measurement by an importance number, adds them all up, and tacks on a baseline. That add-up-the-products move is a dot product. A layer has several neurons, so we glue their importance-lists together into a grid (a matrix) and let matrix multiplication run all the dot products at once. To train fast we don't feed one sample — we stack a whole batch of them as rows and multiply the whole stack, because each row is handled on its own and they never interfere. The only wrinkle: the baseline vector is short, so PyTorch copies it onto every row (broadcasting). If the shared inner dimension doesn't match, or the bias is the wrong length, PyTorch stops and complains — and now you know exactly which number to fix.

Recall Quick checks

What is the shape of $XW$ when $X$ is $(32, 784)$ and $W$ is $(784, 128)$ ? ::: $(32, 128)$ — inner $784$ matches and vanishes. Which dimension of $W$ is summed over in a matmul? ::: The inner one (input features), leaving batch × outputs. Why can bias $(O,)$ be added to output $(N, O)$ ? ::: Broadcasting stretches it to $(1, O)$ then copies down to all $N$ rows. What output shape do you get from an empty batch $(0, 3)$ through $W$ of $(3, 2)$ ? ::: $(0, 2)$ — legal, zero rows.

Where this goes next

Building models in PyTorch wraps this exact formula inside nn.Linear.
Backpropagation runs this multiply in reverse to compute gradients.
Batch normalization and convolutional layers reuse the same broadcast/reduction machinery.