This is the hands-on drill room for the parent tensor note. The parent told you what the operations are. Here we grind through every kind of situation those operations can create — matching shapes, mismatching shapes, empty tensors, memory traps, gradient tracking — until nothing can surprise you.

Before line one: a tensor is just a box of numbers arranged in a grid. A grid can be a single line (1D), a rectangle (2D), a stack of rectangles (3D), and so on. The word shape is the list of sizes along each direction of that grid. If someone writes shape $(2, 3)$ , picture a rectangle with $2$ rows and $3$ columns. That mental picture is all you need — every example below leans on it.

The scenario matrix

Here is the full menu of situations tensor operations throw at you. Each later example is tagged with the cell(s) it covers, so by the end every row is checked off.

#	Case class	What makes it tricky	Example
A	Matching shapes	Plain element-wise add/multiply — the easy baseline	Ex 1
B	Broadcasting (rank mismatch)	One tensor has fewer dimensions than the other	Ex 2
C	Broadcasting (size-1 stretch)	A dimension of size $1$ gets stretched to match	Ex 2
D	Reduction along a chosen axis	`sum`/`mean` with `dim=` — which axis collapses?	Ex 3
E	`keepdim` + broadcast combo	Normalisation: reduce, then subtract back	Ex 4
F	Matrix multiply shape rule	Inner dims must match; outer dims survive	Ex 5
G	Reshape / view / flatten	Rearrange without touching the numbers	Ex 6
H	Degenerate / zero-size inputs	Empty tensor, scalar (0D), single element	Ex 7
I	Memory sharing trap	`from_numpy`, `view` share memory; `clone` breaks it	Ex 8
J	Real-world word problem	A full linear layer forward pass	Ex 9
K	Exam-style twist	Broadcast that looks legal but is illegal	Ex 10

Example 1 — Case A: matching shapes (the baseline)

Forecast: guess the shape and the numbers before reading on. (Both shapes match, so nothing gets stretched.)

Look at the figure: two rows of three boxes, lined up perfectly. Same shape means we just pair up box-by-box.

Add position by position. $[1 + 10, 2 + 20, 3 + 30] = [11, 22, 33]$ . Why this step? When shapes are identical PyTorch does no stretching — it walks both grids in lockstep. This is the simplest case and the yardstick for everything harder.
Multiply position by position. $[1 \cdot 10, 2 \cdot 20, 3 \cdot 30] = [10, 40, 90]$ . Why this step? Element-wise * is not matrix multiplication — it never mixes different positions. This is exactly how activation functions and scaling work.
Output shape stays $(3,)$ — element-wise ops never change shape.

Verify: total of the sum is $11 + 22 + 33 = 66$ , and indeed $\sum x + \sum y = 6 + 60 = 66$ . ✓

Example 2 — Cases B + C: broadcasting from a scalar and a row

Forecast: what shape does adding a lone number to a $(2, 3)$ grid produce? What about adding a single row?

The word broadcasting means: when shapes don't match, PyTorch copies the smaller thing to fill the gap, without actually using extra memory. The figure shows the row $r$ being copied downward onto every row of $M$ .

Scalar case. $100$ has shape $()$ (a 0-dimensional tensor — rank $0$ , no axes at all). By rule 1 it becomes $(1, 1)$ ; every axis is size $1$ , so it legally stretches to cover all $6$ boxes. Result rows: $[101, 102, 103]$ and $[104, 105, 106]$ , shape $(2, 3)$ . Why this step? A scalar broadcasts against anything — this is why x * 2 in the parent note "just works".
Row case — align ranks (rule 1). $M$ has rank $2$ (two axes), $r$ has rank $1$ (one axis). Prepend a $1$ to the smaller rank, so $r$ is treated as shape $(1, 3)$ . Why this step? Broadcasting compares shapes from the right. To compare fairly it first pads the missing left-axes of the lower-rank tensor with $1$ .
Legality check (rule 2), then stretch (rule 3). $(2, 3)$ vs $(1, 3)$ : rightmost axis $3 = 3$ (equal, legal); left axis $2$ vs $1$ — one is $1$ , legal, so the size- $1$ axis is stretched to $2$ . Row $r$ is copied onto both rows of $M$ . Result rows: $[11, 22, 33]$ and $[14, 25, 36]$ , shape $(2, 3)$ . Why this step? This is precisely the mechanism behind adding a bias vector to a batch — one bias row reused for every sample.

Verify: row-1 sum of (b) is $11 + 22 + 33 = 66$ ; from Example 1 that pattern is right. Row-2 sum is $14 + 25 + 36 = 75 = (4 + 5 + 6) + (10 + 20 + 30) = 15 + 60$ . ✓

Example 3 — Case D: reduction — which axis dies?

Forecast: dim=0 sums across... rows or columns? Guess the two answers.

Here is the trap everyone hits: dim=k means "the axis that disappears is axis $k$ ". Axis $0$ is the vertical (row) direction, axis $1$ is the horizontal (column) direction. The figure marks each with a coloured arrow — the arrow points along the axis that gets collapsed.

dim=0 collapses the vertical axis. We add down each column: $[1 + 4, 2 + 5, 3 + 6] = [5, 7, 9]$ . The $2$ rows vanish; $3$ columns remain, shape $(3,)$ . Why this step? Killing axis $0$ removes the row-count from the shape. Loss functions do this to average over a batch: "sum away the batch axis."
dim=1 collapses the horizontal axis. We add along each row: $[1 + 2 + 3, 4 + 5 + 6] = [6, 15]$ . The $3$ columns vanish; $2$ rows remain, shape $(2,)$ . Why this step? Killing axis $1$ removes the feature-count. This is how you'd total each sample's features.

Verify: both reductions must reproduce the grand total. $5 + 7 + 9 = 21$ and $6 + 15 = 21$ , and the full grid sum is $1 + 2 + 3 + 4 + 5 + 6 = 21$ . All three agree. ✓

Example 4 — Case E: `keepdim` and centering (the normalisation move)

Forecast: the mean over dim=0 has shape $(3,)$ . Can you subtract a $(3,)$ tensor from a $(2, 3)$ tensor directly? What does keepdim change?

This is the pattern behind batch normalization: reduce along the batch axis, then subtract the result back off every sample.

Column means, dim=0. Column means are $\frac{1 + 4}{2} = 2.5$ , $\frac{2 + 5}{2} = 3.5$ , $\frac{3 + 6}{2} = 4.5$ . Without keepdim this is shape $(3,)$ . Why this step? We want each feature's average, and features run across columns, so we collapse the row (batch) axis.
Use keepdim=True so the mean has shape $(1, 3)$ instead of $(3,)$ . Why this step? We are about to compute $M - mean$ . With shape $(1, 3)$ the mean broadcasts cleanly against $(2, 3)$ (Case C, size-1 stretch). Shape $(3,)$ also broadcasts here by luck, but the moment you reduce over dim=1 you'd get $(2,)$ , which would not align — keepdim makes the pattern robust every time.
Subtract (broadcasts). Row 1: $[1 - 2.5, 2 - 3.5, 3 - 4.5] = [- 1.5, - 1.5, - 1.5]$ . Row 2: $[4 - 2.5, 5 - 3.5, 6 - 4.5] = [1.5, 1.5, 1.5]$ . Shape $(2, 3)$ . Why this step? After centering, each column's new mean is $0$ — exactly what a normalisation layer wants.

Verify: new column means must be $0$ : column 1 is $\frac{- 1.5 + 1.5}{2} = 0$ . ✓ Sum of the whole centred grid is $0$ (means removed all mass): $- 1.5 \cdot 3 + 1.5 \cdot 3 = 0$ . ✓

Example 5 — Case F: matrix multiplication shape rule

Forecast: which two numbers must match, and which two survive into the output shape?

The rule: for $A @ B$ , the inner dimensions must be equal, and the outer dimensions form the output. Write the shapes side by side: $(outer 2, inner 3) \times (inner 3, outer 4) \to (2, 4)$ . The figure shows the inner $3$ s "touching and cancelling", leaving $(2, 4)$ .

Check the inner dims. $A$ 's last axis is $3$ ; $B$ 's first axis is $3$ . They match, so the multiply is legal. Why this step? Each output entry is a dot product over that shared inner axis — if the lengths differed there'd be nothing to pair up.
Output shape = outer dims $= (2, 4)$ . Why this step? Row $i$ of $A$ dotted with column $j$ of $B$ gives entry $(i, j)$ ; there are $2$ rows and $4$ columns, hence $2 \times 4$ .
Compute entry $(0, 0)$ . Row $0$ of $A$ is $[1, 2, 3]$ ; column $0$ of $B$ is $[1, 1, 1]$ . Dot product $= 1 \cdot 1 + 2 \cdot 1 + 3 \cdot 1 = 6$ . Why this step? Since $B$ is all ones, every entry in row $0$ equals the row sum of $A$ 's row $0$ , which is $6$ .

Verify: because $B$ is all ones, row $0$ of the product should be $[6, 6, 6, 6]$ and row $1$ should be $[15, 15, 15, 15]$ (row sums of $A$ ). Row-1 sum check: $4 + 5 + 6 = 15$ . ✓

Example 6 — Case G: reshape / view / flatten

Forecast: guess the flattened feature count and the value $- 1$ resolves to.

Total elements (this is numel) $= 10 \times 3 \times 28 \times 28$ . Why this step? Reshaping can never create or destroy numbers, so the product of the new shape must equal this. $28 \times 28 = 784$ , times $3 = 2352$ , times $10 = 23520$ .
Flatten per image. Keep the batch axis, merge the rest: x.flatten(start_dim=1) gives shape $(10, 3 \cdot 28 \cdot 28) = (10, 2352)$ . Why this step? A dense layer wants each sample as one long feature vector, but we must not blend different images together — so we flatten from axis $1$ onward, protecting axis $0$ .
The -1 placeholder. view(-1, 4) on $2 \times 3 \times 4 = 24$ elements: the known axis is $4$ , so $- 1$ becomes $24/4 = 6$ , giving $(6, 4)$ . Why this step? $- 1$ means "you do the arithmetic for me." PyTorch fills it so the total ( $24$ ) is preserved.

Verify: each new shape's product equals the original element count. $(10, 2352) \to 23520$ . ✓ $(6, 4) \to 24 = 2 \cdot 3 \cdot 4$ . ✓

Example 7 — Case H: degenerate & zero-size inputs

Forecast: what's the sum of no numbers? What's the rank of a lone $7$ ?

Empty-tensor sum. Shape $(0,)$ holds zero elements (numel $= 0$ ). sum returns the additive identity, $0.0$ . Why this step? The sum of an empty set is defined as $0$ so that splitting-and-recombining sums stays consistent (an empty sub-batch contributes nothing). Never assume it errors — it quietly returns $0$ .
Scalar tensor. torch.tensor(7.0) has shape $()$ (empty tuple) and rank $0$ — a 0D tensor. Its numel is $1$ (one number lives inside, even with no axes). Why this step? A single value still lives in a tensor, but with no axes. This is what a loss value looks like: one number, rank $0$ . .item() pulls out the plain Python float $7.0$ .
Squeeze a $(1, 1, 1)$ tensor. .squeeze() removes every axis of size $1$ , so $(1, 1, 1) \to ()$ , a scalar again holding $5.0$ , rank $0$ . Why this step? squeeze deletes only length- $1$ axes because those carry no real spread of data; here all three qualify, collapsing to a scalar.

Verify: empty sum $= 0.0$ ; scalar numel $= 1$ ; squeezed value $= 5.0$ with rank $0$ . All three are checked below. ✓

Forecast: does editing a leak into b? Into c?

The word view means a second window onto the same numbers — no copy is made. clone makes a genuine separate copy.

b is a view. It points at the same storage as a. After a[0] = 99, reading b[0] gives $99.0$ . Why this step? view never allocates new memory; it only reinterprets the existing block. Editing through one name is visible through the other — this is the #1 silent bug when people expect a copy.
c is a clone. It owns its own storage, frozen at $[1, 2, 3]$ the instant it was made. c[0] is still $1.0$ . Why this step? clone breaks the link deliberately — use it whenever you need to keep an untouched snapshot.

Verify: b[0] == 99.0 (shared) and c[0] == 1.0 (independent). ✓

import numpy as np, torch arr = np.array([1., 2., 3.]) t = torch.from_numpy(arr) # shares memory with arr arr[0] = 99.0 # edit the NumPy side

now t[0] is ALSO 99.0

safe = torch.from_numpy(arr).clone() # independent snapshot

> **Walk it through.** `torch.from_numpy(arr)` does **not** copy — `t` and `arr` point at one shared block of memory (this is the same link discussed in [[2.4.01-numpy-fundamentals|NumPy fundamentals]]). So editing `arr[0]` mutates `t[0]` underneath you: `t[0]` becomes $99.0$ even though you never touched `t`. The reverse is true too — `t.numpy()` shares memory back. **Fix:** append `.clone()` to break the link the instant you need an independent tensor, as with `safe` above.

> [!mistake] The classic leak
> `t.numpy()` and `torch.from_numpy(arr)` **share memory**. Modify the NumPy side and your tensor mutates underneath you — a bug that hides for hours because the two objects look independent. If you need a truly independent tensor, always break the link explicitly: `torch.from_numpy(arr).clone()`.

---

## Example 9 — Case J: real-world word problem (a full linear layer)

> [!example] Statement
> An MNIST batch has $32$ images, each flattened to $784$ features: input $x$ has shape $(32, 784)$. A linear layer maps $784$ inputs to $128$ outputs, using weight $W$ of shape $(784, 128)$ and bias $b$ of shape $(128,)$. Compute the output shape of $y = xW + b$, and confirm the bias broadcasts correctly. This is the forward pass of [[3.3.02-building-modelspytorch|a PyTorch model layer]] feeding into [[3.2.03-backpropagation|backpropagation]].

**Forecast:** what is $y$'s shape, and how does a $(128,)$ bias reach $32$ different samples?

1. **Matmul shape.** $(32, 784) @ (784, 128)$: inner dims $784=784$ match; outer dims give $(32, 128)$ (Case F, and recall `@` = matmul).
   *Why this step?* Each of the $32$ samples (a row of $x$) is turned into a $128$-dimensional output vector — one row of $y$ per sample.
2. **Bias broadcast.** $b$ has shape $(128,)$, i.e. rank $1$; padded to $(1,128)$, then the size-$1$ axis stretches to $32$ (Cases B+C, and the legality check of Example 2 passes: $128=128$, $32$ vs $1$).
   *Why this step?* Every sample receives the **same** bias vector, added to its $128$ outputs. This is why the layer's convention is $xW$ (not $Wx$): it keeps the batch axis on the left so broadcasting the bias is automatic.
3. **Result** $y$ has shape $(32, 128)$: $32$ samples, each now a $128$-feature vector, ready for the next layer / activation.
   *Why this step?* This exact shape is what feeds [[3.5.02-convolutional-layers|later layers]] or a softmax head; every network is a chain of these. On a GPU this runs in parallel (see [[4.2.01-gpu-acceleration|GPU acceleration]]).

**Verify:** dimension bookkeeping: $(32,784)\times(784,128)\to(32,128)$, and $(32,128)+(1,128)\to(32,128)$. The inner $784$ cancels; batch $32$ and output $128$ survive. ✓

---

## Example 10 — Case K: the exam-style twist (illegal broadcast)

> [!example] Statement
> You try `A + B` with $A$ shape $(3,)$ and $B$ shape $(4,)$. Does it broadcast? Now $A$ shape $(2,3)$ and $B$ shape $(2,)$ — legal or not? Fix the second one so it works.

**Forecast:** decide *legal* or *error* for each before reading.

Broadcasting compares shapes **right-aligned** (the rules boxed in Example 2). Two axes are compatible only if they're **equal** or **one of them is $1$**. No other case is allowed.

1. **$(3,)$ vs $(4,)$.** Rightmost axes: $3$ vs $4$. Not equal, neither is $1$. **Illegal** — PyTorch raises a size-mismatch error.
   *Why this step?* There is no consistent way to stretch a length-$3$ line to length $4$; broadcasting refuses rather than guess.
2. **$(2,3)$ vs $(2,)$.** Right-align: compare $3$ (from $A$) with $2$ (from $B$). $3 \ne 2$ and neither is $1$. **Illegal** — this is the trap: the $2$s look like they "match" but they're in *different* positions after right-alignment.
   *Why this step?* Beginners see a $2$ in both shapes and assume it's fine. But alignment is from the right, so $B$'s $2$ lines up with $A$'s $3$, not $A$'s $2$.
3. **Fix it.** Make $B$ shape $(2, 1)$ (via `B.unsqueeze(1)`). Now right-align: $1$ vs $3$ → stretch (size-1 rule); $2$ vs $2$ → match. **Legal**, output $(2,3)$.
   *Why this step?* Reshaping $B$ to $(2,1)$ puts its $2$ on the correct (row) axis so each row of $A$ gets its own scalar from $B$.

**Verify:** compatibility of the fixed pair: axis-by-axis $(2,3)$ vs $(2,1)$ → $[2{=}2,\ 3\text{ vs }1{\to}\text{stretch}]$ → $(2,3)$. Legal. ✓

---

## Recall

> [!recall]- What is the "rank" of a tensor, and what is `numel`?
> Rank = number of axes (length of the shape tuple); `numel` = total number of elements (product of the shape). Shape $(2,3)$ has rank $2$ and `numel` $6$.

> [!recall]- Which axis vanishes with `sum(dim=1)` on a $(2,3)$ tensor, and what shape remains?
> Axis $1$ (the columns) collapses; shape $(2,)$ remains — one total per row.

> [!recall]- Why does the linear layer use $xW$ with $W$ of shape (in, out) rather than $Wx$?
> So the batch axis stays leftmost: $(\text{batch},\text{in})@(\text{in},\text{out})=(\text{batch},\text{out})$, letting the bias broadcast to every sample automatically.

> [!recall]- $A$ shape $(2,3)$, $B$ shape $(2,)$ — legal to add? Fix?
> Illegal (right-aligned $3$ vs $2$). Reshape $B$ to $(2,1)$; then it broadcasts to $(2,3)$.

> [!recall]- `b = a.view(...)` then `a[0]=99`. What is `b[0]`?
> $99$ — a view shares memory. Use `a.clone()` to get an independent copy.

> [!recall]- What does the infix `@` do in PyTorch, and how is it different from `*`?
> `@` is matrix multiplication (same as `torch.matmul`); `*` is element-wise multiplication. They give different shapes and different numbers.

> [!mnemonic] Broadcast in one line
> **Right-align, then each axis must be Equal or contains a 1 — Else it Errors.** (R-E-1-E.)

Cloze drill:
- The output shape of $(m,k)@(k,n)$ is ==$(m,n)$== — inner $k$ cancels.
- `keepdim=True` keeps the reduced axis as size ==$1$== so the result still broadcasts.
- The sum of an empty tensor is ==$0.0$==, the additive identity.
- The ==rank== of a tensor is its number of axes; `numel` is its total element count.
- The infix `@` means ==matrix multiplication== (equivalent to `torch.matmul`).