Broadcasting — rules, why it works, gotchas: Exercises

Recall The rule, restated in one line

Right-align the shapes. Each aligned axis must be equal or contain a 1. Missing leading entries count as 1. Result size = the bigger number.

Before we start, one picture of the mechanic every problem uses — the right-alignment table:

Figure — Broadcasting — rules, why it works, gotchas

What the figure shows (in words): two shapes are written on two rows, A = (5, 1, 4) on top and B = (3, 4) below, pushed so their rightmost numbers sit in the same vertical line (the red bar on the right marks that alignment edge). Because B is shorter, its missing leading slot is filled with a red 1* (the "padded 1"). Then you scan the three aligned axes right-to-left: 4 vs 4 → 4, 1 vs 3 → 3, 5 vs 1 → 5, giving the result row (5, 3, 4). The rule underneath each aligned axis is always the same: equal or contains a 1 → take the bigger number. Read it like this: write both shapes so their last number lines up, pad the shorter one on the left with imaginary 1s, then walk axis by axis from the right.

Level 1 — Recognition

Goal: read two shapes and state the result (or ERROR). No code needed.

L1.1

a.shape == (4,), b.shape == (4,). What is (a + b).shape?

Recall Solution

Right-align: (4) over (4). One aligned axis, both equal → result 4. Answer: (4,). The everyday same-length case; no stretching happens.

L1.2

A.shape == (3, 4), s is the Python scalar 7. What is (A + s).shape?

Recall Solution

A scalar has shape () — no dimensions. Pad on the left with 1s to match length 2 → (1, 1). Right-align against (3, 4): axis 1 → 4 vs 1 → stretch to 4; axis 0 → 3 vs 1 → stretch to 3. Answer: (3, 4). Adding a number to a grid is the simplest broadcast.

L1.3

A.shape == (3, 4), b.shape == (4,). What is (A + b).shape?

Recall Solution

Pad b to (1, 4). Right-align:

axis 1: 4 vs 4 → equal → 4
axis 0: 3 vs 1 → stretch → 3

Answer: (3, 4). b aligns with the trailing (length-4) axis, so it acts as a per-column vector added to every row.

L1.4

A.shape == (3, 4), b.shape == (3,). What is (A + b).shape?

Recall Solution

Pad b to (1, 3). Right-align:

axis 1: 4 vs 3 → not equal, neither is 1 → conflict.

Answer: ERROR. The bare (3,) chased the last axis (length 4), not the rows. This is the single most common broadcasting bug — see the mistake below.

Level 2 — Application

Goal: choose the reshape that makes the operation you want.

L2.1

M.shape == (3, 4). You have w = np.array([1., 2., 3.]) (shape (3,)) and want to multiply row i of M by w[i]. Write the expression and give its shape.

Recall Solution

The rows live on axis 0. To make w align there, give it a trailing size-1 axis: w[:, None] → shape (3, 1).

M * w[:, None]        # (3,4) * (3,1)

Right-align (3, 4) over (3, 1): axis 1 → 4 vs 1 → 4; axis 0 → 3 vs 3 → 3. Answer: (3, 4). Each row i gets the single value w[i] reused across all 4 columns.

L2.2

M.shape == (3, 4). Now w = np.array([10., 20., 30., 40.]) (shape (4,)) and you want to multiply column j by w[j]. Write the expression and give its shape.

Recall Solution

The columns live on axis 1, which is the trailing axis. A bare (4,) already aligns there — no reshape needed.

M * w                 # (3,4) * (4,)  -> (4,) padded to (1,4)

axis 1 → 4 vs 4 → 4; axis 0 → 3 vs 1 → 3. Answer: (3, 4). Row-scaling needs a reshape, column-scaling does not — because "trailing axis is free."

L2.3

X.shape == (100, 3) (100 samples, 3 features). Subtract each column's mean. Write it and confirm the shapes at each step.

Recall Solution

mu = X.mean(axis=0)   # reduce over axis 0 -> shape (3,)
Xc = X - mu           # (100,3) - (3,)

mu has shape (3,) (see Reductions and the axis argument — reducing axis 0 removes it, leaving the 3 features). Pad to (1, 3), right-align against (100, 3):

axis 1: 3 vs 3 → 3
axis 0: 100 vs 1 → 100

Answer: Xc.shape == (100, 3), and every row has the same mu subtracted — exactly per-column centering.

L2.4

Build the $3 \times 4$ table where table[i, j] = i + j using two 1-D ranges. Give the code and result shape.

Recall Solution

col = np.arange(3).reshape(3, 1)   # (3,1): [[0],[1],[2]]
row = np.arange(4).reshape(1, 4)   # (1,4): [[0,1,2,3]]
table = col + row                  # (3,1)+(1,4)

Right-align (3, 1) over (1, 4): axis 1 → 1 vs 4 → 4; axis 0 → 3 vs 1 → 3. Both size-1 axes stretch. Answer: (3, 4) with table[i,j] = i + j. This "column plus row" pattern is the canonical grid builder — see Reshaping and newaxis (None) indexing.

Level 3 — Analysis

Goal: predict result shapes for tricky multi-axis cases, and catch failures.

L3.1

a.shape == (3,), b.shape == (3, 1). What is (a + b).shape?

Recall Solution

Pad a to (1, 3). Right-align against (3, 1):

axis 1: 3 vs 1 → 3
axis 0: 1 vs 3 → 3

Answer: (3, 3). You accidentally built a matrix! result[i, j] = a[j] + b[i, 0]. "Both have 3 elements" does not mean the result has 3 elements.

L3.2

A.shape == (5, 1, 4), B.shape == (3, 4). What is (A + B).shape?

Recall Solution

Pad B (length 2) to (1, 3, 4). Right-align against (5, 1, 4):

axis 2: 4 vs 4 → 4
axis 1: 1 vs 3 → 3
axis 0: 5 vs 1 → 5

Answer: (5, 3, 4). Every aligned axis was either equal or contained a 1, so all pass.

L3.3

A.shape == (8, 1, 6, 1), B.shape == (7, 1, 5). What is (A + B).shape?

Recall Solution

Pad B (length 3) to (1, 7, 1, 5). Right-align against (8, 1, 6, 1):

axis 3: 1 vs 5 → 5
axis 2: 6 vs 1 → 6
axis 1: 1 vs 7 → 7
axis 0: 8 vs 1 → 8

Answer: (8, 7, 6, 5). Each axis picks the bigger number; every axis had a 1 to give way.

L3.4

A.shape == (2, 3), b.shape == (2,). What is (A + b).shape?

Recall Solution

Pad b to (1, 2). Right-align against (2, 3):

axis 1: 3 vs 2 → not equal, neither is 1 → conflict.

Answer: ERROR. Even though the leading numbers "match" (2 and 2), broadcasting never got to compare them — the trailing axis failed first. Fix: if you meant to act on axis 0, use b[:, None] → (2, 1), giving (2, 3).

L3.5 — Empty (zero-length) dimension

A.shape == (0, 4) (an array with no rows — a real thing when you filter everything out) and b.shape == (4,). What is (A + b).shape? And what about A.shape == (0, 4) with b.shape == (3,)?

Recall Solution

A size-0 axis is not special to the compatibility rule — it still only passes when the other side is equal (also 0) or 1. It does not stretch a 1 into a 0-vs-something clash away; the same "equal or 1" test applies.

(0, 4) + (4,): pad b to (1, 4). axis 1: 4 vs 4 → 4. axis 0: 0 vs 1 → the size-1 side stretches to 0. Answer: (0, 4) — a valid empty result (0 rows, each of length 4). No error: an empty array plus anything broadcastable is just empty.
(0, 4) + (3,): pad b to (1, 3). axis 1: 4 vs 3 → neither equal nor 1 → ERROR. The emptiness of axis 0 is irrelevant; the trailing mismatch still fails.

Take-away: treat 0 like any other size — the only wildcard is 1. A size-1 axis broadcasting to 0 gives 0 (the empty case wins in that axis), but 0 vs 2 (or any 0 vs n≥2) is an error just like 3 vs 2.

Level 4 — Synthesis

Goal: combine broadcasting with reshaping to build real computations.

L4.1 — Pairwise distance matrix

Given P.shape == (5, 2) (5 points in 2-D), build D.shape == (5, 5) where D[i, j] is the squared distance between point i and point j. Write it with broadcasting only (no loops), and justify each shape.

Recall Solution

We want, for every ordered pair (i, j), the difference P[i] - P[j]. Broadcasting builds all pairs if we put i on a new axis and j on another:

diff = P[:, None, :] - P[None, :, :]   # (5,1,2) - (1,5,2) -> (5,5,2)
D = (diff ** 2).sum(axis=2)            # sum over the 2 coords -> (5,5)

Shapes step by step:

P[:, None, :] → (5, 1, 2): point index on axis 0, a fresh size-1 axis 1.
P[None, :, :] → (1, 5, 2): point index on axis 1.
Subtract: right-align → axis2 2 vs 2=2, axis1 1 vs 5=5, axis0 5 vs 1=5 → (5, 5, 2). diff[i, j] is the coordinate difference of points i and j.
Square and .sum(axis=2) collapses the 2 coordinates (Reductions and the axis argument) → D.shape == (5, 5).

The diagonal D[i, i] = 0 (a point to itself). No explicit copies were made of the operands — the stretches are stride-0 views. This is vectorization replacing a double for loop.

L4.2 — Normalize each row to unit sum

M.shape == (4, 5), all entries positive. Make each row sum to 1. Give code and the reshape reasoning.

Recall Solution

s = M.sum(axis=1, keepdims=True)   # (4,1)   <- keepdims keeps the axis!
Mn = M / s                          # (4,5) / (4,1) -> (4,5)

keepdims=True makes the reduction return (4, 1) instead of (4,). Why we need that: a bare (4,) would right-align under the length-5 axis → 5 vs 4 ERROR. The (4, 1) aligns axis 1 5 vs 1→5 and axis 0 4 vs 4→4, so each row is divided by its own single sum. Answer: Mn.shape == (4, 5) with every row summing to 1.

Level 5 — Mastery

Goal: reason about memory, writability, and subtle equivalences.

L5.1 — Memory cost

a.shape == (1000,). You compute b = np.broadcast_to(a, (1000, 1000)). How many new elements of a's data are allocated for b? What is b.strides?

Recall Solution

Zero. broadcast_to returns a stride-trick view: the new axis (size 1000) gets stride 0, so all 1000 rows point back to the same 1000 numbers. No copy of a's buffer happens. b.strides == (0, 8) for a float64 array (8 bytes each): moving down a "row" advances 0 bytes (re-read the same row), moving across a column advances 8 bytes. See NumPy Arrays — shape, dtype, strides. The result of an arithmetic op like a + M still allocates its own output, but the broadcast operand is never duplicated.

L5.2 — Writability

Continuing L5.1, is b writable? What happens with b[0, 0] = 99?

Recall Solution

No — b is read-only. Because axis 0 has stride 0, indices b[0, 0], b[1, 0], ... all map to the same memory cell. Writing would be ambiguous (which "copy" did you mean?), so NumPy forbids it: b[0, 0] = 99 raises ValueError: assignment destination is read-only. Fix: b = np.broadcast_to(a, (1000, 1000)).copy() materializes an independent, writable array (now it does cost 1000×1000 elements).

L5.3 — Is broadcasting the same as `np.tile`?

Explain the result-value equivalence and the cost difference between A + w[:, None] (broadcasting) and A + np.tile(w[:, None], (1, A.shape[1])) (explicit replication), for A.shape == (3, 4), w.shape == (3,).

Recall Solution

Values: identical. Both add w[i] to every entry of row i. Cost: np.tile(w[:, None], (1, 4)) builds a real (3, 4) array in memory (np.tile and np.repeat (explicit replication)) — 12 stored numbers — then adds. Broadcasting keeps w[:, None] as a (3, 1) array with a stride-0 axis: it reads w[i] four times without ever storing the duplicates. Same answer, less memory, and no wasted allocation. Broadcasting is "tile, but conceptually only."

L5.4 — Predict and explain

x = np.arange(6).reshape(2, 3), y = np.arange(2).reshape(2, 1), z = np.arange(3). What is (x + y + z).shape and the top-left value (x + y + z)[0, 0]?

Recall Solution

Broadcast is associative in shape: combine left to right.

x + y: (2, 3) + (2, 1) → axis 1 3 vs 1=3, axis 0 2 vs 2=2 → (2, 3).
... + z: (2, 3) + (3,) → pad z to (1, 3) → axis 1 3 vs 3=3, axis 0 2 vs 1=2 → (2, 3).

Shape: (2, 3). Values: x[0,0]=0, y[0,0]=0, z[0]=0, so (x + y + z)[0, 0] = 0. (Sanity check the bottom-right: [1,2] = x[1,2]=5 + y[1,0]=1 + z[2]=2 = 8.)

Connections

NumPy Arrays — shape, dtype, strides
Vectorization vs Python loops
Reshaping and newaxis (None) indexing
Array Memory Layout — C vs Fortran order
Reductions and the axis argument
np.tile and np.repeat (explicit replication)