NumPy broadcasting rules: Worked examples — AI-ML deep dive

You have read the parent note and know the three rules. This page is a drill hall. We line up every kind of shape pairing NumPy can throw at you, then walk each one to the ground so that no shape combination can ever surprise you again.

Before we start, one picture and one reminder to fix everything we will use on every single example.

Formula The three rules, restated here so this page stands alone

Compare two shapes axis by axis, starting from the rightmost. A pair of sizes (a, b) is legal if at least one of these holds:

Rule 1 — Equality: a == b. Neither stretches; the size passes through unchanged.
Rule 2 — Singleton: one of them is 1. That 1 stretches (is virtually copied) up to the other size.
Rule 3 — Missing: one array has no axis in this slot because it has fewer axes. The missing axis is inserted as size 1 — after which it is just an ordinary singleton, so Rule 2 then decides it.

That last sentence is what we mean when we write "Rule 3 then Rule 2": Rule 3 creates the padded 1, and Rule 2 stretches it. Rule 3 never acts alone — it always hands off to Rule 2.

If every axis pair passes one of these, broadcasting succeeds and the output size on each axis is the larger of the two. If any pair passes none — two different sizes with neither equal to 1 — you get ValueError.

Look at the figure: two shape tuples are printed as train carriages, pushed flush against a right-hand wall. The shorter train gets empty carriages (each holding a 1) welded onto its left — that welding is Rule 3. The three coloured verdicts — magenta "equal" (Rule 1), violet "one is 1 → stretch" (Rule 2), orange "clash → ValueError" — are the only three things that can happen to a carriage pair. That is the entire game.

The scenario matrix

Every broadcasting question you will ever meet is one row of this table. The examples below are numbered to hit each cell.

#	Case class	What makes it special	Covered by
A	Scalar vs anything	one array has rank 0, shape `()`	Ex 1
B	Equal shapes	no stretching at all — pure element-wise	Ex 2
C	Singleton in trailing axis	one array has a `1` on the right	Ex 3
D	Both sides stretch (row × col)	`(1,n)` meets `(m,1)` → grid	Ex 4
E	Rank mismatch (missing axes)	short array padded on the left	Ex 5
F	Degenerate: a `0`-length axis	an axis of size 0	Ex 6
G	Hard failure	trailing axes clash, no `1` to save it	Ex 7
H	Real-world word problem	feature normalization on a data table	Ex 8
I	Exam twist: transpose trap	right-alignment lands on the wrong axis	Ex 9
J	Rank > 2 (N-dimensional)	a 3-D batch meets a 2-D operand	Ex 10

We build each one from the shape tuple, never from intuition alone. Intuition lies; the right-alignment picture does not.

The examples

Worked example Example 1 — Cell A: scalar meets a matrix

A = np.array([[1, 2, 3],
              [4, 5, 6]])   # shape (2, 3)
b = 10                       # shape ()
result = A + b

Forecast: What shape comes out, and what is result[1, 2]?

Write the shapes. A is (2, 3), b is (). Why this step? Because broadcasting is decided only by shapes, never by the numbers inside. Always read the tuple first.
Right-align and pad. b has rank 0, A has rank 2, so b gets two padding 1s on the left → effective (1, 1). Why this step? Rule 3 inserts a size-1 axis for each axis b lacks (it lacks both), and padding goes on the left because NumPy compares from the right.
Compare per axis. Trailing: 3 vs 1 → 1 stretches to 3. Next: 2 vs 1 → 1 stretches to 2. Why this step? Each padded 1 is now an ordinary singleton, so Rule 2 stretches it to match its partner — this is the "Rule 3 then Rule 2" hand-off in action.
Result shape = (max(2,1), max(3,1)) = (2, 3). Why this step? By the output rule, each axis takes the larger of the two sizes; since every b-axis is 1, A sets the whole shape. Every element gets 10 added, so result[1, 2] = 6 + 10 = 16.

Verify: A had a 6 at [1,2]; add 10 → 16. Shape unchanged because a scalar cannot enlarge anything. ✓

Worked example Example 2 — Cell B: equal shapes, nothing stretches

P = np.array([[2, 4],
              [6, 8]])      # shape (2, 2)
Q = np.array([[1, 1],
              [2, 2]])      # shape (2, 2)
result = P * Q

Forecast: Guess result before reading on.

Shapes: both (2, 2). Why this step? Equal ranks, so Rule 3 never fires — every carriage already has a partner, no padding needed.
Compare: 2 vs 2 (equal), 2 vs 2 (equal) → Rule 1 on both axes. Why this step? When both sizes match, Rule 1 passes the size through untouched, so nothing is stretched; this is ordinary element-wise multiplication.
Multiply element by element: Why this step? With no stretching anywhere, each output cell is simply the product of the two same-position inputs — the plain definition of an element-wise operation. $[2648] ⊙ [1212] = [212416]$ (the $⊙$ symbol just means "multiply matching cells", not matrix multiplication).

Verify: Position [1,0]: $6 \times 2 = 12$ . Position [1,1]: $8 \times 2 = 16$ . ✓ No broadcasting happened — this is your control case, the "boring" baseline everything else is measured against.

Worked example Example 3 — Cell C: singleton on the trailing axis (row-wise centering)

X = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])   # shape (3, 3)
mean = np.array([[2],
                 [5],
                 [8]])      # shape (3, 1)
centered = X - mean

Forecast: Which axis stretches, and what is row 2 of the answer?

Shapes: X is (3, 3), mean is (3, 1). Same rank, no padding. Why this step? Both are rank 2, so Rule 3 stays quiet — we go straight to comparing sizes from the right.
Trailing axis: 3 vs 1 → Rule 2, the 1 stretches to 3. Why this step? mean is one column; Rule 2 virtually copies that single column across all 3 columns of X. No memory is duplicated.
Next axis: 3 vs 3 (equal) → Rule 1, stays 3. Result shape (max(3,3), max(3,1)) = (3, 3). Why this step? The row axis already matches, so Rule 1 leaves it alone; the output takes the larger size on each axis, giving (3, 3).
Subtract: each row of X loses its own row-mean. Row 2 (index 2) is [7,8,9] - 8 = [-1, 0, 1]. Why this step? After the stretch, mean's value 8 sits in all three columns of row 2, so every entry of that row has 8 subtracted.

Look at the figure: the single violet column of mean fans out into three faded copies (the virtual stretch) so it can meet every cell of the orange X grid.

Verify: Row 0: [1,2,3]-2 = [-1,0,1]. Row 1: [4,5,6]-5=[-1,0,1]. Row 2: [7,8,9]-8=[-1,0,1]. Every row centers to [-1,0,1] — a clean sanity check that our subtraction hit the right axis. ✓

Worked example Example 4 — Cell D: both sides stretch (the outer-product grid)

a = np.array([1, 2, 3])        # shape (3,)
b = np.array([10, 20, 30, 40]) # shape (4,)
grid = a[:, np.newaxis] * b    # a becomes (3, 1)

Forecast: Output shape? And the value at row 2, column 3?

Reshape a. a[:, np.newaxis] inserts a new trailing axis → shape (3, 1). Why this step? We want every pairwise product. To force a 2-D grid, one operand must own the rows and the other the columns.
Right-align b. b is (4,), rank 1; Rule 3 pads it on the left → effective (1, 4). Why this step? Rank mismatch: (3,1) has rank 2, b has rank 1, so Rule 3 inserts one size-1 axis on b's left.
Compare axes. Trailing: 1 vs 4 → Rule 2, a stretches to 4. Next: 3 vs 1 → Rule 2, b stretches to 3. Why this step? This is the double-stretch: both operands have a singleton, so Rule 2 fires on each axis in opposite directions, producing an all-pairs grid — the definition of an outer product.
Result (max(3,1), max(1,4)) = (3, 4). Cell [i,j] = a[i] * b[j]. Row 2, col 3 = 3 * 40 = 120. Why this step? Each axis takes the larger size, so the grid is 3 rows by 4 columns; after both stretches, position [i,j] holds a[i] (from the stretched column) times b[j] (from the stretched row).

The figure shows the magenta column a and the orange row b each fanning out over the empty grid; where the fans overlap, the product is written in navy.

Verify: Full grid:

[[ 10  20  30  40]
 [ 20  40  60  80]
 [ 30  60  90 120]]

Bottom-right corner 3*40 = 120. ✓

Worked example Example 5 — Cell E: rank mismatch, adding a vector to every row

M = np.arange(6).reshape(2, 3)  # [[0,1,2],[3,4,5]], shape (2, 3)
v = np.array([10, 20, 30])      # shape (3,)
result = M + v

Forecast: Does the (3,) line up with the rows or the columns?

Shapes: M is (2, 3), v is (3,). Why this step? Different ranks (2 vs 1) means Rule 3 will fire — but we must first know which axis is missing, and it is always the leftmost.
Right-align. Rule 3 gives v one size-1 axis on the left → effective (1, 3). Why this step? The pad always lands on the left, so v's single 3 sits under M's trailing 3 — under the columns, not the rows.
Compare. Trailing: 3 vs 3 → Rule 1 (equal). Next: 2 vs 1 → Rule 2, v stretches to 2 rows. Why this step? The column axis already matches (Rule 1), and the padded 1 on the row axis is stretched (Rule 2) so v is copied onto both rows — the same vector added to every row.
Result (2, 3): row 0 = [0,1,2]+[10,20,30] = [10,21,32]; row 1 = [3,4,5]+[10,20,30] = [13,24,35]. Why this step? With v now present in both rows, we simply add the aligned column entries position by position.

Verify: Corner [1,2] = 5 + 30 = 35. This is the "add-a-bias-row" pattern used everywhere in matrix ops in neural networks. ✓

Worked example Example 6 — Cell F: the degenerate zero-length axis

E = np.empty((0, 3))    # shape (0, 3): zero rows, 3 columns
v = np.array([1, 2, 3]) # shape (3,)
result = E + v

Forecast: Error, or a valid array? What shape?

Shapes: E is (0, 3), v is (3,) → Rule 3 aligns it to (1, 3). Why this step? A size-0 axis is still a legal size; Rule 3 (missing axis) and the later rules only care whether sizes clash, not whether they are empty.
Compare. Trailing: 3 vs 3 → Rule 1 (equal). Next: 0 vs 1 → Rule 2, the 1 stretches to 0. Why this step? Rule 2 says a singleton matches any size — and 0 counts as "any size". Stretching to 0 just means "make zero copies".
Result (0, 3): a valid but empty array — no rows to fill. Why this step? The output takes the larger size on each axis; max(0,1)=0 keeps the row axis empty, so there are simply no cells to compute.

Verify: result.shape == (0, 3) and result.size == 0. No ValueError fires. The key lesson: 0 is not the clash trigger — only two different, both-non-1 sizes clash. ✓

Worked example Example 7 — Cell G: the hard failure

C = np.array([[1, 2]])     # shape (1, 2)
D = np.array([[1, 2, 3]])  # shape (1, 3)
result = C + D             # raises ValueError

Forecast: Which axis breaks it, and why can't Rule 2 rescue us?

Shapes: both rank 2, no padding: (1, 2) and (1, 3). Why this step? Equal ranks, so Rule 3 stays quiet and we compare sizes directly from the right.
Trailing axis: 2 vs 3. Why this step? Neither is equal (2 ≠ 3, so Rule 1 fails) and neither is 1 (so Rule 2 cannot fire) — there is no singleton to stretch, so NumPy cannot invent a matching size.
Verdict: ValueError: operands could not be broadcast together with shapes (1,2) (1,3). Why this step? When an axis pair passes none of the three rules, broadcasting is declared impossible and NumPy raises immediately.

The figure contrasts a rescue and a crash: (1,2) vs (2,1) (both have a 1, both stretch via Rule 2 → orange grid) beside (1,2) vs (1,3) (trailing 2 vs 3, both non-1 → magenta crash bolt).

Verify: The pairing that works — (1,2) vs (2,1) — gives (2,2); the pairing that fails — (1,2) vs (1,3) — has trailing 2 ≠ 3 with no 1. ✓

Worked example Example 8 — Cell H: word problem, standardizing exam scores

Three students each sat three subjects. We want each column (subject) shifted by its column mean and scaled by its column standard deviation — classic feature standardization.

S = np.array([[50., 80., 90.],
              [60., 70., 30.],
              [70., 60., 60.]])   # shape (3, 3): rows=students, cols=subjects
mu    = S.mean(axis=0)            # shape (3,): per-column mean
sigma = S.std(axis=0)            # shape (3,): per-column std
Z = (S - mu) / sigma

Forecast: What is the mean of each column of Z? (Think before computing.)

Column means. mu = [60, 70, 60] (average down each subject column). Why this step? axis=0 collapses the row axis, leaving one number per column — the per-subject average.
Shape of mu: (3,) → Rule 3 aligns it to (1, 3), sitting under the columns of S. It then broadcasts down every student row. Why this step? We want the same subject-mean subtracted from every student — that is exactly a down-the-rows stretch (Rule 2 on the padded axis, as in Cell E), applied for real.
Column stds. For subject 0 the deviations are [-10, 0, 10], so $σ_{0} = \frac{100 + 0 + 100}{3} = 200/3 \approx 8.165$ . For subject 2 the entries are [90, 30, 60] with mean 60, so the deviations are [30, -30, 0] and $σ_{2} = \frac{900 + 900 + 0}{3} = 600 \approx 24.495$ — a much wider spread than subject 0. Why this step? Standardization divides by spread so every subject ends on the same scale; the spreads genuinely differ, so each column needs its own $σ$ .
Divide. (S - mu) is (3,3); sigma is (3,) → Rule 3 makes it (1,3); Rule 2 broadcasts it down the rows again. Result Z is (3,3). Why this step? The same right-alignment that placed mu under the columns also places sigma there, so each column is divided by its matching std.

Verify: Every column of Z has mean $0$ by construction (we subtracted its mean) and population standard deviation $1$ (we divided by its std). Column-0 entries: $[- 10, 0, 10] /8.165 \approx [- 1.2247, 0, 1.2247]$ , mean $0$ , std $1$ . Column-2 std check: $600 \approx 24.495 \neq = 8.165$ , confirming the columns really do differ. ✓

Worked example Example 9 — Cell I: the exam-twist transpose trap

X = np.random.rand(3, 100)          # 3 features (rows), 100 samples (cols)
weights = np.array([0.5, 0.3, 0.2]) # shape (3,)
bad  = X * weights                  # ValueError!
good = X * weights[:, np.newaxis]   # works

Forecast: Why does the "obvious" version crash when there are clearly three weights and three feature-rows?

Shapes: X is (3, 100), weights is (3,) → Rule 3 aligns it to (1, 3). Why this step? Right-alignment forces weights's 3 under X's trailing axis, which is 100 — not under the feature-axis 3.
Trailing compare: 100 vs 3 → both non-1, different → Rule 1 and Rule 2 both fail → clash → ValueError (this is Cell G in disguise). Why this step? Human intuition matches by meaning ("3 weights = 3 features"); NumPy matches by position from the right. Meaning is irrelevant to the alignment rule.
The fix: weights[:, np.newaxis] makes shape (3, 1). Why this step? Now the 3 sits under X's row-axis (3 vs 3, Rule 1) and the 1 sits under the sample-axis (1 vs 100, Rule 2 stretch). Both axes are now legal.
Result good.shape == (3, 100): each feature-row scaled by its own weight. Why this step? With the 1 stretched to 100, weight w[i] sits across the whole of row i, so every sample of feature i is multiplied by the same w[i].

Verify: bad raises ValueError; good.shape == (3, 100). Reshaping weights to (3,1) moves the 3 off the trailing slot and onto the row slot. ✓

Worked example Example 10 — Cell J: rank > 2, a 3-D batch meets a 2-D operand

batch = np.ones((4, 2, 3))          # shape (4, 2, 3): 4 images, each 2x3
bias  = np.array([[10, 20, 30],
                  [40, 50, 60]])    # shape (2, 3): one bias plane
out = batch + bias

Forecast: Does the same 2-D bias plane get added to all 4 images? What shape?

Shapes: batch is (4, 2, 3) (rank 3), bias is (2, 3) (rank 2). Why this step? Ranks differ (3 vs 2), so Rule 3 must fire once — but only on the single leftmost axis that bias lacks.
Right-align. Rule 3 pads bias to (1, 2, 3). Why this step? Padding goes on the left, so bias's (2, 3) lands under batch's trailing two axes (2, 3), and a fresh size-1 axis appears in the batch slot.
Compare all three axes, right to left. Trailing: 3 vs 3 → Rule 1. Middle: 2 vs 2 → Rule 1. Leading: 4 vs 1 → Rule 2, the padded 1 stretches to 4. Why this step? The two trailing axes already match (Rule 1), and the padded batch axis is a singleton, so Rule 2 copies the single bias plane across all 4 images.
Result (max(4,1), max(2,2), max(3,3)) = (4, 2, 3). Every one of the 4 images gets the same (2,3) bias plane added. Why this step? Each axis takes the larger size; because only the batch axis was a singleton, the bias plane is shared identically across the batch — exactly the "one bias per feature-map, reused per sample" pattern of neural-net matrix ops.

Verify: out.shape == (4, 2, 3), and since every image was all-ones, image k equals 1 + bias, e.g. out[0] and out[3] are both [[11,21,31],[41,51,61]]. Corner out[3,1,2] = 1 + 60 = 61. ✓

Active recall

Recall What is the

only condition that causes a broadcasting ValueError? Two aligned axes have different sizes and neither is 1 — that is, both Rule 1 and Rule 2 fail. ::: Everything else — missing axes (Rule 3), singletons (Rule 2), even size-0 axes — broadcasts fine.

Recall What does "Rule 3 then Rule 2" actually mean?

Rule 3 only inserts a size-1 axis where an array has fewer dimensions; that new 1 is then an ordinary singleton, so Rule 2 is what stretches it. ::: Rule 3 never resolves an axis by itself — it always hands off to Rule 2.

Recall A

(3,) vector times a (3, 100) matrix crashes. Why, in one sentence? Right-alignment puts the vector's 3 under the trailing axis 100 (not under the feature-axis 3), and 3 ≠ 100 with no singleton, so both Rule 1 and Rule 2 fail. ::: Fix with weights[:, np.newaxis] to make it (3, 1).

Recall When a

(4, 2, 3) batch meets a (2, 3) plane, which axis stretches and to what? Rule 3 pads the plane to (1, 2, 3), then Rule 2 stretches that leading 1 to 4, so the one plane is reused across all 4 batch items. ::: Result stays (4, 2, 3).

For the speed payoff of never copying data, see 1.4.04-NumPy-vectorization, 1.4.06-NumPy-performance-optimization, and 1.2.04-Memory-efficient-Python-patterns; for the shape-manipulation tools like np.newaxis, revisit 1.4.03-NumPy-array-indexing-slicing.