Visual walkthrough — NumPy broadcasting rules

This page builds NumPy's broadcasting rule from nothing. We will draw every array as a grid of squares, watch small grids "stretch" to fill big grids, and discover the three rules by seeing exactly when the stretching is honest and when it lies.

Parent: NumPy broadcasting rules · prerequisites: 1.4.03-NumPy-array-indexing-slicing, 1.4.04-NumPy-vectorization.

Before we use a single symbol, let us agree on our pictures.

---

Step 1 — Why "same size" is the only honest rule

WHAT. We want $C=A+B$ where $+$ means: add the number in each square of $A$ to the number in the matching square of $B$.

WHY. "Matching square" only makes sense if every square of $A$ has exactly one partner in $B$. If $A$ is $2\times 3$ and $B$ is $2\times 3$, each of the 6 squares pairs up perfectly. This pairing, one square to one square, is called an element-wise operation.

PICTURE. Two identical grids sit side by side; arrows connect each square to its twin. No ambiguity — every square has exactly one partner.

$C[i,j]=A[i,j]+B[i,j]$

• $i$ ::: which row (top to bottom), the same $i$ picks the row in all three grids.
• $j$ ::: which column (left to right), again shared across all three.
• $A[i,j]$ ::: the single number sitting in row $i$, column $j$ of grid $A$.

The rule "shapes must be equal" is safe but rigid. The rest of this page is about safely bending it.

---

Step 2 — The trick: a size-1 side can repeat

WHAT. Take $A$ of shape $(3,3)$ and $m$ of shape $(3,1)$ — a single column of 3 numbers. We claim $A-m$ makes sense.

WHY this is allowed. Column $m$ has only one square per row. But a whole row of $A$ needs a partner for each of its 3 squares. The honest fix: let that one number repeat across the row. One value, reused 3 times — no new memory, just re-reading the same square.

PICTURE. The thin column on the left; faded "ghost" copies fan out to the right until it is as wide as $A$. The original squares are solid; the repeats are pale.

$m.\text{shape}=(3,\underset{\text{only 1 column}}{\underbrace{1}})⟶\text{stretched to }(3,3)$

• The $1$ ::: the signal "I have only one value here — reuse me."
• The stretched $3$ ::: matches $A$'s 3 columns so pairing works.

---

Step 3 — Why we align from the RIGHT

WHAT. When two shapes have different lengths, e.g. $(2,3)$ and $(3,)$, we must decide how to line them up. NumPy pushes both shapes to the right and compares column-numbers first.

WHY. In memory, a row's numbers sit next to each other (this is row-major storage). The rightmost dimension is the one that changes fastest as you walk through memory. Aligning on the right keeps the fast-moving axis paired with the fast-moving axis — cache-friendly and predictable (1.4.06-NumPy-performance-optimization).

PICTURE. Two shape-strips, both shoved against a right wall. The shorter one leaves an empty slot on the left, which we fill with an invisible "1".

$$\begin{array}{rl}A:& (2,3)\\ b:& (3)\stackrel{\text{pad left}}{\to }(1,3)\end{array}$$

• Right wall ::: the trailing dimension — always compared first.
• The padded $1$ on the left ::: a missing dimension is treated as size 1 (which, by Step 2, can stretch).

So "missing" and "singleton" collapse into the same idea: a 1 that can grow.

---

Step 4 — The three cases for a single dimension pair

WHAT. After right-aligning, we walk each column of the shape strip and ask: are these two numbers compatible? There are exactly three "yes" answers and one "no".

WHY. Every dimension is judged independently. If even one pair fails, the whole operation fails. Knowing the four outcomes lets you predict any broadcast by eye.

PICTURE. Four little panels: EQUAL (green ✓), ONE-IS-1 (blue ✓, arrow shows the stretch), MISSING (blue ✓, it becomes 1 then stretches), and CLASH (red ✗, two unequal numbers neither of which is 1).

For a pair $(a_i,b_i)$ the output size is:

$${\text{out}}_i=\{\begin{array}{llc}{a}_i& \text{if }{a}_i=b_i& \text{(equal — Rule 1)}\\ b_i& \text{if }{a}_i=1& \text{(stretch }a\text{ — Rule 2)}\\ a_i& \text{if }{b}_i=1& \text{(stretch }b\text{ — Rule 2)}\\ \text{ERROR}& \text{otherwise}& \text{(clash)}\end{array}$$

• $a_i,b_i$ ::: the two numbers sitting in column $i$ after right-alignment.
• ${\text{out}}_i$ ::: the resulting size — always the bigger of the two (since the smaller one is a 1 that grew).

This compresses to one clean formula, treating any missing dimension as 1:

${\text{out}}_i=\max (a_i,b_i)\text{— but only if the CLASH case never fires.}$

---

Step 5 — Two grids stretch at once (the outer product)

WHAT. Take $a$ of shape $(3,1)$ (a column) and $b$ of shape $(4,)$ (a row of 4). Multiply: $a\cdot b$.

WHY it's the perfect stress test. Here both operands have a size-1 side, so both must stretch — in different directions. This proves stretching isn't a special "small helps big" trick; it's symmetric.

PICTURE. The 3-tall column stretches rightward to width 4 (blue ghosts). The 4-wide row (padded to $(1,4)$) stretches downward to height 3 (orange ghosts). Where they overlap, every cell is one product $a_i{b}_j$.

$$\underset{a\text{: stretch cols}}{\underbrace{(3,1)}}\times \underset{b\text{: stretch rows}}{\underbrace{(1,4)}}=(3,4)$$

$\text{out}[i,j]=a[i]\cdot b[j]$

• $a[i]$ ::: the $i$-th value of the column, reused across the whole row $i$.
• $b[j]$ ::: the $j$-th value of the row, reused down the whole column $j$.

With $a=[1,2,3{]}^{\top }$ and $b=[10,20,30,40]$:

$$\left[\begin{array}{cccc}10& 20& 30& 40\\ 20& 40& 60& 80\\ 30& 60& 90& 120\end{array}\right]$$

---

Step 6 — The degenerate & failing cases (never get surprised)

WHAT. We deliberately hit the edges: a bare scalar, a genuine clash, and a "looks-different-but-works" mismatch.

WHY. The contract is that you must never meet a scenario we skipped. So we catalogue all of them.

PICTURE. Three stacked strips. Top (green): scalar shape $()$ padded to $(1,1)$, stretches to fill anything. Middle (red): $(1,2)$ vs $(1,3)$ — the rightmost $2$ vs $3$ clash, both bigger than 1, so ✗. Bottom (green): $(3,100)$ vs $(3,)$ — the naive "3 lines up" is a trap, because $(3,)$ aligns as $(1,3)$ against the last axis $100$, giving $100$ vs $3$ → ✗ until you reshape.

---

The one-picture summary

Right-align → fill blanks with 1 → per column keep it if equal, stretch it if it's a 1, explode if it's a clash → the result is the tall/wide envelope of both.

Recall Feynman retelling — say it like a story

You have two grids of numbers and you want to add them square by square. But they're different sizes, so some squares have no partner. Broadcasting is a fair way to fake partners. First you shove both size-lists to the right, because in memory the columns are what sit next to each other. Any empty slot on the left you fill with a 1 — a 1 means "I only brought one value here, so I'll just reuse it as many times as needed." Now you march through each direction: if the two counts are equal, great; if one of them is a 1, it politely copies itself to match the other; but if you get two different numbers and neither is a 1, there's no honest way to pair them, so NumPy stops and complains. The final grid is exactly as big as the bigger side in every direction. That's it — no data is ever really copied; the 1s just get re-read. Scalars are the extreme case: a scalar has no directions at all, so it's all 1s, so it stretches to fit absolutely anything.

Recall Rebuild the rule from scratch

Why align from the right? ::: Because memory stores rows contiguously — the rightmost (fastest-changing) axis must pair with the rightmost axis. What does a size-1 dimension do? ::: It repeats its single slice to match any size, using no extra memory. What is a missing dimension treated as? ::: Size 1 (padded on the left), so it too can stretch. When does broadcasting fail? ::: When a dimension pair has two different sizes and neither is 1. What is the output size of each dimension? ::: The maximum of the two aligned sizes — provided no clash occurs.