Foundations — NumPy arrays and vectorized operations

This page assumes you know nothing. Before you can read the parent note 1.4.4, you must be able to look at every symbol on it and know exactly what picture it names. We build them one at a time, each resting on the one before.

1. A number, a list, and why lists are slow

Start with the most basic thing: a number. Picture a single box with a value written inside it, say 7.

A Python list [1, 2, 3, 4] is a row of such boxes. But — and this is the secret the parent note leans on — in ordinary Python those boxes are scattered all over memory. The list only stores arrows (pointers) telling the computer "the real box is over there."

Look at the figure. The top row (black) is a Python list: little arrows pointing off to boxes flung across memory. The bottom row (red) is a NumPy array: the numbers themselves sit side by side, touching, in one strip.

If you have not met Python lists and loops yet, pause and read 1.4.01-Python-fundamentals and 1.4.02-List-comprehensions-and-generators first — everything below assumes you can picture a for loop.

2. The word element and the symbol $a_i$

The parent writes things like $c_i=a_i+b_i$. Two new symbols hide here.

Picture a row of boxes numbered 0, 1, 2, 3. Then $a_2$ is the box in slot 2. The bold letter $\mathbf{a}$ (bold = the whole row) versus the light $a_i$ (light = one box) is a distinction the parent uses constantly.

Why does the topic need indices? Because to say "do the same thing to every box" you first need a name for a box. The index is that name.

3. Shape: describing the grid

So far a straight row. Now stack rows to make a rectangle — a grid.

In the figure the red frame surrounds the array and the red labels show 3 down the side and 4 across the top — that pair is the shape (3, 4). Each little black box is one element $A_{ij}$: row $i$, column $j$.

Why does the topic need shape? Because vectorized rules ("do these two arrays fit together?") are decided entirely by comparing shapes. No shape, no broadcasting.

4. dtype: every box holds the same kind of thing

Because the parent stressed arrays are homogeneous (all boxes the same type), the computer knows every box is, say, exactly 8 bytes wide. That fixed width is what lets it "walk forward a fixed number of bytes" from Section 1. Mixed types would break the even spacing.

You rarely type strides yourself; know that they exist because the parent lists them, and they are simply "byte-steps," nothing scarier.

5. Element-wise: the heart symbol $⟹$ and the loop it hides

The parent writes: $\mathbf{c}=\mathbf{a}+\mathbf{b}⟹c_i=a_i+b_i$

Two things to decode.

The symbol $⟹$ means "which is the same as saying" — it unpacks the compact left side into the box-by-box truth on the right.

The statement itself says: to add two arrays, add matching boxes. Box 0 with box 0, box 1 with box 1, and so on.

In the figure, array $\mathbf{a}$ sits above array $\mathbf{b}$; the red arrows connect matching positions, and the bottom row is their box-by-box sum $\mathbf{c}$. There is no arrow from box 0 to box 2 — only same-slot pairs meet. That "same-slot only" rule is what element-wise means.

6. The Hadamard product $\cdot$ vs. matrix product — two different pictures

The parent uses two multiply-like symbols and warns not to confuse them.

Two new symbols to earn here:

${\sum }_{k=1}^n$ is the summation sign. It is shorthand for "add up a list of terms." Read it: let $k$ run from 1 to $n$, form each term, and total them. For $n=3$ it means term($k=1$) + term($k=2$) + term($k=3$). It is just a compact for-loop-that-adds.

$A_{ik}$ means "the box in row $i$, column $k$ of grid $A$." Two subscripts because a grid needs two coordinates.

Why does the topic need $\sum$ and $@$? Because neural-network layers are matrix products — see 2.1.03-Matrix-operations-for-ML and 3.1.02-Tensor-operations-in-PyTorch. You cannot read those without this symbol.

7. ufunc and $f(\mathbf{a})$: one function on every box

The parent writes $\mathbf{c}=f(\mathbf{a})⟹c_i=f(a_i)$.

$f$ here is just "some function" — a rule that takes one number and returns one number, like "square it" or $\sin$.

8. Broadcasting: stretching a small array to fit a big one

The parent's "superpower." Its rule uses the word compatible shapes.

Picture a (3, 4) grid and a single row (4,). You want to add the small row to every row of the grid. Broadcasting pretends the small row is copied down 3 times (the red ghosted copies in the figure) — without actually using that extra memory.

Why does the topic need broadcasting? Because in ML you constantly add a small bias row to a big batch of data — exactly the (3,4) + (4,) picture. See 2.2.04-Batch-gradient-descent.

9. Views vs. copies, and BLAS (the last two words)

Prerequisite map

Equipment checklist

Say each answer out loud before revealing it. If any stumps you, reread that section.