Vectorization — avoiding Python loops, speed comparison

WHY does this matter?

WHY is the Python loop slow? Each iteration of a Python loop must:

Fetch the next object (a boxed PyObject, not a raw number).
Check its type at runtime (Python is dynamically typed).
Dispatch to the correct __add__/__mul__ method.
Allocate a new Python object for the result.

That is dozens of machine instructions to add two numbers. A NumPy array stores raw float64 values packed tightly in memory, so the C loop just does result[i] = a[i] + b[i] — one CPU instruction, no boxing, no type checks.

WHAT is the speed model? (Derivation from scratch)

Let's derive the expected speedup instead of memorizing "100×".

Total time for an operation on $N$ elements:

$T = c_{\text{setup}} + N \cdot c_{\text{per-element}}$

For a Python loop: $c^{\text{py}}_{\text{per-element}}$ is large (boxing, type check, dispatch). Call it $\tau_{\text{py}} \approx 50\text{–}200$ ns.
For a vectorized op: $c^{\text{vec}}_{\text{per-element}}$ is just a raw CPU op. Call it $\tau_{\text{vec}} \approx 0.5\text{–}2$ ns. There is a fixed C-call setup $c_{\text{setup}}$ .

Figure — Vectorization — avoiding Python loops, speed comparison

HOW do I vectorize? (Patterns)

Worked example 1. Element-wise arithmetic

# SLOW
out = []
for x in data:
    out.append(x*x + 1)
# FAST
out = data**2 + 1     # data is a np.ndarray

Why this step? data**2 + 1 triggers two compiled C loops (square, add). No Python-level iteration.

Worked example 3. Conditional logic with masks (replace

if in loop)

# SLOW
for i in range(len(a)):
    if a[i] < 0: a[i] = 0     # ReLU
# FAST
a = np.where(a < 0, 0, a)     # or: a[a < 0] = 0

Why? a < 0 builds a boolean mask array in C; np.where selects per-element in C. The branch is gone from Python.

Worked example 4. Broadcasting (avoid nested loops)

# pairwise sums of two vectors, SLOW double loop
M = [[u + v for v in col] for u in row]
# FAST
M = row[:, None] + col[None, :]   # shape (len(row), len(col))

Why? row[:,None] makes a column, col[None,:] a row; NumPy broadcasts them into a full matrix inside C — no Python loops at all.

Common mistakes (Steel-manned)

Forecast-then-Verify

Recall Feynman: explain to a 12-year-old

Imagine you must stamp 1,000,000 envelopes. Way 1 (Python loop): for each envelope you walk to the cupboard, take out the stamp, read the instructions, stamp it, put the stamp back, write a note. Repeat a million times — exhausting! Way 2 (vectorization): you tell a fast robot "stamp ALL of these the same way," and it does the whole pile in one smooth go without re-reading instructions each time. Same result, but the robot doesn't waste time on the boring setup a million times. That's why vectorized code is so much faster — it does the repetitive thinking once, not once per item.

Flashcards

Why is a Python for loop over numbers slow compared to NumPy?

Each iteration boxes objects, does runtime type-checking, method dispatch, and new-object allocation — dozens of instructions per element instead of one CPU op.

What does "vectorization" mean?

Expressing an operation on a whole array at once so the looping runs inside compiled C, not the Python interpreter.

Give the time model for an array op.

T = c_{\text{setup}} + N\,\tau_{\text{per-element}}

; speedup

\to \tau_{py}/\tau_{vec}

for large

N

For very small arrays, is vectorization always faster?

No — the fixed C-call setup

c_{\text{setup}}

dominates, so a scalar/loop can be faster.

Why is np.append inside a loop bad?

It copies the entire array each call →

O(N^2)

. Build a list then np.array, or pre-allocate np.empty.

Does np.vectorize(f) make f fast?

No — it's essentially a Python-level loop; use NumPy ufuncs or numba for real speed.

How do you vectorize a per-element if (e.g. ReLU)?

Use a boolean mask: a[a<0]=0 or np.where(a<0,0,a).

What is broadcasting used for?

Combining arrays of different shapes (e.g. row[:,None]+col[None,:]) to avoid nested Python loops.

Why can iterating for x in np_array be slower than a list?

Each element is unboxed into a np.float64 object, adding overhead beyond a normal Python loop.

What often limits vectorized speed at large N?

Memory bandwidth — the CPU waits on data, not on arithmetic.

Connections

NumPy Arrays vs Python Lists — contiguous typed memory is why vectorization works.
Broadcasting Rules — vectorizing multi-dimensional combinations.
Universal Functions (ufuncs) — the compiled C kernels you call.
Big-O Complexity — why np.append-in-loop is $O(N^2)$ .
Numba and Cython — when even vectorization isn't enough.
Memory Layout and Cache — bandwidth limits on vectorized ops.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, Python ka for loop slow kyun hota hai? Kyunki har ek number ke liye Python ko bahut kaam karna padta hai — type check karo, object box karo, sahi __add__ method dhoondho, aur naya object banao. Yeh sab ek hi addition ke liye! Agar tumhare paas 10 lakh numbers hain, toh yeh boring kaam 10 lakh baar repeat hota hai. Vectorization ka matlab hai: poore array pe ek saath operation karo (jaise a + b), aur asli looping NumPy ke pre-compiled C code ke andar chale — jaha boxing aur type-check ka jhanjhat nahi hota.

Speed model simple hai: $T = c_{setup} + N \times \tau$ . Python loop me $\tau$ bada (~100 ns), vectorized me chhota (~1 ns). Isiliye bade arrays pe speedup ~100× tak jaata hai. Lekin chhote arrays pe woh fixed c_setup (C-call ka overhead) dominate kar deta hai, toh kabhi-kabhi vectorize karna ulta slow bhi ho sakta hai. Yahi reason hai ki hamesha %timeit se measure karo.

Ek important galti: loop ke andar np.append mat karo — woh har baar pura array copy karta hai, $O(N^2)$ ho jaata hai, list se bhi bura! Behtar: Python list me append karo, phir np.array(list) se convert karo, ya np.empty(N) se pehle se jagah bana lo. Aur ek aur trap — np.vectorize(f) naam se lagta hai fast hoga, par andar se woh bhi Python loop hi hai, koi compilation nahi. Asli speed ke liye NumPy ke ufuncs (jaise np.where, data**2, .sum()) use karo, ya numba laga lo.

Yaad rakhne ka mantra: "Loop in C, not in Me." Agar tum khud loop chala rahe ho Python me, toh slow; NumPy ko poore array de do taaki woh C me loop chalaaye — phir code rocket ban jaayega.

Vectorization — avoiding Python loops, speed comparison

WHY does this matter?

WHAT is the speed model? (Derivation from scratch)

HOW do I vectorize? (Patterns)

Common mistakes (Steel-manned)

Forecast-then-Verify

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Scientific Computing (Python)

Connections