Foundations — Autograd and computational graphs in PyTorch

Before you can read a single line of the parent note the Autograd note, you need a small toolbox of ideas. This page builds each one from nothing. Nothing here is assumed. If a symbol appears, it was defined a line earlier and pinned to a picture.

0. The absolute starting point: a number that "remembers"

Normally a number is just a value: 3.0. It sits there. It has no history.

The whole trick of autograd is to use a number that carries a history — it remembers where it came from and what was done to it. In PyTorch that "number-with-history" is a tensor (a container for one or more numbers) that you flag with requires_grad=True.

Look at the picture. A plain number (left) is a lone dot — no arrows in, no arrows out. A tracked tensor (right) has an arrow pointing back to whatever operation produced it. That backward-pointing arrow is the seed of the whole computational graph.

1. Variable, function, and output — $x$, $y$, $f$, and $z$

Before derivatives we need the language of changing quantities.

The picture for a function is an input arrow → box → output arrow. This is exactly the node picture from Section 0 — a function is an operation, and its output is a new tracked number. This is why the topic calls the whole thing a graph of operations.

2. The symbols $+$, $\times$, $x^2$ — operations that become nodes

The parent note writes z = x * y + x**2. Reading the left side with Section 1: $z$ is the output of feeding inputs $x$ and $y$ through three operations:

• $x\times y$ — multiply (in code: x * y)
• $x^2$ — square, meaning $x\times x$ (in code: x**2)
• $+$ — add

So in symbols the output is $z=xy+x^2$.

Each operation is a little machine (a function of Section 1). So $z=xy+x^2$ is really three machines feeding into each other, and $z$ is the final number that drops out the end.

3. What "a tiny nudge" means — the derivative

This is the single most important tool the topic uses, so we build it slowly.

Why do we need a rate and not just the output value? Because training a network means adjusting inputs (the weights) to lower a loss. To adjust wisely you must know which direction lowers the loss and by how much per step. That is precisely a derivative.

Look at the curve $f(x)=x^2$ in the figure. Pick the point at $x=2$. Draw a tiny step of width $h$ to the right (the pale-yellow segment). The output climbs by the pink segment. The slope of the straight line connecting the two points is $\frac{\text{rise}}{\text{run}}=\frac{f(x+h)-f(x)}{h}$. As $h$ shrinks toward zero, that line becomes the blue tangent — the true instantaneous rate. That limiting slope is the derivative.

Let us earn the parent note's result $\frac{d}{dx}{x}^2=2x$ by hand: $\frac{d}{dx}{x}^2={\lim }_{h\to 0}\frac{(x+h{)}^2-x^2}{h}={\lim }_{h\to 0}\frac{x^2+2xh+h^2-x^2}{h}={\lim }_{h\to 0}\frac{2xh+h^2}{h}={\lim }_{h\to 0}(2x+h)=2x$

What we did: expanded $(x+h{)}^2$, cancelled $x^2$, cancelled one $h$ top-and-bottom, then let $h\to 0$ so the leftover $h$ vanished. Why: this is the only honest definition of "instantaneous rate," and it is exactly the rule autograd applies internally for every squaring operation.

4. Many inputs at once — the partial derivative $\partial$

The output $z=xy+x^2$ has two inputs, $x$ and $y$. "Nudge the input" is now ambiguous — nudge which one?

The picture: on a hilly surface with two floor-directions $x$ and $y$, $\frac{\partial z}{\partial x}$ is the steepness you feel if you walk purely in the $x$-direction, ignoring $y$.

Because $y$ is frozen, treat it like a plain constant: $\frac{\partial z}{\partial x}=\frac{\partial }{\partial x}(xy)+\frac{\partial }{\partial x}(x^2)=y+2x,\frac{\partial z}{\partial y}=\frac{\partial }{\partial y}(xy)+\frac{\partial }{\partial y}(x^2)=x+0=x$

5. Chaining machines — the chain rule

Section 2 said operations feed into each other. Suppose the output is built in two stages: an inner machine $g$ turns $x$ into an in-between value we also call $g$, then an outer machine $f$ turns that into the final output $z$. In symbols $z=f(g(x))$, where $g$ names both the inner machine and the number it produces. How does a nudge in $x$ reach $z$?

Why the topic lives on this: the backward pass is nothing but the chain rule applied step by step through the graph, multiplying the little rate stored at each node.

When $x$ reaches $z$ by two different paths (in $z=xy+x^2$, the value $x$ flows into both the $xy$ node and the $x^2$ node), the two ripples add: $\frac{\partial z}{\partial x}=\underset{\text{via }xy}{\underbrace{y}}+\underset{\text{via }{x}^2}{\underbrace{2x}}$

6. Direction of travel — the DAG and "reverse order"

The parent calls the graph a DAG. Two new words:

7. The vector / matrix symbols — $W$, $x^T$, and $y=Wx+b$

The parent's Example 1 uses $y=Wx+b$. Heads-up on naming: here $y$ is the output of the layer — it plays the role $z$ played in Sections 2–5. We keep the parent's letter $y$, but remember $y$ is now "the number the machine produces," not an input. Unpack the rest of the alphabet:

Now the weight gradient. Let $\frac{\partial L}{\partial y}$ be the row of rates of the loss with respect to the layer's output $y$ (Section 8 defines $L$). Since $y=Wx+b$, matrix calculus gives $\frac{\partial L}{\partial W}=x^T\frac{\partial L}{\partial y}.$ Shape check: $x^T$ is $2\times 1$ and $\frac{\partial L}{\partial y}$ is $1\times 2$, so their product is $2\times 2$ — exactly the shape of $W$. That shape agreement is how you know the formula is put together correctly. (The parent note wrote this with $z$ for the pre-activation output; it is the same output-role letter, just renamed.)

You will meet these live when you build layers: 3.4.01-Building-neural-networks-with-torch.nn.Module.

8. The loss $L$ and the summary symbol $\sum$

Now the subtle point the parent raises. A batch of $N$ samples has a batch loss. A common choice is the average: $L_{\text{batch}}=\frac{1}{N}{\sum }_{i=1}^N{L}_i.$

9. The autograd nouns you'll type

Now the code words in the parent are just labels on the ideas above:

These reappear the moment you touch 4.2.03-Batch-normalization, 5.3.01-Transfer-learning, and memory tricks like 6.4.02-Gradient-checkpointing — so lock them in.

Prerequisite map

The picture below shows how each foundation feeds the next, ending at the autograd machinery.

Read it top-to-bottom: raw tensors and functions become graph nodes; derivatives and the chain rule power the backward walk; vectors and matrices supply the layer shapes; a scalar loss seeds the whole thing — and all of it lands in Autograd in PyTorch.

Equipment checklist

Cover the right side and answer out loud. If any stalls you, reread that section before opening the parent note.