Computational graphs and autograd: Visual walkthrough

We use the same running function as the parent so nothing clashes: $f (x, y) = (x + y) \cdot sin (x)$

Step 1 — What "derivative" really means: a wiggle

WHAT. Before any graph, fix the one idea everything rests on. A derivative answers a single question: if I nudge the input by a tiny amount, how big is the nudge that comes out? That ratio (out-nudge ÷ in-nudge) is the derivative.

WHY this tool. Training a network means improving the loss. To improve something you must know which way to push each knob — and by how hard. "How much does the output move when I move this knob" is exactly a derivative. So the derivative is not an abstract symbol; it is literally the steering signal.

PICTURE. Look at the figure. We push the input $x$ right by a tiny step (the small blue arrow $Δ x$ ). The curve carries that push to a tiny vertical step in the output (the pink arrow $Δ f$ ). The slope of the little triangle — pink over blue — is the derivative. Steeper triangle = a small input wiggle causes a big output wiggle.

Figure — Computational graphs and autograd

Step 2 — Break the function into primitive gears

WHAT. We cannot wiggle-analyse a big messy formula directly. So we chop $f$ into the smallest operations we already know how to differentiate — one operation per box.

WHY. We only have simple wiggle-rules memorised: the wiggle-rule for $+$ , for $\times$ , for $sin$ . If the whole computation is nothing but these primitives strung together, we can differentiate each box on its own and later glue them. This chopping is the computational graph the parent defined.

PICTURE. Each round node is a value; each arrow is one operation that builds a new value from its parents. Trace it: $x$ and $y$ flow into a $+$ node making $a = x + y$ ; $x$ also flows into a $sin$ node making $b = sin x$ ; then $a$ and $b$ multiply into $f$ . Notice the pale-yellow node $x$ has two arrows leaving it — remember that; it is the whole plot of this page.

Step 3 — Write the wiggle-rule on every single arrow

WHAT. For each arrow (operation), we compute its local derivative — how much the child wiggles when the parent wiggles, ignoring the rest of the graph.

WHY. Local means "just this one box." A single box is simple enough that we already know its rule by heart. Doing them one at a time is the only reason this is tractable for millions of operations: no box ever needs to know about the whole network.

PICTURE. Each arrow is now labelled with its local wiggle-factor. Read them straight off the boxes:

Each label answers "if the tail of this arrow wiggles by 1, how much does the head wiggle?" — nothing more. (For $f = ab$ : wiggling $a$ scales up by whatever $b$ currently is, so the factor is $b$ . See Sigmoid and activation functions later for the $σ^{'}$ box.)

Step 4 — One path: multiply the factors along it

WHAT. Follow one route from an input to $f$ and ask how a wiggle survives the trip. Take the route $x \to a \to f$ .

WHY. A wiggle passing through a chain of boxes gets rescaled at each box by that box's local factor. Rescale, then rescale again — that's multiplication. This is the chain rule (Chain rule (multivariable calculus)) in its bare mechanical form.

PICTURE. Watch the blue wiggle enter $x$ , get multiplied by $\frac{\partial a}{\partial x} = 1$ at the add box, then by $\frac{\partial f}{\partial a} = b$ at the multiply box. The two factors sit on the two arrows of the highlighted path; the output wiggle is their product.

But this is only one of two routes. Hold that thought.

Step 5 — The forgotten second path, and why we ADD

WHAT. Because $x$ leaves through two arrows (Step 2!), a wiggle in $x$ also travels the bottom route $x \to b \to f$ . The true effect on $f$ is the sum of both routes.

WHY. The two routes are two independent deliveries of the same wiggle. If nudging $x$ raises $f$ a bit through path A and raises it a bit through path B, the total rise is both added together — you cannot get one delivery for free. This "sum over paths" is precisely the multivariable part of the chain rule, and it is the reason autograd uses += not =.

PICTURE. Both routes are lit at once — top (blue) through $a$ , bottom (pink) through $b$ — merging at $f$ . The final derivative is top-factor plus bottom-factor.

Step 6 — Turn "sum over paths" into a backward sweep (the adjoint)

WHAT. Computing every path separately explodes: a deep net has astronomically many paths. Instead we compute, for each node $v$ , a single number $\overset{v}{ˉ}$ = "total sensitivity of $f$ to $v$ ," and pass it backward once.

WHY. Notice the top and bottom paths in Step 5 shared the last arrow into $f$ . Rather than re-walking shared arrows, we store the sensitivity at each node and reuse it. Sweeping output → inputs means every node's $\overset{v}{ˉ}$ is ready exactly when its parents ask for it. This is reverse mode, and it is why cost scales with outputs (one loss), not inputs (millions of weights) — see Automatic differentiation (forward vs reverse mode).

PICTURE. The seed $\overset{ˉ}{f} = 1$ is injected at the output. It flows backward: $f$ hands $\overset{a}{ˉ}$ and $\overset{ˉ}{b}$ to its parents, then $a$ and $b$ hand their shares back into $x$ (which adds them) and $y$ .

Step 7 — Plug in numbers at $x = 1, y = 2$

WHAT. Run the whole machine on concrete values to see the abstract sweep produce a real gradient.

WHY. Numbers expose whether we truly understand the flow, and let us machine-verify. Note the forward pass stores $a, b$ because the backward factors ( $b$ , $a$ , $cos x$ ) need them — the reason we cache values.

PICTURE. Values flow right (forward, yellow), adjoints flow left (backward, pink), on the same graph.

Step 8 — Degenerate & edge cases (never leave a gap)

WHAT. Cases where naive intuition wobbles. Each still obeys the same rule.

WHY. A reader must never hit a scenario we skipped. Autograd's rule is uniform, so let's watch it survive the weird inputs.

PICTURE. Three mini-boards: a zero local factor, a fan-in node, and a dead-flat slope.

Worked example The three edge cases

(a) Zero local derivative — the gradient stops. At $x = \frac{π}{2}$ , $cos x = 0$ , so the bottom path contributes $\overset{ˉ}{b} \cdot 0 = 0$ . The wiggle through $sin$ is blocked here; only the top path survives: $\overset{x}{ˉ} = b \cdot 1 = sin (\frac{π}{2}) = 1$ . Nothing breaks — a $0$ factor simply delivers a $0$ contribution.

(b) A node with many children (fan-in of adjoints). If $x$ fed three boxes, $\overset{x}{ˉ}$ would be a sum of three += terms. The rule scales to any number of arrows; that's why deep nets with heavily reused activations still work.

(c) Flat regions kill learning (vanishing gradient). In Sigmoid and activation functions, $σ^{'} (z) = σ (z) (1 - σ (z)) \to 0$ when $∣ z ∣$ is large. A near-zero local factor means the backward wiggle nearly vanishes — the mathematical face of the vanishing gradient problem in Gradient descent. Same rule, dangerous regime.

The one-picture summary

Everything on one board: forward values glide right and get cached; the seed $\overset{ˉ}{f} = 1$ enters at the output and adjoints glide left, each arrow multiplying by its local factor, each fan-out node summing its incoming adjoints. One forward + one backward sweep = the gradient with respect to every input.

Recall Feynman retelling — the whole walkthrough in plain words

Think of a row of little machines. First I turn the input knobs and let the numbers roll through — each machine writes down the number it saw (that's the forward pass, and it saves those numbers because it'll need them). Now I ask the last machine: "if your output wiggled by one, whose fault is it?" It looks at its two input wires and says "you two are responsible by this much and that much" — those amounts are just how steep that machine is. Each earlier machine takes the blame it received and passes it back, again scaled by its own steepness. If a wire splits and feeds two machines, the knob it came from collects blame from both and adds them up. When the blame reaches the very first knobs, I now know exactly how much each knob matters — so I know which way to turn every one to make the answer better. That backward blame-passing is Backpropagation, and it costs about the same as running the machines forward once.

Active recall

Recall Quick self-test

Which two paths carry $x$ 's wiggle to $f$ , and why add them? ::: Path $x \to a \to f$ and path $x \to b \to f$ ; a wiggle is delivered through both routes, and the multivariable chain rule sums independent deliveries — hence +=. Why must we cache $a$ and $b$ during the forward pass? ::: The backward factors ( $\partial f / \partial a = b$ , $\partial f / \partial b = a$ , $\partial b / \partial x = cos x$ ) are functions of those stored values. What is the seed adjoint and why? ::: $\overset{ˉ}{f} = 1$ , because $\partial f / \partial f = 1$ — the output is perfectly sensitive to itself. At $x = π /2$ what happens to the $sin$ path's contribution? ::: $cos (π /2) = 0$ , so that path delivers $0$ ; the gradient still flows through the add path. Why does reverse mode win for neural nets? ::: One backward sweep gives gradients for all inputs at cost of outputs (one scalar loss), while millions of weights would each need a separate forward-mode pass.

Why do adjoints accumulate with += rather than =?

A node with several outgoing arrows affects the output through several paths; the multivariable chain rule sums those path contributions.

What does a zero local derivative do to backprop?

It blocks that path — its contribution to the parent's adjoint is (adjoint)×0 = 0 — which is exactly the vanishing-gradient mechanism.