Worked examples — Computational graphs and autograd

We first draw the scenario matrix: the complete list of structural situations a computational graph can throw at you. Then every example is tagged with the cell it covers.

---

The scenario matrix

Every computational-graph problem is some combination of these structural situations. A "cell" is one row here.

#	Cell (structural situation)	Why it's tricky	Covered by
A	Single path input → output	The plain chain rule, no summing	Ex 1
B	Fan-out (one node feeds ≥2 children)	Must accumulate ($+=$) adjoints	Ex 2
C	Dead branch (a node not on any path to output)	Its adjoint must be 0	Ex 3
D	Multiplication node with equal inputs ($x\cdot x$)	Both edges carry gradient, easy to halve by mistake	Ex 4
E	Zero / degenerate input killing a gradient	A factor becomes $0$, gradient vanishes	Ex 5
F	Full neuron chain (loss → sigmoid → linear)	Real backprop, several primitives stacked	Ex 6
G	Broadcast / vector sum (one input, many outputs summed)	Adjoints from every output add	Ex 7
H	Limiting behaviour (saturated sigmoid, $z\to \pm \infty$)	Gradient $\to 0$, "vanishing gradient"	Ex 8
I	Word problem (real-world sensitivity)	Translate story → graph → adjoint	Ex 9
J	Exam twist (branch that cancels: $u-u$)	Adjoints from two paths cancel to $0$	Ex 10

We reuse the adjoint rule throughout:

Below, the small diagram shows the reading direction: values flow right (forward), blame flows left (backward).

---

Example 1 — Single path (Cell A)

Forecast: guess first — is $\partial f/\partial x$ bigger or smaller than $6$? Write your guess.

1. Forward, store values. $v=2\cdot 0=0$; $u=e^0=1$; $f=3\cdot 1=3$. Why this step? Local derivatives need these numbers (e.g. $\partial u/\partial v=e^v$ needs $v$).
2. Seed $\bar{f}=1$. Why? $\partial f/\partial f=1$ by definition.
3. Edge $u\to f$: $\bar{u}=\bar{f}\cdot \frac{\partial f}{\partial u}=1\cdot 3=3$. Why? $f=3u$ so the local slope is $3$.
4. Edge $v\to u$: $\bar{v}=\bar{u}\cdot \frac{\partial u}{\partial v}=3\cdot e^v=3\cdot 1=3$. Why? $\frac{d}{dv}{e}^v=e^v$, evaluated at the stored $v=0$.
5. Edge $x\to v$: $\bar{x}=\bar{v}\cdot \frac{\partial v}{\partial x}=3\cdot 2=6$. Why? $v=2x$ is linear, slope $2$.

Answer: $\partial f/\partial x=6$.

Verify: analytically $f=3e^{2x}$, so $f^{\prime }=6e^{2x}$, at $x=0$ gives $6e^0=6$. ✓ (Single path = one product of slopes, no summing needed.)

---

Example 2 — Fan-out, must accumulate (Cell B)

Forecast: how many paths does $x$ take to reach $f$? Guess the count before continuing.

Look at the figure: the amber node $x$ splits into two cyan edges — one to $a$, one to $b$. That split is the fan-out. Both branches must contribute blame.

1. Forward. $a=2+1=3$; $b=2^2=4$; $f=3\cdot 4=12$. Why? Store $a,b$ — the product's local derivatives are the other factor.
2. Seed $\bar{f}=1$.
3. Edge $a\to f$: $\bar{a}=\bar{f}\cdot b=1\cdot 4=4$. Why? $\partial (ab)/\partial a=b$.
4. Edge $b\to f$: $\bar{b}=\bar{f}\cdot a=1\cdot 3=3$. Why? symmetric, $\partial (ab)/\partial b=a$.
5. Edge $x\to a$: $\bar{x}+=\bar{a}\cdot 1=4$. Why? $a=x+1$, slope $1$. First contribution.
6. Edge $x\to b$: $\bar{x}+=\bar{b}\cdot 2x=3\cdot 4=12$. Why? $b=x^2$, slope $2x=4$; accumulate — this is the second path. → $\bar{x}=4+12=16$.

Answer: $\partial f/\partial x=16$.

Verify: $f=(x+1)x^2=x^3+x^2$, so $f^{\prime }=3x^2+2x$; at $x=2$: $12+4=16$. ✓ If you had overwritten instead of accumulating you'd get $12$ — wrong. This is the classic mistake.

---

Example 3 — Dead branch (Cell C)

Forecast: does $y$ affect $f$ at all? Guess $\bar{y}$.

1. Forward. $f=\sin (0)=0$; $q=0+y$ is computed but unused by $f$. Why? We evaluate the whole program, but only what feeds $f$ matters for blame.
2. Seed $\bar{f}=1$.
3. Edge $x\to f$: $\bar{x}=\bar{f}\cdot \cos x=1\cdot 1=1$. Why? $\frac{d}{dx}\sin x=\cos x$, at $x=0$ is $1$.
4. Node $q$: it has no child on the path to $f$, so $\bar{q}=0$. Why? Blame only flows backward from the output; $q$ received no seed and no adjoint from any child.
5. Edge $y\to q$: $\bar{y}+=\bar{q}\cdot 1=0\cdot 1=0$.

Answer: $\bar{y}=0$.

Verify: $f=\sin x$ contains no $y$, so $\partial f/\partial y=0$ for all $x,y$. ✓ (Autograd sets adjoints of unreachable nodes to zero automatically — they simply never get pushed.)

---

Example 4 — Multiply a node by itself (Cell D)

Forecast: slope of $x^2$ is $2x=10$. Will autograd give $10$ or accidentally $5$?

1. Forward. Treat the two inputs of the multiply as $u\to f$ and $w\to f$ where $u=w=x$. $f=5\cdot 5=25$. Why? A multiply node has two input slots; both happen to be fed by $x$ — that is a fan-out.
2. Seed $\bar{f}=1$.
3. Left leg $u\to f$: $\bar{u}=\bar{f}\cdot w=1\cdot 5=5$. Why? $\partial (uw)/\partial u=w$.
4. Right leg $w\to f$: $\bar{w}=\bar{f}\cdot u=1\cdot 5=5$. Why? $\partial (uw)/\partial w=u$.
5. Edge $x\to u$: $\bar{x}+=5\cdot 1=5$.
6. Edge $x\to w$: $\bar{x}+=5\cdot 1=5$. Accumulate. → $\bar{x}=10$.

Answer: $\partial f/\partial x=10$.

Verify: $f=x^2$, $f^{\prime }=2x=10$ at $x=5$. ✓ The factor of $2$ comes entirely from summing both legs — the derivative rule for $x^2$ is autograd's fan-out at work.

---

Example 5 — Zero input kills the gradient (Cell E)

Forecast: one of these adjoints will be zero. Which one, and why?

1. Forward. $a=7$; $b=0$; $f=7\cdot 0=0$.
2. Seed $\bar{f}=1$.
3. Edge $a\to f$: $\bar{a}=\bar{f}\cdot b=1\cdot 0=0$. Why? $\partial (ab)/\partial a=b=0$ here.
4. Edge $b\to f$: $\bar{b}=\bar{f}\cdot a=1\cdot 7=7$. Why? $\partial (ab)/\partial b=a=7$.
5. $\bar{x}=\bar{a}\cdot 1=0$; $\bar{y}=\bar{b}\cdot 1=7$.

Answer: $\bar{x}=0,\bar{y}=7$.

Verify: $\partial f/\partial x=y=0$ and $\partial f/\partial y=x=7$ at $(7,0)$. ✓ A zero value on one input zeroes the other input's gradient — a degenerate but perfectly valid case. Frameworks handle it with no special code.

---

Example 6 — Full neuron chain (Cell F)

Forecast: since $t=0$ and $\hat{y}>0$, the loss wants to decrease $\hat{y}$. Will $\partial L/\partial w$ be positive or negative?

1. Forward. $z=0.5\cdot 2+0=1$; $\hat{y}=\sigma (1)=\frac{1}{1+e^{-1}}\approx 0.7311$; $L=\frac{1}{2}(0.7311-0{)}^2\approx 0.2672$. Why store $z,\hat{y}$? ${\sigma }^{\prime }$ needs $\hat{y}$; the linear slope needs $x$.
2. Seed $\bar{L}=1$.
3. Edge $\hat{y}\to L$: $\bar{\hat{y}}=(\hat{y}-t)=0.7311$. Why? $\frac{d}{d\hat{y}}\frac{1}{2}(\hat{y}-t{)}^2=\hat{y}-t$.
4. Edge $z\to \hat{y}$: $\bar{z}=\bar{\hat{y}}\cdot \sigma (z)(1-\sigma (z))=0.7311\cdot (0.7311)(0.2689)\approx 0.1437$. Why? ${\sigma }^{\prime }=\sigma (1-\sigma )$, the sigmoid identity.
5. Edge $w\to z$: $\bar{w}=\bar{z}\cdot x=0.1437\cdot 2\approx 0.2874$. Why? $z=wx+b$, so $\partial z/\partial w=x$.

Answer: $\partial L/\partial w\approx 0.2874$ (positive).

Verify: positive gradient means Gradient descent will decrease $w$, lowering $z$, lowering $\hat{y}$ toward $t=0$ — exactly what we want. Numerically the closed form $(\hat{y}-t)\hat{y}(1-\hat{y})x$ gives the same $0.2874$. ✓

---

Example 7 — Broadcast / summed outputs (Cell G)

Forecast: $f=6x$, so the answer is obviously $6$ — but watch how the adjoints stack.

1. Forward. $u_1=x,u_2=2x,u_3=3x$; $f=x+2x+3x=6x$.
2. Seed $\bar{f}=1$.
3. Sum node: $\partial f/\partial u_i=1$ for each term. So ${\bar{u}}_1={\bar{u}}_2={\bar{u}}_3=1$. Why? Derivative of a sum w.r.t. each summand is $1$.
4. Edge $x\to u_1$: $\bar{x}+=1\cdot 1=1$.
5. Edge $x\to u_2$: $\bar{x}+=1\cdot 2=2$.
6. Edge $x\to u_3$: $\bar{x}+=1\cdot 3=3$. → $\bar{x}=1+2+3=6$.

Answer: $\bar{x}=6$.

Verify: $f=6x$, $f^{\prime }=6$. ✓ Broadcasting one variable into many summed uses = sum the adjoints from every use — the general form of the fan-out rule (this is exactly how a bias reused across a batch gets its gradient in PyTorch autograd / TensorFlow GradientTape).

---

Example 8 — Limiting behaviour: saturated sigmoid (Cell H)

Forecast: the sigmoid is nearly flat far from the origin. Guess: will $\bar{z}$ be near $1$ or near $0$?

The figure shows $\sigma (z)$ flattening: at $z=10$ the curve is almost horizontal, so a wiggle in $z$ barely moves $\hat{y}$.

1. Forward. $\hat{y}=\sigma (10)\approx 0.9999546$.
2. Local slope ${\sigma }^{\prime }(10)=\hat{y}(1-\hat{y})\approx 0.9999546\cdot 0.0000454\approx 4.54\times 10^{-5}$. Why? Sigmoid identity again.
3. Edge $z\to \hat{y}$: $\bar{z}=\bar{\hat{y}}\cdot {\sigma }^{\prime }(10)=1\cdot 4.54\times 10^{-5}\approx 4.54\times 10^{-5}$.

Answer: $\bar{z}\approx 4.5\times 10^{-5}$ — essentially zero.

Verify: as $z\to +\infty$, $\sigma \to 1$, $\sigma (1-\sigma )\to 0$, so $\bar{z}\to 0$. Same as $z\to -\infty$ ($\sigma \to 0$). ✓ This is the vanishing-gradient case: saturated neurons pass almost no blame backward — a real reason deep nets stall, and why alternatives to sigmoid exist (Sigmoid and activation functions).

---

Example 9 — Word problem: cost sensitivity (Cell I)

Forecast: guess whether cutting the price or the count gives more cost reduction per unit nudge.

1. Model as a graph. $z=p\cdot n$ (multiply), $C=z+f$ (add). Why? Translate the story into add/multiply primitives.
2. Forward. $z=3\cdot 100=300$; $C=300+50=350$.
3. Seed $\bar{C}=1$ (we ask sensitivity of total cost).
4. Edge $z\to C$: $\bar{z}=1\cdot 1=1$; also $\bar{f}=1$. Why? Add node passes blame unchanged.
5. Edge $p\to z$: $\bar{p}=\bar{z}\cdot n=1\cdot 100=100$. Why? $\partial z/\partial p=n$.
6. Edge $n\to z$: $\bar{n}=\bar{z}\cdot p=1\cdot 3=3$. Why? $\partial z/\partial n=p$.

Answer: $\bar{p}=100$ dollars-per-(dollar-of-price), $\bar{n}=3$ dollars-per-part.

Verify: dropping price by $1 saves $100 (all 100 parts get cheaper); dropping count by 1 saves $3 (one part's price). Cutting price moves cost far faster. Units check: $\bar{p}$ has units $/($/part)·... = dollars saved per unit price change. ✓ This is exactly a gradient in cost space.

---

Example 10 — Exam twist: branches that cancel (Cell J)

Forecast: $f=x^2-x^2=0$ always. But do the intermediate adjoints vanish, or do they cancel at the end? Guess.

1. Forward. $a=16$; $b=16$; $f=16-16=0$.
2. Seed $\bar{f}=1$.
3. Edge $a\to f$: $\bar{a}=\bar{f}\cdot (+1)=1$. Why? $\partial (a-b)/\partial a=+1$.
4. Edge $b\to f$: $\bar{b}=\bar{f}\cdot (-1)=-1$. Why? $\partial (a-b)/\partial b=-1$ — the sign matters.
5. Edge $x\to a$: $\bar{x}+=\bar{a}\cdot 2x=1\cdot 8=8$.
6. Edge $x\to b$: $\bar{x}+=\bar{b}\cdot 2x=-1\cdot 8=-8$. Accumulate. → $\bar{x}=8+(-8)=0$.

Answer: $\bar{x}=0$.

Verify: $f=0$ identically, so $f^{\prime }=0$. ✓ Note the adjoints did not vanish individually ($\bar{a}=+1,\bar{b}=-1$); they cancelled at the accumulation step. This is the subtractive twin of Example 2 — same fan-out, opposite signs — and shows why keeping edge signs correct is critical.

---

Recall Self-test on the cells

Which cell is $x\cdot x$? ::: Cell D — fan-out in disguise, both legs summed to give $2x$. Why is Example 10's answer zero even though $\bar{a},\bar{b}\ne 0$? ::: The two path adjoints ($+8$ and $-8$) cancel at the accumulation step. In Example 8, what physical phenomenon does $\bar{z}\approx 0$ represent? ::: The vanishing gradient of a saturated sigmoid. In the word problem, why is $\bar{p}>\bar{n}$? ::: $\bar{p}=n=100$ (many parts share the price) while $\bar{n}=p=3$.