Computational graphs and autograd: Exercises — AI-ML deep dive

Throughout, the adjoint of a node $v$ is written $\overset{v}{ˉ}$ and means "how much does the final scalar output change when $v$ wiggles a little." The rule we keep applying is the reverse-mode update: for each edge $u \to v$ , $\overset{u}{ˉ} + = \overset{v}{ˉ} \cdot \frac{\partial v}{\partial u}, seed \overset{ˉ}{f} = 1.$

Level 1 — Recognition

Exercise 1.1 (L1)

Here is a small computational graph for $f = (x + y) \cdot z$ . Name the nodes, name the operations on the edges, and confirm it is a DAG (no cycles).

Figure — Computational graphs and autograd

Recall Solution

Nodes: the inputs $x, y, z$ ; the intermediate $a = x + y$ ; the output $f = a \cdot z$ . Edge operations:

$x \to a$ and $y \to a$ : the add operation ( $a = x + y$ ).
$a \to f$ and $z \to f$ : the multiply operation ( $f = a \cdot z$ ). DAG? Every edge points strictly "forward" (input → intermediate → output). You can never return to a node you left, so there are no cycles. Directed + acyclic = DAG. ✓

Exercise 1.2 (L1)

For $f = a \cdot b$ , write down the two local derivatives $\frac{\partial f}{\partial a}$ and $\frac{\partial f}{\partial b}$ — the numbers autograd stores on those edges.

Recall Solution

$f = a \cdot b$ is a product. Treat the other factor as a constant: $\frac{\partial f}{\partial a} = b, \frac{\partial f}{\partial b} = a .$ Why these matter: in the backward pass the multiply edge sends $\overset{a}{ˉ} + = \overset{ˉ}{f} \cdot b$ and $\overset{ˉ}{b} + = \overset{ˉ}{f} \cdot a$ . The multiply node literally "swaps" the values.

Level 2 — Application

Exercise 2.1 (L2)

Function: $f (x, y) = (x + y) \cdot z$ with $x = 2, y = 3, z = 4$ . Do a full forward pass (store all values) and a full backward pass to get $\overset{x}{ˉ}, \overset{y}{ˉ}, \overset{z}{ˉ}$ .

Recall Solution

Forward (fill values):

$a = x + y = 2 + 3 = 5$
$f = a \cdot z = 5 \cdot 4 = 20$

Backward (blame each node):

Seed $\overset{ˉ}{f} = 1$ .
Multiply edge: $\overset{a}{ˉ} = \overset{ˉ}{f} \cdot z = 1 \cdot 4 = 4$ , and $\overset{z}{ˉ} = \overset{ˉ}{f} \cdot a = 1 \cdot 5 = 5$ .
Add edge ( $a = x + y$ , so $\partial a / \partial x = \partial a / \partial y = 1$ ): $\overset{x}{ˉ} = \overset{a}{ˉ} \cdot 1 = 4$ , $\overset{y}{ˉ} = \overset{a}{ˉ} \cdot 1 = 4$ .

Answers: $\overset{x}{ˉ} = 4, \overset{y}{ˉ} = 4, \overset{z}{ˉ} = 5$ . Sanity check by hand: $f = (x + y) z$ , so $\partial f / \partial x = z = 4$ ✓, $\partial f / \partial z = x + y = 5$ ✓.

Exercise 2.2 (L2)

For $g = exp (u)$ with $u = 0.7$ , compute $g$ and the adjoint $\overset{u}{ˉ}$ given the incoming adjoint $\overset{g}{ˉ} = 2$ .

Recall Solution

Why exp is special: its derivative is itself, $\frac{d}{d u} e^{u} = e^{u}$ .

Forward: $g = e^{0.7} \approx 2.01375$ .
Local derivative of the $exp$ edge: $\partial g / \partial u = g = 2.01375$ .
Backward: $\overset{u}{ˉ} = \overset{g}{ˉ} \cdot g = 2 \cdot 2.01375 = 4.02750$ .

Note the reuse: the backward step reuses the forward value $g$ . That is exactly why the forward pass stores intermediates.

Level 3 — Analysis

Exercise 3.1 (L3)

Function with a shared input: $f = sin (x) \cdot (x + y)$ at $x = 1, y = 2$ . Notice $x$ feeds two branches. Compute $\overset{x}{ˉ}$ and $\overset{y}{ˉ}$ by the backward pass, and show the accumulation explicitly.

Recall Solution

Break into primitives: $b = sin x$ , $a = x + y$ , $f = b \cdot a$ . Forward:

$b = sin 1 \approx 0.841471$
$a = 1 + 2 = 3$
$f = 0.841471 \cdot 3 = 2.524413$

Backward:

$\overset{ˉ}{f} = 1$ .
Multiply: $\overset{ˉ}{b} = \overset{ˉ}{f} \cdot a = 3$ , $\overset{a}{ˉ} = \overset{ˉ}{f} \cdot b = 0.841471$ .
Path through $b = sin x$ : $\overset{x}{ˉ} + = \overset{ˉ}{b} \cdot cos x = 3 \cdot cos 1 = 3 \cdot 0.540302 = 1.620907$ .
Path through $a = x + y$ : $\overset{x}{ˉ} + = \overset{a}{ˉ} \cdot 1 = 0.841471$ .
Accumulate: $\overset{x}{ˉ} = 1.620907 + 0.841471 = 2.462378$ .
$\overset{y}{ˉ} = \overset{a}{ˉ} \cdot 1 = 0.841471$ .

Check against the closed form $\frac{\partial f}{\partial x} = cos x (x + y) + sin x$ : $= 0.540302 \cdot 3 + 0.841471 = 2.462378$ ✓.

Exercise 3.2 (L3)

Explain, using this same graph, why reverse mode gives both $\overset{x}{ˉ}$ and $\overset{y}{ˉ}$ in one backward pass, while forward mode would need one pass per input.

Recall Solution

Reverse mode seeds the single output ( $\overset{ˉ}{f} = 1$ ) and propagates sensitivities backward. Every input's adjoint drops out of that one sweep — here $\overset{x}{ˉ}$ and $\overset{y}{ˉ}$ both. Cost $\propto$ number of outputs (one).
Forward mode seeds one input's perturbation (e.g. $\overset{x}{˙} = 1, \overset{y}{˙} = 0$ ) and pushes it forward to see $\dot{f} = \partial f / \partial x$ . To also get $\partial f / \partial y$ you must re-seed ( $\overset{x}{˙} = 0, \overset{y}{˙} = 1$ ) and run again. Cost $\propto$ number of inputs.
For a loss (one output, many weights), reverse mode wins massively. See Automatic differentiation (forward vs reverse mode).

Level 4 — Synthesis

Exercise 4.1 (L4) — one neuron end-to-end

$L = \frac{1}{2} (\overset{y}{^} - t)^{2}$ , $\overset{y}{^} = σ (z)$ , $z = w x + b$ , with $σ (z) = \frac{1}{1 + e ^{- z}}$ . Values: $w = 0.5, x = 2, b = - 0.3, t = 1$ . Forward + backward to get $\partial L / \partial w$ , $\partial L / \partial b$ , $\partial L / \partial x$ .

Recall Solution

Forward:

$z = 0.5 \cdot 2 + (- 0.3) = 0.7$
$\overset{y}{^} = σ (0.7) = \frac{1}{1 + e ^{- 0.7}} \approx 0.668188$
$L = \frac{1}{2} (0.668188 - 1)^{2} = \frac{1}{2} (- 0.331812)^{2} \approx 0.055050$

Backward (chain, each edge's local derivative):

$\overset{ˉ}{\overset{y}{^}} = \frac{\partial L}{\partial y ^} = \overset{y}{^} - t = - 0.331812$ .
Sigmoid edge: $σ^{'} (z) = σ (z) (1 - σ (z)) = 0.668188 \cdot 0.331812 = 0.221718$ . $\overset{z}{ˉ} = \overset{ˉ}{\overset{y}{^}} \cdot σ^{'} (z) = - 0.331812 \cdot 0.221718 = - 0.073568$ .
Linear edge $z = w x + b$ :
- $\overset{w}{ˉ} = \overset{z}{ˉ} \cdot x = - 0.073568 \cdot 2 = - 0.147136$
- $\overset{ˉ}{b} = \overset{z}{ˉ} \cdot 1 = - 0.073568$
- $\overset{x}{ˉ} = \overset{z}{ˉ} \cdot w = - 0.073568 \cdot 0.5 = - 0.036784$

Answers: $\partial L / \partial w \approx - 0.147136, \partial L / \partial b \approx - 0.073568, \partial L / \partial x \approx - 0.036784$ . Interpretation: all negative → increasing $w, b, x$ decreases the loss, so a Gradient descent step (which subtracts the gradient) would increase them. Makes sense: $\overset{y}{^} < t$ , we want $z$ bigger.

Exercise 4.2 (L4)

Take one gradient-descent step on $w$ with learning rate $η = 0.1$ , using $\overset{w}{ˉ}$ from 4.1. Compute the new $w$ .

Recall Solution

Update rule: $w \leftarrow w - η \overset{w}{ˉ}$ . $w_{new} = 0.5 - 0.1 \cdot (- 0.147136) = 0.5 + 0.0147136 = 0.5147136.$ $w$ increased, exactly the direction that pushes $\overset{y}{^}$ toward $t = 1$ . See Loss functions and Gradient descent.

Level 5 — Mastery

Exercise 5.1 (L5) — deeper shared path

$f = a^{2} + sin (a)$ , where $a = x y$ , at $x = 3, y = 2$ . Here $a$ feeds two children ( $a^{2}$ and $sin a$ ), and $a$ itself depends on both $x$ and $y$ . Get $\overset{x}{ˉ}$ and $\overset{y}{ˉ}$ .

Recall Solution

Primitives: $a = x y$ ; $p = a^{2}$ ; $q = sin a$ ; $f = p + q$ . Forward: $a = 3 \cdot 2 = 6$ ; $p = 36$ ; $q = sin 6 \approx - 0.279415$ ; $f = 36 - 0.279415 = 35.720585$ .

Backward:

$\overset{ˉ}{f} = 1$ .
Add node $f = p + q$ : $\overset{p}{ˉ} = 1$ , $\overset{q}{ˉ} = 1$ .
$p = a^{2} \Rightarrow \partial p / \partial a = 2 a$ : $\overset{a}{ˉ} + = \overset{p}{ˉ} \cdot 2 a = 1 \cdot 12 = 12$ .
$q = sin a \Rightarrow \partial q / \partial a = cos a$ : $\overset{a}{ˉ} + = \overset{q}{ˉ} \cdot cos 6 = 1 \cdot 0.960170 = 0.960170$ .
Accumulate: $\overset{a}{ˉ} = 12 + 0.960170 = 12.960170$ .
$a = x y$ : $\overset{x}{ˉ} = \overset{a}{ˉ} \cdot y = 12.960170 \cdot 2 = 25.920340$ ; $\overset{y}{ˉ} = \overset{a}{ˉ} \cdot x = 12.960170 \cdot 3 = 38.880510$ .

Check $\frac{\partial f}{\partial x} = (2 a + cos a) \cdot y$ : $(12 + 0.960170) \cdot 2 = 25.920340$ ✓.

Exercise 5.2 (L5) — design the backward pass yourself

Given $L = (uv + exp (u))$ , $u = 0.5, v = 3$ . Design and run the full backward pass; report $\overset{u}{ˉ}$ (note $u$ appears in both the product and the exponential).

Recall Solution

Primitives: $m = u \cdot v$ ; $e = exp (u)$ ; $L = m + e$ . Forward: $m = 0.5 \cdot 3 = 1.5$ ; $e = e^{0.5} \approx 1.648721$ ; $L = 1.5 + 1.648721 = 3.148721$ .

Backward:

$\overset{ˉ}{L} = 1$ .
Add: $\overset{m}{ˉ} = 1$ , $\overset{e}{ˉ} = 1$ .
Product $m = uv$ : $\overset{u}{ˉ} + = \overset{m}{ˉ} \cdot v = 1 \cdot 3 = 3$ ; $\overset{v}{ˉ} + = \overset{m}{ˉ} \cdot u = 1 \cdot 0.5 = 0.5$ .
Exp $e = e^{u}$ (local deriv $= e$ ): $\overset{u}{ˉ} + = \overset{e}{ˉ} \cdot e = 1 \cdot 1.648721 = 1.648721$ .
Accumulate: $\overset{u}{ˉ} = 3 + 1.648721 = 4.648721$ ; $\overset{v}{ˉ} = 0.5$ .

Check $\frac{\partial L}{\partial u} = v + e^{u} = 3 + 1.648721 = 4.648721$ ✓.