Exercises — Autograd and computational graphs in PyTorch

The figure below draws the graph, but here is the same thing in words so you can follow even without the picture: there are three boxes. Two boxes on the left — x and y — are the inputs you typed in; these are the nodes that hold data. One box on the right — z — is the result of the + operation. The two arrows (edges) run from x and from y into z; each arrow means "this data flows into that operation." Data flows left-to-right in the forward pass, and gradients will later flow right-to-left.

(a) Leaves with a .grad: ==x and y==. A leaf is a tensor you created directly (not the output of an operation) that has requires_grad=True. In the picture, leaves are the two boxes with no arrow pointing into them. Both x and y were typed in by hand, so both are leaves and both will hold a gradient after .backward().

(b) z.grad_fn: z is the result of an addition — the box that has arrows pointing into it — so PyTorch attaches grad_fn=<AddBackward0>. This little object is the "recipe card" that remembers how z was made, so it can later be undone.

(c) Does x have a grad_fn? ==No — it is None==. x was not computed from anything; it is a raw input (no arrow enters its box). Only tensors that come out of an operation carry a grad_fn. Leaves do not.

• (i) a → Not tracked. Default requires_grad=False.
• (ii) b → Tracked. You explicitly asked for it.
• (iii) c → Not tracked. .detach() returns a copy that shares the data but is cut off from the graph — its requires_grad is False. Think of it as a photocopy of the number with no memory of where it came from.
• (iv) d → Tracked. If any input to an operation requires grad, the output requires grad. Since b requires grad, d does too, with grad_fn=<MulBackward0>.

What we do: differentiate term by term. $\frac{dz}{dx}=\frac{d}{dx}\left(x^2\right)+\frac{d}{dx}\left(5x\right)=2x+5$

Why $2x$ for $x^2$? The derivative measures "output change per tiny input nudge." Let $h$ be that tiny nudge — a small number we imagine adding to $x$ and then shrink toward zero (it is the "hair" we push $x$ by). Nudging $x$ to $x+h$ changes $x^2$ into $(x+h{)}^2=x^2+2xh+h^2$, a change of $2xh+h^2$. Because $h$ is tiny, $h^2$ is tinier still and negligible, so the change is about $2xh$ — meaning the rate (change per unit of $h$) is $2x$. Likewise, nudging $x$ changes $5x$ by exactly $5h$, rate $5$.

Plug in $x=3$: $2(3)+5=\boxed{11}$.

PyTorch prints tensor([11.]).

Here x feeds two operations at once — it goes into xy and into x^2. In words, the graph has five boxes: input x, input y, an operation box x * y, an operation box x ** 2, and the final z = sum. There are two arrows leaving x — one into x * y, one into x ** 2 — and that fan-out is the whole point: when gradients flow back, each arrow carries its own share and we add them. The figure shows this fan-out:

Gradient w.r.t. x — sum both branches (multivariate chain rule): $\frac{\partial z}{\partial x}=\underset{\text{from }xy}{\underbrace{y}}+\underset{\text{from }{x}^2}{\underbrace{2x}}=3+2(2)=\boxed{7}$

Gradient w.r.t. y — y only appears in xy (only one arrow leaves y): $\frac{\partial z}{\partial y}=x=\boxed{2}$

So x.grad = tensor([7.]), y.grad = tensor([2.]).

The subtle bit: ReLU is $\max (0,z)$. Its slope is $1$ for $z>0$ and $0$ for $z<0$, but at exactly $z=0$ there is a corner — no single slope exists mathematically. PyTorch makes a convention: at $z=0$ it uses the left derivative, giving ${\text{ReLU}}^{\prime }(0)=0$.

So the chain gives $\frac{dL}{dx}=\underset{1}{\underbrace{\frac{dL}{dy}}}\cdot \underset{0\text{ at }z=0}{\underbrace{{\text{ReLU}}^{\prime }(z)}}\cdot \underset{-1}{\underbrace{\frac{dz}{dx}}}=1\cdot 0\cdot (-1)=\boxed{0}$

x.grad = tensor([0.]).

What .backward() does to .grad: it does not overwrite — it adds: $\text{x.grad}\leftarrow \text{x.grad}+\frac{dy}{dx}$

Each pass: $\frac{dy}{dx}=2x=2(2)=4$. There is no .zero_(), so three passes add up: $4+4+4=\boxed{12}$

x.grad = tensor([12.]). To get the "expected" 4, you must call x.grad.zero_() at the top of each loop. This is exactly why training loops always contain optimizer.zero_grad().

Why it errors: PyTorch builds the graph dynamically and, to save memory, frees the intermediate buffers as soon as the first .backward() finishes. The second call finds the graph already destroyed → RuntimeError: Trying to backward through the graph a second time.

Fix: ask PyTorch to keep the graph alive on the first call:

y.backward(retain_graph=True)
y.backward()   # now works

Numeric check: $\frac{dy}{dx}=3x^2=3(9)=27$ per call. After two accumulated backwards (no zeroing) x.grad = tensor([54.]).

The cut: b = a.detach() makes b a constant (value $4$) with no path back to x. So although a came from x, autograd treats b as a frozen number.

The only live path from c to x is the explicit x in c = b * x: $c=4\cdot x\Rightarrow \frac{dc}{dx}=4$

The would-be second contribution through b (which secretly holds $x^2$) is severed. So x.grad = tensor([4.]), not the $\frac{d}{dx}(x^2\cdot x)=\frac{d}{dx}{x}^3=3x^2=12$ you'd get without the detach.

Forward. $z_1=1(0.5)+2(0.2)+0.1=1.0,z_2=1(-0.3)+2(0.4)-0.2=0.3$ Both positive, so ReLU passes them through: $y=[1.0,0.3]$, and $L=1.0+0.3=1.3$.

Backward through the sum. $L=y_1+y_2\Rightarrow \frac{\partial L}{\partial y}=[1,1]$.

Backward through ReLU. Both $z_i>0$ so ${\text{ReLU}}^{\prime }=1$: $\frac{\partial L}{\partial z}=[1,1]$.

Backward into W — why it becomes an outer product. Write out one entry of the forward. Since $z_j={\sum }_i{x}_i{W}_{ij}$, the weight $W_{ij}$ appears in exactly one place: it multiplies $x_i$ inside $z_j$. So nudging $W_{ij}$ only changes $z_j$, at rate $\frac{\partial z_j}{\partial W_{ij}}=x_i$. Chaining with the loss: $\frac{\partial L}{\partial W_{ij}}=\frac{\partial L}{\partial z_j}\cdot \frac{\partial z_j}{\partial W_{ij}}=\frac{\partial L}{\partial z_j}\cdot x_i$ Look at that formula: row index $i$ comes only from $x$, column index $j$ comes only from $\frac{\partial L}{\partial z}$. Two indices that vary independently and get multiplied is exactly the definition of an outer product — a column vector times a row vector fills a whole matrix. Hence, stacking all $i,j$: $\frac{\partial L}{\partial W}=x^{\top }\frac{\partial L}{\partial z}=\left[\begin{array}{c}1\\ 2\end{array}\right][1,1]=\left[\begin{array}{cc}1& 1\\ 2& 2\end{array}\right]$ Shape sanity check: x is $1\times 2$ so $x^{\top }$ is $2\times 1$; $\frac{\partial L}{\partial z}$ is $1\times 2$; $(2\times 1)(1\times 2)=2\times 2$ — the same shape as W. A gradient must always match the shape of the thing it differentiates, and it does.

Backward into x. $\frac{\partial L}{\partial x}=\frac{\partial L}{\partial z}{W}^{\top }$: $[1,1]\left[\begin{array}{cc}0.5& 0.2\\ -0.3& 0.4\end{array}\right]=[0.5-0.3,0.2+0.4]=[0.2,0.6]$

So W.grad = [[1,1],[2,2]] and x.grad = [0.2, 0.6].

Why the vector argument? .backward() needs a scalar to start from ($\frac{\partial L}{\partial L}=1$). When the output y is a vector, PyTorch cannot guess how to collapse it, so you supply the upstream gradient $v$ (the "seed").

The Jacobian $J$. Since each output depends on its own input only ($y_i=x_i^2$, and $y_i$ does not depend on $x_{j\ne i}$), the matrix of all partials $J_{ij}=\frac{\partial y_i}{\partial x_j}$ is diagonal: $J=\left[\begin{array}{ccc}2x_1& 0& 0\\ 0& 2x_2& 0\\ 0& 0& 2x_3\end{array}\right]=\left[\begin{array}{ccc}2& 0& 0\\ 0& 4& 0\\ 0& 0& 6\end{array}\right]$ (Off-diagonal entries are $0$ because $y_1$ ignores $x_2$, etc.)

How the seed contracts against $J$. .backward(v) computes the vector–Jacobian product $v^{\top }J$ (a row vector times the matrix). For our diagonal $J$: $x.\text{grad}=v^{\top }J=[v_1\cdot 2x_1,v_2\cdot 2x_2,v_3\cdot 2x_3]$ With $v=[1,1,1]$: $x.\text{grad}=[2(1),2(2),2(3)]=\boxed{[2,4,6]}$ This equals y.sum().backward(), because summing means "weight every output by $1$."

Edge case — a non-ones seed. If instead $v=[1,0,2]$, the contraction reweights each output: $x.\text{grad}=[1\cdot 2,0\cdot 4,2\cdot 6]=[2,0,12]$ The middle component is killed (its seed is $0$) and the last is doubled (its seed is $2$). So the seed vector lets you ask "how much does each output matter to me?" before backpropagating.

Gradient. $\frac{dL}{dw}=2(w-3)=2(1-3)=-4$.

Why the minus in the update? We want to lower the loss. The gradient points uphill (direction of steepest increase), so we step the opposite way — that's the $-\eta \nabla$ in gradient descent: $w\leftarrow 1.0-0.1(-4)=1.0+0.4=\boxed{1.4}$

The parameter moved from $1.0$ toward the minimum at $w=3$. Good — descent is working.

Code that produces it:

w = torch.tensor([1.0], requires_grad=True)
L = (w - 3) ** 2
L.backward()
with torch.no_grad():
    w -= 0.1 * w.grad
# w is now 1.4

Note the with torch.no_grad(): — the update itself must not be recorded, or autograd would try to differentiate through your optimizer step.

Operation A — with torch.no_grad(): wraps a block; no operation inside builds graph nodes at all. Cheapest — saves memory. Use it for whole frozen forward passes / inference.

with torch.no_grad():
    out = model_backbone(p)   # nothing here is recorded

Operation B — p.detach() produces a single new tensor sharing p's data but severed from the graph, while the rest of your computation still records normally.

frozen = p.detach()           # only this value is cut
out = head(frozen) + other_live_term

Key difference: no_grad() disables tracking for everything in the block; detach() surgically cuts one tensor and leaves the surrounding graph intact. Choose no_grad() for "freeze this whole region," detach() for "freeze just this value but keep the rest live."

2.1 ::: x.grad = 11 2.2 ::: x.grad = 7, y.grad = 2 2.3 ::: x.grad = 0 (ReLU'(0)=0 convention) 3.1 ::: x.grad = 12 (three un-zeroed backwards of 4) 3.2 ::: after fix, x.grad = 54 (two accumulated backwards of 27) 3.3 ::: x.grad = 4 (detach cuts the $x^2$ path) 4.1 ::: W.grad = [[1,1],[2,2]], x.grad = [0.2, 0.6] 4.2 ::: x.grad = [2, 4, 6]; with seed [1,0,2] → [2, 0, 12] 5.1 ::: new w = 1.4