Autograd and computational graphs in PyTorch: Worked examples

This page is the practice deep-dive for Autograd and computational graphs in PyTorch. The parent note built the machine; here we run it on every kind of input it can face. Before any code, we map out the full space of cases so nothing surprises you later.

If a word here feels unfamiliar (gradient, chain rule, .backward()), the parent note defines it — this page assumes you have read it and now want to feel the mechanics through examples.

The scenario matrix

Autograd is a small idea (record operations, replay them backward) but it has many corners. A gradient can be positive, negative, or exactly zero; a graph can be used once or many times; a branch can be alive or cut off. The table below lists every corner. Each later example is tagged with the cell it lands in.

#	Case class	What is special about it	Covered by
A	Positive gradient	derivative $> 0$ : output rises when input rises	Ex 1
B	Negative gradient	derivative $< 0$ : output falls when input rises	Ex 2
C	Zero gradient (dead branch)	derivative $= 0$ at the operating point (ReLU on a negative, saturated point)	Ex 3
D	Multiple paths to one node	a variable feeds two operations — gradients must sum	Ex 4
E	Gradient accumulation across calls	`.backward()` twice without zeroing — the `.grad` adds up	Ex 5
F	Detached / stopped gradient	`.detach()` cuts the graph — no gradient flows past it	Ex 6
G	Non-scalar output	`.backward()` on a vector needs a seed vector (Jacobian-vector product)	Ex 7
H	Real-world word problem	linear layer + loss, compute weight gradients	Ex 8
I	Exam twist / degenerate	`requires_grad=False` leaf, or in-place error	Ex 9

Every numeric answer below is machine-checked in the verify block.

Example 1 — Positive gradient (Cell A)

Steps.

Forward: $z = x^{2} + 3 x = 4 + 6 = 10$ . Why this step? We must run the forward pass so PyTorch records the graph (a PowBackward and an AddBackward node).
Hand-derivative: $\frac{d z}{d x} = 2 x + 3$ . Why this step? Power rule on $x^{2}$ gives $2 x$ ; the linear term $3 x$ gives $3$ . Adds because derivative of a sum is the sum of derivatives.
Plug $x = 2$ : $2 (2) + 3 = 7$ . Why this step? Autograd evaluates the derivative at the current point, not symbolically.

x = torch.tensor([2.0], requires_grad=True)
z = x**2 + 3*x
z.backward()
print(x.grad)   # tensor([7.])

Verify: $7 > 0$ , so $z$ increases with $x$ — matches the intuition that both $x^{2}$ and $3 x$ rise for $x > 0$ . ✓

Example 2 — Negative gradient (Cell B)

Steps.

Forward: $z = 4 (1) - 1 = 3$ . Why? Record the graph.
Derivative: $\frac{d z}{d x} = 4 - 3 x^{2}$ . Why? $4 x \to 4$ ; $x^{3} \to 3 x^{2}$ by the power rule, with the minus sign carried through.
Plug $x = 1$ : $4 - 3 (1) = 1$ . Still positive here. Try $x = 2$ : $4 - 3 (4) = - 8$ — negative. Why the second point? To actually hit Cell B we need a point where the slope is negative; $x = 2$ delivers it.

x = torch.tensor([2.0], requires_grad=True)
z = 4*x - x**3
z.backward()
print(x.grad)   # tensor([-8.])

Verify: at $x = 2$ , increasing $x$ shrinks $z$ (the $- x^{3}$ term dominates). Sign $- 8 < 0$ confirms Cell B. ✓

Example 3 — Zero gradient / dead branch (Cell C)

Figure — Autograd and computational graphs in PyTorch

Steps.

Forward: $y = max (0, - 1.5) = 0$ . Why? ReLU clamps negatives to zero — look at the red flat segment on the left of the figure.
Slope of ReLU: it is $1$ for $x > 0$ and $0$ for $x < 0$ . Why? On the left the graph is a flat horizontal line — flat means zero slope. On the right it is the line $y = x$ — slope $1$ .
At $x = - 1.5$ we sit on the flat part, so $\frac{d y}{d x} = 0$ . Why this matters? A zero gradient means no learning signal flows back through this unit — the "dying ReLU" problem.

x = torch.tensor([-1.5], requires_grad=True)
y = torch.relu(x)
y.backward()
print(x.grad)   # tensor([0.])

Verify: the branch is dead — moving $x$ a tiny bit still gives $y = 0$ , so the change per unit input is $0$ . ✓

Example 4 — One variable, multiple paths (Cell D)

Steps.

Draw the graph (figure): $x$ splits into a square path ( $x^{2}$ ) and a linear path ( $x$ ), then an Add merges them. Why? The red node $x$ has two outgoing edges — this is exactly the multi-path case.
Path 1 gradient: $\frac{\partial}{\partial x} (x^{2}) = 2 x = 6$ . Path 2 gradient: $\frac{\partial}{\partial x} (x) = 1$ . Why separately? The multivariate chain rule says each path from $x$ to $z$ contributes independently.
Sum the paths: $6 + 1 = 7$ . Why sum, not multiply? The paths run in parallel into the same output. Parallel contributions add (they'd multiply only if they were in series, one after the other).

x = torch.tensor([3.0], requires_grad=True)
z = x*x + x
z.backward()
print(x.grad)   # tensor([7.])

Verify: direct derivative $\frac{d}{d x} (x^{2} + x) = 2 x + 1 = 7$ matches the sum-of-paths answer. ✓

Example 5 — Accumulation across two backward calls (Cell E)

Steps.

First backward: x.grad $= 2 x = 4$ . Why? .grad starts empty, so it just stores $4$ .
Second backward (graph retained): PyTorch adds to the existing .grad. Why add? By default .backward() accumulates so that mini-batch gradients sum automatically (parent note, Step 3).
Result: $4 + 4 = 8$ .

x = torch.tensor([2.0], requires_grad=True)
y = x**2
y.backward(retain_graph=True)   # x.grad = 4
y.backward()                    # x.grad = 4 + 4 = 8
print(x.grad)   # tensor([8.])

Verify: $2 \times (2 x) = 2 \times 4 = 8$ . ✓

Example 6 — Detaching to stop gradient flow (Cell F)

Steps.

Compute the value: $a = 4$ , $b = 4$ (detach copies the value, drops the history). Why? .detach() returns a tensor equal in value but with no grad_fn — see the red scissors cutting the edge in the figure.
Now $z = b \cdot x$ where $b$ is treated as a constant $4$ . Why? Autograd cannot see through $b$ back to $x$ ; that path is severed.
So $\frac{d z}{d x} = b = 4$ (the visible $x$ only), not $12$ . Why? Only the un-detached occurrence of $x$ contributes.

x = torch.tensor([2.0], requires_grad=True)
a = x**2
b = a.detach()
z = b * x
z.backward()
print(x.grad)   # tensor([4.])

Verify: treating $b = 4$ as constant, $\frac{d}{d x} (4 x) = 4$ . Detach removed the $8$ that the $x^{2}$ path would have added ( $12 - 4 = 8$ ). ✓

Example 7 — Non-scalar output needs a seed (Cell G)

Steps.

.backward() on a scalar means "start with $\frac{\partial z}{\partial z} = 1$ ". A vector has no single output to start from. Why the error? Autograd needs to know how much each output matters — a weight per component.
Supply that weight vector $v = [1, 1, 1]$ . Autograd computes the Jacobian-vector product $J^{⊤} v$ . Why this tool (Jacobian-vector product) and not a full Jacobian? Storing the whole Jacobian is wasteful; deep learning only ever needs $J^{⊤} v$ (that is exactly what the chain rule multiplies), so autograd computes it directly.
Each component: $\frac{\partial y _{i}}{\partial x _{i}} = 2 x_{i}$ , weighted by $v_{i} = 1$ : $[2 (1), 2 (2), 2 (3)] = [2, 4, 6]$ .

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x**2
y.backward(torch.tensor([1.0, 1.0, 1.0]))
print(x.grad)   # tensor([2., 4., 6.])

Verify: each entry equals $2 x_{i}$ : $[2, 4, 6]$ . Using $v = [1, 1, 1]$ is the same as calling y.sum().backward(). ✓

Example 8 — Real-world word problem: a linear layer (Cell H)

Steps.

Forward: $z = 0.5 (1) + (- 0.3) (2) + 0.1 = 0.5 - 0.6 + 0.1 = 0.0$ . Why? Standard weighted sum — the forward pass of one neuron.
Since $L = z$ , $\frac{\partial L}{\partial z} = 1$ . Why? Identity loss; its derivative is $1$ .
Chain to each weight: $\frac{\partial L}{\partial w _{i}} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial w _{i}} = 1 \cdot x_{i}$ . So $\frac{\partial L}{\partial w _{1}} = x_{1} = 1$ , $\frac{\partial L}{\partial w _{2}} = x_{2} = 2$ , $\frac{\partial L}{\partial b} = 1$ . Why input? In $w_{i} x_{i}$ , differentiating w.r.t. $w_{i}$ leaves $x_{i}$ ; the bias has coefficient $1$ .

x = torch.tensor([1.0, 2.0])
w = torch.tensor([0.5, -0.3], requires_grad=True)
b = torch.tensor([0.1], requires_grad=True)
z = (w * x).sum() + b
z.backward()
print(w.grad)   # tensor([1., 2.])
print(b.grad)   # tensor([1.])

This is the atom of 3.2.04-Backpropagation-algorithm and of every layer in 3.4.01-Building-neural-networks-with-torch.nn.Module.

Verify: $w . grad = [x_{1}, x_{2}] = [1, 2]$ , $b . grad = 1$ , independent of the (zero) value of $z$ . ✓

Example 9 — Exam twist: a non-tracked leaf (Cell I)

Steps.

Only tensors with requires_grad=True are tracked. Why? Autograd records operations on tracked leaves; untracked ones are treated as constants (efficiency — you rarely need gradients w.r.t. fixed data).
$\frac{\partial z}{\partial w} = x = 2$ , so w.grad = 2. Why? In $w \cdot x$ , differentiating w.r.t. $w$ leaves the constant $x$ .
x.grad = None, because $x$ was never marked for tracking. Why not zero? None means "no gradient buffer was ever created", which is different from a computed value of $0$ .

x = torch.tensor([2.0], requires_grad=False)
w = torch.tensor([3.0], requires_grad=True)
z = w * x
z.backward()
print(w.grad)   # tensor([2.])
print(x.grad)   # None

Verify: $w . grad = x = 2$ ; $x . grad$ is None. ✓

Recall

Recall Why do parallel paths from one node

add their gradients? Because they are separate routes into the same output; the multivariate chain rule sums contributions along parallel paths (series paths would multiply). ::: See Example 4.

Recall What is the difference between

x.grad = None and x.grad = 0? None means the tensor was never tracked (requires_grad=False); 0 means it was tracked but the slope at that point is zero (e.g. ReLU on a negative). ::: Examples 3 and 9.

Recall Why does

y.backward() fail for a vector y? A vector has no single scalar to seed $\frac{\partial z}{\partial z} = 1$ ; you must pass a weight vector $v$ so autograd can compute $J^{⊤} v$ . ::: Example 7.

Connections: gradient flow feeds 3.5.02-Gradient-descent-variants; detaching is central to 4.2.03-Batch-normalization running stats and 5.3.01-Transfer-learning frozen backbones; retaining/rebuilding graphs relates to 6.4.02-Gradient-checkpointing. Tensor basics: 3.301-Introduction-to-PyTorch-tensors-and-operations.