Autograd and computational graphs in PyTorch: Question bank

This is a rapid-fire trap bank for Autograd and computational graphs in PyTorch. Each item hides a specific misconception. Cover the answer side, say your reasoning out loud, then reveal. If you can only say "true/false" without the why, you don't know it yet.

Before we start, the words we lean on everywhere below, defined plainly:

Definition The words this page assumes

A tensor is just PyTorch's name for a grid of numbers (a single number, a list, a matrix, or higher). See PyTorch tensors.
The computational graph is the recorded chain of operations linking your inputs to your final number, built while the forward pass runs.
A gradient is a slope: "if I nudge this input a tiny bit, how much does the output move?" .backward() fills in every slope by walking the graph backward (the backpropagation algorithm).
==requires_grad== is a true/false flag on a tensor. When it is True, PyTorch watches every operation you do with that tensor and records it into the graph so it can later compute that tensor's slope. When False, the tensor is treated as a fixed constant — no watching, no slope.
Throughout this page, when we write a symbol like $L$ (or "the loss") we mean the single final number you called .backward() on. Its own slope with respect to itself is trivially $1$ — nudging a number by a tiny amount changes it by exactly that amount — which is the seed the backward walk starts from.
An optimizer is the small helper object (e.g. torch.optim.SGD) that owns your parameters and, on command, nudges them downhill using their .grad. Its optimizer.zero_grad() method resets every parameter's .grad to zero. More in gradient descent variants.

The picture below is the mental model behind almost every answer: a forward chain of tensors and operations, with slopes flowing back along the same edges.

Figure — Autograd and computational graphs in PyTorch

True or false — justify

TF1. A tensor created with requires_grad=False can never end up in the computational graph.

False. If it is combined with a requires_grad=True tensor, the resulting tensor tracks gradients — the graph forms wherever at least one input needs a gradient.

TF2. Calling .backward() on a tensor that is not a scalar works with no extra arguments.

False. The backward walk needs a single starting slope: for the final number

L

(the one you call .backward() on) that seed is

\frac{\partial L}{\partial L} = 1

. A non-scalar has many outputs, so there is no single seed — you must pass a gradient= vector of the same shape saying how to weight each element.

TF3. .grad holds the value from your most recent .backward() call.

False. .grad accumulates — each .backward() adds to whatever was already there. That is why x.grad.zero_() exists.

TF4. A leaf tensor (one you created directly, not from an operation) always has grad_fn=None.

True. grad_fn records which operation produced a tensor. A leaf was produced by you, not by an op, so it has no grad_fn; its gradient lands in .grad instead.

TF5. Reassigning x = x + 1 on a requires_grad=True leaf keeps x a leaf.

False. The name x now points to a new tensor made by the + op (it has a grad_fn, so it is a non-leaf). Its .grad stays None; only the original leaf tensor (which you can no longer reach by the name x) would ever accumulate a gradient. See the sketch below.

TF6. Every time you run the forward pass, PyTorch reuses the same graph.

False. PyTorch is define-by-run: a fresh graph is built each forward pass and (by default) freed after .backward(). This is what lets control flow like if/loops change the graph run to run.

TF7. with torch.no_grad(): makes tensors lose their existing .grad values.

False. It only stops new operations from being recorded. Existing .grad values are untouched; you just can't backprop through anything computed inside the block.

TF8. A tensor's .grad and the tensor's data have the same shape.

True. A gradient answers "how does the loss change per unit change in each element", so there is exactly one slope per element — same shape as the tensor.

Spot the error

SE1. "I'll call loss.backward() every mini-batch and my weights will update correctly without zeroing grads." — What breaks?

Without optimizer.zero_grad() (the optimizer's reset method, or the raw .grad.zero_()), gradients from previous batches keep piling on, so each step uses a stale sum of many batches instead of the current one. The sketch below shows the pile-up. See gradient descent variants.

SE2. "y.backward() then y.backward() again on the same y — should be fine." — What breaks?

The second call errors: RuntimeError: graph freed. The graph is used-once by default. Pass retain_graph=True on the first call only if you truly need to backprop twice.

SE3. "I set requires_grad=True on my input images so the model trains." — What's confused here?

Training updates parameters (weights/biases), not inputs. Inputs normally have requires_grad=False; you set it on the learnable tensors, which torch.nn handles for you (see nn.Module).

SE4. "I did x.grad += something to nudge gradients, and got a graph-tracking error." — Why?

In-place edits on a tensor that is needed for backward can corrupt saved values. Wrap manual .grad edits in with torch.no_grad(): so they aren't recorded.

SE5. "My detached tensor still updates the encoder through backprop." — Why is that impossible?

.detach() returns a tensor sharing the same data but cut off from the graph (no grad_fn). Gradients stop dead there, so nothing upstream (the encoder) receives them — the sketch below shows the severed edge.

SE6. "I converted a tensor with .numpy() mid-graph and gradients vanished." — What happened?

.numpy() only works on tensors outside the graph, so you were forced to .detach() first (or it errored). NumPy has no autograd, so any path through it is severed.

SE7. "ReLU has no derivative at 0, so autograd crashes there." — Why doesn't it?

PyTorch just picks a subgradient (it uses 0 for the input-

\leq 0

branch). It won't crash; it silently chooses a valid value, which is standard for ReLU at the kink.

Why questions

WHY1. Why does the backward pass sum contributions over all children of a node?

A node like

x

can feed several operations at once (say

x^{2}

and

x y

, as in the figure below). Each branch is a separate way

x

influences the loss, and the multivariate chain rule says total slope = sum of the per-branch slopes.

WHY2. Why must the thing you call .backward() on be a scalar (by default)?

The backward walk starts from the loss

L

(the single number you called .backward() on) with seed slope

\frac{\partial L}{\partial L} = 1

. With many outputs there is no single "1" to start from — you'd have to say how much each output matters, i.e. supply a weighting vector.

WHY3. Why does PyTorch build the graph during the forward pass rather than ahead of time?

Define-by-run means the graph mirrors whatever Python code actually ran — loops, ifs, variable lengths. This makes debugging and dynamic architectures natural, at the cost of rebuilding each pass.

WHY4. Why does PyTorch free the graph after .backward() by default?

To save memory. The intermediate values saved for backward can be large; releasing them right after use keeps training memory-lean. Techniques like gradient checkpointing push this trade-off even further.

WHY5. Why does gradient accumulation actually help mini-batch training instead of being a bug?

The batch gradient is a sum (then average) of per-sample gradients:

\frac{\partial L _{batch}}{\partial w} = \frac{1}{N} \sum_{i} \frac{\partial L _{i}}{\partial w}

. Accumulation implements that sum for free — you just remember to zero between update steps.

WHY6. Why do frozen layers in transfer learning set requires_grad=False instead of just skipping them?

Setting requires_grad=False tells autograd not to record ops or compute grads for those weights, so backprop stops there — saving compute and guaranteeing they never update.

WHY7. Why can .backward() fill in gradients for parameters you never mentioned in the call?

Because those parameters are wired into the graph through grad_fn edges. The backward walk visits every reachable node with requires_grad=True, not just the ones you named.

Edge cases

EC1. What is x.grad for a requires_grad=True tensor before you ever call .backward()?

It is None. No backward pass has run, so no slope has been computed or accumulated yet.

EC2. A tensor is used in computing the loss but its value happens not to change the loss (e.g. multiplied by a branch that is zero) — what does its .grad come out as?

A genuine 0.0 tensor. It was reached by the backward walk, so it gets a real gradient, and that gradient correctly says "nudging it changes the loss by nothing." (Contrast: a tensor never reached at all stays None, as in EC1.)

EC3. What happens to gradients if the loss is exactly constant (e.g. loss = torch.tensor(3.0) with no graph)?

.backward() errors by default — a constant with no grad_fn has no path back to any parameter to differentiate. (If you truly needed to seed it, you would have to pass an explicit gradient= argument, but with no graph there is still nothing downstream to receive it.)

EC4. During evaluation you forget torch.no_grad(). What still goes wrong even if you never call .backward()?

Every forward op still builds graph nodes and stores intermediates, wasting memory and time. Correctness is fine, but you pay for tracking you never use.

EC5. In batch normalization, running mean/variance buffers are updated during forward. Do they get gradients?

No. Those buffers are marked non-learnable (no requires_grad); they are updated by running statistics, not by backprop, so autograd leaves them alone.

EC6. In Python you write x = torch.tensor([2.0], requires_grad=True) then y = x ** 2 (so y is computed from x), and you call y.backward() twice with retain_graph=True and no zeroing in between. What is x.grad after the second call?

8

. First call adds

\frac{d y}{d x} = 2 x = 4

; second call adds another

4

; accumulation gives

4 + 4 = 8

. This shows exactly why zeroing matters.

EC7. A tensor is built from two branches where one branch used .detach(). Does the other branch still get gradients?

Yes. Detaching only severs that branch; the live branch keeps its grad_fn path, so gradients still flow through it normally.

Recall Fastest self-check

If you can answer these three you own the page: What does .grad do between two .backward() calls without zeroing? Why must the backward root be scalar? What does .detach() cut? .grad accumulates (adds), the root needs a single seed slope of 1, and .detach() cuts the graph link so gradients stop there. ::: Correct if you said accumulate / scalar seed / cut-the-link.