Computational graphs and autograd: Question bank

True or false — justify

Forward mode and reverse mode always give the same numerical gradient.

True — they compute the same exact derivatives; they differ only in the order of multiplication and therefore in cost, not in the answer.

Reverse mode is always faster than forward mode.

False — reverse mode wins when outputs ≪ inputs (like a scalar loss with millions of weights); if you had one input and many outputs, forward mode would be cheaper. See Automatic differentiation (forward vs reverse mode).

A computational graph can contain a cycle.

False — it must be a directed acyclic graph; a cycle would mean a value depends on itself, so there'd be no well-defined order to evaluate or differentiate.

Autograd is just a fancy name for symbolic differentiation like Mathematica does.

False — symbolic diff builds a giant closed-form expression; autograd applies local derivative rules numerically at the current point, avoiding expression blow-up.

Autograd computes derivatives by nudging inputs and measuring the output change.

False — that's finite differences, which has step-size error. Autograd uses exact local rules (

\partial (uv) / \partial u = v

, etc.) with no

h

Backpropagation and reverse-mode autodiff are two different algorithms.

False — Backpropagation is reverse-mode autodiff applied to neural networks; same algorithm, domain-specific name.

The backward pass needs the values computed in the forward pass.

True — most local derivatives depend on values (e.g.

\frac{d}{d z} σ (z) = σ (z) (1 - σ (z))

needs

σ (z)

), which is exactly why the forward pass stores them.

If a node feeds into three later operations, its adjoint is the average of the three contributions.

False — it's the sum, not the average; the multivariable chain rule adds one term per path (see Chain rule (multivariable calculus)).

Seeding the output with

\overset{ˉ}{f} = 1

is an arbitrary convention we could set to any number.

False — it's forced:

\overset{ˉ}{f} = \partial f / \partial f = 1

. Any other seed would scale every gradient by that factor.

The gradient of the loss tells you the direction of steepest increase of the loss.

True — that's what the gradient means; Gradient descent steps in the opposite direction to decrease the loss.

Spot the error

"I overwrote

\overset{x}{ˉ} = \overset{ˉ}{b} \cdot cos x

because

x

's derivative is a single number."

The error:

x

flows through two paths (

a = x + y

and

b = sin x

), so you must accumulate (

+ =

) both contributions, not overwrite.

"My loss increased every epoch, but I never called optimizer.zero_grad() — that shouldn't matter."

It matters: adjoints accumulate into .grad, so old batches' gradients pile onto new ones, corrupting the step. You must reset each iteration (see PyTorch autograd / TensorFlow GradientTape).

"I differentiated the whole nested function by hand into one formula, so I don't need a graph."

For a real network that formula is astronomically large and error-prone; the graph's whole point is to reuse local derivatives mechanically without ever writing the monster expression.

"I only stored the output of the forward pass to save memory, then ran backward."

You need the intermediate activations too, because local derivatives depend on them; dropping them breaks the backward pass (this is the memory cost that motivates gradient checkpointing).

"Since

\frac{\partial z}{\partial w} = x

for

z = w x + b

, the gradient w.r.t.

w

is just

x

That's only the local derivative for one edge; the full

\partial L / \partial w

multiplies it by the upstream adjoint

\overset{z}{ˉ}

(the accumulated sensitivity of the loss to

z

"To get

\partial L / \partial w_{1}

and

\partial L / \partial w_{2}

I run two separate backward passes."

One backward pass already produces adjoints for every input simultaneously — that's the core efficiency of reverse mode.

Why questions

Why must the graph be acyclic rather than any directed graph?

A cycle has no valid topological order, so neither the forward evaluation nor the reverse accumulation would have a well-defined sequence.

Why does reverse mode pair so perfectly with training neural nets specifically?

Training minimizes a scalar loss over millions of weights — one output, many inputs — which is precisely the regime where one backward pass yields all gradients cheaply.

Why do we differentiate local operations instead of the whole function at once?

We only reliably know derivatives of primitives (

+, \times, exp, sin, σ

); the chain rule then glues these known local pieces into the full derivative.

Why is the sigmoid's derivative

σ (1 - σ)

so convenient for autograd?

Because it's expressed entirely in terms of the already-stored forward value

σ (z)

, so the backward pass reuses it with no new exp computation (see Sigmoid and activation functions).

Why does adding a node with a huge adjoint but tiny local derivative still possibly give a small gradient?

The contribution is the product

\overset{v}{ˉ} \cdot \frac{\partial v}{\partial u}

; a near-zero local derivative shrinks it regardless of how large the upstream adjoint is — this is exactly how saturated activations cause vanishing gradients.

Why can't autograd give the gradient of a function that has an if-branch we never executed?

Autograd only records the operations that actually ran; the untaken branch never entered the graph, so it has no edges to differentiate through.

Edge cases

What is the adjoint of a node whose value is never used to compute the output?

Zero — it has no path to the output, so no term in the chain-rule sum, so wiggling it changes nothing.

For the constant node in

z = w x + b

, what is

\partial z / \partial (constant 2)

if we hardcoded a literal 2?

There is no derivative — a hardcoded literal isn't a graph node with requires_grad, so autograd never tracks or differentiates it.

z = 0

, is the sigmoid's local derivative maximal or minimal?

Maximal (

σ (0) = 0.5

, so

σ^{'} (0) = 0.25

); far from zero the derivative shrinks toward

0

, causing tiny adjoints (vanishing gradient).

If the loss is exactly flat at the current point (gradient all zeros), what does one backward pass return?

All-zero adjoints for the inputs — a valid answer meaning "no direction improves the loss locally," which Gradient descent reads as a stationary point.

What happens to the accumulated gradient of a weight shared across two layers (weight tying)?

Its single adjoint sums the contributions from both usage sites — the accumulation rule handles shared parameters automatically, no special code needed.

Does detaching a node (stopping gradient flow) change the forward value?

No — detach only cuts the backward edges so no adjoint flows through it; the forward number computed is identical.

Recall One-line summary

Most autograd mistakes reduce to forgetting that adjoints accumulate over paths, that the backward pass reuses stored forward values, and that reverse mode is a cost choice (outputs ≪ inputs) not a correctness one.