Computational graphs and autograd: Foundations

This page assumes nothing. We collect every symbol, arrow, and word the parent note (Computational graphs and autograd) leans on, and we build each from a picture before you ever need it.

0. What a function really is (the machine picture)

Before any fancy notation, picture a machine with a knob and a screen. You set the knob to a number $x$ ; the machine grinds and shows a number on the screen. We write this as $f (x)$ — read "f of x", meaning "the output the machine $f$ produces when fed $x$ ".

Figure — Computational graphs and autograd

Why the topic needs this: the whole point of the parent note is that a neural network is a function — just a giant one made of many small machines wired together. If you can picture one machine, you can picture a thousand wired in a chain.

1. Variables and the arrow $\to$

The parent writes things like $a = x + y$ and $x \to a$ . Two pieces of notation hide here.

$a = x + y$ names the result of an operation. Think of $a$ as a labelled wire carrying the value $x + y$ . It is a variable — a box that currently holds a number.
$x \to a$ (an arrow) means " $x$ is used to build $a$ ". The arrow points in the direction the value flows: from an ingredient to the thing made from it.

Why the topic needs this: the entire computational graph is drawn from these arrows. Forward = follow arrows; backward = go against them.

2. The graph, the DAG, and "nodes / edges"

Why acyclic matters: if the graph had a loop, a value would depend on itself and there would be no order to compute in. No loops ⇒ there is always a clean left-to-right ordering, which is exactly what lets the forward pass and backward pass proceed step by step.

Why the topic needs this: "computational graph = DAG" is the parent's first definition. Every other idea (forward pass, backward pass) is walking this DAG in one direction or the other.

3. The primitives: $+$ , $\times$ , $sin$ , $exp$

The parent breaks a big function into elementary operations (primitives). You already know most:

Symbol	Plain words	Picture
$x + y$	add the two numbers	two lengths laid end to end
$x \cdot y$ or $x y$	multiply	area of a rectangle, sides $x$ and $y$
$sin x$	height on a spinning circle	vertical position of a point going round a unit circle at angle $x$
$e^{- z}$	the exponential $exp (- z)$	a curve that shrinks fast toward 0 as $z$ grows

Why the topic needs this: we only know how to differentiate simple things. Chop the monster into primitives, differentiate each one locally, then glue — that gluing is the chain rule (Section 6).

4. The derivative $\frac{df}{d x}$ — "how much wiggle out per wiggle in"

This is the heart. Wiggle the input knob by a tiny amount; watch how much the screen moves.

The picture: draw the output curve, zoom into one point until the curve looks like a straight line, and measure its slope = rise over run. A steep slope means a small input wiggle causes a big output wiggle.

The known local derivatives you'll reuse:

5. Partial derivatives $\frac{\partial f}{\partial x}$ — the curly d

When a machine has two knobs $f (x, y)$ , "the derivative" is ambiguous — derivative with respect to which knob? The curly $\partial$ answers that.

Why the topic needs this: a weight in a network is one knob among millions. We want the wiggle-effect of each knob separately — that is a partial derivative for every weight.

6. The chain rule — gluing local wiggles

Two machines in a row: $x \to (make u) \to (make f)$ . If wiggling $x$ moves $u$ by a factor $\frac{\partial u}{\partial x}$ , and wiggling $u$ moves $f$ by a factor $\frac{\partial f}{\partial u}$ , then wiggling $x$ moves $f$ by the product.

When $x$ reaches $f$ through several paths, you add the paths (this is the multivariable version):

Why multiply along a path, add across paths? Multiply: sensitivities compound through a chain (like gear ratios in series). Add: independent routes each contribute their own wiggle, and total wiggle is the sum. This single sentence is the mathematical seed of the entire Backpropagation algorithm — see also Chain rule (multivariable calculus).

7. The adjoint / bar $\overset{v}{ˉ}$ and the $+ =$ notation

The parent writes $\overset{v}{ˉ}$ and $\overset{u}{ˉ} + = \overset{v}{ˉ} \cdot \frac{\partial v}{\partial u}$ . Now everything is in place to read it.

$\overset{v}{ˉ}$ ("v-bar", the adjoint) is short for $\frac{\partial f}{\partial v}$ — the sensitivity of the final output $f$ to node $v$ . It is just a partial derivative with a nickname because we compute so many of them.
$+ =$ (read "plus-equals", borrowed from programming) means "add this to what's already stored", i.e. $\overset{u}{ˉ} \leftarrow \overset{u}{ˉ} + (new bit)$ . It appears because a shared node's total sensitivity is the sum over paths from Section 6.

Why the topic needs this: the whole backward pass is: seed $\overset{ˉ}{f} = 1$ (the output is perfectly sensitive to itself), then push bars backward through each edge, accumulating with $+ =$ . That is reverse-mode automatic differentiation.

8. The sigmoid $σ (z) = \frac{1}{1 + e ^{- z}}$

The neuron example uses $σ$ . Read it as a squashing machine: feed any real number $z$ , get a number strictly between $0$ and $1$ (a "probability-like" value).

Its derivative has a famously clean form used in the parent's neuron: $σ^{'} (z) = σ (z) (1 - σ (z))$ Why the topic cares: this tidy derivative is exactly the "known local derivative" the backward pass multiplies in.

Prerequisite map

Read top-to-bottom: pictures of functions and arrows build the graph; slope and chain rule build the derivative machinery; together they power autograd, which drives training.

Equipment checklist

Test yourself — cover the right side and answer aloud.

I can read

f (x, y)

and say what a two-input function is

A rule turning each pair of inputs into one output; picture: a machine with two knobs and one screen.

I know what the arrow

u \to v

means

v

is built from

u

"; the arrow shows the direction the value flows.

I can state why a computational graph must be acyclic

No loops means every value has a computation order; otherwise a value would depend on itself.

I can explain a derivative in wiggle language

How many times bigger the output wiggle is than a tiny input wiggle; it is the slope of the output curve.

I know why

\partial

is curly, not straight

It signals we wiggle one variable while freezing all others (a partial derivative).

I can state the chain rule for one path and for many paths

One path: multiply local derivatives; many paths: also sum across the paths.

I can decode

\overset{v}{ˉ}

and

+ =

\overset{v}{ˉ} = \partial f / \partial v

(sensitivity of output to

v

);

+ =

adds a new contribution because shared nodes sum over paths.

I know the primitive derivatives of

+, \times, sin, e^{- z}

1

; the other factor;

cos x

;

- e^{- z}

I can describe the sigmoid's shape and range

An S-curve squashing any real number into

(0, 1)

, equal to

\frac{1}{2}

z = 0

; derivative

σ (1 - σ)

Recall Why this page comes before everything else

Every line of the parent note reuses these nine symbols. If any one is fuzzy, the backward-pass algebra becomes memorized noise. Once each symbol is a picture, the parent's chain-rule bookkeeping reads like plain English.