Noesis
3.1.10Neural Network Fundamentals

Computational graphs and autograd

1,938 words9 min readdifficulty · medium4 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHAT is a computational graph?

WHY decompose? Because we only know how to differentiate simple operations easily. If we express the whole computation as a chain of simple ops, the chain rule glues the local derivatives together into the full derivative.

Example function: f(x,y)=(x+y)⋅sin⁡(x)f(x,y) = (x + y)\cdot \sin(x)

Break it into primitives:

  • a=x+ya = x + y
  • b=sin⁡(x)b = \sin(x)
  • f=a⋅bf = a \cdot b
Figure — Computational graphs and autograd

HOW autograd works: forward + backward

Forward pass (WHAT it computes)

Evaluate the graph left-to-right, storing every intermediate value. Why store them? The local derivatives usually depend on the values (e.g. ddxsin⁡x=cos⁡x\frac{d}{dx}\sin x = \cos x needs xx).

Backward pass (the chain rule, systematized)

Deriving the backward rule from first principles

Take f=a⋅bf = a\cdot b, a=x+ya = x+y, b=sin⁡xb = \sin x. We want ∂f∂x\frac{\partial f}{\partial x}.

By the multivariable chain rule, xx influences ff through two paths (x→ax\to a and x→bx\to b): ∂f∂x=∂f∂a∂a∂x+∂f∂b∂b∂x\frac{\partial f}{\partial x} = \frac{\partial f}{\partial a}\frac{\partial a}{\partial x} + \frac{\partial f}{\partial b}\frac{\partial b}{\partial x}

Now compute each local derivative (why local? each is a single primitive):

  • ∂f∂a=b\frac{\partial f}{\partial a} = b (since f=abf=ab)
  • ∂f∂b=a\frac{\partial f}{\partial b} = a
  • ∂a∂x=1\frac{\partial a}{\partial x} = 1 (since a=x+ya=x+y)
  • ∂b∂x=cos⁡x\frac{\partial b}{\partial x} = \cos x

So: ∂f∂x=b⋅1+a⋅cos⁡x=sin⁡x+(x+y)cos⁡x\frac{\partial f}{\partial x} = b\cdot 1 + a\cdot\cos x = \sin x + (x+y)\cos x

WHY reverse mode (not forward mode)?

Worked Example 1 — full trace

Compute f=abf = a b with the values above at x=1,y=2x=1, y=2.

Forward (Why? store values):

  • a=1+2=3a = 1+2 = 3
  • b=sin⁡(1)≈0.841b = \sin(1) \approx 0.841
  • f=3×0.841=2.524f = 3 \times 0.841 = 2.524

Backward:

  1. Seed fˉ=1\bar f = 1. Why? ∂f/∂f=1\partial f/\partial f = 1.
  2. aˉ=fˉ⋅b=1⋅0.841=0.841\bar a = \bar f\cdot b = 1\cdot 0.841 = 0.841. Why? local deriv of abab w.r.t. aa is bb.
  3. bˉ=fˉ⋅a=1⋅3=3\bar b = \bar f\cdot a = 1\cdot 3 = 3. Why? symmetric.
  4. xˉ+=aˉ⋅∂a∂x=0.841⋅1=0.841\bar x \mathrel{+}= \bar a\cdot \frac{\partial a}{\partial x} = 0.841\cdot 1 = 0.841. Why? a=x+ya=x+y.
  5. xˉ+=bˉ⋅cos⁡x=3⋅cos⁡1=3⋅0.540=1.621\bar x \mathrel{+}= \bar b\cdot \cos x = 3\cdot\cos 1 = 3\cdot0.540 = 1.621. Why? b=sin⁡xb=\sin x; accumulate because xx also flows through bb. → xˉ=0.841+1.621=2.462\bar x = 0.841 + 1.621 = 2.462.
  6. yˉ=aˉ⋅1=0.841\bar y = \bar a\cdot 1 = 0.841.

Check against formula: sin⁡1+3cos⁡1=0.841+1.621=2.462\sin 1 + 3\cos 1 = 0.841 + 1.621 = 2.462. ✓

Worked Example 2 — a tiny neuron

L=12(y^−t)2L = \tfrac12(\hat y - t)^2, y^=σ(z)\hat y = \sigma(z), z=wx+bz = wx + b, with σ(z)=11+e−z\sigma(z)=\frac{1}{1+e^{-z}}.

Goal: ∂L/∂w\partial L/\partial w. Chain through the graph: ∂L∂w=(y^−t)⏟∂L/∂y^⋅σ(z)(1−σ(z))⏟∂y^/∂z⋅x⏟∂z/∂w\frac{\partial L}{\partial w} = \underbrace{(\hat y - t)}_{\partial L/\partial\hat y}\cdot\underbrace{\sigma(z)(1-\sigma(z))}_{\partial\hat y/\partial z}\cdot\underbrace{x}_{\partial z/\partial w}

  • Why ∂L/∂y^=y^−t\partial L/\partial\hat y = \hat y - t? Derivative of 12(y^−t)2\tfrac12(\hat y-t)^2.
  • Why σ′(z)=σ(1−σ)\sigma'(z)=\sigma(1-\sigma)? Standard sigmoid identity (derive: σ′=e−z(1+e−z)2=σ(1−σ)\sigma'=\frac{e^{-z}}{(1+e^{-z})^2}=\sigma(1-\sigma)).
  • Why ∂z/∂w=x\partial z/\partial w = x? z=wx+bz=wx+b is linear in ww.

Autograd computes exactly these three numbers in the backward pass and multiplies them. No calculus by you.

Recall Feynman: explain to a 12-year-old

Imagine a machine made of tiny gears (add, multiply, sin). You turn the input knob and out pops an answer. Now you ask: "If I wiggle this one gear a little, how much does the final answer wiggle?" Autograd answers this by starting at the output and walking backward gear-by-gear, at each gear multiplying by how sensitive it is to its neighbor. If one gear connects to two others, you add up both wiggle-effects. Do this once and you learn how every knob affects the answer — so you know which way to nudge each knob to make the answer better.

Active recall

What is a computational graph?
A directed acyclic graph whose nodes are variables and whose edges are elementary operations with known local derivatives; it decomposes a complex function into differentiable primitives.
Why store intermediate values during the forward pass?
Local derivatives usually depend on those values (e.g. d/dx sin x = cos x needs x), so backprop reuses them.
What is the adjoint (bar) of a node?
The sensitivity ∂(final output)/∂(that node) — how much the output changes if the node wiggles.
State the reverse-mode update rule.
For each edge u→v: uˉ+=vˉ⋅∂v/∂u\bar u \mathrel{+}= \bar v \cdot \partial v/\partial u, seeded with fˉ=1\bar f = 1.
Why do we accumulate (+=) adjoints?
A node feeding multiple children affects the output through several paths; the multivariable chain rule sums those contributions.
Why is reverse mode preferred for neural nets?
Cost scales with number of outputs; loss has one scalar output but millions of inputs, so one backward pass gives all gradients ≈ cost of one forward pass.
Is autograd the same as finite differences?
No. Autograd applies exact local derivative rules (no step size h), so it has no truncation error; finite differences approximate.
Relationship between backprop and autograd?
Backpropagation is reverse-mode automatic differentiation applied to neural networks — same algorithm.
Derivative of the sigmoid σ(z)?
σ(z)(1−σ(z)).
Why zero gradients each training step?
Because .grad accumulates (+=), leftover gradients from previous batches would otherwise add up incorrectly.

Connections

Concept Map

needs

solved by

is a

nodes are

edges are

each has

walked by

step 1

step 2

glued by

systematized as

computes

makes

Neural network as nested function

Gradient wrt every weight

Computational graph

Directed acyclic graph

Variables

Elementary operations

Known local derivative

Autograd

Forward pass stores values

Backward pass

Chain rule

Adjoints accumulated

Deep learning trainable at scale

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek neural network basically ek bahut bada nested function hai — chhote-chhote operations (add, multiply, sin, exp) ek ke baad ek lage hue hote hain. Is puri cheez ko ek picture ki tarah likh lo jise computational graph kehte hain: nodes matlab variables, aur edges matlab operations. Har chhote operation ka derivative hume aata hai, bas unko chain rule se jodna hai.

Autograd ka kaam do steps mein hota hai. Pehle forward pass — left se right chalke saare intermediate values calculate karke store kar lo (store isliye kyunki cos x jaise derivatives ko x ki value chahiye hoti hai). Fir backward pass — output se shuru karo, seed lagao fˉ=1\bar f = 1, aur peeche ki taraf har node ko batao "tu final answer ke liye kitna zimmedar hai" (isko adjoint kehte hain). Rule simple hai: uˉ+=vˉ⋅∂v/∂u\bar u \mathrel{+}= \bar v \cdot \partial v/\partial u.

Sabse important baat: agar ek node do jagah use ho raha hai, to uske gradients ko add karo (isliye +=), kyunki wo output ko do raaste se affect karta hai — yahi multivariable chain rule hai. Aur ek badi galti: log samajhte hain autograd numerical (finite difference) hai — nahi bhai, ye exact hai, koi hh wala approximation nahi.

Isse kyun farak padta hai? Loss ek scalar (ek output) hota hai lekin weights millions mein. Reverse mode ek hi backward pass mein saare weights ke gradients de deta hai — cost lagbhag ek forward pass jitni. Yahi wajah hai ki deep learning practically train ho pata hai. Backprop koi alag jaadu nahi — ye bas neural net pe lagaya gaya reverse-mode autograd hai.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 2,009 wordsD2Visual walkthroughThe derivation in pictures · 2,296 wordsD3Worked examplesEvery scenario covered · 2,805 wordsD4ExercisesGraded, full solutions · 2,187 wordsD5Question bankConcept traps, quick reveals · 1,373 words

Test yourself — Neural Network Fundamentals

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Chain rule for multivariable functions — all casesMaths · 4.4.7Backpropagation — chain rule, gradient computationCoding · 5.6.8Gradient descent and variants — convergence analysisMaths · 4.10.20Activation functions - sigmoid, tanhAI-ML · 3.1.4Chain rule for multivariate functions (backprop foundation)AI-ML · 1.2.14Backpropagation algorithm derivationAI-ML · 3.1.9Edge deployment and ONNXAI-ML · 5.3.17Backpropagation — chain rule, gradient computationCoding · 5.6.8