Visual walkthrough — Recurrent neural networks — hidden state, BPTT

We keep everything scalar (single numbers, not matrices) so nothing hides behind linear algebra. Once you see it in scalars, the matrix version is the same story with $\odot$ and transposes bolted on.

---

Step 1 — The chain of memory (WHAT we are differentiating)

WHAT. A recurrent net reads a sequence one item at a time. At step $t$ it holds a memory number $h_t$ (the hidden state). It updates that memory using the new input $x_t$ and the old memory $h_{t-1}$, always with the same three numbers $w_{hh},w_{xh},b_h$:

$$\underset{\text{new memory}}{\underbrace{h_t}}=\tanh (\underset{\text{memory→memory}}{\underbrace{w_{hh}}}\underset{\text{old memory}}{\underbrace{h_{t-1}}}+\underset{\text{input→memory}}{\underbrace{w_{xh}}}\underset{\text{new input}}{\underbrace{x_t}}+\underset{\text{offset}}{\underbrace{b_h}})$$

WHY. We want the loss $L$ to go down, so we need to know how much each weight is to blame for the error. The blame is a derivative $\partial L/\partial w$. Before we can chase that derivative we must see the road the signal travels — that road is the chain below.

PICTURE. Time runs left to right. Each box is a memory $h_t$; each black arrow is one use of the same weight $w_{hh}$; each red arrow drops an output $y_t$ that gets compared to a target.

---

Step 2 — Name the pre-activation (WHY we introduce $a_t$)

WHAT. The $\tanh$ wraps a plain linear combination. Give that inside-part its own name:

$$a_t=\underset{\text{the linear part}}{\underbrace{w_{hh}{h}_{t-1}+w_{xh}{x}_t+b_h}},h_t=\tanh (a_t)$$

WHY. The chain rule works one simple function at a time. If we lump the linear part and the squash together we have to differentiate a monster. Splitting at $a_t$ gives us two easy links:

$$\text{weights}⟶a_t\stackrel{\tanh }{\to }{h}_t$$

WHY tanh at all? It squashes any number into $(-1,1)$, so the memory can recirculate forever without blowing up. It also passes through the origin, so a zero input gives a zero-ish memory (a fair starting point). Compare the alternatives in the picture.

PICTURE. Three curves. Only $\tanh$ (red) is both bounded and centered at zero — the property we need for a memory that loops.

Recall Why not just drop tanh and stay linear?

Question: what breaks if $h_t=a_t$ (no squash)? ::: The whole unrolled net collapses to one big linear map ($h_T$ becomes a constant times $h_0$ plus input terms), and the gradient becomes pure $w_{hh}^T$ — guaranteed to vanish or explode with no nonlinearity to control it. See Vanishing and Exploding Gradients.

---

Step 3 — The single-link derivative: $\tanh$'s twin (WHAT ${\tanh }^{\prime }$ is)

WHAT. To move blame through the squash we need the slope of $\tanh$. A beautiful fact:

$$\frac{d}{da}\tanh (a)=1-{\tanh }^2(a)=1-h^2$$

So the slope is written using the output we already computed — no new evaluation needed.

WHY. During the backward pass we are standing at $h_t$ and want to step back to $a_t$. The chain rule says multiply by this slope. Because it equals $1-h_t^2$, and $h_t\in (-1,1)$, this slope always lies in $(0,1]$: each backward step through $\tanh$ can only shrink the signal, never grow it. Hold that thought for Step 6.

PICTURE. The red curve is $1-h^2$. It is tallest (=1) at the middle and squashed to 0 at the edges ($h=\pm 1$). A memory that has saturated (pushed to $\pm 1$) passes almost no gradient — it has gone "deaf."

---

Step 4 — Blame arriving at a hidden state (the heart of BPTT)

WHAT. Define the blame on memory $h_t$ as ${\delta }_t\equiv \frac{\partial L}{\partial h_t}$: how much the total loss changes if we nudge $h_t$. Because $h_t$ forked into two roads (Step 1), its blame is the sum of blame coming back along each road:

$${\delta }_t=\underset{\text{road 1: its own output }{y}_t}{\underbrace{\frac{\partial L_t}{\partial h_t}}}+\underset{\text{road 2: blame from the future}}{\underbrace{\frac{\partial h_{t+1}}{\partial h_t}{\delta }_{t+1}}}$$

Term by term:

• $\frac{\partial L_t}{\partial h_t}$ — the local error this step made through its own output.
• ${\delta }_{t+1}$ — blame already computed for the next memory (we work right-to-left, so it's ready).
• $\frac{\partial h_{t+1}}{\partial h_t}$ — the bridge that carries future blame back one step. From Steps 2–3 it is $(1-h_{t+1}^2)w_{hh}$: pass through $\tanh$'s twin, then through the weight.

WHY the sum? This is the multivariable chain rule: a quantity that influences the answer along several paths contributes the sum of the paths. $h_t$ has exactly two paths, so exactly two terms. This is the idea of "back-propagation through time": blame from step $t+1$ is handed to step $t$, then step $t$ hands its total to step $t-1$, and so on backward.

PICTURE. Red arrows flow right to left. At $h_t$ two red arrows merge: a short one from its own output, a long one arriving from $h_{t+1}$. Their sum is ${\delta }_t$.

---

Step 5 — Blaming the shared weight: sum over all its jobs

WHAT. Now the payoff. The weight $w_{hh}$ was used at every step (every black arrow in Step 1's picture). Its total blame is the sum of what it did at each step. At step $t$ its job was to multiply $h_{t-1}$ inside $a_t$, so its local contribution is $(1-h_t^2){\delta }_t$ (blame at $h_t$, pushed back through the twin to $a_t$) times the number it multiplied, $h_{t-1}$:

$$\boxed{\frac{\partial L}{\partial w_{hh}}=\sum _{t=1}^T\underset{\text{twin: }{h}_t\to a_t}{\underbrace{(1-h_t^2)}}\underset{\text{blame at }{h}_t}{\underbrace{{\delta }_t}}\underset{\text{what }{w}_{hh}\text{ multiplied}}{\underbrace{h_{t-1}}}}$$

WHY sum again? Same weight-sharing logic as the sum in Step 4, but now over parameter uses instead of paths. One knob, many jobs $\Rightarrow$ add up the blame from every job. This single boxed line is the whole point of the BPTT derivation.

PICTURE. One knob labelled $w_{hh}$ in the center; wires reach out to each time step; the total turn on the knob is the sum of the pulls coming back along every wire.

---

Step 6 — Edge case: many steps back = a product that dies or explodes

WHAT. How does blame from the final loss reach an early memory? Chain the bridges together. Going from step $t$ all the way to step $T$ multiplies one bridge per hop:

$$\frac{\partial h_T}{\partial h_t}=\prod _{k=t+1}^T\underset{\in (0,1]}{\underbrace{(1-h_k^2)}}\underset{\text{fixed}}{\underbrace{w_{hh}}}$$

Each factor is roughly $w_{hh}$ scaled by a number $\le 1$. Call the typical size of one factor $\lambda$. Then a gap of $n=T-t$ steps multiplies the blame by about ${\lambda }^n$.

WHY it matters — cover both signs of $\log \lambda$:

• $\lambda <1$ (e.g. small $w_{hh}$, or saturated memories): ${\lambda }^n\to 0$. Blame from far away vanishes — the net cannot learn long-range dependencies. This is the common case, and why LSTM and GRU exist (they add a near-identity memory path where $\lambda \approx 1$).
• $\lambda >1$ (large $w_{hh}$, unsaturated): ${\lambda }^n\to \infty$. Blame explodes — training diverges. Fixed by gradient clipping.
• $\lambda =1$ (the knife-edge): blame is preserved — exactly the regime gated cells try to engineer.

PICTURE. Three red curves of ${\lambda }^n$ against the time-gap $n$: one decaying to zero, one flat at 1, one shooting up. Only the flat one keeps memory alive.

---

Step 7 — Sanity check on real numbers (degenerate + normal case)

WHAT. Run the tiny scalar RNN end-to-end: $w_{hh}=0.5,w_{xh}=1,b_h=0$, inputs $x_1=x_2=1$, start $h_0=0$, target $1$ at the last step with $L=\frac{1}{2}(h_2-1{)}^2$.

Forward:

• $a_1=0.5\cdot 0+1\cdot 1=1\Rightarrow h_1=\tanh (1)=0.7616$
• $a_2=0.5\cdot 0.7616+1=1.3808\Rightarrow h_2=\tanh (1.3808)=0.8811$

Backward (right to left):

• ${\delta }_2=\partial L/\partial h_2=h_2-1=-0.1189$.
• bridge $\frac{\partial h_2}{\partial h_1}=(1-h_2^2)w_{hh}=(1-0.7763)(0.5)=0.1118$.
• ${\delta }_1=\underset{\text{no output loss at step 1}}{\underbrace{0}}+0.1118\cdot (-0.1189)=-0.01329$.

The degenerate first term. Because $h_0=0$, the step-1 contribution to $\partial L/\partial w_{hh}$ is multiplied by zero and vanishes — a clean illustration that the weight only earns blame where it actually multiplied a nonzero memory:

$$\frac{\partial L}{\partial w_{hh}}=\underset{=0(\text{since }{h}_0=0)}{\underbrace{(1-h_1^2){\delta }_1{h}_0}}+\underset{\text{the only live term}}{\underbrace{(1-h_2^2){\delta }_2{h}_1}}$$ $$=0+(0.2237)(-0.1189)(0.7616)=-0.02025$$

WHY. Every symbol here was defined in Steps 1–5. This number is the boxed formula of Step 5 in action, and it confirms the "sum over steps" structure with one term switched off by the zero initial state.

PICTURE. The two candles of the sum drawn side by side: a grey zero candle (step 1, killed by $h_0=0$) and a red live candle (step 2). Their heights sum to $-0.02025$.

---

The one-picture summary

Everything on one canvas: forward $\tanh$ chain across the top, red blame flowing backward along the bottom, and the shared weight collecting a sum of pulls from every step.

Recall Feynman: the whole walkthrough in plain words

You read a sequence one item at a time and keep a single memory number that you refresh with the same rule every step — that rule squashes with $\tanh$ so the memory can loop forever without exploding. When the ending is wrong, you walk backward through the whole sequence handing blame from each step to the one before it. At each hand-off you pass the blame through "$\tanh$'s twin" ($1-h^2$, always between 0 and 1) and then through the weight. Because that same weight was used at every step, you add up its blame from all of them — "same weight, sum the blame." The catch: walking back many steps multiplies many of those between-0-and-1 factors, so distant blame usually fades to nothing (vanishing gradient) — which is exactly the leak that LSTM and GRU plug. See also Time Series Forecasting and Sequence Modeling in Flight Data for where this memory earns its keep.