Recurrent Neural Networks (RNN) architecture: Worked examples

This page is the "get your hands dirty" companion to Recurrent Neural Networks (RNN) architecture. We take the two equations of an RNN cell and run them by hand through every kind of situation they can meet: the very first step with no memory, a middle step, the sign behaviour of $tanh$ , saturation toward both $+ 1$ and $- 1$ , the tiny-weight and huge-weight limits, negative recurrence, a nonzero hidden bias, a nonzero output bias, a real word problem, and an exam-style twist.

Before anything else, let us make sure every symbol is earned.

Definition The two RNN equations (in plain words)

An RNN reads a sequence one item at a time. At time step $t$ : $h^{(t)} = tanh (W_{hh} h^{(t - 1)} + W_{x h} x^{(t)} + b_{h})$ $y^{(t)} = W_{h y} h^{(t)} + b_{y}$

$x^{(t)}$ — the input at step $t$ (one number here, or a small vector). Think "the word/character you just read."
$h^{(t)}$ — the hidden state, the network's memory after reading step $t$ . A vector of numbers.
$h^{(t - 1)}$ — the memory from one step ago. At the very first step there is none, so we set $h^{(0)} = 0$ .
$W_{hh}$ — the recurrent weight: how strongly old memory carries forward.
$W_{x h}$ — the input weight: how strongly the new input pushes the memory.
$W_{h y}$ — turns memory into an output $y^{(t)}$ .
$b_{h}$ — the hidden bias, a constant nudge added inside the $tanh$ .
$b_{y}$ — the output bias, a constant nudge added to the output after the readout.

Figure — Recurrent Neural Networks (RNN) architecture

Read the figure like this: the horizontal axis is $z^{(t)}$ (the pre-activation), the vertical axis is $h^{(t)} = tanh (z^{(t)})$ . The pale-yellow curve is $tanh$ ; the two blue dashed lines are the hard ceilings at $+ 1$ and $- 1$ ; the pink dotted line is $y = z$ , showing that near the origin $tanh$ barely bends. Every worked example is really just "find where your $z^{(t)}$ lands on this curve."

To keep the arithmetic honest and checkable, all worked examples use 1-dimensional state (one number for $h$ , one for $x$ ). The logic is identical for vectors — you just replace numbers with matrix–vector products.

Our fixed toy cell (unless a problem overrides it): $W_{hh} = 0.5, W_{x h} = 1.0, b_{h} = 0, W_{h y} = 2.0, b_{y} = 0.$

The scenario matrix

Every situation the two equations can throw at you falls into one of these cells. The examples below are labelled with the cell they cover.

Cell	Situation	Why it's a distinct case
A	First step, $h^{(0)} = 0$	No memory term — tests the "cold start"
B	Middle step, memory feeds in	$W_{hh} h^{(t - 1)}$ now non-zero
C	Positive vs. negative input (sign of $tanh$ )	$tanh$ is odd: flips sign with input
D	Saturating input, both tails ( $z \to + \infty$ and $z \to - \infty$ )	$tanh$ flattens → memory "maxed out" at $\pm 1$
E	Limiting weight $W_{hh} \to 0$ (memoryless)	RNN collapses to feedforward
F	Limiting weight $W_{hh}$ large (gradient blow-up)	Foreshadows exploding gradients
G	Real-world word problem (sentiment, many-to-one)	Only final state used
H	Exam twist: identical inputs, does $h$ settle?	Fixed-point / convergence reasoning
I	Negative recurrent weight $W_{hh} < 0$	Sign flips each step → oscillation
J	Nonzero hidden bias $b_{h} \neq = 0$ and output bias $b_{y} \neq = 0$	Bias shifts the whole $z$ -axis / the output line

Cell A + B — cold start, then a middle step

Worked example Example 1 (Cell A & B): run "H, E" through the toy cell

Inputs (pretend one-hot collapsed to a scalar): $x^{(1)} = 1.0$ , then $x^{(2)} = 1.0$ . Use $W_{hh} = 0.5, W_{x h} = 1.0, b_{h} = 0$ .

Forecast: before computing — will $h^{(2)}$ be larger or smaller than $h^{(1)}$ ? (Same input twice, but now memory adds a positive push...)

Step 1 — first step, no memory. $z^{(1)} = W_{x h} x^{(1)} = 1.0 \cdot 1.0 = 1.0$ . Then $h^{(1)} = tanh (1.0) \approx 0.7616$ . Why this step? At $t = 1$ there is no $h^{(0)}$ , so we drop the $W_{hh} h^{(0)}$ term (it is $0.5 \cdot 0 = 0$ ). This is the cold start, Cell A.

Step 2 — output. $y^{(1)} = W_{h y} h^{(1)} = 2.0 \cdot 0.7616 = 1.5232$ . Why this step? Output is a plain linear readout of memory — no squashing here.

Step 3 — second step, memory feeds in. $z^{(2)} = W_{hh} h^{(1)} + W_{x h} x^{(2)} = 0.5 (0.7616) + 1.0 (1.0) = 1.3808$ . Then $h^{(2)} = tanh (1.3808) \approx 0.8811$ . Why this step? Now Cell B: the term $0.5 h^{(1)}$ carries the "I saw H" signal forward and adds to the fresh input.

Verify: Forecast confirmed — $h^{(2)} = 0.8811 > h^{(1)} = 0.7616$ , because the extra memory push raised $z$ . But notice it grew less than the input alone would suggest: $tanh$ is flattening near 1 (peek at the S-curve). Both $h$ values sit inside $(- 1, 1)$ ✓.

About the next figure: it draws Example 1 unrolled — two yellow $tanh$ boxes side by side (one per time step), blue arrows feeding inputs up from below, and a pink arrow carrying the memory $h^{(1)} = 0.7616$ rightward from the first box into the second. That pink arrow is the recurrence.

Cell C — the sign of the input

Worked example Example 2 (Cell C): flip the input negative

Same cell, but $x^{(1)} = - 1.0$ at the cold start.

Forecast: guess the sign of $h^{(1)}$ .

Step 1. $z^{(1)} = W_{x h} (- 1.0) = - 1.0$ , so $h^{(1)} = tanh (- 1.0) \approx - 0.7616$ . Why this step? $tanh$ is an odd function: $tanh (- z) = - tanh (z)$ . Flipping the input sign flips the memory sign, same magnitude. Look at the S-curve — it has point symmetry through the origin.

Step 2. Feed $x^{(2)} = + 1.0$ : $z^{(2)} = 0.5 (- 0.7616) + 1.0 = 0.6192$ , $h^{(2)} = tanh (0.6192) \approx 0.5505$ . Why this step? Negative memory now fights the positive input, pulling $z$ below 1. Memory and input can cancel — this is how an RNN "changes its mind."

Verify: $h^{(1)} = - 0.7616$ is exactly the negative of Example 1's $h^{(1)}$ ✓ (oddness). And $h^{(2)} = 0.5505 < 0.8811$ from Example 1 — the opposing memory reduced it, as expected ✓.

Cell D — saturation, BOTH tails

Worked example Example 3 (Cell D): a huge positive AND a huge negative input

Positive tail: set $x^{(1)} = 50$ with $W_{x h} = 1.0$ . Negative tail (same run): then feed $x^{(2)} = - 50$ .

Forecast: what are $h^{(1)}$ and $h^{(2)}$ , roughly? Do they hug the ceilings $+ 1$ and $- 1$ ?

Step 1 — positive saturation. $z^{(1)} = 50$ . $h^{(1)} = tanh (50) \approx 1.0000000000$ (to 10 decimals it is 1). Why this step? This is the flat right tail of the S-curve. Once $z$ is past ~3, $tanh$ is glued to $+ 1$ ; making $z$ bigger changes almost nothing.

Step 2 — negative saturation. $z^{(2)} = 0.5 (1.0) + 1.0 (- 50) = - 49.5$ , so $h^{(2)} = tanh (- 49.5) \approx - 1.0000000000$ . Why this step? Now we hit the flat left tail: a large negative $z$ is glued to $- 1$ . Both ceilings of the S-curve are reached, one per step — the complementary saturation cases.

Step 3 — the milder danger case. Restart the memory at the saturated $h^{(1)} = 1.0$ and feed only a moderate opposing input $x = - 2$ : $z = 0.5 (1.0) + 1.0 (- 2) = - 1.5$ , $h = tanh (- 1.5) \approx - 0.9051$ . Why this step? Even a saturated positive memory can be overridden, because $W_{hh} h^{(t - 1)}$ is bounded by $∣ W_{hh} ∣ = 0.5$ while the input term is unbounded.

Verify: $tanh (50)$ rounds to $+ 1$ and $tanh (- 49.5)$ rounds to $- 1$ ✓ — both tails covered. In exact math $∣ h ∣ < 1$ strictly; the derivative $tanh^{'} (z) = 1 - tanh^{2} (z) \approx 0$ at both ends, which is precisely where gradients die — the link to Vanishing and Exploding Gradients.

Cell E & F — the two weight limits

Worked example Example 5 (Cell F): large

$W_{hh}$ foreshadows exploding gradients Set $W_{hh} = 3.0$ , $W_{x h} = 1.0$ , $b_{h} = 0$ . Feed $x^{(1)} = 0.6$ , $x^{(2)} = 0$ , $x^{(3)} = 0$ .

Forecast: with no new input after step 1, will memory fade or persist?

Step 1. $z^{(1)} = 1.0 \cdot 0.6 = 0.6$ , so $h^{(1)} = tanh (0.6) \approx 0.5370$ . Why this step? Cold start; only the input contributes since $h^{(0)} = 0$ .

Step 2. $z^{(2)} = 3.0 (0.5370) + 0 = 1.6110$ , $h^{(2)} = tanh (1.6110) \approx 0.9226$ . Why this step? Input is now zero, so all of $z$ comes from the recurrent term. The large $W_{hh} = 3.0$ amplifies the old memory instead of shrinking it.

Step 3. $z^{(3)} = 3.0 (0.9226) + 0 = 2.7679$ , $h^{(3)} = tanh (2.7679) \approx 0.9922$ . Why this step? Same amplification again — memory is being driven toward the saturating region purely by the strong recurrence.

Verify: memory grew $0.5370 \to 0.9226 \to 0.9922$ ✓. In the forward pass $tanh$ caps the value at 1, but the backward pass multiplies by $W_{hh} = 3$ each step — three-to-the-power-of-steps growth. That is the exploding gradient seen in Backpropagation Through Time (BPTT), tamed in practice by Gradient Clipping.

Cell I — negative recurrent weight (oscillation)

Worked example Example 6 (Cell I):

$W_{hh} < 0$ makes the memory sign flip each step Set $W_{hh} = - 0.9$ , $W_{x h} = 0$ , $b_{h} = 0$ , and start with a "planted" memory $h^{(0)} = 0.5$ (imagine one strong input already happened). Feed zeros afterward.

Forecast: will $h^{(t)}$ stay positive, or bounce between $+$ and $-$ ?

Step 1. $z^{(1)} = - 0.9 (0.5) + 0 = - 0.45$ , so $h^{(1)} = tanh (- 0.45) \approx - 0.4219$ . Why this step? A negative $W_{hh}$ multiplies a positive memory into a negative pre-activation, so $tanh$ hands back a negative state — the sign flipped.

Step 2. $z^{(2)} = - 0.9 (- 0.4219) = 0.3797$ , so $h^{(2)} = tanh (0.3797) \approx 0.3625$ . Why this step? Now the memory is negative, so $- 0.9 \times (negative)$ is positive again: the sign flips back. This is the oscillation.

Step 3. $z^{(3)} = - 0.9 (0.3625) = - 0.3263$ , so $h^{(3)} = tanh (- 0.3263) \approx - 0.3151$ . Why this step? Third flip. Because $∣ W_{hh} ∣ = 0.9 < 1$ the flips also shrink in size, so the oscillation decays toward zero rather than growing.

Verify: the signs run $+, -, +, -$ ( $h^{(0)} = + 0.5, h^{(1)} < 0, h^{(2)} > 0, h^{(3)} < 0$ ) ✓ and the magnitudes shrink $0.5 \to 0.4219 \to 0.3625 \to 0.3151$ ✓. Sign of $W_{hh}$ decides direction; its magnitude decides growth vs. decay.

Cell G — real-world word problem (many-to-one)

Worked example Example 7 (Cell G): sentiment score of a 3-word review

Words map (via Word Embeddings) to scalars: "not" $\to - 0.8$ , "very" $\to 0.5$ , "good" $\to 0.9$ . Cell: $W_{hh} = 0.5, W_{x h} = 1.0, b_{h} = 0$ . Classify with $p = σ (w h^{(3)})$ , $w = 1.5$ , where $σ (u) = \frac{1}{1 + e ^{- u}}$ . We only read the final state (many-to-one).

Forecast: the review is "not very good" — negative overall. Do you expect $p < 0.5$ ?

Step 1. "not": $z^{(1)} = 1.0 \cdot (- 0.8) = - 0.8$ , so $h^{(1)} = tanh (- 0.8) \approx - 0.6640$ . Why this step? Cold start; the negation word alone drives memory negative.

Step 2. "very": $z^{(2)} = 0.5 (- 0.6640) + 1.0 (0.5) = 0.1680$ , $h^{(2)} = tanh (0.1680) \approx 0.1664$ . Why this step? The weak recurrence ( $0.5$ ) only carries a fraction of the negative memory forward, while the fresh positive word dominates — memory has almost forgotten "not" already.

Step 3. "good": $z^{(3)} = 0.5 (0.1664) + 1.0 (0.9) = 0.9832$ , $h^{(3)} = tanh (0.9832) \approx 0.7548$ . Why this step? Another positive word plus the now-positive carry pushes memory firmly positive — the final summary of the sentence.

Step 4 — classify. $u = w h^{(3)} = 1.5 \cdot 0.7548 = 1.1322$ ; $p = σ (1.1322) \approx 0.7563$ . Why this step? $σ$ squashes the score into a probability in $(0, 1)$ .

Verify: $p \approx 0.756 > 0.5$ — the model says positive, contradicting the true negative sentiment ✓ (the arithmetic is right; the model is wrong). This is a genuine failure mode: the negation "not" decayed away through the weak $W_{hh} = 0.5$ recurrence. Great motivation for gated architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU).

Cell J — nonzero biases shift the dynamics

Worked example Example 8 (Cell J): a hidden bias

$b_{h} = 0.7$ AND an output bias $b_{y} = - 0.3$ Cell: $W_{hh} = 0.5, W_{x h} = 1.0, b_{h} = 0.7$ , readout $W_{h y} = 2.0, b_{y} = - 0.3$ . Feed $x^{(1)} = 0$ , $x^{(2)} = 0$ (no input at all).

Forecast: with zero input, a bias-free cell would keep $h = 0$ forever. What does $b_{h}$ do to the memory, and what does $b_{y}$ do to the output?

Step 1 — hidden bias creates memory from nothing. $z^{(1)} = 0.5 (0) + 1.0 (0) + 0.7 = 0.7$ , so $h^{(1)} = tanh (0.7) \approx 0.6044$ . Why this step? $b_{h}$ is a constant nudge added inside $tanh$ : even with no input and no memory, $z$ is shifted right to $0.7$ , so memory becomes nonzero on its own.

Step 2 — output bias shifts the readout. $y^{(1)} = W_{h y} h^{(1)} + b_{y} = 2.0 (0.6044) + (- 0.3) = 0.9088$ . Why this step? $b_{y}$ lives outside $tanh$ , on the output line: it slides every $y^{(t)}$ down by a constant $0.3$ regardless of memory. Compare to the bias-free readout $2.0 (0.6044) = 1.2088$ — exactly $0.3$ higher.

Step 3 — bias keeps pushing memory each step. $z^{(2)} = 0.5 (0.6044) + 0.7 = 1.0022$ , so $h^{(2)} = tanh (1.0022) \approx 0.7625$ . Why this step? $b_{h}$ pushes again; combined with the recurrent carry, memory climbs toward a resting value that the bias sets.

Step 4 — where does memory settle? The fixed point solves $h_{⋆} = tanh (0.5 h_{⋆} + 0.7)$ . Iterating from Step 3 lands at $h_{⋆} \approx 0.7987$ . Why this step? With no input, the only forces are the recurrent carry and $b_{h}$ ; since $∣ W_{hh} ∣ = 0.5 < 1$ the map contracts to a single resting state.

Verify: $h^{(1)} = 0.6044$ , $y^{(1)} = 0.9088$ (exactly $0.3$ below the no-bias readout $1.2088$ ✓), $h^{(2)} = 0.7625$ , and the settled state $h_{⋆} \approx 0.7987$ satisfies $tanh (0.5 \cdot 0.7987 + 0.7) = tanh (1.0994) \approx 0.8004 \approx h_{⋆}$ ✓. $b_{h}$ slides the S-curve input (moving where memory rests); $b_{y}$ slides the output line (moving every $y$ by a constant).

Cell H — exam twist: does the state settle?

Worked example Example 9 (Cell H): feed the SAME input forever — does

$h$ converge? Cell $W_{hh} = 0.5, W_{x h} = 1.0, b_{h} = 0$ , constant input $x = 1.0$ at every step. A fixed point $h_{⋆}$ satisfies $h_{⋆} = tanh (0.5 h_{⋆} + 1.0)$ .

Forecast: will $h^{(t)}$ keep climbing, oscillate, or approach one value?

Step 1 — iterate. From Example 1: $h^{(1)} = 0.7616$ , $h^{(2)} = 0.8811$ . Continue: $h^{(3)} = tanh (0.5 \cdot 0.8811 + 1.0) = tanh (1.4406) \approx 0.8938$ , $h^{(4)} = tanh (0.5 \cdot 0.8938 + 1.0) = tanh (1.4469) \approx 0.8951$ . Why this step? Repeatedly applying the same map is how you find a fixed point — the value that maps to itself.

Step 2 — check the contraction condition. The map $g (h) = tanh (0.5 h + 1)$ has slope $g^{'} (h) = 0.5 (1 - tanh^{2} (\cdot))$ . Since $∣1 - tanh^{2} ∣ \leq 1$ , we get $∣ g^{'} ∣ \leq 0.5 < 1$ : $g$ is a contraction, so iterates converge to a unique $h_{⋆}$ . Why this step? A slope below 1 in absolute value means each step shrinks the gap to $h_{⋆}$ — guaranteed settling, no oscillation.

Verify: the sequence $0.7616, 0.8811, 0.8938, 0.8951, \dots$ is climbing but by shrinking steps toward $h_{⋆} \approx 0.8953$ , and indeed $tanh (0.5 \cdot 0.8953 + 1.0) \approx 0.8953$ ✓. This is exactly why $∣ W_{hh} ∣ < 1$ keeps things stable but also why old memory fades — the same reason Example 7 forgot "not".

About the last figure: it plots $h^{(t)}$ (yellow dots, joined) against the time step, with the blue dashed line marking the fixed point $h_{⋆}$ . Watch the dots take smaller and smaller vertical jumps as they hug the dashed line — that shrinking gap is the contraction in action.

Tying it back together

Every one of the ten cells is just the same two equations — $h^{(t)} = tanh (W_{hh} h^{(t - 1)} + W_{x h} x^{(t)} + b_{h}), y^{(t)} = W_{h y} h^{(t)} + b_{y}$ — read under a different lighting:

Cell A/B showed the raw mechanics: cold start ( $h^{(0)} = 0$ ) then memory feeding in.
Cell C/D were about where $z$ lands on the S-curve: sign (odd symmetry) and both saturating tails ( $\pm 1$ ).
Cell E/F/I were about what $W_{hh}$ does: zero → feedforward, large → blow-up, negative → oscillation. Sign chooses direction, magnitude chooses decay vs. growth.
Cell J separated the two biases: $b_{h}$ moves the memory's resting point (inside $tanh$ ), $b_{y}$ moves the output line (outside).
Cell G/H were the payoff: a real many-to-one classifier, and the fixed-point reasoning that explains why memory fades when $∣ W_{hh} ∣ < 1$ .

The single thread through all of them: because $tanh$ squashes and $∣ W_{hh} ∣ < 1$ contracts, a vanilla RNN is stable but forgetful — great for short dependencies, weak for long ones. That tension is exactly what Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), and later the Attention Mechanism and Transformers were built to resolve.

Recall Quick self-check

First step uses which memory value? ::: $h^{(0)} = 0$ , so the $W_{hh} h^{(t - 1)}$ term drops out. What is $z^{(t)}$ ? ::: The pre-activation $z^{(t)} = W_{hh} h^{(t - 1)} + W_{x h} x^{(t)} + b_{h}$ — the sum before $tanh$ squashes it. Why is $tanh$ chosen over sigmoid for the hidden state? ::: It is zero-centred, range $(- 1, 1)$ , keeping positive/negative signals balanced and gradients healthier. What does $W_{hh} = 0$ turn the RNN into? ::: A memoryless feedforward network applied per step. What does a negative $W_{hh}$ do? ::: Flips the memory's sign every step, causing oscillation; magnitude $< 1$ makes it decay. What is the difference between $b_{h}$ and $b_{y}$ ? ::: $b_{h}$ is added inside $tanh$ and shifts the memory's resting point; $b_{y}$ is added outside on the output line and shifts every $y^{(t)}$ by a constant. Why does memory settle when $∣ W_{hh} ∣ < 1$ ? ::: The update map is a contraction (slope $< 1$ ), so iterates converge to a unique fixed point — but old signals also decay. Which cell foreshadows exploding gradients? ::: Cell F, large $W_{hh}$ : forward $tanh$ caps values, but backward each step multiplies by $W_{hh}$ .