Exercises — Recurrent Neural Networks (RNN) architecture

This page is a self-test ladder for Recurrent Neural Networks (RNN) architecture. Each rung is harder than the last: you start by recognising the formula, and end by synthesising a full training-time reasoning chain. Every solution is hidden inside a collapsible box — try the problem first, then reveal.

Before we begin, here is the single equation everything rests on. We define each symbol in words so a reader who has never seen it can follow.

We also name the quantity inside the $\tanh$: call it the pre-activation $z^{(t)}$, defined as $z^{(t)}=W_{hh}{h}^{(t-1)}+W_{xh}{x}^{(t)}+b_h,h^{(t)}=\tanh (z^{(t)}).$ So $z^{(t)}$ is "the raw weighted sum before squashing." We will need it when we differentiate $\tanh$ in Level 3.

The same three matrices are reused at every step — this is called weight sharing. That fact drives most of the counting problems below.

Figure s01 — the unrolled RNN. Read it left to right: each blue box is the same cell drawn three times (weight sharing). An orange arrow feeds the input $x^{(t)}$ up into the cell; a green arrow carries the output $y^{(t)}$ out the top; the red arrow between boxes is the memory $h^{(t)}$ handed from one step to the next. The parameter count never grows with the number of boxes.

---

Level 1 — Recognition

Goal: can you read the equation and identify its parts?

Exercise 1.1

An RNN has input dimension $d_x=100$, hidden dimension $d_h=256$, output dimension $d_y=10$. State the shape (rows × columns) of each of $W_{xh}$, $W_{hh}$, $W_{hy}$.

Recall Solution

Read the shapes straight off the definition. A matrix that maps a vector of size $a$ into a vector of size $b$ has $b$ rows and $a$ columns (output-dim × input-dim).

• $W_{xh}$ maps input ($100$) → hidden ($256$): $256\times 100$.
• $W_{hh}$ maps hidden ($256$) → hidden ($256$): $256\times 256$.
• $W_{hy}$ maps hidden ($256$) → output ($10$): $10\times 256$.

What we did: matched each arrow in the figure to a matrix. Why: the shape is forced by "the output vector must come out the right size."

Exercise 1.2

True or false, and say why: "An RNN needs different weight matrices for each of its 20 time steps, so a 20-step sequence has 20 copies of $W_{hh}$."

Recall Solution

False. There is exactly one $W_{hh}$, reused at every step. The picture of the unrolled chain shows many boxes, but they are the same box drawn again — like frames of a film showing one actor, not twenty actors. That is weight sharing.

---

Level 2 — Application

Goal: plug numbers in and turn the crank.

Exercise 2.1

With $d_x=100$, $d_h=256$, $d_y=10$, count the total number of trainable parameters (include the biases $b_h$ of size $d_h$ and $b_y$ of size $d_y$).

Recall Solution

Add up every matrix and bias.

• $W_{xh}$: $256\times 100=25600$
• $W_{hh}$: $256\times 256=65536$
• $W_{hy}$: $10\times 256=2560$
• $b_h$: $256$
• $b_y$: $10$

Total $=25600+65536+2560+256+10=93962$.

Why the count is independent of $T$: weight sharing again — the same $93962$ numbers describe a 3-step or a 3000-step sequence.

Exercise 2.2

A tiny RNN has $d_h=1$ (a single-number memory), $W_{hh}=0.5$, $W_{xh}=1$, $b_h=0$, and initial state $h^{(0)}=0$. The input sequence is $x^{(1)}=1,x^{(2)}=1,x^{(3)}=1$. Compute $h^{(1)},h^{(2)},h^{(3)}$ (round to 4 decimals).

Recall Solution

Apply $h^{(t)}=\tanh (0.5h^{(t-1)}+1\cdot x^{(t)})$ step by step (here the pre-activation is $z^{(t)}=0.5h^{(t-1)}+x^{(t)}$).

• $h^{(1)}=\tanh (0.5\cdot 0+1)=\tanh (1)=0.7616$
• $h^{(2)}=\tanh (0.5\cdot 0.7616+1)=\tanh (1.3808)=0.8814$
• $h^{(3)}=\tanh (0.5\cdot 0.8814+1)=\tanh (1.4407)=0.8940$

What it looks like (see figure s02): the memory climbs but flattens — each new input pushes $h$ higher, yet $\tanh$ caps it below 1. That ceiling is exactly what keeps a single RNN neuron from blowing up.

Figure s02 — the blue curve is the hidden state $h^{(t)}$ over 8 identical $+1$ inputs; the numbers printed at the first three points are the values computed above. The dashed red line at $1$ is the $\tanh$ ceiling — notice the curve bends toward it and never crosses. Even with input piling up forever, the memory saturates.

---

Level 3 — Analysis

Goal: reason about why things behave, especially gradients.

Before the exercises, two pieces of notation used in the gradient formula:

• $z^{(i)}$ = the pre-activation at step $i$ (defined above): the raw weighted sum fed into $\tanh$. We need it because the slope of $\tanh$ depends on where on the curve we are, and that location is $z^{(i)}$.
• ${\tanh }^{\prime }(z)=1-{\tanh }^2(z)$ = the slope of $\tanh$ at input $z$; it peaks at $1$ (when $z=0$) and shrinks toward $0$ as $|z|$ grows.
• $\text{diag}(v)$ = the operation that takes a vector $v$ and places its entries on the diagonal of an otherwise-zero square matrix. It appears because differentiating the element-wise $\tanh$ scales each hidden component by its own slope — a per-component multiply, which as a matrix is a diagonal matrix. In the 1-D case (a single hidden number) $\text{diag}(\cdot )$ is just that one number, so the whole Jacobian collapses to an ordinary product of scalars.

Where the Jacobian comes from (a 3B1B-style sketch)

We are about to multiply many copies of $W_{hh}^{\top }\text{diag}({\tanh }^{\prime }(z^{(i)}))$ together. Before trusting that formula, let us see why those two pieces — and no others — show up when we ask "if I nudge $h^{(i-1)}$ a little, how much does $h^{(i)}$ move?"

Recall $h^{(i)}=\tanh (z^{(i)})$ with $z^{(i)}=W_{hh}{h}^{(i-1)}+W_{xh}{x}^{(i)}+b_h$. A tiny nudge $\delta$ in $h^{(i-1)}$ travels through two gates in series:

1. Through the matrix $W_{hh}$: the nudge first hits the linear map that turns old memory into the pre-activation. So $z^{(i)}$ changes by $W_{hh}\delta$. When we later push the error backward (as BPTT does) instead of pushing the input forward, this same linear map acts as its transpose $W_{hh}^{\top }$ — that is simply what "running a matrix in reverse" means for gradients.

2. Through the $\tanh$ squash: the changed $z^{(i)}$ then enters $\tanh$. But $\tanh$ is not straight — its steepness is ${\tanh }^{\prime }(z^{(i)})$ at that exact spot. So the change is scaled by the local slope, component by component. Packaging one slope per hidden unit onto a diagonal gives $\text{diag}({\tanh }^{\prime }(z^{(i)}))$.

Multiply the two gate-effects and you get exactly one factor $W_{hh}^{\top }\text{diag}({\tanh }^{\prime }(z^{(i)}))$ for one step. Chaining $t-k$ steps means the nudge passes through the same pair of gates over and over — hence the product of identical-looking factors. The figure below makes the "two gates in series, repeated" picture concrete.

Figure s04 — one BPTT step as two gates in series. A gradient entering from the right (red) first passes the blue $W_{hh}^{\top }$ gate (linear rescale by the recurrent matrix), then the orange $\text{diag}({\tanh }^{\prime })$ gate (multiply by the local slope of $\tanh$, always $\le 1$). Chaining many steps stacks these gate-pairs — every extra step multiplies in another factor, which is precisely why the product either shrinks to nothing or blows up.

Exercise 3.1

Recall the through-time Jacobian product that appears in Backpropagation Through Time (BPTT): $\frac{\partial h^{(t)}}{\partial h^{(k)}}={\prod }_{i=k+1}^t{W}_{hh}^{\top }\text{diag}({\tanh }^{\prime }(z^{(i)}))$ Suppose (1-D case) $W_{hh}=0.9$ and ${\tanh }^{\prime }(z^{(i)})=0.7$ at every step. What is $\partial h^{(t)}/\partial h^{(k)}$ across a gap of $t-k=20$ steps? Is this vanishing or exploding?

Recall Solution

In 1-D, $W_{hh}^{\top }$ is just the number $0.9$ and $\text{diag}({\tanh }^{\prime }(z^{(i)}))$ is just the number $0.7$, so the product becomes a plain power: each factor is $0.9\times 0.7=0.63$, multiplied $20$ times. $\frac{\partial h^{(t)}}{\partial h^{(k)}}=(0.9\cdot 0.7{)}^{20}=0.63^{20}\approx 9.70\times 10^{-5}.$ Since $0.63<1$, raising it to the 20th power drives the gradient toward zero: this is a vanishing gradient. A signal from 20 steps back arrives roughly $10000\times$ weaker.

Why this is the whole story of RNN failure: the derivative of $\tanh$ is at most $1$ (and usually well below it), so unless $W_{hh}$ compensates by being large, the per-step factor is $<1$ and long-range memory decays exponentially.

Exercise 3.2

Now set $W_{hh}=1.5$ with ${\tanh }^{\prime }(z^{(i)})=0.9$. Over $t-k=20$ steps, does the gradient vanish or explode? Give the magnitude.

Recall Solution

Per-step factor $=1.5\times 0.9=1.35>1$. Over 20 steps: $1.35^{20}\approx \mathrm{424.9.}$ This explodes — the gradient grows about $425\times$, which in higher dimensions produces NaN losses. The standard patch is Gradient Clipping (rescale the gradient if its norm exceeds a threshold).

Figure s03 — the gradient magnitude versus how many steps back you look, on a log vertical axis. The blue line (per-step factor $0.63$) plunges toward zero (vanishing); the red line (factor $1.35$) rockets upward (exploding). The gray dashed line at height $1$ is the knife-edge: below it gradients die, above it they blow up.

Exercise 3.3

Take $W_{hh}=-0.8$ (a negative recurrent weight) with ${\tanh }^{\prime }(z^{(i)})=1$ at every step. Compute the 1-D through-time factor for gaps of $t-k=1,2,3,4$ and describe the pattern.

Recall Solution

Each per-step factor is $-0.8\times 1=-0.8$, so the gap-$n$ factor is $(-0.8{)}^n$:

• $n=1$: $(-0.8{)}^1=-0.8$
• $n=2$: $(-0.8{)}^2=+0.64$
• $n=3$: $(-0.8{)}^3=-0.512$
• $n=4$: $(-0.8{)}^4=+0.4096$

The sign flips every step while the magnitude still shrinks ($|-0.8|<1$). So a negative $W_{hh}$ produces an oscillating, decaying gradient: the influence of a past step alternates between pushing the loss up and pulling it down as the gap grows. Vanishing/exploding is governed by the magnitude $|W_{hh}\cdot {\tanh }^{\prime }|$; the sign of $W_{hh}$ adds this alternation on top.

---

Level 4 — Synthesis

Goal: combine architecture choices to design or diagnose a full system.

Two definitions we will lean on here:

• The logistic sigmoid $\sigma$ is the squashing function $\sigma (u)=\frac{1}{1+e^{-u}}$, which takes any real number $u$ and returns a value in $(0,1)$. That output range is exactly what we want for a probability.

Exercise 4.1

You must classify a movie review (a variable-length sentence) as positive or negative. Which RNN type (many-to-many, many-to-one, one-to-many, encoder-decoder) is correct, which hidden state feeds the classifier, and why do we ignore the intermediate outputs?

Recall Solution

Type: many-to-one (sequence → single label). You feed the whole sentence word-by-word using Word Embeddings as inputs $x^{(t)}$, then read only the final hidden state $h^{(T)}$ into a small classifier head: $p(\text{positive})=\sigma (w_c^{\top }{h}^{(T)}+b_c).$ Here $w_c\in {\mathbb{R}}^{d_h}$ is a new weight vector (one weight per hidden component, so its length is $d_h$) and $b_c\in \mathbb{R}$ is a new scalar bias. These belong to the classifier head and are distinct from the RNN's own $W_{hy},b_y$ — in a pure classifier you replace the sequence-output layer $(W_{hy},b_y)$ with this single-number head $(w_c,b_c)$. The dot product $w_c^{\top }{h}^{(T)}$ collapses the $d_h$-vector to one number, and $\sigma$ turns that number into a probability in $(0,1)$. Why only $h^{(T)}$: by construction $h^{(T)}$ has "read" every earlier word (each $h^{(t)}$ folded in the previous memory), so it is already a summary of the entire review. The intermediate $y^{(t)}$ would be per-word predictions we simply do not need for a single sentence-level verdict.

Exercise 4.2

For machine translation French → English, sentences differ in length between the two languages. Explain why a plain many-to-many RNN (one output per input step) fails, and what architecture fixes it.

Recall Solution

A synchronous many-to-many RNN emits exactly one output per input step, forcing $T_{\text{out}}=T_{\text{in}}$. But sentence pairs almost never match length: "Je t'aime" (3 tokens) → "I love you" (3) happens to align, yet "Il pleut" (2) → "It is raining" (3) does not. Any fixed one-output-per-input rule cannot produce a different number of output tokens than input tokens, so this architecture cannot even express most translations.

Fix: the encoder–decoder (see Sequence-to-Sequence Models). It splits the job across two separate RNNs:

1. Encoder RNN — reads the French sentence step by step and does not emit any output. Its only job is to fold the whole sentence into its final hidden state, called the context vector $c=h^{(T_{\text{enc}})}$. Think of $c$ as a fixed-size summary of the meaning.
2. Decoder RNN — a second RNN whose initial hidden state is loaded from $c$ (a one-to-many pattern). It then generates English tokens autoregressively: at each step it takes the token it just produced as the next input, updates its memory, emits the next token, and repeats. It keeps going until it emits a special end-of-sentence token, so the number of output steps $T_{\text{out}}$ is decided by the decoder at run time and is free to differ from $T_{\text{in}}$.

This decoupling — encode into $c$, then freely decode — is exactly what lets input and output lengths differ. When a single fixed $c$ becomes a bottleneck for long sentences (it must hold the entire meaning in one vector), we add an Attention Mechanism so the decoder can look back at all encoder states rather than just the last one; that idea ultimately grew into Transformers.

---

Level 5 — Mastery

Goal: full end-to-end derivations, catching every case and edge condition.

Exercise 5.1

Redo Exercise 2.2 but with a negative input burst: $W_{hh}=0.5$, $W_{xh}=1$, $b_h=0$, $h^{(0)}=0$, and $x^{(1)}=-2,x^{(2)}=-2,x^{(3)}=-2$. Compute $h^{(1)},h^{(2)},h^{(3)}$ and verify the state stays inside $[-1,1]$.

Recall Solution

$h^{(t)}=\tanh (0.5h^{(t-1)}-2)$:

• $h^{(1)}=\tanh (-2)=-0.9640$
• $h^{(2)}=\tanh (0.5(-0.9640)-2)=\tanh (-2.4820)=-0.9862$
• $h^{(3)}=\tanh (0.5(-0.9862)-2)=\tanh (-2.4931)=-0.9865$

Every value lies in $(-1,0)$, confirming the case coverage point: $\tanh$ bounds the memory in $[-1,1]$ for any sign or size of input. Negative inputs push the memory toward $-1$ (the floor) exactly as positive inputs push it toward $+1$ (the ceiling), and it can never escape the band.

Exercise 5.2

Consider the degenerate zero case: $x^{(t)}=0$ for all $t$, $b_h=0$, and any $W_{hh},W_{xh}$, with $h^{(0)}=0$. Prove that $h^{(t)}=0$ for all $t$, and explain what this tells you about an untrained / un-driven RNN.

Recall Solution

Induction. Base: $h^{(0)}=0$ (given). Step: assume $h^{(t-1)}=0$. Then $h^{(t)}=\tanh (W_{hh}\cdot 0+W_{xh}\cdot 0+0)=\tanh (0)=0.$ So $h^{(t)}=0$ for all $t$. Interpretation: with no input and no bias, the network has nothing to remember — the memory sits at the origin forever. The state only becomes non-trivial once real inputs or a non-zero bias drive it. This is the sanity-check baseline: any non-zero hidden state must be traceable to actual data flowing in.

Exercise 5.3

A per-step factor in the through-time gradient is $\gamma =|W_{hh}|\cdot \max {\tanh }^{\prime }$. Since ${\max }_z{\tanh }^{\prime }(z)=1$ (attained at $z=0$), find the exact scalar $W_{hh}$ that keeps the gradient magnitude constant over any gap when ${\tanh }^{\prime }$ sits at its maximum, and explain why this "perfect" value is still unusable in practice.

Recall Solution

We want $\gamma =1$. With $\max {\tanh }^{\prime }=1$, that forces $|W_{hh}|=1$. $|W_{hh}|\cdot 1=1\Rightarrow |W_{hh}|=1.$ Why it still fails: ${\tanh }^{\prime }(z)=1$ only at $z=0$. As soon as the pre-activation $z^{(i)}$ moves away from 0 (which it does the moment real data arrives), ${\tanh }^{\prime }(z^{(i)})<1$, so the actual factor drops below 1 and the gradient vanishes again. There is no fixed scalar that keeps $\gamma =1$ across the changing $z^{(i)}$ of a real sequence — precisely the impossibility that motivates gated architectures (Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU)).

---

Quick self-check

Predicted-answer cloze — cover the right side.