Recurrent Neural Networks (RNN) architecture: Question bank

This page is a misconception hunt. Every item below is a place where students of RNN architecture reliably slip. Read each prompt, commit to an answer out loud, THEN reveal. If your reasoning matched the justification, great — if only your verdict matched but your reason didn't, that's a trap you nearly fell into.

Before we start, one shared vocabulary reminder so no symbol is used unexplained:

Recall What each symbol means (tap to expand)

$x^{(t)}$ — the input vector fed in at time step $t$ (one word, one character, one frame).
$h^{(t)}$ — the hidden state: the network's "memory so far", a vector carried forward.
$y^{(t)}$ — the output produced at step $t$ .
$W_{x h}, W_{hh}, W_{h y}$ — the three weight matrices (input→hidden, hidden→hidden, hidden→output). The same three are reused at every step.
$tanh$ — the squashing function that keeps $h^{(t)}$ inside $[- 1, 1]$ .
$T$ — the length of the sequence.

True or false — justify

TF1. "An RNN has a separate set of weights for each time step."

False. The same

W_{x h}, W_{hh}, W_{h y}

are shared across all steps; that weight-sharing is exactly what lets one network handle sequences of any length with a fixed parameter count.

TF2. "Because weights are shared, an RNN cannot behave differently at step 1 versus step 50."

False. The weights are fixed but

h^{(t)}

changes every step, so the same function produces different outputs — the memory, not the weights, carries the position-specific behaviour.

TF3. "The hidden state

h^{(t)}

and the output

y^{(t)}

are always the same thing."

False.

y^{(t)} = W_{h y} h^{(t)} + b_{y}

is a separate linear projection, so the output dimension

d_{y}

can differ from the hidden dimension

d_{h}

, and the hidden state can stay richer than what is exposed.

TF4. "An RNN is a feedforward network — it just has more layers."

Partly, but the key difference is weight-tying. When you unfold it, it looks like a deep feedforward net, but every "layer" reuses identical weights and each layer also receives a fresh external input

x^{(t)}

; a plain deep net has independent weights per layer and one input.

TF5. "Using

tanh

instead of sigmoid completely solves the vanishing gradient problem."

False.

tanh

is zero-centered and has slightly healthier gradients, but its derivative is still

\leq 1

, so long products of Jacobians still shrink. Solving vanishing gradients properly needs gated cells like LSTM or GRU.

TF6. "If

∥ W_{hh} ∥ > 1

the gradients will vanish."

False — it's the opposite. Repeated multiplication by a matrix with spectral norm above 1 makes gradients explode; norm below 1 makes them vanish. See Vanishing and Exploding Gradients.

TF7. "For a many-to-one classifier you must produce an output

y^{(t)}

at every time step."

False. You compute every hidden state but only read out the final

h^{(T)}

as the summary; intermediate outputs are simply not used for the loss.

TF8. "RNNs process a whole sequence in parallel like a fully-connected layer."

False. The recurrence

h^{(t)} = tanh (W_{hh} h^{(t - 1)} + \dots)

makes step

t

depend on step

t - 1

, so computation is inherently sequential — this is exactly what Transformers later remove with attention.

TF9. "Cross-entropy is the only valid loss for an RNN."

False. Cross-entropy suits classification/next-token prediction; regression targets use MSE. The loss just sums per-step contributions

L = \sum_{t} L (y^{(t)}, \overset{y}{^}^{(t)})

TF10. "One-hot inputs and word embeddings are interchangeable with no downstream effect."

False. One-hot vectors are sparse and orthogonal (no similarity notion); embeddings place similar words nearby, so the RNN starts from a far richer representation.

Spot the error

SE1. "I carried the last hidden state of training example A into example B as

h^{(0)}

, so the net learns cross-example links."

Error: distinct training examples are independent; carrying state invents correlations that don't exist. Reset

h^{(0)} = 0

(or a learned init) at each new sequence.

SE2. "My BPTT gradient for

W_{hh}

only uses

\partial L^{(t)} / \partial h^{(t)}

at the same step

t

Error:

h^{(t)}

depends on

h^{(t - 1)}, h^{(t - 2)}, \dots

, so the gradient must sum contributions back to

k = 1

through

\frac{\partial h ^{(t)}}{\partial h ^{(k)}}

. Ignoring them drops all long-range learning. See Backpropagation Through Time (BPTT).

SE3. "To stop exploding gradients I lowered the learning rate a lot."

Error: a small learning rate hides the symptom but the gradient direction is still corrupted by huge magnitudes. The right tool is Gradient Clipping, which rescales the gradient vector while keeping its direction.

SE4. "I used a different

W_{x h}

at each time step to give the model more capacity."

Error: that breaks weight-sharing — parameters now scale with sequence length, generalization across positions is lost, and variable-length sequences become impossible.

SE5. "For sentiment analysis I applied softmax over the whole hidden-state vector

h^{(T)}

to get the class."

Error:

h^{(T)}

isn't a probability distribution over classes. You must first project it, e.g.

σ (w^{⊤} h^{(T)} + b)

, then interpret; softmaxing raw hidden units is meaningless.

SE6. "The recurrent matrix is

W_{hh} \in R^{d_{x} \times d_{x}}

Error:

W_{hh}

maps hidden→hidden, so it is

d_{h} \times d_{h}

. The one that touches the input dimension

d_{x}

W_{x h} \in R^{d_{h} \times d_{x}}

SE7. "In an encoder–decoder, the decoder reads every encoder hidden state directly at each step."

Error (for the vanilla model): the plain encoder–decoder passes only the encoder's final

h^{(T_{e n c})}

as one context vector. Reading all encoder states is precisely the attention upgrade to plain seq2seq.

SE8. "I forgot the bias

b_{h}

, but that's fine since

tanh (0) = 0

anyway."

Error: without

b_{h}

the pre-activation is forced through the origin, removing the network's ability to shift the activation threshold — a real loss of expressive power, not a harmless simplification.

Why questions

WHY1. Why is the same weight matrix reused at every time step instead of one per step?

So the model has a fixed parameter count independent of sequence length, and any pattern learned at one position transfers to all others — encoding the prior "the rules of the sequence don't change over time."

WHY2. Why does

tanh

appear rather than a raw linear combination?

A pure linear recurrence would collapse to a single linear map no matter how many steps; the non-linearity lets the network represent complex, non-linear temporal patterns.

WHY3. Why do vanishing/exploding gradients arise specifically in RNNs and not obviously in shallow feedforward nets?

Because

\frac{\partial h ^{(t)}}{\partial h ^{(k)}}

is a product of

t - k

Jacobians

W_{hh}^{⊤} diag (tanh^{'})

; repeated multiplication drives the magnitude toward

0

\infty

exponentially in the gap length.

WHY4. Why does this gradient problem cap vanilla RNN memory at roughly ten steps?

Once the Jacobian product has shrunk below numerical relevance, gradients from distant steps stop influencing the weights, so dependencies beyond that horizon simply can't be learned.

WHY5. Why prefer taking only

h^{(T)}

for classification instead of averaging all

h^{(t)}

The final state has "read" the entire sequence and can act as a running summary; averaging is a valid alternative but the final-state design assumes the recurrence already accumulates the needed information.

WHY6. Why is

y^{(t)}

computed as a separate projection rather than reusing

h^{(t)}

directly?

To decouple the internal memory size

d_{h}

from the required output size

d_{y}

, letting the hidden state stay high-dimensional and rich while the output matches the task.

WHY7. Why is BPTT called through time rather than just backpropagation?

Because the network is unfolded across time steps and the error signal is propagated backward along the temporal chain of hidden states, not merely through stacked spatial layers.

WHY8. Why does gradient clipping fix explosion but not vanishing?

Clipping only caps large gradients; it cannot amplify a gradient that has already decayed to near zero. Vanishing needs architectural fixes (gates) instead.

Edge cases

EC1. What is

h^{(0)}

at the very first step, and why?

Conventionally

h^{(0)} = 0

(or a learned vector) because there is no prior context; the first hidden state

h^{(1)} = tanh (W_{x h} x^{(1)} + b_{h})

then depends on the input alone.

EC2. What happens if you feed a sequence of length

T = 1

The recurrent term

W_{hh} h^{(0)}

contributes nothing (with

h^{(0)} = 0

), so the RNN degenerates into an ordinary feedforward layer on that single input.

EC3. What if two sequences in a batch have different lengths?

You typically pad to a common length and mask the padded positions out of the loss, so pad tokens don't contribute gradients — otherwise the model learns from meaningless filler.

EC4. In a one-to-many generator, where does the single input enter if there's only one

x

It initializes

h^{(0)}

(or is fed at step 1); subsequent steps run autoregressively, feeding each generated output back in as the next input.

EC5. If

W_{hh}

were exactly the zero matrix, how would the RNN behave?

Memory is severed —

h^{(t)} = tanh (W_{x h} x^{(t)} + b_{h})

depends only on the current input, making it a memoryless per-step feedforward net with no sequence modelling at all.

EC6. What does an all-ones input (or constant repeated token) reveal about the network?

The hidden state still evolves because each step re-applies

W_{hh}

to the previous state, so even identical inputs can produce changing outputs — the memory dynamics, not the input, drive the change.

EC7. What is the largest and smallest possible value of any component of

h^{(t)}

, and why does that bound matter?

Strictly between

- 1

and

1

because of

tanh

; this bounded range is what keeps activations from blowing up forward, even though it does not by itself fix gradient vanishing backward.

Recall Fast self-test before moving on

Can you, without looking, state (a) which weight matrix is $d_{h} \times d_{h}$ , (b) which condition on $∥ W_{hh} ∥$ causes explosion, (c) why $h^{(0)}$ is zeroed between examples, and (d) which tool fixes exploding but not vanishing gradients? Answers: (a) $W_{hh}$ , (b) $∥ W_{hh} ∥ > 1$ , (c) examples are independent, (d) gradient clipping.