Recurrent Neural Networks (RNN) architecture

Overview

Recurrent Neural Networks (RNs) are neural network architectures designed to process sequential data by maintaining an internal hidden state that acts as memory. Unlike feedforward networks that treat each input independently, RNNs have connections that loop back on themselves, allowing information to persist across time steps.

Figure — Recurrent Neural Networks (RNN) architecture

Core Architecture Components

Definition RNN Cell

An RNN processes a sequence $\mathbf{x} = (x^{(1)}, x^{(2)}, \ldots, x^{(T)})$ one element at a time. At each time step $t$ : $h^{(t)} = \tanh(W_{h} h^{(t-1)} + W_{xh} x^{(t)} + b_h)$ $y^{(t)} = W_{hy} h^{(t)} + b_y$

Where:

$x^{(t)} \in \mathbb{R}^{d_x}$ = input at time $t$
$h^{(t)} \in \mathbb{R}^{d_h}$ hidden state at time $t$ (the "memory")
$y^{(t)} \in \mathbb{R}^{d_y}$ = output at time $t$
$W_{xh} \in \mathbb{R}^{d_h \times d_x}$ = input-to-hidden weights
$W_{hh} \in \mathbb{R}^{d_h \times d_h}$ = recurrent weights (hidden-to-hidden)
$W_{hy} \in \mathbb{R}^{d_y \times d_h}$ = hidden-to-output weights
$b_h, b_y$ = bias vectors

Derivation from First Principles

WHY this architecture?

Start with the problem: we want a function $f$ that maps a sequence to outputs, but the function should "remember" previous inputs.

Step 1: Encode memory as a vector

Let $h^{(t)}$ be our memory at time $t$
$h^{(t)}$ should depend on current input $x^{(t)}$ AND previous memory $h^{(t-1)}$

Step 2: Combine inputs linearly

Most general linear combination: $z = W_{h} h^{(t-1)} + W_{xh} x^{(t)} + b_h$
WHY linear? We can learn any non-linear transformation by composing linear maps with non-linearities

Step 3: Add non-linearity

Apply activation function: $h^{(t)} = \sigma(z)$
WHY $\tanh$ ? Outputs in $[-1, 1]$ , keeps gradients healthier than sigmoid during backprop, zero-centered

Step 4: Produce output

Linear projection: $y^{(t)} = W_{hy} h^{(t)} + b_y$
WHY separate from hidden state? Allows hidden dimension to differ from output dimension; hidden state can be richer

Unfolding Through Time

The "recurrence" can be visualized by unfolding the network through time:

x^(1)    x^(2)    x^(3)
 |        |        |
 ↓        ↓        ↓
[RNN] → [RNN] → [RNN]
  ↓        ↓        ↓
 y^(1)   y^(2)   y^(3)

Each box shares the SAME weights. The arrows between boxes represent $h^{(t-1)} \to h^{(t)}$ .

Worked example Example 1: Character-Level Language Model

Task: Predict next character in "HELLO" Setup:

Vocabulary: {H, E, L, O ...}, size $|V| = 26$
One-hot encode each character: $x^{(t)} \in \mathbb{R}^{26}$
Hidden dimension: $d_h = 128$
Output: probability distribution over next character $P(c_{t+1} | c_1, \ldots, c_t)$

Forward pass for "HEL": $t=1$ : Input "H" (one-hot: $x^{(1)} = [1, 0, 0, \ldots]$ )

$h^{(1)} = \tanh(W_{xh} x^{(1)} + b_h)$ (no previous state, $h^{(0)} = \mathbf{0}$ )
$y^{(1)} = W_{hy} h^{(1)} + b_y$ → softmax → predict "E"

WHY this step? Initialize hidden state from scratch since no prior context.

$t=2$ : Input "E" (one-hot: $x^{(2)} = [0, 1, 0, \ldots]$ )

$h^{(2)} = \tanh(W_{h} h^{(1)} + W_{xh} x^{(2)} + b_h)$
$y^{(2)} = W_{hy} h^{(2)} + b_y$ → softmax → predict "L"

WHY this step? $W_{hh} h^{(1)}$ carries information about seeing "H"; combined with new "E" input.

$t=3$ : Input "L"

$h^{(3)} = \tanh(W_{hh} h^{(2)} + W_{xh} x^{(3)} + b_h)$
$y^{(3)} = W_{hy} h^{(3)} + b_y$ → softmax → predict "L"

WHY this step? Now $h^{(3)}$ encodes "seen HE so far", helps predict second "L".

Worked example Example 2: Sequence Classification (Sentiment Analysis)

Task: Classify "This movie rocks!" as positive

Setup:

Process word by word: ["This", "movie", "rocks", "!"]
Use word embeddings: $x^{(t)} \in \mathbb{R}^{300}$ (e.g., Word2Vec)
Hidden state: $d_h = 256$
Output: Take ONLY final hidden state $h^{(T)}$ → linear layer → sigmoid → positive/negative

Forward pass:

$t=1$ : "This" → $h^{(1)} = \tanh(W_{xh} x^{(1)} + b_h)$ $t=2$ : "movie" → $h^{(2)} = \tanh(W_{hh} h^{(1)} + W_{xh} x^{(2)} + b_h)$ $t=3$ : "rocks" → $h^{(3)} = \tanh(W_{hh} h^{(2)} + W_{xh} x^{(3)} + b_h)$ $t=4$ : "!" → $h^{(4)} = \tanh(W_{hh} h^{(3)} + W_{xh} x^{(4)} + b_h)$ Classification: $p(\text{positive}) = \sigma(w^T h^{(4)} + b)$

WHY only final state? $h^{(4)}$ has "read" the entire sentence, acts as a summary. We don't need intermediate predictions for classification.

Training: Backpropagation Through Time (BPTT)

Deriving Gradient for $W_{hh}$ :

By chain rule: $\frac{\partial \mathcal{L}}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial L^{(t)}}{\partial W_{hh}}$

For time step $t$ : $\frac{\partial L^{(t)}}{\partial W_{hh}} = \frac{\partial L^{(t)}}{\partial y^{(t)}} \frac{\partial y^{(t)}}{\partial h^{(t)}} \frac{\partial h^{(t)}}{\partial W_{hh}}$

But $h^{(t)}$ depends on $h^{(t-1)}$ , which depends on $h^{(t-2)}$ , ... back to $h^{(1)}$ !

Full gradient: $\frac{\partial L^{(t)}}{\partial W_{hh}} = \sum_{k=1}^{t} \frac{\partial L^{(t)}}{\partial y^{(t)}} \frac{\partial y^{(t)}}{\partial h^{(t)}} \frac{\partial h^{(t)}}{\partial h^{(k)}} \frac{\partial h^{(k)}}{\partial W_{hh}}$

Where: $\frac{\partial h^{(t)}}{\partial h^{(k)}} = \prod_{i=k+1}^{t} \frac{\partial h^{(i)}}{\partial h^{(i-1)}} = \prod_{i=k+1}^{t} W_{h}^T \cdot \text{diag}(\tanh'(z^{(i)}))$

WHY this matters? This product of Jacobians causes vanishing/exploding gradients.

RNN Variants and Types

Common Mistakes and Fixes

Common mistake Mistake 1: Forgetting to Reset Hidden State Between Sequences

Wrong idea: Reuse $h^{(T)}$ from previous training example as $h^{(0)}$ for next example. Why it feels right: "Maybe the network can learn connections across examples?"

The reality: Training examples are independent. Carrying over state creates spurious correlations. The network tries to find patterns where none exist.

Fix: Always initialize $h^{(0)} = \mathbf{0}$ (or learned vector) at the start of each new sequence.

# Wrong
for batch in dataset:
    h = model(batch, h)  # h carries over!
 
# Right
for batch in dataset:
    h = torch.zeros(hidden_size)  # Reset!
    h = model(batch, h)

Common mistake Mistake 2: Not Truncating BPTT for Long Sequences

Wrong idea: Backpropagate through entire sequence no matter how long.

Why it feels right: "Full gradients are more accurate."

The reality: For sequences of length 1000+, memory explodes (need to store all intermediate activations) and vanishing gradients kill learning anyway.

Fix: Truncated BPTT: Backprop only through $k$ steps (e.g., $k=20$ ). Still forward-prop through entire sequence, but detach gradient every $k$ steps.

for t in range(0, T, truncate_len):
    loss = 0
    for in range(truncate_len):
        h = rnn_step(x[t+i], h)
        loss += criterion(h, target[t+i])
    loss.backward()  # Only backprop through truncate_len steps
    h = h.detach()  # Detach for next chunk

Key Properties

Property	Description
Parameter Sharing	Same weights across all time steps → handles variable-length sequences
Hidden State	$h^{(t)}$ acts as memory, summary of sequence up to time $t$
Inductive Bias	Assumes nearby elements are more relevant than distant ones
Limitation	Vanishing gradients → poor long-term dependencies (fixed by LSTM/GRU)

Computational Complexity

For a sequence of length $T$ :

Forward pass: $O(T \cdot d_h^2)$ (matrix multiplications at each step)
Backward pass (BPTT): $O(T \cdot d_h^2)$ (same, but gradients flow backward)
Space: $O(T \cdot d_h)$ (store hidden states for backprop)

Comparison to feedforward:

Feedforward: $O(1)$ for one input, no sequential dependency
RNN: $O(T)$ sequential operations, cannot parallelize across time

This sequential nature is RNN's key limitation compared to Transformers.

Recall Explain to a 12-year-old

Imagine you're reading a detective story. Each page gives you a clue. A normal robot brain (feedforward network) would look at each page separately and forget what it read before—useless for solving the mystery!

An RNN is like a detective's notebook. When you read page1, you write down important stuff in your notebook. On page 2, you read the NEW clue AND check your notebook to remember what you learned before. You update your notebook with this combined information. Page 3? Same thing—read new clue, check notebook, update.

The notebook is the "hidden state" ( $h$ ). The detective's brain uses the same thinking rules (weights $W$ ) for every page, but the notebook contents change as you learn more. By the last page, your notebook has clues from the ENTIRE book, helping you say "The butler did it!"

Problem: If the book is 1000 pages long, the notebook might forget stuff from page 1 by page 1000—that's the vanishing gradient problem. Better detectives (LSTM, GRU) have special notebooks that can remember longer!

Connections

Long Short-Term Memory (LSTM) - Solves vanishing gradient with gating mechanisms
Gated Recurrent Unit (GRU) - Simpler alternative to LSTM with fewer parameters
Backpropagation Through Time (BPTT) - Training algorithm for RNNs
Vanishing and Exploding Gradients - Core problem limiting vanilla RNNs
Sequence-to-Sequence Models - Encoder-decoder architecture using RNNs
Attention Mechanism - Allows focusing on relevant parts of input sequence
Transformers - Modern alternative that paralelizes sequence processing
Word Embeddings - Common input representation for NLP with RNNs
Gradient Clipping - Technique to prevent exploding gradients

#flashcards/ai-ml

What is the purpose of the hidden state in RNNs? :: The hidden state $h^{(t)}$ acts as memory, encoding information about all previous time steps, allowing the network to maintain context across the sequence.

Why do RNNs share weights across time steps?

Weight sharing allows RNNs to handle variable-length sequences with a fixed number of parameters, and provides an inductive bias that the sequence rules don't change over time (analogous to spatial weight sharing in CNNs).

Write the RNN hidden state update equation

h^{(t)} = \tanh(W_{hh} h^{(t-1)} + W_{xh} x^{(t)} + b_h)

where

W_{hh}

are recurrent weights,

W_{xh}

are input weights, and

\tanh

is the activation function.

What causes vanishing gradients in RNNs?

During backpropagation through time, gradients involve products of Jacobians

\prod_{i} W_{hh}^T \cdot \text{diag}(\tanh')

. If

||W_{hh}|| < 1

, repeated multiplication shrinks gradients exponentially, making it impossible to learn long-term dependencies.

What is the difference between many-to-one and many-to-many RNN architectures?

Many-to-one processes an input sequence and produces a single output (e.g., sentiment classification using final hidden state). Many-to-many processes an input sequence and produces an output sequence (e.g., machine translation), with potentially different input/output lengths.

Why initialize

h^{(0)} = \mathbf{0}

at the start of each new sequence?

Training examples are independent. Carrying over hidden state from a previous sequence creates spurious correlations, causing the network to learn patterns that don't actually exist across different sequences.

What is truncated BPTT and why use it?

Truncated Backpropagation Through Time limits gradient computation to

k

time steps instead of the full sequence. This reduces memory usage and computation for long sequences, while still allowing forward propagation through the entire sequence.

How do you produce output from an RNN hidden state?

Apply a linear projection:

y^{(t)} = W_{hy} h^{(t)} + b_y

, then apply task-specific activation (softmax for classification, none for regression). This allows hidden dimension to differ from output dimension.

What is the computational complexity of RNN forward pass for sequence length

T

O(T \cdot d_h^2)

where

d_h

is hidden dimension. This is due to matrix multiplications at each time step. Unlike feedforward networks, RNN operations are sequential and cannot be parallelized across time.

Why use

\tanh

activation instead of sigmoid RNN hidden state?

\tanh

outputs in range

[-1, 1]

(zero-centered) instead of sigmoid's

[0, 1]

, which helps maintain healthier gradients during backpropagation and prevents bias shift issues in the hidden state.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, RNN ka core idea bahut simple hai — jab hum ek sentence padhte hain word by word, to "mat" tak pahuchne se pehle humein "cat" aur "sat" yaad rehta hai. Normal feedforward networks mein yeh memory nahi hoti, woh har input ko alag-alag treat karte hain. Isliye RNN ek hidden state (basically ek memory vector) rakhta hai jo time ke saath aage travel karta hai. Har time step par network current input ke saath purani memory ko combine karta hai, ek formula se: naya hidden state = tanh(purani memory + current input). Yahi loop-back connection RNN ko "sequential" data ke liye perfect banata hai.

Ab sabse important baat jo yaad rakhni hai woh hai weight sharing. Matlab jo weight matrices hain ( $W_{xh}$ , $W_{hh}$ , $W_{hy}$ ), woh har time step par same rehte hain — badalte nahi. Yeh isliye zaroori hai kyunki isse humein sirf ek fixed set of parameters chahiye, chahe sequence kitna bhi lamba ho. Aur jo pattern network position 1 par seekhta hai, woh position 5 par bhi apply ho jaata hai. Yeh CNN ke spatial weight sharing jaisa hi hai, bas yahan hum time ke across share kar rahe hain. Iska matlab hai ek strong assumption — ki "sequence ke rules time ke saath change nahi hote".

Yeh concept kyun matter karta hai? Kyunki real-world mein bahut saara data sequential hota hai — text, speech, stock prices, weather, sab kuch. In sab mein order aur context important hota hai, aur RNN humein woh "memory" deta hai jisse machine bhi context samajh sake. Language models, translation, speech recognition — yeh sab RNN (aur uske advanced versions jaise LSTM, GRU) ki foundation par khade hain. Toh agar aap sequence models ko samajhna chahte ho, to yeh hidden state aur weight sharing ka intuition aapki base hai — ise achhe se pakad lo, aage sab kuch isi par build hoga.

Recurrent Neural Networks (RNN) architecture

Overview

Core Architecture Components

Derivation from First Principles

Unfolding Through Time

Training: Backpropagation Through Time (BPTT)

RNN Variants and Types

Common Mistakes and Fixes

Key Properties

Computational Complexity

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Sequence Models

Connections