Visual walkthrough — Bidirectional RNNs

---

Step 1 — One word at a time: what "reading left to right" means

WHAT. Imagine a sentence laid out as beads on a string: $x_1,x_2,x_3,\dots ,x_T$. Each $x_t$ is one word turned into a little list of numbers (an embedding) — think of it as the word's "fingerprint." The subscript $t$ is just the position: $t=1$ is the first word, $t=T$ is the last ($T$ = total number of words).

WHY. Before any mathematics, we must agree on what we are sliding our finger over. Everything later is "what does my finger know so far?" — so we first draw the finger.

PICTURE. The blue arrow is the finger. It sits on word $x_3$. Everything to its left (lavender) it has already read; everything to its right (grey) it has not seen yet.

---

Step 2 — Memory: the hidden state ${\overrightarrow{h}}_t$

WHAT. We want a running summary of everything the finger has read so far. Call it ${\overrightarrow{h}}_t$ — the little right-pointing arrow on top means "this summary was built reading left→right." It is itself a short list of numbers; say it has dimension $d$ (we get to choose $d$ — a bigger $d$ means a roomier memory).

WHY tanh, and why this exact recipe? We need to fold the new word into the old summary. The natural recipe is:

${\overrightarrow{h}}_t=\tanh (\underset{\text{re-use old memory}}{\underbrace{W_{\overrightarrow{h}}{\overrightarrow{h}}_{t-1}}}+\underset{\text{stir in new word}}{\underbrace{W_{\overrightarrow{x}}{x}_t}}+\underset{\text{a nudge}}{\underbrace{b_{\overrightarrow{h}}}})$

Term by term, right where each lives:

• ${\overrightarrow{h}}_{t-1}$ = the summary one word ago (memory of the past); a list of $d$ numbers.
• $W_{\overrightarrow{h}}$ = a matrix (a grid of learned numbers) that reshapes that old memory — "how much of yesterday do I carry forward?" It must turn a $d$-list into a $d$-list, so its shape is $d\times d$.
• $x_t$ = the new word's fingerprint; a list of $d_{\text{embed}}$ numbers.
• $W_{\overrightarrow{x}}$ = a matrix that reshapes the new word — "how does this word change my summary?" It turns a $d_{\text{embed}}$-list into a $d$-list, so its shape is $d\times d_{\text{embed}}$.
• $b_{\overrightarrow{h}}$ = a constant bias vector of length $d$, a small default push.
• $\tanh (\cdot )$ = the squash. It takes any number and gently bends it into the range $[-1,1]$.

Why $\tanh$ and not just addition? Without a squash, adding word after word could let the numbers grow without bound (explode) or the model could only ever draw straight-line relationships. $\tanh$ answers the question "how do I keep values bounded AND let the model bend, not just add?" Its S-shape saturates: huge inputs flatten out near $\pm 1$.

PICTURE. Left: the S-curve of $\tanh$ — see how big positive inputs flatten toward $+1$, big negatives toward $-1$. Right: the "fold" — old memory (lavender) and new word (coral) pour into the box, tanh squashes, out comes the new memory.

---

Step 3 — The forward pass grows a summary, word by word

WHAT. Start the finger before the first word with an empty memory ${\overrightarrow{h}}_0=0$ (all zeros — "I know nothing yet"). Then apply Step 2's recipe once per word.

WHY start at zero? At $t=1$ there is no previous word, so ${\overrightarrow{h}}_0$ must be a neutral placeholder. Zero means "no prior belief."

Watch what each summary contains:

• ${\overrightarrow{h}}_1$ = knows $x_1$ only.
• ${\overrightarrow{h}}_2$ = knows $x_1,x_2$ (because it was built from ${\overrightarrow{h}}_1$).
• ${\overrightarrow{h}}_T$ = knows the whole past $x_1,\dots ,x_T$.

PICTURE. A chain: each box passes its memory (lavender arrow) rightward while a new word (coral) drops in from below. The shaded region under each box shows how much of the sentence that summary has absorbed — it grows left to right.

Recall

Why does ${\overrightarrow{h}}_2$ "know" $x_1$? ::: Because it is computed from ${\overrightarrow{h}}_1$, which was computed from $x_1$. Memory chains backward through the arrows.

---

Step 4 — The blind spot: the forward finger cannot see the future

WHAT. Consider tagging the first word of "Apple released iOS." At $t=1$ the forward summary ${\overrightarrow{h}}_1$ has read only the word "Apple." Is it a fruit or a company? The word alone cannot say.

WHY this is the whole motivation. The forward RNN is causally blind: at position $t$ it has zero information about $x_{t+1},\dots ,x_T$. For tasks where the answer depends on later words (as in Named Entity Recognition), this is fatal.

PICTURE. The forward finger on "Apple" — the words after it are behind a fog. The label bubble wavers between FRUIT and ORG.

---

Step 5 — The backward pass: a second finger reading right to left

WHAT. A mirror RNN, with its own separate weights, starts at the last word and marches left. Its summaries wear a left-pointing arrow: ${\overleftarrow{h}}_t$, also of length $d$.

${\overleftarrow{h}}_t=\tanh (\underset{\text{memory of the future}}{\underbrace{W_{\overleftarrow{h}}{\overleftarrow{h}}_{t+1}}}+\underset{\text{current word}}{\underbrace{W_{\overleftarrow{x}}{x}_t}}+\underset{\text{nudge}}{\underbrace{b_{\overleftarrow{h}}}})$

The only change from Step 2 is the index: ${\overrightarrow{h}}_{t-1}$ became ${\overleftarrow{h}}_{t+1}$. The "previous" summary is now the one to the right, because this finger walks the other way. The shapes match Step 2 exactly: $W_{\overleftarrow{h}}$ is $d\times d$ and $W_{\overleftarrow{x}}$ is $d\times d_{\text{embed}}$.

WHY separate weights $W_{\overleftarrow{h}},W_{\overleftarrow{x}}$? Reading a language backward is a different skill from reading it forward. Forcing both fingers to share one set of weights would make each learn a blurry compromise. Independent parameters let each specialise.

Where does it start? ${\overleftarrow{h}}_{T+1}=0$ — an empty memory just past the end of the sentence, the mirror of ${\overrightarrow{h}}_0$.

Now the summaries absorb the future:

• ${\overleftarrow{h}}_T$ = knows $x_T$ only.
• ${\overleftarrow{h}}_{T-1}$ = knows $x_{T-1},x_T$.
• ${\overleftarrow{h}}_1$ = knows the whole sentence from the right.

PICTURE. The lower chain (coral finger) walks right→left; its shaded region grows from the right edge inward — the mirror image of Step 3.

---

Step 6 — The handshake: concatenation gives $h_t$

WHAT. At every position $t$ we now have two summaries: ${\overrightarrow{h}}_t$ (past) and ${\overleftarrow{h}}_t$ (future). Glue them end to end:

$h_t=[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$

The semicolon $[\cdot ;\cdot ]$ means concatenation — simply lay the second list right after the first. If each arrow-summary is a list of $d$ numbers, then $h_t$ is a list of $2d$ numbers — the past block, then the future block.

WHY concatenate rather than add? Adding, ${\overrightarrow{h}}_t+{\overleftarrow{h}}_t$, blends past and future into one soup — a big past value can cancel a big future value and you lose which was which. Concatenation keeps both blocks intact, so the next layer can decide, per feature, how much to trust past vs. future. It answers "how do I combine two summaries without losing either?"

PICTURE. At position 1 ("Apple"), the lavender past-block (${\overrightarrow{h}}_1$, weak: "just saw Apple") and the mint future-block (${\overleftarrow{h}}_1$, strong: "…released iOS in California") snap together into one long bar $h_1$.

---

Step 7 — Decide: softmax turns $h_t$ into a labelled guess

WHAT. Push $h_t$ through one more matrix and a softmax to get a probability for each possible label:

$y_t=\text{softmax}(\underset{\text{score per label}}{\underbrace{W_y{h}_t}}+b_y)$

• $W_y{h}_t+b_y$ = a raw score for each class. If there are $C$ classes, $W_y$ has shape $C\times 2d$ (it swallows the length-$2d$ vector $h_t$ and produces one score per class), and $b_y$ has length $C$.
• $\text{softmax}$ = turns those raw scores into positive numbers that add up to 1 — a probability distribution.

WHY softmax? We are choosing one label out of several. We need the outputs to be non-negative and to sum to 1 so they read as probabilities. Softmax answers exactly "how do I turn arbitrary scores into a legal probability distribution?" by exponentiating (making everything positive) and dividing by the total (making it sum to 1).

PICTURE. For "Apple": raw scores (grey bars) → softmax → probabilities (coloured bars). B-ORG towers at $0.85$; the model commits.

---

Step 8 — The edge and degenerate cases

Every reader must know what happens at the boundaries. The figure below has four mini-panels, top-left to bottom-right, matching Cases A, B, C, D in order.

Case A — the very first / very last word (top-left panel). At $t=1$: ${\overrightarrow{h}}_1$ uses ${\overrightarrow{h}}_0=0$, so it sees only $x_1$. At $t=T$: ${\overleftarrow{h}}_T$ uses ${\overleftarrow{h}}_{T+1}=0$, so it sees only $x_T$. But the combined $h_1$ and $h_T$ are still full-context, because the other direction supplies the rest — the top-left panel shows the mint future-arrow filling in behind the lone word $x_1$. No position is ever context-starved — that is the payoff.

Case B — a one-word sentence, $T=1$ (top-right panel). Both fingers see the same single word. ${\overrightarrow{h}}_1$ and ${\overleftarrow{h}}_1$ are computed from $x_1$ alone (with different weights), then concatenated. The architecture still works; there is simply no future or past to add — the top-right panel shows the single shared word.

Case C — streaming / real-time input (bottom-left panel). The backward finger needs the last word to start. If words arrive one at a time (live speech, live typing), $\overleftarrow{h}$ cannot be computed until the sentence ends. The bottom-left panel draws a clock with the backward arrow greyed out and blocked. Bidirectional RNNs are therefore unusable for strictly causal, streaming tasks — a genuine limitation, not a bug. This is one reason later models like the Attention Mechanism and Transformers were developed.

Case D — the empty sequence, $T=0$ (bottom-right panel). Nothing to read; both chains return their zero initial states and no $h_t$ is produced. A well-guarded implementation short-circuits here — the bottom-right panel is an empty, dashed box.

---

The one-picture summary

Two fingers, opposite directions, meeting at every word to shake hands into one long summary that then votes on a label.

Recall Feynman retelling — say it like a story

Picture a mystery novel and two friends. Friend One reads from page one forward, keeping a running "here's what I know so far" note — that note is ${\overrightarrow{h}}_t$, and it grows as she reads (Steps 2–3). But on any single page she has no idea how the story ends (Step 4 — the blind spot). So Friend Two reads the same book backwards from the last page, keeping his own note ${\overleftarrow{h}}_t$ that captures "here's what's still to come" (Step 5). At every page the two friends staple their notes together side by side — that stapled pair is $h_t$, and stapling (concatenation) rather than averaging means neither friend's insight is smudged out (Step 6). Finally a judge reads the stapled note and, using softmax, announces the most probable label for that page — like B-ORG, the beginning of a company name (Step 7). The only catch: Friend Two can't start until the book is finished — so this trick only works when you already have the whole sequence, not live word-by-word (Step 8, Case C). That single caveat is why streaming systems reach instead for Attention Mechanism and Transformers. See also how these summaries feed Sequence-to-Sequence Models and the Encoder-Decoder Architecture, and how gated cells like LSTM and GRU Cells can replace the plain $\tanh$ box in each direction.

---