Worked examples — Bidirectional RNNs

This page is the "practice arena" for Bidirectional RNNs. The parent note built the machinery: a forward RNN reading left→right, a backward RNN reading right→left, and a way to glue their hidden states together. Here we drill that machinery against every kind of input it can meet — clean sequences, the two edge timesteps, length-one sequences, all-zero inputs, the difference between concatenation and summation, and one exam-style twist.

Everything is tiny on purpose. All numbers are hand-sized so you can check every arithmetic step yourself (and the Verify blocks re-check them by machine).

---

Notation refresher (nothing new, just naming the pieces)

We will use these symbols. Each is defined in words first.

• $x_t$ — the input at position $t$, a small list of numbers (a word's embedding). Picture it as an arrow of numbers dropped into the machine at step $t$.
• $x_t^{(a)},x_t^{(b)}$ — when we compare two different sentences in the same example, the parenthesised superscript is just a label naming which sentence (sentence $a$ vs sentence $b$). So $x_3^{(a)}$ means "the input at position 3 in sentence a." The superscript is not a power and never means multiplication.
• ${\overrightarrow{h}}_t$ — the forward hidden state: the forward RNN's memory after it has read words $1$ through $t$. The little right-arrow means "built while moving right."
• ${\overleftarrow{h}}_t$ — the backward hidden state: memory after reading words $T$ down to $t$. Left-arrow = "built while moving left."
• $T$ — the length of the sequence (how many words).
• $d_h$ — the hidden-state size: how many numbers make up one hidden state. If $d_h=3$ the memory is a list of three numbers; if $d_h=1$ it is a single number. We introduce this name only so we can say "how big is the memory" in one symbol.
• $\tanh (z)$ — a squashing function that takes any number $z$ and gently bends it into the range $-1$ to $+1$. We use it (not, say, plain addition) because without a squash the memory could blow up to huge values as it accumulates step after step.
• $[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$ — concatenation: literally lay the two lists end to end into one longer list.

To keep arithmetic by hand possible, every example uses scalar hidden states ($d_h=1$), so a "hidden state" is a single number, and the weights $W$ are single numbers too. The BiRNN recurrence in scalar form is:

Unless a problem says otherwise, our shared "toy weights" are: $w_{\overrightarrow{h}}=0.5,w_{\overrightarrow{x}}=1.0,b_{\overrightarrow{h}}=0,w_{\overleftarrow{h}}=0.5,w_{\overleftarrow{x}}=1.0,b_{\overleftarrow{h}}=0.$

The figure below is the map for the whole page — refer back to it as you work each example.

Figure 1 — the BiRNN wiring over three words (Cells A, B, C at a glance). Alt-text: three input boxes $x_1,x_2,x_3$ along the bottom; an orange forward chain of circles with rightward arrows (${\overrightarrow{h}}_1\to {\overrightarrow{h}}_2\to {\overrightarrow{h}}_3$); a teal backward chain of circles with leftward arrows (${\overleftarrow{h}}_3\to {\overleftarrow{h}}_2\to {\overleftarrow{h}}_1$); and a plum output box $h_t$ above each position fed by both chains.

How to read this figure. The bottom row is the three input words $x_1,x_2,x_3$. The orange row is the forward chain: its arrows point rightward, showing memory built left→right (${\overrightarrow{h}}_1\to {\overrightarrow{h}}_2\to {\overrightarrow{h}}_3$). The teal row is the backward chain: its arrows point leftward (${\overrightarrow{h}}_3\leftarrow \cdots$, i.e. ${\overleftarrow{h}}_3\to {\overleftarrow{h}}_2\to {\overleftarrow{h}}_1$). At each position the plum box on top is the stapled output $h_t=[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$, fed by both an orange arrow (from below) and a teal arrow. Notice the two chains flow in opposite directions but share the same inputs — that single picture is Cell A (interior), Cell B (leftmost column) and Cell C (rightmost column) all at once.

---

The scenario matrix

Every worked example below is tagged with a cell from this table. Together the ten examples touch every cell. ("$|x|$" below just means the size of the input, ignoring its sign — e.g. $|-5|=5$.)

Cell	Scenario class	What's tricky about it
A	Interior timestep, normal input	Both neighbours exist — the "textbook" step
B	Left boundary $t=1$	Forward uses ${\overrightarrow{h}}_0=0$; backward has already crossed the whole sentence
C	Right boundary $t=T$	Backward uses ${\overleftarrow{h}}_{T+1}=0$; forward has crossed everything
D	Length-one sequence $T=1$	Both directions collapse to a single input; are they equal?
E	All-zero / degenerate input	What memory survives when $x_t=0$ everywhere?
F	Negative-valued input	Sign flows through $\tanh$; check it stays in $(-1,1)$
G	Combine rule: concat vs sum	Same states, two glue rules, different downstream size
H	Real-world word problem (NER)	Future context flips a label
I	Limiting behaviour (large input size $	x
J	Exam twist: causal vs bidirectional	Why a BiRNN cannot be used to predict the next word

---

Example 1 — Interior timestep (Cell A)

Step 1 — compute ${\overrightarrow{h}}_1$ first (we need the previous memory). ${\overrightarrow{h}}_1=\tanh (0.5\cdot 0+1.0\cdot 0.4)=\tanh (0.4)\approx 0.3799$. Why this step? ${\overrightarrow{h}}_2$ depends on ${\overrightarrow{h}}_1$; the forward chain forces us to build memory in order, left to right.

Step 2 — feed ${\overrightarrow{h}}_1$ and $x_2$ into the cell. ${\overrightarrow{h}}_2=\tanh (0.5\cdot 0.3799+1.0\cdot 1.0)=\tanh (1.1900)\approx 0.8306$. Why this step? This is the actual interior recurrence: past memory ($0.5\cdot 0.3799$) plus fresh input ($1.0$), squashed.

Verify: the squash means $|{\overrightarrow{h}}_2|<1$, so it is smaller than the raw input $1.0$ — matches the forecast if you guessed "smaller." Numerically $0.8306<1$. ✓

---

Example 2 — Left boundary (Cell B)

Step 1 — forward part. From Example 1, ${\overrightarrow{h}}_1=\tanh (0.4)\approx 0.3799$. Why this step? At $t=1$ the previous state is the seed ${\overrightarrow{h}}_0=0$, so the forward memory knows only word 1.

Step 2 — backward part, which must start at $t=3$. ${\overleftarrow{h}}_3=\tanh (1.0\cdot (-0.5))=\tanh (-0.5)\approx -0.4621$ (uses seed ${\overleftarrow{h}}_4=0$). Why this step? The backward RNN begins at the far right, so we cannot get ${\overleftarrow{h}}_1$ without walking leftward from $t=3$.

Step 3 — march backward to $t=2$, then $t=1$. ${\overleftarrow{h}}_2=\tanh (0.5\cdot (-0.4621)+1.0\cdot 1.0)=\tanh (0.7690)\approx 0.6462$. ${\overleftarrow{h}}_1=\tanh (0.5\cdot 0.6462+1.0\cdot 0.4)=\tanh (0.7231)\approx 0.6183$. Why this step? Each leftward step folds one more future word into the backward memory.

Step 4 — staple. $h_1=[0.3799;0.6183]$.

Verify: the forward number saw 1 word; the backward number (by construction) has absorbed words 3, 2, and 1 — the whole future including the present. That asymmetry at the boundary is exactly why bidirectional is powerful at position 1. ✓

---

Example 3 — Right boundary (Cell C)

Step 1 — forward, march to $t=3$. ${\overrightarrow{h}}_2\approx 0.8306$ (Example 1). Then ${\overrightarrow{h}}_3=\tanh (0.5\cdot 0.8306+1.0\cdot (-0.5))=\tanh (-0.0847)\approx -0.0845$. Why this step? At the right boundary the forward memory has crossed the entire past (words 1–3).

Step 2 — backward at $t=3$. From Example 2, ${\overleftarrow{h}}_3\approx -0.4621$ — it used the seed ${\overleftarrow{h}}_4=0$, so it knows only word 3. Why this step? $t=T$ is where the backward RNN starts, so its memory is freshest (least context) here.

Step 3 — staple. $h_3=[-0.0845;-0.4621]$.

Verify: mirror image of Cell B — now the forward half carries the whole sentence and the backward half sees one word. Both boundaries are covered. ✓

---

Example 4 — Length-one sequence (Cell D)

Step 1 — forward. ${\overrightarrow{h}}_1=\tanh (0.5\cdot 0+1.0\cdot 0.7)=\tanh (0.7)\approx 0.6044$. Step 2 — backward. ${\overleftarrow{h}}_1=\tanh (0.5\cdot 0+1.0\cdot 0.7)=\tanh (0.7)\approx 0.6044$. Why these steps? With $T=1$ there are no neighbours, so both recurrences reduce to $\tanh (w_x{x}_1)$.

Step 3 — but note the general case. They are equal only because our two directions share the same weight values. In a real BiRNN the forward and backward weights are independent, so even at $T=1$ you would generally get ${\overrightarrow{h}}_1\ne {\overleftarrow{h}}_1$.

Verify: with identical toy weights both equal $\tanh (0.7)\approx 0.6044$. The forecast is conditionally right — equal here, not equal in general. ✓

---

Example 5 — Degenerate all-zero input (Cell E)

Step 1 — forward chain. ${\overrightarrow{h}}_1=\tanh (0.5\cdot 0+0)=\tanh (0)=0$. Then ${\overrightarrow{h}}_2=\tanh (0.5\cdot 0)=0$, and ${\overrightarrow{h}}_3=0$. Step 2 — backward chain. Symmetrically ${\overleftarrow{h}}_3={\overleftarrow{h}}_2={\overleftarrow{h}}_1=0$. Why this step? $\tanh (0)=0$, and with zero input every term inside is zero, so the memory can never leave the origin.

Verify: all $h_t=[0;0]$. This is the sanity anchor: no input, no signal. If your implementation ever produces nonzero states from all-zero inputs and zero bias, you have a bug (likely a nonzero bias). ✓

---

Example 6 — Negative input, sign through tanh (Cell F)

Step 1 — pre-activation. $z=0.5\cdot 0+1.0\cdot (-2.0)=-2.0$. Step 2 — squash. ${\overrightarrow{h}}_1=\tanh (-2.0)\approx -0.9640$. Why these steps? $\tanh$ is an odd function: negative in, negative out, but bounded strictly between $-1$ and $+1$.

Verify: $-0.9640$ keeps the sign of the input yet obeys $|\cdot |<1$. Negative inputs are safe — no special case needed. ✓

---

Example 7 — Concatenate vs sum (Cell G)

Step 1 — concatenation. $h_2^{\text{concat}}=[0.8306;0.6462]$, a 2-number list. Why? Concatenation keeps past and future memories side by side, so a downstream layer can weight them independently.

Step 2 — summation. $h_2^{\text{sum}}=0.8306+0.6462=1.4768$, a 1-number value. Why? Summation is cheaper (half the downstream weights) but blends the two memories, so the model can no longer tell "how much came from the future."

Verify: concat length is $2d_h=2$; sum length is $d_h=1$; and $0.8306+0.6462=1.4768$. Both rules are legitimate; concatenation is the default in LSTM and GRU Cells-based BiRNNs. ✓

---

Example 8 — Real-world NER: future context flips the label (Cell H)

We model the evidence signal as a scalar: let the embedding of the word after "Jordan" drive the backward state. Use $x_3^{(a)}=+1.0$ ("dunked" ≈ action/person cue) and $x_3^{(b)}=-1.0$ ("soon" ≈ travel/place cue). To isolate that future cue we set $x_2^{(a)}=x_2^{(b)}=0$ (the word "Jordan" itself contributes nothing distinguishing). Toy weights throughout, seed ${\overleftarrow{h}}_4=0$.

Step 1 — backward state at position 3. (a) ${\overleftarrow{h}}_3^{(a)}=\tanh (1.0\cdot (+1.0))=\tanh (1)\approx 0.7616$. (b) ${\overleftarrow{h}}_3^{(b)}=\tanh (1.0\cdot (-1.0))=\tanh (-1)\approx -0.7616$. Why this step? This is the first future cue the backward RNN encodes, and the two sentences already differ in sign here.

Step 2 — carry the cue leftward into position 2 (using $x_2=0$ so only the future cue shows). (a) ${\overleftarrow{h}}_2^{(a)}=\tanh (0.5\cdot 0.7616+1.0\cdot 0)=\tanh (0.3808)\approx +0.3635$. (b) ${\overleftarrow{h}}_2^{(b)}=\tanh (0.5\cdot (-0.7616)+1.0\cdot 0)=\tanh (-0.3808)\approx -0.3635$. Why this step? At position 2 (the word "Jordan") the backward memory now carries an opposite-signed signal for the two sentences. The forward memory at position 2, by contrast, has only read words 1–2 and cannot see the deciding future word at all.

Step 3 — staple to get the two combined states. Suppose the forward halves are (near) identical for both sentences, say ${\overrightarrow{h}}_2\approx 0.30$ (same first two words modelled as a shared cue). Then $h_2^{(a)}=[0.30;+0.3635]$ and $h_2^{(b)}=[0.30;-0.3635]$. Why this step? This is the vector a downstream classifier actually receives; the only coordinate that differs between the two sentences is the backward one.

Step 4 — read off the decision. The classifier sees $+0.3635$ (push toward PERSON) for sentence (a) vs $-0.3635$ (push toward LOCATION) for sentence (b). Same word "Jordan," opposite future, opposite label.

Verify: $\tanh (1)\approx 0.7616$; the position-2 backward signals are $+0.3635$ and $-0.3635$ — equal magnitude, opposite sign. Their difference is $0.3635-(-0.3635)=0.7270\ne 0$, exactly the separation a forward-only model cannot produce (its position-2 states are identical). This is the NER story of Named Entity Recognition, made numeric. ✓

---

Example 9 — Limiting behaviour: saturation (Cell I)

Step 1 — pre-activation for each. With ${\overrightarrow{h}}_0=0$ and $w_{\overrightarrow{x}}=1.0$, the value inside $\tanh$ is just $x_1$ itself: $z=5$, $z=10$, $z=50$. Why this step? Stripping the recurrence to one step lets us watch $\tanh$ alone.

Step 2 — squash each. ${\overrightarrow{h}}_1(5)=\tanh (5)\approx 0.99991$. ${\overrightarrow{h}}_1(10)=\tanh (10)\approx 0.99999999996$. ${\overrightarrow{h}}_1(50)=\tanh (50)\approx 1$ (to machine precision). Why this step? $\tanh$ saturates: once past a modest magnitude, multiplying the input by ten barely moves the output — it is already glued to the ceiling of $+1$.

Step 3 — read the consequence for learning. Near saturation the slope ${\tanh }^{\prime }(z)=1-{\tanh }^2(z)$ is almost $0$: at $z=5$ it is about $1-0.99991^2\approx 0.00018$. A nearly-flat slope means gradients passing through this unit nearly vanish — the same vanishing-gradient pressure that motivates the gated cells in LSTM and GRU Cells.

Verify: the three states are strictly increasing yet all pinned below $1$ ($0.99991<0.99999999996\le 1$), and the slope at $z=5$ is tiny ($\approx 0.00018$). The memory is bounded no matter how large the input — the whole point of the squash. ✓

---

Example 10 — Exam twist: why not use a BiRNN for next-word prediction (Cell J)

Step 1 — expand what $h_t$ contains. $h_t=[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$, and by definition ${\overleftarrow{h}}_t$ was built from $x_t,x_{t+1},\dots ,x_T$. Why this step? We must see exactly which inputs leak into the output before trusting it.

Step 2 — spot the leak. ${\overleftarrow{h}}_t$ already contains $x_{t+1}$ — the very word we are supposed to predict. Concretely for $t=2$ in a length-5 sentence, the backward state's input set is $\{2,3,4,5\}$, which includes index $3=t+1$. The model would be "predicting" an answer it has literally read. That is label leakage; the training accuracy is meaningless and it cannot run at test time (the future does not exist yet).

Step 3 — state the rule. BiRNNs require the full sequence available up front (tagging, translation encoders in Encoder-Decoder Architecture, classification). For generation you need a causal (forward-only) model, or the masked self-attention of Transformers.

Verify: the backward state's input set $\{t,t+1,\dots ,T\}$ contains index $t+1$ (checked for $t=2$: $3\in \{2,3,4,5\}$), contradicting the causality any predictor of position $t+1$ must obey. Hence the answer is no. ✓

---

Recall Self-check

At the last position $t=T$, which half of $h_T$ has seen only one word? ::: The backward half ${\overleftarrow{h}}_T$ (it just started, seeded by ${\overleftarrow{h}}_{T+1}=0$). With all-zero inputs and zero bias, what is every hidden state? ::: Exactly $0$, because $\tanh (0)=0$ propagates. Why can't a BiRNN predict the next token? ::: Its backward state already contains $x_{t+1}$, the target — label leakage. Concatenation vs summation: which preserves "how much came from the future"? ::: Concatenation (it keeps the two memories separate).

Related deep contexts: Attention Mechanism, Sequence-to-Sequence Models, and the Hinglish walkthrough 3.5.06 Bidirectional RNNs (Hinglish).