Exercises — Bidirectional RNNs

• $x_t$ ::: the input vector at timestep $t$ (a word embedding, a vector of numbers).
• ${\overrightarrow{h}}_t$ ::: the forward hidden state, built from $x_1,\dots ,x_t$ (the past).
• ${\overleftarrow{h}}_t$ ::: the backward hidden state, built from $x_T,\dots ,x_t$ (the future).
• $[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$ ::: concatenation — stack the two vectors end to end.
• $\tanh$ ::: a squashing function that maps any real number into $(-1,1)$.
• $\text{softmax}$ ::: turns a vector of scores into probabilities that sum to $1$.

Need a BiRNN: (b) and (d). Both label a word using context that can appear after it, and the full sequence is available.

• (a) is causal — you only have the past by definition, so a forward RNN is correct.
• (c) is online / real-time: the future frames have not arrived yet, so a BiRNN cannot be used (it would need to wait for the whole utterance).
• (b), (d): full sequence present, answer depends on later words → BiRNN wins.

• Concatenation stacks them: dimension $=5+5=10$.
• Summation adds element-wise: both must already match ($5$ and $5$), and the result stays $5$. Concatenation keeps past and future separate; summation mixes them into one 5-vector.

• ${\overrightarrow{h}}_0=0$ seeds the forward RNN before the first word, so ${\overrightarrow{h}}_1$ depends only on $x_1$.
• ${\overleftarrow{h}}_{T+1}=0$ seeds the backward RNN after the last word, so ${\overleftarrow{h}}_T$ depends only on $x_T$. Look at figure s01: the two zero-boxes sit at the two ends of the strip.

Start ${\overrightarrow{h}}_0=0$.

• ${\overrightarrow{h}}_1=\tanh (0.5\cdot 0+1.0\cdot 0.5)=\tanh (0.5)=0.4621$
• ${\overrightarrow{h}}_2=\tanh (0.5\cdot 0.4621+1.0\cdot (-1.0))=\tanh (-0.7689)=-0.6462$
• ${\overrightarrow{h}}_3=\tanh (0.5\cdot (-0.6462)+1.0\cdot 0.2)=\tanh (-0.1231)=-0.1225$ What we did: fed each input plus the decayed memory of the previous state. What it looks like: each state is a point on the tanh S-curve, pulled toward the input's sign.

Start ${\overleftarrow{h}}_4=0$, march right-to-left.

• ${\overleftarrow{h}}_3=\tanh (0.5\cdot 0+1.0\cdot 0.2)=\tanh (0.2)=0.1974$
• ${\overleftarrow{h}}_2=\tanh (0.5\cdot 0.1974+1.0\cdot (-1.0))=\tanh (-0.9013)=-0.7176$
• ${\overleftarrow{h}}_1=\tanh (0.5\cdot (-0.7176)+1.0\cdot 0.5)=\tanh (0.1412)=0.1403$

• $h_1=[{\overrightarrow{h}}_1;{\overleftarrow{h}}_1]=[0.4621,0.1403]$
• $h_3=[{\overrightarrow{h}}_3;{\overleftarrow{h}}_3]=[-0.1225,0.1974]$ The second coordinate of $h_1$ is ${\overleftarrow{h}}_1=0.1403$; it carries information about $x_2,x_3$ — the future of position 1. The first coordinate (${\overrightarrow{h}}_1$) knows only $x_1$.

Summation collapses a 2-vector to a scalar, so it is many-to-one. From L2, $h_1$ concatenated is $[0.4621,0.1403]$, summing to $0.6024$. But the pair $[0.5024,0.1000]$ also sums to $0.6024$ — a different balance of past/future that the downstream layer can no longer distinguish. Concatenation would keep them apart. Conclusion: summation is lossy; concatenation lets the output layer learn how to weight past vs future.

The forward RNN learns how information flows from the past; the backward RNN learns how it flows from the future — these are genuinely different statistical patterns (e.g. "a determiner precedes a noun" vs "an object follows a verb"). Sharing $W$ would force one compromise matrix to model both, degrading both. Separate parameters let each direction specialize. (Cost: roughly double the recurrent parameters.)

The classifier sees $h_t$: dimension $d$ (uni) or $2d$ (bi).

• Uni: $W_y$ is $C\times d=4\times 3=12$, plus $b_y$ of size $4$ → $16$ params.
• Bi: $W_y$ is $C\times 2d=4\times 6=24$, plus $b_y$ of size $4$ → $28$ params. The BiRNN's classifier grows because it reads a doubled feature vector.

• Encoder — yes, bidirectional. The whole source sentence is available, and encoding word 3 may need word 7 (word order differs across languages). See Encoder-Decoder Architecture and Sequence-to-Sequence Models.
• Decoder — no. The decoder generates the target one token at a time; the future target tokens do not exist yet, so it is inherently causal. A bidirectional decoder is impossible at generation time. This asymmetry (bi-encoder, causal-decoder) is exactly what modern attention-based Transformers preserve.

Replace each $\tanh$ RNN cell with a gated cell — an LSTM or GRU — in both directions, giving a Bi-LSTM / Bi-GRU. The gates provide a near-linear "carry" path so gradients survive long sequences. See LSTM and GRU Cells. The bidirectional wiring (two passes, concatenate) is unchanged — only the cell internals change.

• Forward produces $T=5$ states, backward produces $T=5$ states → $10$ hidden vectors total.
• They combine pairwise into $5$ concatenated states $h_1,\dots ,h_5$, each of dim $2d$, one per word, each getting a softmax tag.

$h_2=[{\overrightarrow{h}}_2;{\overleftarrow{h}}_2]$.

• ${\overrightarrow{h}}_2=-0.6462$ was built from ${\overrightarrow{h}}_1$ (which saw $x_1$) plus $x_2$ → carries $\{x_1,x_2\}$ (the past).
• ${\overleftarrow{h}}_2=-0.7176$ was built from ${\overleftarrow{h}}_3$ (which saw $x_3$) plus $x_2$ → carries $\{x_2,x_3\}$ (the future). Union: $\{x_1,x_2,x_3\}$ — all three inputs influence $h_2$. Numerically $h_2=[-0.6462,-0.7176]$. This is precisely why a BiRNN can tag word 2 using full-sentence context.

Logits $z=W_y{h}_2+b_y$:

• $z_1=1\cdot (-0.6462)+(-1)\cdot (-0.7176)=0.0714$
• $z_2=0\cdot (-0.6462)+1\cdot (-0.7176)=-0.7176$ Softmax: $p_i=e^{z_i}/(e^{z_1}+e^{z_2})$.
• $e^{0.0714}=1.0740,e^{-0.7176}=0.4880,$ sum $=1.5620$
• $p_1=1.0740/1.5620=0.6876$
• $p_2=0.4880/1.5620=0.3124$ Class 1 wins ($0.6876>0.3124$). The softmax turned the raw logit gap into a valid probability distribution summing to $1$.

• Live autocomplete: unidirectional (causal) — future words don't exist yet at typing time; a BiRNN cannot run.
• Offline NER: bidirectional (ideally a Bi-LSTM) — the whole sentence is present and a token's tag depends on later words. This is the canonical BiRNN use case from the parent note.

A BiRNN needs the whole sequence available — true or false? ::: True; it cannot compute any backward state until the last input is seen. Concatenation vs summation — which preserves more info? ::: Concatenation (keeps past and future separate). Which fix keeps bidirectionality but stops vanishing gradients? ::: Swap the cell for an LSTM/GRU → Bi-LSTM.