Bidirectional RNNs: Foundations — AI-ML deep dive

Before you can trust that one idea, you must own every symbol the parent note throws at you. This page builds each one from nothing: plain words → the picture → why the topic needs it, in an order where each brick rests on the last.

1. A sequence and its index $t$

Picture a row of boxes on a shelf, numbered left to right. Box $t = 1$ holds the first word, box $t = T$ holds the last.

Why the topic needs it. "Bidirectional" only means anything because there is a direction — a first box and a last box. Without an ordered sequence, "forward" and "backward" would be meaningless. The whole page is built on walking this row of boxes in two opposite ways.

2. A vector: turning a word into numbers ( $x_{t}$ as an arrow of numbers)

A neural network cannot chew on the letters "Apple". It needs numbers. So each word in the box gets replaced by a short list of numbers called a vector.

Picture each number as a coordinate. Two numbers = a dot on a flat map; three numbers = a dot floating in a room. More than three we can't draw, but the idea is identical: a vector is a point / arrow in a space of $n$ dimensions.

Why the topic needs it. Every arrow in the network is a vector; every "encode Apple + released" is really "move to a new point in this space". You cannot read the worked example without knowing that $[\cdot, \cdot, \cdot]$ is a location, and that its length is what must match up when matrices multiply it.

Recall Why can't we just use the word itself?

Networks multiply and add; they need numbers, not letters. ::: A word is first mapped (embedded) to a numeric vector so arithmetic becomes possible.

3. The embedding: the map from word to vector

Picture a dictionary where the definition of each word is not text but a point on the map from §2. Similar words (king / queen) land near each other.

Why the topic needs it. Step 1 of the worked example is literally "embed the words". The vectors $x_{1}, \dots, x_{5}$ the parent lists come out of this table. This foundation feeds directly into everything downstream. (Depth on this is deferred; here we only need "word → fixed vector".)

4. Weights $W$ , bias $b$ , and matrix–vector multiply

The network's job is to transform one vector into another. The tool for a linear transform is a matrix.

Picture $W v$ as stretching, rotating and squishing the arrow $v$ into a new arrow; then $+ b$ slides that arrow to a new spot.

Definition The specific weight/bias names used later

The generic $W$ and $b$ get subscripts to say which transform they are. In §6 and in the parent you will meet:

$W_{h}$ , $W_{x h}$ , $b_{h}$ — the matrices/bias of a single (plain) RNN cell: $W_{h}$ stirs the old memory, $W_{x h}$ stirs the new word, $b_{h}$ shifts the sum. (Read the double subscript $x h$ as "from $x$ into $h$ ".)
$W_{h}$ , $W_{x h}$ , $b_{h}$ and $W_{h}$ , $W_{x h}$ , $b_{h}$ — the same idea, but one independent copy for the forward RNN and one for the backward RNN. Note the parallel naming: $W_{x h}$ is the forward "from $x$ into $h$ " matrix, exactly matching $W_{x h}$ of the plain cell.
$W_{y}$ , $b_{y}$ — the transform that turns a hidden state into raw label-scores at the output (used with softmax in §9).

Every one of these is just "a $W$ and a $b$ " from the box above; the subscript is only a label naming its job.

Definition The shapes of the matrices (so the multiply is well-posed)

A matrix $W$ that turns a length- $p$ vector into a length- $q$ vector must be $q \times p$ (write $W \in R^{q \times p}$ : $q$ rows, $p$ columns). Matching each name to its job, with $d$ = hidden length (§6) and $d_{e mb e d}$ = word length (§2):

$W_{h} \in R^{d \times d}$ — takes the length- $d$ memory $h_{t - 1}$ to a length- $d$ result.
$W_{x h} \in R^{d \times d_{e mb e d}}$ — takes the length- $d_{e mb e d}$ word $x_{t}$ to a length- $d$ result.
$b_{h} \in R^{d}$ — added to the length- $d$ sum.
Forward/backward copies have identical shapes: $W_{h}, W_{h} \in R^{d \times d}$ and $W_{x h}, W_{x h} \in R^{d \times d_{e mb e d}}$ .
$W_{y} \in R^{C \times 2 d}$ — takes the length- $2 d$ concatenated state (§8) to $C$ label-scores, where $C$ = number of classes.

These shapes are exactly what makes $W_{h} h_{t - 1} + W_{x h} x_{t}$ add up: both terms land in $R^{d}$ , so the sum is defined.

Why the topic needs it. Every line of the RNN update is a matrix–vector multiply plus a bias. The parent uses separate matrices for memory ( $W_{h}$ ) and for the new word ( $W_{x h}$ ) precisely because a matrix is how you turn one meaning-vector into another, and each job needs its own grid of numbers — and its own shape.

5. $tanh$ : the squashing function

After stirring and shifting we get raw numbers that could be huge. We tame them with $tanh$ .

Picture an S-shaped curve lying flat at $- 1$ on the far left, passing through the origin, and flattening at $+ 1$ on the far right.

Why THIS tool and not just $W v + b$ ? Two reasons. (1) Bounding: if we only ever multiplied and added, values could explode across many timesteps; $tanh$ caps every hidden number in $[- 1, 1]$ , keeping the memory stable — that is exactly the parent's remark "tanh squashes to $[- 1, 1]$ to prevent exploding values." (2) Nonlinearity: stacking pure matrices collapses to one matrix (still just a line); $tanh$ bends the space so the network can learn curved patterns.

Recall Why is a nonlinearity needed at all?

Because a chain of linear maps is itself linear — no extra expressive power. ::: $tanh$ bends the transformation so the network can represent complex, non-straight relationships.

6. The hidden state $h_{t}$ : the memory vector

Now the star of the show.

Picture a notepad you update after each box: read box $t$ , glance at your last note $h_{t - 1}$ , scribble a new note $h_{t}$ that blends old memory with the new word.

Why the topic needs it. The entire bidirectional idea is "keep two separate notepads". Understanding $h_{t}$ as memory that flows in a direction is what lets §7's arrows make sense.

7. The direction arrows $h_{t}$ and $h_{t}$

The parent writes little arrows over the letter $h$ . These are not vectors-vs-scalars; they mark which way we walked the shelf.

Picture two friends: one starts at box 1 and shuffles right, another starts at box $T$ and shuffles left. They meet at every box $t$ and each hands over a note.

Why the topic needs it. This is bidirectionality. $h_{t}$ answers "given all words before me, what am I?"; $h_{t}$ answers "given all words after me, what am I?". The whole point of the parent — distinguishing "Apple" the company from "apple" the fruit — needs $h_{1}$ to know "released iOS" comes next.

8. Concatenation $[\cdot; \cdot]$

Picture gluing two short trains of boxes into one longer train — nothing is added or averaged; both lists survive intact, just side by side.

Why the topic needs it. The parent combines past and future by concatenation, not summation, "to preserve both". Knowing $[;]$ = stacking (so $R^{d}$ and $R^{d}$ give $R^{2 d}$ ) is required to read $h_{1} \in R^{6}$ in the example (two length-3 notes glued).

9. $softmax$ : numbers into a probability decision

Finally we must decide a label for each word (is it an organization? a location?).

Picture a bar chart of raw scores; softmax reshapes the bars so their total height is exactly 1, letting you read each bar as "chance this is the answer".

Why THIS tool? We are classifying — choosing one of several tags. A classification needs a probability per class, and softmax is the standard way to get one from raw scores. That is exactly the parent's line "we're making a classification decision at each timestep." The output $[B-ORG : 0.85, \dots]$ is a softmax result: all four numbers sum to $1$ .

How the foundations feed the topic

Each foundation is a prerequisite of the next; the last three (direction arrows, concatenation, softmax) are the parts unique to going bidirectional. These same bricks reappear when you meet LSTM and GRU Cells, the Encoder-Decoder Architecture, the Attention Mechanism, and Transformers — so time spent here pays forward.

Equipment checklist

Test yourself — you should be able to answer each before reading the parent's derivation.

What does

T

stand for in

x = (x_{1}, \dots, x_{T})

The total number of slots (words) in the sequence.

What does

R^{n}

mean in one phrase?

The set of all lists of

n

real numbers — an

n

-dimensional space of points/arrows.

What is the difference between

d_{e mb e d}

and

d

d_{e mb e d}

is the length of an input word-vector;

d

is the length of a hidden-state (memory) vector.

Where does the vector

x_{1} = [0.8, - 0.3, 0.5, 0.1]

come from?

From the embedding table that maps each word to a fixed numeric vector.

What single operation does

W v + b

perform on a vector?

A linear transform ("stir": stretch/rotate/squish) followed by a shift (add bias).

What shape must

W_{x h}

have, and why?

R^{d \times d_{e mb e d}}

— it maps a length-

d_{e mb e d}

word into a length-

d

hidden vector, so it has

d

rows and

d_{e mb e d}

columns.

h_{t} = tanh (W_{h} h_{t - 1} + W_{x h} x_{t} + b_{h})

, what job does each subscripted matrix do?

W_{h}

(shape

d \times d

) stirs the old memory

h_{t - 1}

;

W_{x h}

(shape

d \times d_{e mb e d}

) stirs the new word

x_{t}

;

b_{h}

shifts the sum — all landing in

R^{d}

before

tanh

Give the two reasons

tanh

is applied.

(1) It bounds values into

(- 1, 1)

to prevent explosion; (2) it adds nonlinearity so the network can learn curved patterns.

In words, what is the hidden state

h_{t}

A running-summary memory vector of everything read up to slot

t

What does the arrow in

h_{t}

tell you?

The RNN walked left→right, so this note encodes the past (boxes

1

t

What does

h_{t}

encode and where does it start blank?

The future (boxes

t

T

); it starts blank at the right,

h_{T + 1} = 0

What does

[h_{t}; h_{t}]

do, and what length results?

Stacks the two vectors end to end; if each is length

d

the result is length

2 d

What produces the raw scores that softmax turns into probabilities?

The output transform

z = W_{y} h_{t} + b_{y}

(with

W_{y} \in R^{C \times 2 d}

), then softmax normalizes them over the label classes.