Bidirectional RNNs: Question bank — AI-ML deep dive

The whole page rests on one picture in your head: two separate readers of the same sentence. One reads left→right and stores "everything before word $t$ " in $h_{t}$ (the right-arrow means forward). The other reads right→left and stores "everything after word $t$ " in $h_{t}$ (the left-arrow means backward). At each word we glue them: $h_{t} = [h_{t}; h_{t}]$ , where the semicolon means "stack the two lists of numbers into one longer list." Keep that image; every trap below bends some part of it.

True or false — justify

A BiRNN can predict the next word in a language model as you type

False — the backward reader needs the whole sentence including future words, which don't exist yet during generation; BiRNNs need the full sequence available up front.

The forward and backward RNNs share the same weight matrices

False — they use independent parameters (

W_{h}

W_{h}

, etc.); sharing would force one set of weights to model both past- and future-dependencies, a lossy compromise.

At timestep

t

h_{t}

already contains information about future words

False —

h_{t}

only sees

x_{1}, \dots, x_{t}

; future context enters only through

h_{t}

, which is why we concatenate the two.

The backward RNN processes the reversed input, so it is just the forward RNN run on a flipped list

True in mechanics but with its own weights — you reverse the sequence, run an ordinary RNN, then re-align outputs to the original positions; the flip is real, the shared-weights part is not.

A BiRNN with hidden size

d

per direction produces a per-timestep vector of size

2 d

when concatenating

True — concatenation stacks

h_{t} \in R^{d}

and

h_{t} \in R^{d}

into

R^{2 d}

; the output layer

W_{y}

must therefore have

2 d

input columns.

Making an RNN bidirectional roughly doubles the parameter and compute cost

True — you now train two full recurrent stacks; this is the price you pay for future context, and it's often worth it for tagging tasks.

Because it reads both directions, a BiRNN removes the vanishing-gradient problem of RNNs

False — each direction is still a plain RNN over long distances; use LSTM and GRU Cells gates for that. Bidirectionality fixes access to future context, not gradient flow.

You can stack BiRNN layers just like unidirectional RNNs

True — the

2 d

-wide

h_{t}

from one BiRNN layer becomes the input sequence to the next BiRNN layer; deep BiRNNs are standard for speech.

Spot the error

"Use a BiRNN as the decoder in a translation model so it can see the whole target sentence."

The decoder generates the target left-to-right and must not peek at future target tokens (they don't exist yet); a BiRNN belongs in the encoder. See Encoder-Decoder Architecture.

h_{1}

encodes only

x_{1}

because it's the first timestep."

Reversed —

h_{1}

is computed last in the backward pass and encodes

x_{1}, \dots, x_{T}

; it's

h_{1}

that sees only

x_{1}

"We can start the backward loop before the forward loop finishes since they're independent."

The two directions are independent of each other, but within the backward loop

h_{t}

depends on

h_{t + 1}

, so it must run right-to-left in order; you can, however, run the forward and backward passes in parallel across directions.

h_{t} = h_{t} + h_{t}

is always preferred because it keeps the vector small."

Summation mixes past and future into the same coordinates and can cancel information; concatenation keeps them separable so downstream layers learn the weighting — smaller isn't automatically better.

"For NER on a fixed corpus we should avoid BiRNNs because they can't be causal."

NER runs on complete sentences at inference, so non-causality is fine — that's exactly the setting BiRNNs are built for. See Named Entity Recognition.

"The output

y_{t}

is computed from

h_{T}

, the final forward state."

For per-token tagging,

y_{t}

uses the per-timestep combined state

h_{t} = [h_{t}; h_{t}]

, not just the last forward state; that final-state trick belongs to seq-classification, not sequence labeling.

"Set

h_{0} = 0

to initialize the backward RNN."

The backward RNN is seeded at the right end:

h_{T + 1} = 0

. There is no meaningful

h_{0}

before you've reached

t = 1

Why questions

Why not just train one RNN and feed it the sentence twice (once reversed)?

A single shared-weight net would average two conflicting objectives; two separate directions with their own weights specialize, and concatenation lets the model use both at every position simultaneously.

Why is concatenation the default combine rule rather than averaging or a learned gate?

Concatenation is lossless and lets the next layer learn any linear mix (including averaging) if it wants; it's the most flexible default. Gated fusion exists but adds parameters for often-marginal gain.

Why does the "This movie is not bad" sentiment case specifically motivate bidirectionality?

"not" appears before "bad", so a forward-only model tagging "bad" has already passed "not"; but to reinterpret the whole phrase at the "not" position you'd want the future word "bad" — bidirectional context lets both readings meet.

Why do BiRNNs pair naturally with attention and eventually get replaced by Transformers?

BiRNNs give every position a past+future summary but still process sequentially; Attention Mechanism lets any position directly query any other, and Transformers drop recurrence entirely for full parallelism while keeping bidirectional context.

Why can't we simply add more forward RNN layers instead of going bidirectional?

Stacking forward layers deepens the past-only representation but never lets a token see words to its right; no depth of causal layers manufactures future context. Direction is orthogonal to depth.

Why must the output weight matrix

W_{y}

change shape when we switch a model from unidirectional to bidirectional?

Its input dimension grows from

d

2 d

because

h_{t}

is now the concatenation; forgetting to resize

W_{y}

is a common shape-mismatch bug.

Edge cases

What is

h_{1}

for a length-1 sequence (

T = 1

h_{1}

sees only

x_{1}

and

h_{1}

also sees only

x_{1}

(its neighbor set is empty), so both directions encode the same lone word — bidirectionality adds nothing for a single token.

What does the backward pass contribute at the last timestep

t = T

h_{T}

is seeded from

h_{T + 1} = 0

and sees only

x_{T}

, so at the final word the backward reader has no future context — symmetric to the forward reader having none at

t = 1

What happens to a BiRNN if you feed it a token stream in real time, one token at a time?

It cannot emit

y_{t}

until the sequence ends, because

h_{t}

needs all later tokens; for streaming you must fall back to a causal (unidirectional) model or a windowed approximation.

If forward and backward hidden states are identical at some position, is that a bug?

Not necessarily — for a length-1 sequence, or a highly symmetric input, they can legitimately coincide; it's only suspicious if it happens for every position on varied data, hinting at accidentally tied weights.

For very long sequences, why might a plain BiRNN still tag early words poorly despite "seeing the future"?

The future information must travel through many recurrent steps into

h_{t}

, and a vanilla RNN forgets over distance; you'd swap in LSTM and GRU Cells to preserve long-range future signal.

Does reversing the input change which word each output aligns to?

No — after computing the backward states you re-index them to the original positions so

h_{t}

pairs with

x_{t}

; skipping this re-alignment mismatches every label to the wrong word.

Recall One-line self-test

Cover everything: state (a) which direction sees the future, (b) why a BiRNN can't be a language-model decoder, and (c) why concatenation beats summation. Answer ::: (a) the backward RNN via $h_{t}$ ; (b) generation has no future tokens to read; (c) concatenation preserves past and future separately, summation can cancel them.

Related building blocks: Sequence-to-Sequence Models · Encoder-Decoder Architecture · Attention Mechanism · Transformers.