5.6.12 · D5Machine Learning (Aerospace Applications)
Question bank — Recurrent neural networks — hidden state, BPTT
Before you start, a one-screen refresher of the symbols so nothing below is unearned:
- = the input at time step (e.g. one airspeed reading).
- = the hidden state, a vector "sticky note" summarising everything seen up to .
- = the three weight matrices, the same at every step (weight sharing).
- = the pre-activation, before squashes it.
- = total loss; = "blame arriving at step ."
- = element-wise multiply; = a matrix with vector on its diagonal.
- BPTT = backprop run on the network unrolled into one layer per time step.
True or false — justify
Every item is a claim. Decide true/false, then justify in one sentence.
"BPTT is a fundamentally different algorithm from ordinary backpropagation."
False. It is ordinary backprop, just run on the unrolled graph; the only twist is that a shared weight appears many times, so its gradient becomes a sum over steps.
"Making the update linear, , would simplify the model without real harm."
False. A stack of linear maps collapses into one linear map, so the net loses all expressive power and its gradients become pure — guaranteed to vanish or explode with no way to gate.
"Truncated BPTT loses no information as long as you keep the recent steps."
False. Cutting the backward pass at steps forces every gradient older than to exactly zero, so the net cannot learn any dependency longer than — it's a compute trade-off, not a free lunch.
"Because the same is reused, the RNN has fewer parameters than an equivalent unrolled feed-forward net."
True. Weight sharing means one matrix serves every step, so parameter count is independent of sequence length, unlike a feed-forward net that would need fresh weights per slot.
"."
False. You forgot the derivative; the true Jacobian is , because the state passes through before feeding forward.
"Weight sharing lets one trained RNN handle a longer sequence than it saw in training without retraining."
True. One rule is applied step by step regardless of length, so a net that learned "airspeed rising 3 in a row" already generalises to a 5-sample stream.
"If only has a loss term at the final step , then for all ."
False. Only the direct term is zero for ; the indirect term still carries the final-step blame backward through time.
"The vanishing-gradient problem means the forward hidden states also shrink to zero."
False. It's the backward gradient signal that decays; the forward states stay bounded in thanks to and can be perfectly non-trivial.
"Gradient clipping cures the vanishing-gradient problem."
False. Clipping only tames exploding gradients by capping their size; vanishing gradients are too small already, so clipping does nothing — that needs gated cells like LSTM and GRU.
Spot the error
Each line states a claim with a hidden mistake. Reveal the correction.
"Someone writes using only the last step."
Error: missing the sum. Since is used at every step, its gradient is — "same weight, sum the blame."
"A student claims sigmoid and are interchangeable in the update since both squash."
Error: centering. is centered at (output in ) which keeps gradients healthier, while sigmoid is centered at and biases activations positive, hurting gradient flow.
"To get they write (a product)."
Error: it's a sum, not a product. The multivariate chain rule adds the direct path and the future path: .
"Someone uses in the backward pass."
Error: argument confusion. It's , i.e. one minus the output squared, not one minus the pre-activation squared.
"A note says the recurrent Jacobian's largest singular value must be exactly 1 for stable training."
Error: it need not be exactly 1. You want the effective factor (singular value of times ) near 1; well below 1 vanishes, well above 1 explodes, but a healthy band around 1 is fine.
"Someone treats (output weights) as also getting summed contributions from future hidden states."
Error: only reads locally. Its gradient at step depends only on that step's output, though it is still summed over because it too is shared across time.
Why questions
Answer with the reason, not just the fact.
"Why must the gradient of a shared weight be summed rather than averaged or taken from one step?"
Because each use of the weight is an independent path from parameter to loss, and the chain rule adds all path contributions; averaging would silently scale the true gradient by .
"Why does unrolling turn an RNN into something ordinary backprop can handle?"
Unrolling replaces the loop with a finite chain of tied layers, which is just a (deep) feed-forward graph, and backprop is defined on any such acyclic graph.
"Why does a product of many similar Jacobians cause vanishing/exploding gradients?"
Multiplying near-identical matrices behaves like (with the dominant factor), and any raised to a large power races to or .
"Why do LSTM/GRU cells help where a plain RNN struggles?"
They introduce a gated, near-identity memory path so the backward Jacobian is close to along that path, keeping the long-range gradient product from collapsing — see LSTM and GRU and Vanishing and Exploding Gradients.
"Why is the hidden state called a summary rather than a store of the sequence?"
It is a fixed-size vector compressing an arbitrarily long history, so it necessarily discards detail and keeps only what past updates deemed relevant — it cannot losslessly store everything.
"Why does weight sharing act as a useful prior for flight data?"
It bakes in time-invariance — "the same physics applies at every instant" — so the model learns one rule for streams like attitude or airspeed instead of memorising position-specific patterns; compare Sequence Modeling in Flight Data.
Edge cases
Push each rule to its boundary.
"What is when and the input is zero, , with ?"
so ; the state stays at the origin, meaning "no information yet" is a genuine fixed point of the update.
"For a length-1 sequence (), what does BPTT reduce to?"
It reduces to ordinary single-step backprop with no backward-in-time term, since there is no future step to propagate from.
"If , what happens to memory across steps?"
The state ignores its own past entirely (), so the RNN degenerates into a per-step feed-forward net with no memory.
"As a hidden unit saturates, — what happens to the gradient flowing through it?"
The factor , so that unit's backward gradient is throttled toward zero — saturation is a local cause of vanishing gradients even before any long product.
"If the target only exists at the final step, can early inputs still be trained?"
Yes, provided the sequence is short enough that the summed backward gradient hasn't vanished; the final-step blame reaches through the recurrent path, though weakly for long sequences.
"What happens to at step 1 if ?"
That step's contribution is zero because multiplies it out — the first step simply adds nothing to the recurrent weight's gradient.