Foundations — Recurrent neural networks — hidden state, BPTT

This page is the toolbox. Before you can read the parent note (RNN — hidden state & BPTT) you need to own every symbol it throws at you. We build each one from zero: plain words → a picture → why the topic needs it. Nothing is used before it is defined.

0. The characters, in reading order

We meet the symbols in the exact order the parent note uses them, so each rests on the last.

1. A subscript with time: $t$, $x_t$, $h_t$

Think of a filmstrip: each frame is one $t$.

• $x_t$ = the input at step $t$ — the thing the network reads right now (e.g. this second's sensor reading). It is a bundle of numbers; the precise word for "a bundle of numbers," and the notation for how many, come next in Section 2.
• $h_t$ = the hidden state at step $t$ — the memory note after reading $x_t$.

Why does the topic need $h_t$ specifically? Because the answer to "what will happen next?" depends on the history, not just the current frame. $h_t$ is where that history is compressed into a fixed-size summary.

2. Vectors and ${\mathbb{R}}^d$

Now we can be precise about Section 1: $x_t$ is a vector too (say $x_t\in {\mathbb{R}}^n$, $n$ inputs per step), and $h_t\in {\mathbb{R}}^d$ is a $d$-slot memory vector.

Why does the topic need this? The memory note isn't one number — it's a bundle of $d$ features ("airspeed trend", "attitude drift", ...). Storing $d$ numbers lets the state remember several things at once. The picture: a small stack of dials, each dial one component of the vector.

Building block — the humble arrow view first appears in Feed-forward Neural Networks, where every layer's activation is such a vector.

3. The output vector $y_t$ and the initial state $h_0$

Why the topic needs $y_t$: the loss compares $y_t$ to a target, so $y_t$ is the thing that gets judged. Without naming it there is nothing to be "wrong," and no gradient to send back.

Why the topic needs it: in the parent's worked example $h_0=0$ is plugged straight into $a_1$. Every recurrence needs a seed; $h_0$ is that seed.

4. Weights, matrices and the subscripts $W_{hh},W_{xh},W_{hy}$

The double-letter subscript is a from→to label, read right-to-left in the product:

Why does the topic need three separate matrices? Because three different journeys happen at each step: the old memory must be reshaped, the new input must be folded in, and the memory must be turned into the output $y_t$. Each journey gets its own tunable grid. The picture: three arrows arriving at the memory dial-stack, each arrow a matrix.

Matrix–vector multiplication is the workhorse of Feed-forward Neural Networks; here the same grids reappear at every step (Section 7).

5. Biases $b_h$, $b_y$ and the pre-activation $a_t$

Why the topic needs the name: in the derivation we differentiate "through $\tanh$" and "through the linear part" separately. Naming the boundary point $a_t$ makes those two steps readable.

6. The squashing function $\tanh$, its slope ${\tanh }^{\prime }$ — and why this one

You do not need to compute $\tanh$ by hand — recognise the S-curve, and remember: forward uses $\tanh$, backward uses $1-{\tanh }^2$.

The reason the slope matters for training is Vanishing and Exploding Gradients: it controls how signal shrinks going backward (Section 9).

7. Weight sharing — the same $W$ every step

Why the topic needs it: aerospace signals have no fixed length, and the laws of motion don't change between second 3 and second 4. Sharing bakes that fact in as a built-in assumption (a "prior"). This idea is what makes an RNN different from just stacking many Feed-forward Neural Networks layers with separate weights.

8. The gradient symbol $\frac{\partial L}{\partial W}$ and the chain rule

Why the topic needs it: the whole of Section 3 in the parent is one long, careful application of these two facts. Everything else is bookkeeping. This machinery is developed for plain nets in Backpropagation.

9. Two small pieces of matrix shorthand: $\odot$, $\mathrm{diag}$, and ${}^{\top }$

Why the topic needs these: the backward formulas are literally "$\tanh$-slope $\odot$ signal, then push through $W_{hh}^{\top }$." Recognising the three symbols makes the boxed BPTT equations readable at a glance.

10. Putting the vocabulary together

Now every symbol in the update rule is earned: $h_t=\tanh (\underset{\text{§4 grid}}{\underbrace{W_{hh}}}\underset{\text{§1 old memory, seeded by }{h}_0\text{ §3}}{\underbrace{h_{t-1}}}+\underset{\text{§4 grid}}{\underbrace{W_{xh}}}\underset{\text{§1 input}}{\underbrace{x_t}}+\underset{\text{§5 shift}}{\underbrace{b_h}}),\underset{\text{§3}}{\underbrace{y_t}}=W_{hy}{h}_t+b_y$ Read it aloud: "mix the old memory and the new input with two grids, add a shift, squash to keep it bounded — that's the new memory; then read the prediction off it." Every word links to a section above.

Prerequisite map

Related destinations once you own these: LSTM and GRU, Time Series Forecasting, Sequence Modeling in Flight Data, and the classical cousin Kalman Filter.

Equipment checklist

Cover the right side and answer out loud before opening the parent note.