Question bank — Gated Recurrent Units (GRU)

False — GRU has two gates ($z_t$ update, $r_t$ reset). LSTM's forget and input gates are fused into the single $z_t$, so there is no separate output gate either.

False — GRU drops the cell state entirely; there is only one state vector $h_t$ that plays both roles. That is the whole "LSTM-lite" saving.

True — $\sigma (u)=1/(1+e^{-u})$ squashes any real number into $(0,1)$, and it acts slot-by-slot, so each of the $d_h$ dimensions gets its own independent gate value.

True — plug in: $(1-1)\odot {\tilde{h}}_t+1\odot h_{t-1}=0+h_{t-1}$. The candidate is completely discarded and memory is copied through untouched.

True — $(1-0)\odot {\tilde{h}}_t+0\odot h_{t-1}={\tilde{h}}_t$. The old memory is fully overwritten by the candidate.

False — $r_t$ only appears inside the candidate ${\tilde{h}}_t=\tanh (W_h[r_t\odot h_{t-1},x_t]+b_h)$. The final line $h_t=(1-z_t)\odot {\tilde{h}}_t+z_t\odot h_{t-1}$ uses only $z_t$.

False — they are coupled: once $z_t$ is known, $(1-z_t)$ is fixed. This coupling is exactly how GRU compresses LSTM's separate forget/input gates into one.

False — GRU has fewer parameters and often trains faster, but on tasks needing very fine-grained memory control LSTM can win. "Nearly as well, usually cheaper" is the honest claim, not "always better."

True — $z_t$ is a vector; dimension 5 might be $0.9$ (keep) while dimension 6 is $0.1$ (replace). The gate is per-feature, not one scalar for the whole state.

The $z_t$ and $(1-z_t)$ are swapped. The standard convention has $z_t$ multiply the old state $h_{t-1}$; here $z_t\approx 1$ would mean "take the candidate," inverting the whole "keep-vs-write" meaning.

The reset gate. It should be $r_t\odot h_{t-1}$ inside, not raw $h_{t-1}$. Without $r_t$ the network loses its ability to ignore irrelevant past when forming a candidate.

${\tilde{h}}_t$ is not a gate — it comes from $\tanh$, so it lives in $[-1,1]$ and can be negative. Only $z_t$ and $r_t$ (from $\sigma$) are gates in $(0,1)$.

Wrong input. The correct form is $z_t=\sigma (W_z[h_{t-1},x_t]+b_z)$ using the raw concatenation, independently of $r_t$. Only the candidate ${\tilde{h}}_t$ uses $r_t\odot h_{t-1}$.

No — $\odot$ is the Hadamard (element-wise) product, returning a vector of the same length. A dot product would collapse the state to a scalar and destroy the memory.

The reset gate multiplies only $h_{t-1}$, never $x_t$. The correct form is $W_h[r_t\odot h_{t-1},x_t]$ — the current input always enters at full strength; resetting the present input would make no sense.

Because the two weights $(1-z_t)$ and $z_t$ are both in $[0,1]$ and sum to exactly 1. Any such weighted average is a convex combination — $h_t$ is guaranteed to lie "between" the old state and the candidate.

The term $z_t\odot h_{t-1}$ contributes $z_t$ to the Jacobian $\partial h_t/\partial h_{t-1}$. When $z_t\approx 1$ the gradient passes back nearly unscaled, so over $T$ steps it scales like $\prod z_t$ instead of decaying like $\prod {\tanh }^{\prime }$. See 3.5.02-Vanishing-and-Exploding-Gradients.

Gates need to answer "how much?" — a fraction in $(0,1)$, so $\sigma (u)=1/(1+e^{-u})$ is perfect. The candidate is a proposed state value that should be able to push memory positive or negative, so it needs the symmetric $[-1,1]$ range of $\tanh$.

GRU uses 2 gate weight matrices ($W_r,W_z$) instead of LSTM's 3, and has no separate cell-state pathway. Fewer gates and one merged state vector means fewer $W$ and $b$ to learn.

A basic RNN multiplies the state by the same weight/derivative every step, so gradients shrink (or blow up) exponentially. GRU's gated $z_t\odot h_{t-1}$ pathway gives a near-identity route the gradient can travel unscaled.

In LSTM the forget and input gates are often anti-correlated anyway ("forget old ⇔ write new"). GRU bakes that correlation in via $(1-z_t)$ vs $z_t$, losing a little flexibility for a big simplicity/parameter win.

They act at different places. $r_t$ shapes what candidate gets proposed (can the past even contribute?), while $z_t$ decides how much of that candidate to accept afterwards. Resetting before the candidate lets GRU forget history for the proposal yet still keep old memory in the final blend if $z_t$ is high.

The candidate becomes ${\tilde{h}}_t=\tanh (W_h[0,x_t]+b_h)$ — it depends only on the current input $x_t$. The network proposes a fresh state as if it had amnesia about the past, useful right after a topic change.

The state is copied through unchanged: $h_t=h_{t-1}=\dots$. Memory is perfectly preserved and the current inputs are ignored — this is the "gradient highway" that carries information (like a name) across long gaps.

Yes, and the result is subtle: a fresh candidate is computed (ignoring the past), but then $z_t=1$ throws that candidate away and keeps $h_{t-1}$. So the wasted computation of ${\tilde{h}}_t$ has zero effect on $h_t$. The gates are independent, so contradictory-looking combinations are legal.

$h_0$ is initialised (usually to zeros). Then $h_1=(1-z_1)\odot {\tilde{h}}_1+z_1\odot 0=(1-z_1)\odot {\tilde{h}}_1$, so the first state is just a scaled candidate — sensible, since there is no history to keep.

It takes a plain 50/50 average of the old value and the candidate for that feature — neither committing to memory nor fully overwriting. Real trained gates rarely sit exactly here; they push toward 0 or 1 to make crisp keep/write decisions.

No. GRU has no output gate — $h_t$ is exposed directly as both the state and the output. $z_t$ only controls how $h_t$ is formed, not how it is read out. This differs from 3.5.03-Long-Short-Term-Memory-(LSTM).