Worked examples — State-space models (Mamba, S4)

This page is a drill. The parent note built the machinery; here we run it through every situation it can face. Before each example, you forecast the answer — guess the sign, the size, the behaviour — then we grind the steps.

Everything below uses only three moving parts you already met:

• a state $h_k$ (a short list of numbers the model carries forward — its memory),
• a decay $\bar{\mathbf{A}}$ (how much of yesterday's memory survives to today),
• an input push $\bar{\mathbf{B}}{x}_k$ (how much the new token adds to memory),

and one readout $y_k=\mathbf{C}{h}_k$ (what we actually output). If any of those words feels shaky, reread the parent's State-Space Foundation first.

---

The scenario matrix

Every question this topic throws at you lands in one of these cells. Each example below is tagged with the cell it hits.

Cell	What makes it special	Example
A. Stable decay ($0<\bar{A}<1$)	memory fades smoothly	Ex 1
B. Perfect memory ($\bar{A}=1$)	nothing forgotten, running sum	Ex 2
C. Dead memory ($\bar{A}=0$)	forgets instantly, no history	Ex 3
D. Unstable / diverging ($\bar{A}>1$)	memory blows up	Ex 4
E. Sign-flip / oscillation ($\bar{A}<0$)	memory alternates sign	Ex 5
F. Multi-dim state ($N=2$, mixed rates)	fast + slow channels together	Ex 6
G. Convolution view (kernel from recurrence)	unroll to $\bar{\mathbf{K}}$	Ex 7
H. Discretization (continuous $\to$ discrete)	turn $A,\Delta$ into $\bar{A}$	Ex 8
I. Selective (Mamba) ($\Delta$ per token)	model chooses to remember	Ex 9
J. Real-world word problem	reason about behaviour, not just arithmetic	Ex 10

Look at the red curve: it is the impulse response — feed a single spike at time 0, then watch how the state remembers it. Slow decay = long red tail; $\bar{A}=1$ = flat red line (never forgets); $\bar{A}<0$ = red bars flipping above/below zero. Keep this picture in mind for every example.

---

Cell A — Stable decay ($0<\bar{A}<1$)

1. State after the push. $h_1=0.8\cdot 0+0.5\cdot 1=0.5$. Why this step? Only $x_1=1$ is nonzero, so this is the only injection of new information.
2. Silent steps decay the state. $h_2=0.8\cdot 0.5=0.4$, $h_3=0.8\cdot 0.4=0.32$, $h_4=0.8\cdot 0.32=0.256$. Why this step? With $x_k=0$ the input term vanishes; each step just multiplies by $\bar{A}=0.8$.
3. Readout. $\mathbf{C}=1$, so $y_k=h_k$: $(0.5,0.4,0.32,0.256)$. Why this step? The observation matrix is identity-like here, so output equals state.

Verify: $h_k=0.5\cdot 0.8^{k-1}$. At $k=4$: $0.5\cdot 0.512=0.256$. Matches the forecast (~0.25). Geometric decay confirmed — this is the classic "leaky memory."

---

Cell B — Perfect memory ($\bar{A}=1$)

1. $h_1=1\cdot 0+2=2$. Why? Full carry-over of past ($\times 1$) plus new input.
2. $h_2=1\cdot 2+(-1)=1$. Why? The $-1$ input subtracts from stored memory.
3. $h_3=1\cdot 1+4=5$. Why? Still no decay; keep accumulating.

Verify: $y_3=\sum x_i=2-1+4=5$. When $\bar{A}=1$ an SSM is a prefix-sum machine — the degenerate boundary between forgetting and diverging.

---

Cell C — Dead memory ($\bar{A}=0$)

1. $h_1=0\cdot 7+3\cdot 1=3$. Why? The $0\cdot h_0$ term wipes the past — $h_0=7$ is annihilated.
2. $h_2=0\cdot 3+3\cdot 2=6$. Why? Again no memory; output tracks the current input only.

Verify: $y_k=\bar{B}{x}_k=3x_k=(3,6)$. The 7 vanished, confirming forecast. With $\bar{A}=0$ the SSM degenerates to a per-token linear layer — no sequence modelling at all.

---

Cell D — Unstable / diverging ($\bar{A}>1$)

1. $h_1=1.5\cdot 0+1=1$. Why? The single injected spike.
2. $h_2=1.5\cdot 1=1.5$, $h_3=1.5\cdot 1.5=2.25$, $h_4=1.5\cdot 2.25=3.375$. Why? Each silent step amplifies because $\bar{A}>1$.

Verify: $h_k=1.5^{k-1}$, so $y_4=1.5^3=3.375$. Over 100 steps this reaches $1.5^{99}\approx 10^{17}$ — numerical explosion. This is exactly why S4/Mamba constrain the real part of $A$ to be negative so that $|\bar{A}|<1$.

---

Cell E — Sign-flip / oscillation ($\bar{A}<0$)

1. $h_1=-0.5\cdot 0+1=1$. Why? The spike enters.
2. $h_2=-0.5\cdot 1=-0.5$; $h_3=-0.5\cdot (-0.5)=0.25$; $h_4=-0.5\cdot 0.25=-0.125$. Why? Multiplying by a negative each step swings the state above and below zero while $|\bar{A}|=0.5$ shrinks it.

Verify: $h_k=(-0.5{)}^{k-1}=(1,-0.5,0.25,-0.125)$. Sign alternates, magnitude halves — a damped oscillation. Real SSMs use complex $A$ so a single channel can oscillate and decay smoothly; this real negative case is the crudest version of that.

---

Cell F — Multi-dim state, mixed rates ($N=2$)

1. Decay each channel independently: $\left[\begin{array}{c}0.9\cdot 0.3\\ 0.5\cdot 0.1\end{array}\right]=\left[\begin{array}{c}0.27\\ 0.05\end{array}\right]$. Why? $\bar{\mathbf{A}}$ is diagonal, so channels do not talk to each other during decay — each has its own memory length.
2. Add the input push $\bar{\mathbf{B}}{x}_1=\left[\begin{array}{c}0.2\\ 0.4\end{array}\right]$: $h_1=\left[\begin{array}{c}0.47\\ 0.45\end{array}\right]$. Why? Same scalar input $x_1=2$ enters both channels but scaled differently by $\bar{\mathbf{B}}$.
3. Readout: $y_1=1\cdot 0.47+(-0.5)\cdot 0.45=0.47-0.225=0.245$. Why? The negative weight lets the output say "high when channel 1 high and channel 2 low."

Verify: $0.47-0.225=0.245$. Matches. Two channels = two memory timescales in parallel; stack $N$ of them and you get a rich filter bank.

---

Cell G — Convolution view (kernel from the recurrence)

1. Entry 0: $\mathbf{C}\bar{B}=3\cdot 2=6$. Why? This is the immediate response of the output to a spike at the current step.
2. Entry 1: $\mathbf{C}\bar{A}\bar{B}=3\cdot 0.5\cdot 2=3$. Why? How much a spike one step ago still contributes.
3. Entries 2,3: $\mathbf{C}{\bar{A}}^2\bar{B}=1.5$, $\mathbf{C}{\bar{A}}^3\bar{B}=0.75$. Why? Older spikes contribute ${\bar{A}}^i$ less — the kernel is the impulse response.

So $\bar{\mathbf{K}}=(6,3,1.5,0.75)$.

4. Cross-check via the recurrence. Feed a spike $x=(1,0,0,0)$: outputs are $y_k=\mathbf{C}{\bar{A}}^{k-1}\bar{B}$, i.e. exactly $\bar{\mathbf{K}}$. Why? Convolving any signal with $\bar{\mathbf{K}}$ gives the same output as running the recurrence — the two views agree.

Verify: $(6,3,1.5,0.75)$ from both the kernel formula and the impulse response. This equivalence is why S4 can train with a parallel FFT convolution but generate with the cheap recurrence.

The red stems are $\bar{\mathbf{K}}$: a decaying comb. Convolving the input with this comb slides the comb along the sequence — no per-token loop needed.

---

Cell H — Discretization (continuous $\to$ discrete)

1. $\bar{A}=e^{-1\cdot 0.5}=e^{-0.5}\approx 0.6065$. Why exponential and not $1+A\Delta$? The matrix exponential is the exact solution of $h^{\prime }=Ah$ for constant input; Euler ($1+A\Delta =0.5$) is only a first-order approximation that drifts over long sequences.
2. $\bar{B}=A^{-1}(e^{A\Delta }-1)B=\frac{e^{-0.5}-1}{-1}\cdot 1=1-e^{-0.5}\approx 0.3935$. Why this form? It is the exact integral ${\int }_0^{\Delta }{e}^{As}ds\cdot B$ — the accumulated effect of a constant input over one timestep.

Verify: $\bar{A}+\bar{B}=e^{-0.5}+(1-e^{-0.5})=1$. (For $B=1$ this "gain sums to 1" is a useful sanity check.) $\bar{A}=0.6065\in (0,1)$ — stable, exactly why Mamba parametrizes $A<0$. Because $\bar{A}=e^{A\Delta }$, a larger $\Delta$ pushes $\bar{A}$ toward 0 (fast forgetting) and a smaller $\Delta$ pushes $\bar{A}$ toward 1 (long memory) — the exact knob Mamba turns per token (Ex 9).

---

Cell I — Selective (Mamba, per-token $\Delta$)

1. Important token: $\bar{A}=e^{-1\cdot 0.05}=e^{-0.05}\approx 0.9512$. Why? Small $\Delta$ means we barely advance the continuous clock, so almost nothing decays — the state holds this token's contribution for many future steps.
2. Filler token: $\bar{A}=e^{-1\cdot 2}=e^{-2}\approx 0.1353$. Why? A big $\Delta$ jumps the clock far forward, so $e^{A\Delta }$ shrinks toward 0 — memory of this token evaporates.
3. Interpretation. Same $A$, but $\Delta$ is now a function of the input ($\Delta ={\text{Linear}}_{\Delta }(x)$). This is precisely the parent's "John" ($\bar{A}\approx 0.98$) vs "and" ($\bar{A}\approx 0.3$) split. Why it matters: fixed S4 must pick one decay for all tokens; Mamba tunes decay token-by-token, which is why it beats S4 on selective-copy and long-context tasks.

Verify: $e^{-0.05}=0.9512$, $e^{-2}=0.1353$. The important token retains $95\%$ of memory per step, the filler only $14\%$ — the "choose what to remember" behaviour, made of nothing but the exponential from Ex 8.

---

Cell J — Real-world word problem

1. Model the decay. After $k$ silent steps the memory is ${\bar{A}}^k=0.98^k$. Why? No new input on this channel, so each step just multiplies by $\bar{A}$ (Cell A behaviour).
2. Set the threshold. We want $0.98^k<0.10$. Why? "Below 10% strength" is the retention cutoff the problem asks for.
3. Solve with logs. Take $\ln$ of both sides (logs turn the exponent into a product we can isolate): $k>\frac{\ln 0.10}{\ln 0.98}=\frac{-2.302}{-0.0202}\approx 114.$ Why logarithms? The unknown $k$ is stuck in an exponent; the logarithm is the exact tool that "brings it down."
4. Round up. After about 114 tokens the fact has faded below 10%.

Verify: $0.98^{114}=e^{114\ln 0.98}=e^{-2.30}\approx 0.099<0.10$ ✓, while $0.98^{113}\approx 0.101>0.10$. So 114 is the first token past the threshold. A "half-life" version: $0.98^k=0.5\Rightarrow k\approx 34$ tokens — memory halves every ~34 tokens. This is the concrete meaning of "long-range memory."

---

Recall

Recall What single number decides whether a scalar SSM forgets, remembers forever, or explodes?

The decay $\bar{A}$ ::: $|\bar{A}|<1$ forgets (stable), $\bar{A}=1$ is a running sum, $|\bar{A}|>1$ diverges, $\bar{A}<0$ oscillates.

Recall Why does Mamba parametrize

$\bar{A}=e^{A\Delta }$ with $A<0$? It guarantees $0<\bar{A}<1$ for any positive $\Delta$ ::: so the model is always stable, and $\Delta$ becomes a clean per-token memory-length knob.

Recall How do the recurrent and convolutional views relate?

The convolution kernel $\bar{\mathbf{K}}$ is the impulse response $\mathbf{C}{\bar{A}}^i\bar{B}$ ::: convolving with it gives identical output to running the recurrence — used for parallel training vs cheap generation.

See also: RNNs and LSTMs (recurrence and gating), Convolutional Neural Networks (the kernel view), Linear Attention (another $O(n)$ route).