Worked examples — Catastrophic forgetting

This page runs the Catastrophic forgetting machinery through every kind of situation you can meet: both signs of a weight, the "zero importance" degenerate case, the limiting values of the knobs $\lambda$ and $\eta$, a real-world LLM word problem, and an exam-style twist. Before any numbers, we lay out the grid of cases so you can see nothing is skipped.

Before we start, one plain-words recap of every symbol that appears in the core formula below — nothing will be used before it is defined here:

The scenario matrix

The grid below has nine cells (C1–C9), one per distinct situation. We cover all nine with eight worked examples — Example 5 deliberately handles the two paired limits C5 and C6 together (they are the two ends of the same $\lambda$ dial), so eight examples fill nine cells. Every cell is labelled on the example that hits it.

To turn "where does the weight end up" into arithmetic we need one small formula. We derive it once, then reuse it.

Why a weighted average? Two springs fighting over one bead always settle where their pulls cancel — that is exactly a weighted average of their anchor points, weighted by stiffness. The next figure draws exactly this tug-of-war.

Example 1 — Cell C1 (new task pulls the weight UP)

Forecast: guess before computing — is the answer closer to $2$ or to $6$? ($F$ times $\lambda =3$ pulls home, $g=1$ pulls new. Home pulls 3× harder → expect closer to 2.)

1. Identify the pulls. Home-pull weight $\lambda F=1\times 3=3$; new-pull weight $g=1$. Why this step? The equilibrium is a weighted average; we need both weights first.
2. Plug into the balance point. ${\theta }_{\text{end}}=\frac{g{\theta }_{\text{new}}+\lambda F{\theta }^{\ast }}{g+\lambda F}=\frac{1\cdot 6+3\cdot 2}{1+3}=\frac{12}{4}=3.$ Why this step? This is the value where the two spring forces exactly cancel — the resting place (the navy dot in figure s01).
3. Interpret the sign. ${\theta }_{\text{end}}=3>{\theta }^{\ast }=2$, so the weight moved up toward the new task, but only a little of the way ($3$ out of the $6-2=4$ possible). Why this step? Confirms the direction matches C1 (upward drift) and shows the spring resisted most of it.

Verify: slope at $\theta =3$: $g(3-6)+\lambda F(3-2)=1(-3)+3(1)=0$. Why this check? A resting point is exactly where the total slope is flat — the two pulls cancel. Getting $0$ confirms ${\theta }_{\text{end}}=3$ is a genuine equilibrium, not just a value we hoped for. ✓

Example 2 — Cell C2 (new task pulls the weight DOWN)

Forecast: by symmetry with Example 1, home pulls 3× harder, so expect a small drop below 2, not a plunge to $-2$.

1. Same weights. $\lambda F=3$, $g=1$. Why this step? Only the anchor ${\theta }_{\text{new}}$ flipped sign; the stiffnesses are unchanged.
2. Balance point. ${\theta }_{\text{end}}=\frac{1\cdot (-2)+3\cdot 2}{1+3}=\frac{-2+6}{4}=\frac{4}{4}=1.$ Why this step? Same formula — the physics is sign-blind; it just averages positions.
3. Interpret. ${\theta }_{\text{end}}=1<{\theta }^{\ast }=2$: the weight drifted down, exactly $1$ unit, mirroring the up-case that drifted $1$ unit ($2\to 3$). Why this step? Shows EWC is symmetric — it resists motion equally in both directions, because the penalty $(\theta -{\theta }^{\ast }{)}^2$ is a symmetric parabola.

Verify: slope at $\theta =1$: $1(1-(-2))+3(1-2)=3-3=0$. Why this check? Zero total slope means the downward new-task pull and the upward spring pull exactly cancel at $\theta =1$ — so $1$ is the true equilibrium, confirming the drift stopped there. ✓

Example 3 — Cell C3 (degenerate: $F=0$, unimportant weight)

Forecast: $\lambda$ is enormous, so it feels like the weight should be locked at 2. But $F=0$...

1. Compute the home-pull. $\lambda F=1000\times 0=0$. Why this step? The spring stiffness is $\lambda F$ — both factors matter. Zero importance kills the spring no matter how big $\lambda$ is.
2. Balance point. ${\theta }_{\text{end}}=\frac{1\cdot 6+0\cdot 2}{1+0}=\frac{6}{1}=6.$ Why this step? With no old-task spring, the new task wins completely — the weight goes all the way to ${\theta }_{\text{new}}$.
3. Interpret. The new task learns freely in this weight. This is EWC's "spare capacity": unimportant weights stay fully trainable. Why this step? Directly refutes the myth that EWC "blocks learning".

Verify: slope at $\theta =6$: $1(6-6)+0=0$. Why this check? With the spring gone, equilibrium must sit at the new-task minimum; a zero slope there confirms nothing resists the move to $6$. ✓

Example 4 — Cell C4 (degenerate: $F$ huge, critical weight)

Forecast: home-pull $=1000$, new-pull $=1$. Expect the weight glued near $2$.

1. Weights. $\lambda F=1000$, $g=1$.
2. Balance point. ${\theta }_{\text{end}}=\frac{1\cdot 6+1000\cdot 2}{1+1000}=\frac{6+2000}{1001}=\frac{2006}{1001}\approx \mathrm{2.004.}$ Why this step? The old spring is 1000× stiffer, so the resting place sits essentially on top of ${\theta }^{\ast }$.
3. Interpret. The weight barely budged (from $2$ to $\approx 2.004$). The old skill this weight encodes is protected. Why this step? This is the "critical weight" corner — high $F$ overrides the new task's wishes.

Verify: $\frac{2006}{1001}\approx 2.00399$, which is within $0.005$ of ${\theta }^{\ast }=2$. Why this check? An equilibrium pinned to home means the weight essentially did not move; confirming ${\theta }_{\text{end}}\approx {\theta }^{\ast }$ proves the old skill is protected. ✓

Example 5 — Cell C5 & C6 (limiting knobs $\lambda \to 0$ and $\lambda \to \infty$)

Forecast: $\lambda =0$ → no springs → plain fine-tuning → full forgetting. $\lambda \to \infty$ → infinitely stiff → frozen at home.

1. General expression in $\lambda$. ${\theta }_{\text{end}}(\lambda )=\frac{6+3\lambda \cdot 2}{1+3\lambda }=\frac{6+6\lambda }{1+3\lambda }.$ Why this step? Keeping $\lambda$ symbolic lets us take both limits from one formula.
2. Limit $\lambda \to 0$ (Cell C5). $\frac{6+0}{1+0}=6$. Why this step? No EWC term ⇒ objective is only the new task ⇒ the weight lands on ${\theta }_{\text{new}}=6$. This IS naive fine-tuning: maximum drift = maximum forgetting.
3. Limit $\lambda \to \infty$ (Cell C6). Divide top and bottom by $\lambda$: $\frac{6/\lambda +6}{1/\lambda +3}\to \frac{6}{3}=2$. Why this step? Infinite spring stiffness ⇒ the weight is pinned to ${\theta }^{\ast }=2$ ⇒ same as freezing it (parameter isolation). But then the new task learns nothing on this weight.
4. The trade-off. Between these extremes ($\lambda =1$ gave $3$ in Example 1) is the useful regime. Why this step? Shows $\lambda$ is a dial from "forget everything" to "learn nothing".

Verify: at $\lambda =1$, $\frac{6+6}{1+3}=\frac{12}{4}=3$ (matches Example 1). Why this check? The interior value must join the two endpoints continuously; recovering Example 1's $3$ at $\lambda =1$ confirms the swept formula and its limits are consistent. ✓ Endpoints: $6$ and $2$.

Example 6 — Cell C7 (learning rate $\eta$ alone)

Forecast: small $\eta$ ⇒ tiny drift ⇒ little forgetting; $\eta =1$ ⇒ jumps straight to the new minimum ⇒ full forgetting in one step.

1. Gradient at the start. $\nabla =\theta -6=2-6=-4$. Why this step? Gradient descent needs the slope; here the slope points strongly toward $6$.
2. Step with $\eta =0.1$. $\theta \leftarrow 2-0.1(-4)=2+0.4=2.4$. Why this step? Small step ⇒ weight only nudged $0.4$ from home ⇒ old skill mostly intact.
3. Step with $\eta =1.0$. $\theta \leftarrow 2-1.0(-4)=2+4=6$. Why this step? A step of size = curvature lands exactly on the minimum ⇒ maximal drift in a single update.
4. Interpret. Same task, same start — only $\eta$ differs, and it alone controls how far we drift from ${\theta }^{\ast }$. This is why "lower LR, fewer epochs" is a forgetting mitigation. Why this step? Connects the arithmetic to Learning Rate & Optimization.

Verify: $\eta =0.1\Rightarrow 2.4$; $\eta =1.0\Rightarrow 6.0$. Why this check? Drift from home ($2\to 2.4$ vs $2\to 6$) is the forgetting; confirming the two step sizes land at these values shows $\eta$ alone controls how far we leave ${\theta }^{\ast }$. ✓

Example 7 — Cell C8 (real-world word problem: LLM medical fine-tune)

Forecast: which fix makes the base weights literally unable to move? That is the only one that can guarantee the score.

1. Turn the requirement into a number. $90\%$ of $80=0.9\times 80=72$ points. The naive run's $32$ points is far below this, confirming the problem is real. Why this step? "Keep 90%" is vague; a concrete pass/fail threshold ($\ge 72$) lets us test each option instead of hand-waving.
2. Score option (c) — lower LR, fewer epochs. From Example 6, a smaller $\eta$ over fewer epochs shrinks the drift from ${\theta }^{\ast }$ but does not zero it; the coding weights still move some, landing near, say, $60$ points. Below $72$ — not guaranteed. Why this step? Example 6 already proved $\eta$ only scales drift; it never pins the weight, so no guarantee is possible.
3. Score option (a) — rehearsal. Mixing $8\%$ general data puts the general/coding loss back into the gradient, pulling the coding weights partway home to roughly $70$ points. Better, but the base weights still moved, and $70<72$ — still not a guarantee. Why this step? Rehearsal reduces forgetting (the general loss now defends itself) but cannot certify a floor, since the amount of protection depends on buffer size and mixing ratio.
4. Score option (b) — LoRA. LoRA freezes every base weight and learns only in small added modules. This is the $\lambda \to \infty$ / Cell C6 corner: ${\theta }_{\text{end}}={\theta }^{\ast }$ for the base coding weights, so they cannot move at all. Why this step? Frozen base weights $\Rightarrow$ the original coding pathway is byte-for-byte unchanged $\Rightarrow$ the score stays at its pretrained value by construction, not by luck.
5. Pick (b). Original coding capability sits in frozen base weights, so it holds at $80$, and $80\ge 72$. Requirement met with margin. ✓ Why this step? Only parameter isolation gives a guarantee; (a) and (c) give reductions that here fall short of the hard floor. Matches the mnemonic "FREEZE the base".

Verify: threshold $=0.9\times 80=72$; LoRA preserves $80\ge 72$ (pass), rehearsal's $\approx 70<72$ (no guarantee), low-LR's $\approx 60<72$ (no guarantee). Why this check? The word "guaranteed" demands the option whose preserved score provably clears $72$; only the frozen-weight case does, so the arithmetic confirms LoRA is the unique correct pick.

Example 8 — Cell C9 (exam twist: two weights, spare capacity)

Forecast: the network should route learning into the free weight (weight 2) and leave weight 1 near home.

1. Weight 1 balance point. Home-pull $\lambda F_1=1\times 100=100$, new-pull $g=1$: ${\theta }_{1,\text{end}}=\frac{g{\theta }_{1,\text{new}}+\lambda F_1{\theta }_1^{\ast }}{g+\lambda F_1}=\frac{1\cdot 5+100\cdot 0}{1+100}=\frac{5}{101}\approx \mathrm{0.0495.}$ Why this step? High importance ⇒ strong spring ⇒ weight 1 barely moves from home $0$ (the Cell C4 corner, now inside a two-weight problem).
2. Weight 2 balance point. Home-pull $\lambda F_2=1\times 0=0$, new-pull $g=1$: ${\theta }_{2,\text{end}}=\frac{1\cdot 5+0\cdot 0}{1+0}=\frac{5}{1}=5.$ Why this step? Zero importance ⇒ no spring ⇒ weight 2 goes all the way to its new minimum (the Cell C3 corner).
3. Interpret — the exam punchline. New-task learning was absorbed almost entirely by weight 2 ($0\to 5$, a full 5-unit move) while weight 1 stayed put ($0\to 0.05$, a $1\%$ nudge). EWC did not block learning; it redirected it into the unimportant direction — this is "learning in the spare capacity". Why this step? Directly refutes the classic EWC misconception "the penalty freezes learning", the trap most exam questions set here.

Verify: weight 1: $\frac{5}{101}\approx 0.0495$; weight 2: $5$. Why this check? The two equilibria together show the total 5-unit learning displacement landed almost entirely on the free weight 2 and almost none on the protected weight 1 — confirming EWC redirects rather than blocks learning. ✓

Connections

• Fine-Tuning LLMs — the setting for Example 7.
• LoRA and Adapters — the $\lambda \to \infty$ / freeze corner (C6, C8).
• Fisher Information — the $F$ that scales every spring here.
• Laplace Approximation — where the quadratic penalty comes from.
• Continual Learning — the sequential-task backdrop.
• Multi-task Learning — the rehearsal ideal in C8.
• Learning Rate & Optimization — the $\eta$ dial in Example 6.