Foundations — Catastrophic forgetting

This page assumes nothing. Before you can understand why forgetting happens or how EWC fixes it, you need to be able to read every squiggle in the parent note. We build each one from a picture, in an order where each rests on the one before.

---

0. What even is a "weight"? (the object everything acts on)

Picture a giant control panel covered in sliders. Each slider is one number. Slide them to different positions and the machine behaves differently — recognises cats, writes code, does arithmetic. That whole panel of numbers is what a network knows.

Why we need this: the parent note's central claim is that there is no separate memory slot per task — cats, code and arithmetic all live on the same panel. That is only sayable once you see $\theta$ as one shared list.

• $\theta$ (no subscript) ::: the entire panel of sliders (a vector).
• ${\theta }_i$ ::: one specific slider, the $i$-th number.

---

1. The function the sliders control: $f_{\theta }(x)$

The machine takes an input $x$ (a question, an image) and produces an output — a prediction.

The picture: $x$ enters on the left, flows through the panel, an answer $f_{\theta }(x)$ pops out on the right. We compare that answer to the truth $y$.

Why we need it: forgetting is a change in what $f_{\theta }(x)$ does — same $x$, but after training the slider positions $\theta$ moved, so the answer got worse. You cannot describe "worse" without naming input, output, and truth.

---

2. How wrong were we? The loss $\ell$ and $\mathcal{L}$

We need one number that says "how bad is this answer." Small = good, big = bad.

We rarely have one example — we have a whole pile of them. That pile is a dataset.

Now average the one-example penalty over the whole bag to get the total badness:

${\mathcal{L}}_{\text{new}}(\theta )=\frac{1}{|D_{\text{new}}|}{\sum }_{(x,y)\in D_{\text{new}}}\ell (f_{\theta }(x),y)$

Let us read each piece, left to right:

Why the topic needs it: "training on the new task" means pushing ${\mathcal{L}}_{\text{new}}$ downward. And catastrophic forgetting is precisely: ${\mathcal{L}}_{\text{new}}$ goes down while the un-mentioned ${\mathcal{L}}_{\text{old}}$ silently goes up.

---

3. Which way is downhill? The gradient ${\nabla }_{\theta }$

We want to make $\mathcal{L}$ small. So we ask: if I nudge each slider a tiny bit, does the badness rise or fall? The tool that answers "which way is downhill, and how steep" is the gradient.

The update rule that "does the walking":

$\theta \leftarrow \theta -\eta {\nabla }_{\theta }{\mathcal{L}}_{\text{new}}(\theta )$

Why the topic needs it: the parent's whole "WHY it happens" hangs on one observation — this update contains only ${\mathcal{L}}_{\text{new}}$. Nothing here even mentions ${\mathcal{L}}_{\text{old}}$, so the walk cheerfully strolls out of the old-task region. And "lower LR reduces drift" is literally "make $\eta$ smaller so each step leaves ${\theta }^{\ast }$ less." See Learning Rate & Optimization.

---

4. Two valleys on one map: the loss landscape

Since $\mathcal{L}(\theta )$ turns any slider setting into one badness number, imagine plotting badness as height above the slider-settings. You get a landscape of hills and valleys.

The key picture (matches the parent): draw two landscapes over the same horizontal axis of slider settings.

• ${\mathcal{L}}_{\text{old}}$ has its valley bottom at ${\theta }^{\ast }$.
• ${\mathcal{L}}_{\text{new}}$ has its valley bottom somewhere else.

Gradient descent on ${\mathcal{L}}_{\text{new}}$ slides $\theta$ toward the new bottom — which means climbing out of the old valley. That climb-out is the forgetting.

Why the topic needs it: this two-valley picture is the mental model for every fix. Rehearsal = add the old valley back into the terrain you descend. EWC = staple a spring to ${\theta }^{\ast }$. LoRA = don't move the base sliders at all.

---

5. "How much did each slider matter?" — importance & the Fisher $F_i$

The parent's fix (EWC) needs to know which sliders were critical for the old task and which were idle. Picture the old valley again: in some directions it is a steep narrow gorge (move a little, loss shoots up → that slider mattered a lot); in others it is a flat wide floor (move freely, loss barely changes → that slider was idle).

The formula the parent writes:

$F_i=\mathbb{E}[({\partial }_{{\theta }_i}\log p{)}^2]$

You do not need to derive this here — the full story lives in Fisher Information and its Gaussian-fitting cousin Laplace Approximation. You only need: $F_i$ = importance of slider $i$.

---

6. The spring penalty: $\lambda$ and $({\theta }_i-{\theta }_i^{\ast }{)}^2$

Now assemble the last pieces of the boxed EWC loss.

Put importance × distance-squared together and you get a per-slider spring of stiffness $F_i$:

${\mathcal{L}}_{\text{EWC}}(\theta )={\mathcal{L}}_{\text{new}}(\theta )+\frac{\lambda }{2}{\sum }_i{F}_i({\theta }_i-{\theta }_i^{\ast }{)}^2$

Read it now that every symbol is earned: "be good at the new task (${\mathcal{L}}_{\text{new}}$), plus pay a penalty ($\frac{\lambda }{2}{\sum }_i$) that pulls every important ($F_i$) slider back toward its old home (${\theta }_i^{\ast }$)." Idle sliders ($F_i\approx 0$) feel no pull — the new task learns in the leftover room. This is why the parent's Multi-task Learning and Continual Learning links matter: EWC is a cheap stand-in for training everything at once.

---

7. The prerequisite map

Here is how every foundation above feeds the topic. Read bottom-up: symbols at the top are needed to state the ones below them.

---

Equipment checklist

Cover the right side; can you answer each before reading on? If not, reread its section.

• What does $\theta$ stand for, and what is ${\theta }_i$? ::: $\theta$ = the whole list of the network's weights (all sliders); ${\theta }_i$ = the single $i$-th weight.
• What does the subscript in $f_{\theta }(x)$ tell you? ::: The output depends on the slider settings $\theta$ — move the weights and the same input $x$ gives a different answer.
• Difference between $\ell$ and $\mathcal{L}$? ::: $\ell$ = penalty for one example; $\mathcal{L}$ = the average penalty over a whole dataset.
• What do $\sum$, $\in$, and $|D|$ mean? ::: $\sum$ = add up; $\in$ = "is inside the set"; $|D|$ = the number of pairs in dataset $D$.
• What does the gradient ${\nabla }_{\theta }\mathcal{L}$ point toward, and which way do we step? ::: It points uphill (fastest loss increase); we step the opposite way to go downhill.
• In the update $\theta \leftarrow \theta -\eta \nabla \mathcal{L}$, what is $\eta$? ::: The learning rate — the size of each step; smaller $\eta$ means less drift from the old optimum.
• What is ${\theta }^{\ast }$ in the loss-landscape picture? ::: The weight setting at the bottom of the old task's valley — where we landed after training the old task.
• Why do "two valleys" cause forgetting? ::: The new task's valley bottom sits elsewhere, so descending the new loss climbs $\theta$ out of the old valley.
• What does a large $F_i$ tell you about slider $i$? ::: It was important for the old task (steep curvature) — moving it hurts the old skill a lot, so EWC penalizes moving it.
• What role does $\lambda$ play in the EWC loss? ::: It scales overall protectiveness — big $\lambda$ = stiff springs defending old knowledge, small $\lambda$ = freer new-task learning.
• Why does $({\theta }_i-{\theta }_i^{\ast }{)}^2$ behave like a spring? ::: It is zero at the old home and grows with the square of displacement — exactly a spring's energy that pulls back harder the further you stray.

---

Connections

• Catastrophic Forgetting — the parent topic these foundations unlock.
• Fisher Information — the full story of the importance $F_i$.
• Laplace Approximation — where the Gaussian-around-${\theta }^{\ast }$ picture comes from.
• Learning Rate & Optimization — the step size $\eta$ and why smaller reduces drift.
• Fine-Tuning LLMs — where these symbols get used in practice.
• LoRA and Adapters — the "freeze the base sliders" alternative.
• Continual Learning · Multi-task Learning — the broader settings EWC approximates.