Worked examples — Continual and lifelong learning

This is a worked-examples deep dive for 6.5.7 Continual and lifelong learning. The parent note built the ideas (EWC, replay, progressive nets, forgetting metrics). Here we drill them: we enumerate every case a continual-learning problem can throw at you, then solve one example per case, guessing first, computing carefully, and checking the answer.

Every symbol below was defined in the parent. If you forgot one:

Recall Quick symbol refresher

• $\theta$ ::: the vector of all network weights (one number per knob).
• ${\mathcal{L}}_t(\theta )$ ::: the loss (error) on task $t$ for weights $\theta$ — smaller is better.
• $F_j$ ::: Fisher information of weight $j$ — how much task performance reacts if you nudge weight $j$. Big $F_j$ = important weight.
• $T$ ::: the total number of tasks in the stream. Task indices run $1,\dots ,T$.
• ${\text{Acc}}_{j,i}$ ::: accuracy on task $i$ measured after finishing training through task $j$.
• ${\text{BWT}}_i={\text{Acc}}_{T,i}-{\text{Acc}}_{i,i}$ ::: backward transfer; negative means we forgot task $i$.

---

The scenario matrix

Think of a continual-learning problem as a machine with dials. Each dial has extreme settings, and each combination is a "cell" you must be able to handle. This table lists every cell class this topic contains. Each row maps one-to-one to exactly one example.

#	Case class	The "extreme / sign" being tested	Example that hits it
C1	Forgetting sign: negative	model got worse on old task → $\text{BWT}<0$	Ex 1
C2	Forgetting sign: positive (backward transfer)	model got better on old task → $\text{BWT}>0$	Ex 2
C3	Forgetting sign: zero (no change)	model untouched → $\text{BWT}=0$	Ex 3
C4	Zero / degenerate input	an unimportant weight, $F_j\approx 0$	Ex 4
C5	Large limit	$F_j$ huge or $\lambda \to \infty$ → total rigidity	Ex 5
C6	EWC full numeric	compute the penalty term end-to-end	Ex 6
C7	Replay sampling probability	reservoir sampling, "every example equal"	Ex 7
C8	Replay budget split	buffer allocation across tasks	Ex 8
C9	Real-world word problem	self-driving / medical stream	Ex 9
C10	Architecture: parameter count growth	progressive nets, cost vs. zero forgetting	Ex 10
C11	Exam twist	which method fails, and why	Ex 11

The figures below carry the arguments; every example that needs geometry points at a specific coloured curve.

Figure 1 — EWC penalty bowls. The red narrow bowl is an important weight (large Fisher); the green wide bowl is a free weight (tiny Fisher). Referenced by Ex 4 and Ex 5.

---

The examples

Ex 1 — Negative backward transfer (cell C1)

Forecast: guess — did task 1 get worse? By roughly how much?

1. ${\text{BWT}}_1={\text{Acc}}_{3,1}-{\text{Acc}}_{1,1}=0.60-0.90=-0.30$. Why this step? BWT compares final accuracy (after all $T=3$ tasks, and here the total number of tasks is $T=3$) to the accuracy right when the task was first mastered. The $-0.30$ sign tells us: catastrophic forgetting happened.
2. Average forgetting uses the peak accuracy, not the first: for task $i$, take ${\max }_{j\in \{i,\dots ,T-1\}}{\text{Acc}}_{j,i}-{\text{Acc}}_{T,i}$.
   • Task 1: peak over $j\in \{1,2\}$ is $\max (0.90,0.70)=0.90$; final is $0.60$ → forgetting $=0.30$.
   • Task 2: peak over $j\in \{2\}$ is $0.88$; final is $0.75$ → forgetting $=0.13$. Why max, not first? Because a task might briefly improve before it decays; forgetting is loss relative to the best we ever had.
3. Average $=\frac{1}{T-1}(0.30+0.13)$. With $T=3$ this is $\frac{1}{2}(0.43)=\frac{0.43}{2}=0.215$. Why $\frac{1}{T-1}$? We average over the $T-1$ old tasks (task $T$ itself cannot be forgotten yet), so the denominator is $T-1=2$.

Verify: BWT is a single before/after difference, so it should be within $[-1,1]$: $-0.30$ ✓. Average forgetting is a mean of non-negative numbers here, all $\le 1$: $0.215$ ✓.

---

Ex 2 — Positive backward transfer (cell C2)

Forecast: could learning task 2 make task 1 better?

1. ${\text{BWT}}_1={\text{Acc}}_{2,1}-{\text{Acc}}_{1,1}=0.87-0.80=+0.07$. Why this step? Same formula as before — nothing special. The novelty is the positive sign.
2. Interpret: positive BWT = backward transfer. Task 2 taught a feature (say, better edge detectors) that also helped task 1. This is the dream of transfer flowing backward in time.

Verify: $0.87>0.80$ so the difference must be positive ✓, and $\le 0.20$ so it's a plausible small gain, not a bug.

---

Ex 3 — Zero backward transfer, the "no change" edge case (cell C3)

Forecast: is BWT ever exactly $0$? Guess what design would guarantee it.

1. ${\text{BWT}}_1={\text{Acc}}_{2,1}-{\text{Acc}}_{1,1}=0.82-0.82=0$. Why this step? Same subtraction — but the two numbers are identical, so we land on the boundary between forgetting ($<0$) and backward transfer ($>0$).
2. Interpret: exactly zero means task 1 was neither harmed nor helped by learning task 2. This is the degenerate middle case the sign axis must cover.
3. Which family guarantees it? Architecture-based methods (e.g. progressive nets) freeze task 1's parameters, so task 1's forward pass is byte-for-byte unchanged → $\text{BWT}=0$ by construction (no better, no worse). Regularization and replay only approximate zero.

Verify: $0.82-0.82=0$ sits exactly on the boundary $[-1,1]$, neither $<0$ nor $>0$ ✓. Consistent with "frozen weights ⇒ identical predictions ⇒ identical accuracy" ✓.

---

Ex 4 — A degenerate (unimportant) weight (cell C4)

Forecast: big penalty or tiny? Guess before computing.

1. EWC penalty for one weight is $\frac{\lambda }{2}{F}_2(w_2-{\theta }_2^{\ast }{)}^2$. Why this term? It is the quadratic Taylor bowl approximating the old loss — steepness set by $F_2$.
2. Plug in. Why this step? We substitute the given numbers and expand the squared distance explicitly so no intermediate is hidden:
   • distance $=5.0-0.1=4.9$,
   • squared $=4.9^2=4.9\times 4.9=24.01$,
   • penalty $=\frac{1000}{2}\cdot 0.25\cdot 24.01=500\cdot 0.25\cdot 24.01=125\cdot 24.01=3001.25$. Wait — that's not tiny! But compare it against a truly important weight (Ex 5). Here $F_2$ is small, so per unit of movement the cost is small; it only got large because we moved a huge distance.
3. Per-unit cost: $\frac{\lambda }{2}{F}_2=500\cdot 0.25=125$ per unit². Why divide out the distance? To separate "how stiff" (the $125$ factor) from "how far we moved" (the $24.01$). For an important weight with $F=100$ that same factor is $500\cdot 100=50000$ — 400× stiffer. That's the point of Fisher: it lets free weights roam while pinning important ones.

Verify: the bowl is symmetric, so moving $+4.9$ or $-4.9$ costs the same: $500\cdot 0.25\cdot (-4.9{)}^2=3001.25$ ✓. In Figure 1 this weight is the green wide bowl — its shallow slope near $w=0.1$ means cheap movement, matching the small per-unit factor.

---

Ex 5 — Limiting case: rigidity as $F\to \infty$ (cell C5)

Forecast: closer to $2.0$ (old) or $2.5$ (new)? Guess the fraction.

1. Total objective: $\frac{1}{2}g(w_1-2.5{)}^2+\frac{\lambda }{2}{F}_1(w_1-2.0{)}^2$. Set derivative to zero. Why this step? Two springs pull $w_1$: the new-task spring (stiffness $g$) toward $2.5$, the old-memory spring (stiffness $\lambda F_1$) toward $2.0$. The minimum is the balance point. In Figure 1 the old-memory spring is the steep red bowl centred at $2.0$; the new-task spring is shallow.
2. Take the derivative and set it to zero: $g(w_1-2.5)+\lambda F_1(w_1-2.0)=0$. Now unpack the algebra step by step (this is the WHY):
   • Expand: $g{w}_1-g\cdot 2.5+\lambda F_1{w}_1-\lambda F_1\cdot 2.0=0$.
   • Collect the $w_1$ terms on the left, constants on the right: $(g+\lambda F_1)w_1=g\cdot 2.5+\lambda F_1\cdot 2.0$.
   • Divide both sides by $(g+\lambda F_1)$: $w_1=\frac{g\cdot 2.5+\lambda F_1\cdot 2.0}{g+\lambda F_1}.$ Why is this a weighted average? Each spring "votes" for its own target ($2.5$ or $2.0$) with weight equal to its stiffness ($g$ or $\lambda F_1$). The result is the stiffness-weighted mean — the stiffer spring wins.
3. Plug in: $\frac{20\cdot 2.5+100000\cdot 2.0}{20+100000}=\frac{200050}{100020}\approx 2.0001$.
4. Limit: as $\lambda F_1\to \infty$, the old-memory weight dominates the average, so $w_1\to 2.0$ exactly. The old memory spring is so stiff the new task barely moves it — total stability, zero plasticity. That is the extreme end of the stability–plasticity dilemma.

Verify: the result must lie between the two targets $2.0$ and $2.5$ (a convex combination): $2.0\le 2.0001\le 2.5$ ✓. It sits far nearer $2.0$ because $\lambda F_1\gg g$ ✓.

---

Ex 6 — Full EWC penalty across all weights (cell C6)

Forecast: which of the three weights dominates the penalty?

1. Compute each term with the shared factor $\frac{\lambda }{2}=500$. Why per-term? The penalty is a sum of independent bowls — one per weight (diagonal Fisher) — so we can add them separately.
   • $w_1$: $500\cdot 100\cdot (2.1-2.0{)}^2=500\cdot 100\cdot 0.01=500$.
   • $w_2$: $500\cdot 0.25\cdot (0.6-0.1{)}^2=500\cdot 0.25\cdot 0.25=31.25$.
   • $b$: $500\cdot 100\cdot (-0.4-(-0.5){)}^2=500\cdot 100\cdot 0.01=500$.
2. Total $=500+31.25+500=1031.25$.
3. Notice $w_2$ moved the most (0.5) yet costs the least (31.25). EWC "forgives" big moves in unimportant directions — exactly what we want for plasticity.

Verify: total must equal sum of parts $500+31.25+500=1031.25$ ✓; every term $\ge 0$ since it's a square times positive $F$ ✓.

---

Ex 7 — Reservoir sampling probability (cell C7)

Forecast: early examples get overwritten a lot — will #37's survival be much below $500/10000$?

1. The rule. When example $n$ arrives (with $n>k$), we keep it with probability $k/n$; if kept, it evicts a uniformly random current occupant. Why this rule at all? We want to end with a uniform random subset of size $k$ without knowing $N$ in advance — this is the only rule that keeps the buffer uniform at every step.
2. Why the survival math cancels — the intuitive derivation. Example #37 enters the buffer immediately (since $37\le 500$). Now walk forward. When example $n=501$ arrives it is accepted with probability $\frac{500}{501}$, and if accepted it kicks out a random one of the 500, so #37 dies with probability $\frac{500}{501}\cdot \frac{1}{500}=\frac{1}{501}$ — i.e. #37 survives step 501 with probability $1-\frac{1}{501}=\frac{500}{501}$. In general #37 survives step $n$ with probability $\frac{n-1}{n}$.
3. Multiply all survival steps from $n=501$ to $N=10000$: $P=\frac{500}{501}\cdot \frac{501}{502}\cdot \frac{502}{503}\cdots \frac{9999}{10000}.$ This is a telescoping product: every numerator cancels the previous denominator, leaving only the first numerator and last denominator: $P=\frac{500}{10000}=\mathrm{0.05.}$ That is the "cancel perfectly" claim, now shown explicitly: being accepted early (advantage) is exactly balanced by facing more eviction rounds (disadvantage), and telescoping proves it equals $k/N$ for every index.

Verify: $500/10000=0.05\in [0,1]$ ✓. Because the product telescopes to $k/N$ regardless of the starting index, example #9000 also gets $0.05$ — uniform, as Figure 2 shows.

Figure 2 — reservoir vs. naive keep-first. The flat orange line at height $0.05$ is reservoir sampling: same survival probability for early and late examples. The dashed gray curve is a naive "keep the first 500" scheme whose survival collapses for later examples — the bias reservoir sampling fixes.

---

Ex 8 — Replay budget split across tasks (cell C8)

Forecast: guess how many task-1 examples get evicted.

1. Equal split over 3 tasks: $600/3=200$ per task. Why this step? Equal rehearsal time per task keeps forgetting balanced — no task starves, so no old task decays for lack of reminders.
2. Task 1 currently has 400, target 200 → drop $400-200=200$. Task 2 has 200, target 200 → keep all. Task 3 needs to fill 200 fresh slots. Why drop exactly this many? The buffer is a fixed 600 slots; to make room for task 3's 200 slots while leaving task 2 untouched, the over-represented task 1 must shed exactly the surplus $400-200$.
3. Check total: $200+200+200=600$, which equals the buffer size — no overflow, no wasted slots. The 200 slots freed by task-1 eviction exactly refill task 3's 200 slots.

Verify: kept $=200\cdot 3=600$ ✓; dropped from task 1 $=400-200=200$ ✓; freed slots $200$ exactly refill task 3's $200$ ✓.

---

Ex 9 — Real-world word problem (cell C9)

Forecast: replay is blocked — which family survives the privacy constraint?

1. Forgetting on A $=0.92-0.61=0.31$ (a ${\text{BWT}}_A=-0.31$). Why this step? Same before/after subtraction as Ex 1 — the metric doesn't care about the domain.
2. Constraint check: raw replay (Catastrophic Forgetting fix #2) needs stored A scans → forbidden by privacy. So choose either:
   • Regularization (EWC): stores only weights + Fisher, no patient data. Legal.
   • Generative replay: a generator makes synthetic A-like images; no real scans stored. Also legal, and links to Knowledge Distillation when you distill old outputs.
3. Recommendation: EWC or generative replay; raw experience replay is ruled out.

Verify: forgetting $0.92-0.61=0.31\in [0,1]$ ✓; it is positive because $0.92>0.61$, confirming a genuine drop ✓.

---

Ex 10 — Progressive nets: parameter growth (cell C10)

Forecast: does cost grow linearly or quadratically in $T$?

1. Column $t$ adds $P$ base params plus $L\times (t-1)$ lateral params (connects to all $t-1$ earlier columns). Why this step? Lateral connections $U^{(t\leftarrow j)}$ exist for every earlier column $j<t$ — that is what reuses old frozen features with zero forgetting.
2. Total $={\sum }_{t=1}^5(P+L(t-1))=5P+L(0+1+2+3+4)=5P+10L$.
3. Plug in: $5\cdot 10^6+10\cdot 2\cdot 10^5=5,000,000+2,000,000=7,000,000$. The lateral part $\sum (t-1)=T(T-1)/2$ grows quadratically — the price of guaranteed no-forgetting. This motivates Neural Architecture Search to prune.

Verify: ${\sum }_{t=1}^5(t-1)=0+1+2+3+4=10=5\cdot 4/2$ ✓; total $5\cdot 10^6+10\cdot 2\cdot 10^5=7\cdot 10^6$ ✓.

---

Ex 11 — Exam twist: which method fails? (cell C11)

Forecast: which single method survives an unbounded task stream with a hard memory cap?

1. Progressive nets: add a column per task → params grow (at least) linearly, laterals quadratically (Ex 10). With thousands of tasks and a memory cap this explodes → infeasible. Failure mode: unbounded parameter growth. Why? No mechanism ever frees old columns.
2. Experience replay: fixed buffer $k$ spread over thousands of tasks → each task gets $\approx k/T\to 0$ examples. Failure mode: rehearsal starves as $T$ grows; forgetting creeps back.
3. EWC — does it survive? Yes, on memory. It stores only ${\theta }^{\ast }$ and the diagonal Fisher $F$ — a fixed-size vector, so its footprint is constant in $T$ (does not grow with the number of tasks). This is why EWC is the only one of the three that fits the hard memory cap for an unbounded stream.
4. EWC's subtler failure mode. Feasible on memory does not mean flawless. Each new task adds another Fisher-weighted penalty term, and these penalties accumulate and stack. As more and more directions become "important", the summed rigidity keeps rising until the network hits the total-rigidity limit of Ex 5 ($\lambda F\to \infty \Rightarrow$ weights pinned). At that point plasticity dies: the model can no longer learn new tasks even though it never forgot. So EWC's failure mode is creeping loss of plasticity, not memory blow-up.
5. Verdict. Ranking (best → worst for this exact setting): EWC > Replay > Progressive nets.
   • EWC: survives the memory cap; fails eventually by saturating rigidity (loses plasticity).
   • Replay: fails gradually as per-task rehearsal $k/T\to 0$.
   • Progressive nets: fails immediately via parameter explosion. Escaping EWC's rigidity trap is exactly what Online Learning and Meta-Learning approaches target.

Verify: the $k/T\to 0$ claim: with $k=500$, $T=5000$, per-task share $=500/5000=0.1$ example — effectively nothing ✓. EWC memory $=|\theta |+|\theta |$ is independent of $T$ ✓.

---

Recall Self-test

BWT sign that means forgetting? ::: negative. BWT exactly zero means, and which family guarantees it? ::: no change on the old task; architecture-based (frozen weights). A weight with $F_j\approx 0$ under EWC is... ::: free to move (unimportant), tiny penalty. As $\lambda F_j\to \infty$, EWC keeps the weight at... ::: its old value ${\theta }^{\ast }$ (pure stability). Reservoir sampling gives each of $N$ examples buffer probability... ::: $k/N$, because the survival product telescopes regardless of arrival time. Progressive-net lateral connections grow how in $T$? ::: quadratically, $T(T-1)/2$. Which family survives an unbounded task stream with fixed memory, and how does it eventually fail? ::: EWC (constant memory); it saturates rigidity and loses plasticity.

See also: Regularization Techniques (the penalty machinery behind EWC) and Neuroscience - Memory Consolidation (biology's own replay-and-consolidate strategy).