Continual and lifelong learning: Exercises — AI-ML deep dive

This page tests the machinery of catastrophic forgetting, regularization-based methods, replay, and architecture methods. A few problems are geometric — look at the chalkboard figures as you read.

Level 1 — Recognition

Goal: can you name the pieces and read the notation?

L1.1 — Reading the forgetting metric

We write $Acc_{j, i}$ = the accuracy on task $i$ measured after the model has finished training through task $j$ . So $Acc_{3, 1}$ means "how well does the final-ish model (trained up to task 3) still do on task 1?"

Given this table:

	Acc on Task 1	Acc on Task 2	Acc on Task 3
after Task 1	0.90	—	—
after Task 2	0.72	0.88	—
after Task 3	0.60	0.75	0.85

Compute the backward transfer on task 1, defined as $BWT_{1} = Acc_{T, i} - Acc_{i, i}$ with final task $T = 3$ , task index $i = 1$ .

Recall Solution

$BWT_{1}$ compares the final accuracy on task 1 to the accuracy on task 1 right when it was learned ( $Acc_{1, 1}$ ).

$Acc_{3, 1} = 0.60$ (bottom-left cell: task 1 accuracy after all training).
$Acc_{1, 1} = 0.90$ (top-left cell).

$BWT_{1} = 0.60 - 0.90 = - 0.30$

Negative means the model got worse on task 1 — it forgot. The magnitude $0.30$ says it lost 30 percentage points.

L1.2 — Which family is which?

Match each one-line description to its family: Regularization-based, Replay-based, or Architecture-based.

(a) "Freeze old columns, add a new column with lateral connections for the new task." (b) "Penalize moving weights that had high Fisher information on the old task." (c) "Keep 500 old examples in a buffer and mix them into new mini-batches."

Recall Solution

(a) → Architecture-based (this is Progressive Neural Networks — see Neural Architecture Search for the design-space idea). Old parameters never updated ⇒ zero forgetting by construction.
(b) → Regularization-based (Elastic Weight Consolidation, EWC). The Fisher information $F_{j}$ measures how sharply the log-likelihood changes when you nudge weight $θ_{j}$ ; high $F_{j}$ = important weight to protect.
(c) → Replay-based (Experience Replay). Related to Online Learning streams and biologically to hippocampal replay.

Level 2 — Application

Goal: plug numbers into the formulas correctly.

L2.1 — Fisher information for logistic regression

For a logistic-regression head, the Fisher information of weight $j$ is

$F_{j} = \sum_{i = 1}^{n} p_{i} (1 - p_{i}) x_{ij}^{2},$

where $p_{i} = σ (θ^{⊤} x_{i})$ is the model's predicted probability for example $i$ , and $x_{ij}$ is feature $j$ of example $i$ . Here $σ$ is the sigmoid (squashes any number into $(0, 1)$ ), and $p (1 - p)$ is largest ( $= 0.25$ ) when the model is maximally uncertain ( $p = 0.5$ ).

Suppose task 1 has $n = 200$ examples, average $p_{i} (1 - p_{i}) = 0.20$ , mean-square feature values $\overline{x_{1}^{2}} = 9$ and $\overline{x_{2}^{2}} = 0.04$ . Compute $F_{1}$ and $F_{2}$ .

Recall Solution

Because we're using averages, the sum $\sum_{i} p_{i} (1 - p_{i}) x_{ij}^{2} \approx n \cdot \overline{p (1 - p)} \cdot \overline{x_{j}^{2}}$ .

$F_{1} \approx 200 \times 0.20 \times 9 = 360$ $F_{2} \approx 200 \times 0.20 \times 0.04 = 1.6$

Reading it: $F_{1} ≫ F_{2}$ , so weight 1 is roughly $360/1.6 = 225 \times$ more "important" for task 1. EWC will make weight 1 much stiffer than weight 2 when learning task 2.

L2.2 — The EWC penalty cost

The EWC objective is

$L_{t}^{EWC} (θ) = L_{t} (θ) + \frac{λ}{2} \sum_{j} F_{j} (θ_{j} - θ_{t - 1, j}^{*})^{2} .$

The last term is a spring: moving weight $j$ away from its old value $θ_{t - 1, j}^{*}$ costs energy proportional to stiffness $F_{j}$ . Using $F_{1} = 360$ , $F_{2} = 1.6$ from L2.1, $λ = 100$ , and old optimum $θ_{1}^{*} = [w_{1}, w_{2}] = [2.0, 0.1]$ :

Compute the penalty cost of (a) moving $w_{1}$ by $0.1$ , (b) moving $w_{2}$ by $0.1$ .

Recall Solution

Penalty per weight $= \frac{λ}{2} F_{j} (Δ θ_{j})^{2}$ .

(a) $\frac{100}{2} \times 360 \times (0.1)^{2} = 50 \times 360 \times 0.01 = 180.$

(b) $\frac{100}{2} \times 1.6 \times (0.1)^{2} = 50 \times 1.6 \times 0.01 = 0.8.$

The same displacement of $0.1$ costs $180$ on the stiff weight but only $0.8$ on the loose one — a ratio of $225$ , exactly the Fisher ratio. EWC automatically puts the rigidity where forgetting would hurt.

L2.3 — Reservoir sampling probability

In reservoir sampling we keep a fixed buffer of $k$ examples out of a stream of $N$ total. The rule guarantees every example ends up in the buffer with probability $k / N$ . Also, when example number $n$ (with $n > k$ ) arrives, it is accepted into the buffer with probability $k / n$ .

With $k = 500$ and a stream of $N = 20000$ : (a) what is the final probability any given example sits in the buffer? (b) what is the acceptance probability for the $1000$ -th arriving example?

Recall Solution

(a) By the reservoir guarantee, $P (in buffer) = \frac{k}{N} = \frac{500}{20000} = 0.025$ (2.5%).

(b) Acceptance probability $= \frac{k}{n} = \frac{500}{1000} = 0.5$ (50%).

Note (b) $>$ (a): early arrivals get accepted often but face many later chances to be evicted, and the two effects balance out to the uniform $k / N$ .

Level 3 — Analysis

Goal: reason about why methods behave as they do.

L3.1 — When do two gradients cause forgetting?

Consider a 2-weight model. At the task-1 optimum $θ^{*} = (0, 0)$ , the task-1 loss looks like a bowl $L_{1} (θ) = \frac{1}{2} (θ_{1}^{2} + θ_{2}^{2})$ . A single SGD step for task 2 uses the task-2 gradient $g_{2} = \nabla L_{2} (θ^{*})$ . We step $θ \leftarrow θ^{*} - η g_{2}$ .

Show, using a first-order change in $L_{1}$ , that whether task 1 gets worse depends on the dot product of $g_{2}$ with the task-1 gradient direction — and evaluate for $g_{2} = (3, - 1)$ , $η = 0.1$ .

Recall Solution

The dot product $a \cdot b = a_{1} b_{1} + a_{2} b_{2}$ measures alignment: positive if two vectors point the same general way. We use it here because the first-order change in any function along a step is exactly (gradient)·(step).

Change in task-1 loss for a small step $Δ θ = - η g_{2}$ : $Δ L_{1} \approx \nabla L_{1} (θ^{*})^{⊤} Δ θ = g_{1} \cdot (- η g_{2}) = - η (g_{1} \cdot g_{2}) .$ But at the optimum $θ^{*} = (0, 0)$ , $g_{1} = \nabla L_{1} = (θ_{1}, θ_{2}) = (0, 0)$ ! So to first order $Δ L_{1} = 0$ .

Forgetting at the exact minimum is a second-order effect. Compute $L_{1}$ at the new point directly. Step: $Δ θ = - 0.1 (3, - 1) = (- 0.3, 0.1)$ , new $θ = (- 0.3, 0.1)$ . $L_{1} (θ_{new}) = \frac{1}{2} ((- 0.3)^{2} + (0.1)^{2}) = \frac{1}{2} (0.09 + 0.01) = 0.05.$ We rose from $L_{1} = 0$ to $0.05$ : forgetting happened, driven by the curvature (the Hessian, here the identity), which is exactly what EWC's Fisher term protects. See figure below.

Figure — Continual and lifelong learning

L3.2 — Why replay changes the gradient

With plain SGD the task-2 update uses only $\nabla L_{2}$ . Experience Replay minimises

$L_{t}^{replay} = E_{D_{t}} [ℓ] + β E_{M} [ℓ],$

where $M$ is the buffer of old examples and $β$ weights their influence. Suppose the pure new-task gradient is $g_{new} = (3, - 1)$ (which we saw raises $L_{1}$ ) and the buffered old-task gradient is $g_{old} = (- 2, 0)$ . With $β = 1$ , compute the combined gradient and the resulting change in $L_{1} (θ) = \frac{1}{2} (θ_{1}^{2} + θ_{2}^{2})$ after a step $η = 0.1$ from $θ^{*} = (0, 0)$ .

Recall Solution

Combined gradient: $g = g_{new} + β g_{old} = (3, - 1) + (- 2, 0) = (1, - 1)$ .

Step: $Δ θ = - 0.1 (1, - 1) = (- 0.1, 0.1)$ , new $θ = (- 0.1, 0.1)$ .

$L_{1} (θ_{new}) = \frac{1}{2} ((- 0.1)^{2} + (0.1)^{2}) = \frac{1}{2} (0.02) = 0.01.$

Compare to $0.05$ from L3.1 without replay: forgetting dropped from $0.05$ to $0.01$ — a 5× reduction. The old-task gradient pulled the step back toward the region that keeps $L_{1}$ low. This is the replay "compromise" made concrete. See figure.

Level 4 — Synthesis

Goal: combine pieces into a design or a proof.

L4.1 — Budgeted replay allocation

You have a total buffer of $B = 900$ examples and have already finished tasks 1 and 2, currently holding $500$ from task 1 and $400$ from task 2. Task 3 now arrives with $3000$ fresh examples. You decide on an equal-per-task buffer after task 3.

(a) How many examples per task should the final buffer hold? (b) A mini-batch of size $30$ is drawn uniformly across the buffer's tasks. Roughly how many come from each task? (c) If instead you keep replay proportional to buffer counts before rebalancing (500 : 400 : 0), why does task 3 risk never being rehearsed, and what's the fix?

Recall Solution

(a) Three tasks share $B = 900$ equally: $900/3 = 300$ examples per task. So we subsample task 1 down from 500→300 and task 2 from 400→300, and admit 300 from task 3.

(b) Uniform across 3 tasks ⇒ $30/3 = 10$ examples per task in each mini-batch.

(c) With weights $500 : 400 : 0$ , task 3 has zero replay weight (it isn't in the "old" buffer while it's the current task, and the given proportional scheme allocates 0). The model would train on task 3's stream but never rehearse it against the buffer's balance, so once task 4 arrives, task 3 is under-protected. Fix: insert task 3 examples into the buffer as they stream in (reservoir/class-balanced), and rebalance to equal shares — exactly part (a). This mirrors the class-balanced buffer idea and connects to online updating.

L4.2 — Design a hybrid and predict forgetting order

Rank these four setups by expected final forgetting on task 1 (lowest forgetting first), and justify with one mechanism each:

(i) Plain SGD, no protection. (ii) EWC with large $λ$ . (iii) Experience Replay, buffer 300/task. (iv) Progressive Networks (frozen old columns).

Recall Solution

Predicted order (lowest forgetting → highest): $(iv) Progressive < (iii) Replay \approx (ii) EWC large λ < (i) Plain SGD .$

(iv) Old parameters are never updated ⇒ zero forgetting by construction — strictly the best on retention (cost: model grows each task; see Neural Architecture Search).
(iii) Replay directly reinjects old-task gradients (L3.2 showed 5× reduction), strong retention, bounded by buffer size and quality; ties to replay in the brain.
(ii) EWC only protects along Fisher-important directions via a quadratic approximation; large $λ$ boosts stability but sacrifices plasticity and errs where the quadratic approximation is poor. Related: Regularization Techniques.
(i) No memory, no penalty ⇒ worst forgetting; this is the baseline that motivates all of Continual and lifelong learning.

(Replay vs strong-EWC ordering can swap in practice; the key defensible claim is (iv) best, (i) worst.)

Level 5 — Mastery

Goal: derive, generalise, and reason at the limits.

L5.1 — Derive the EWC update and its fixed point

Starting from $L^{EWC} (θ) = L_{t} (θ) + \frac{λ}{2} \sum_{j} F_{j} (θ_{j} - θ_{j}^{*})^{2}$ , derive the gradient-descent update for a single weight, then find the fixed point (the $θ_{j}$ the update converges to) assuming the new-task loss is locally quadratic $L_{t} (θ) = \frac{1}{2} a_{j} (θ_{j} - c_{j})^{2}$ per weight, where $a_{j} > 0$ is the new-task curvature and $c_{j}$ its preferred value.

Recall Solution

Gradient of the EWC objective w.r.t. weight $j$ : $\frac{\partial L ^{EWC}}{\partial θ _{j}} = a_{j} (θ_{j} - c_{j}) + λ F_{j} (θ_{j} - θ_{j}^{*}) .$ Update: $θ_{j} \leftarrow θ_{j} - η [a_{j} (θ_{j} - c_{j}) + λ F_{j} (θ_{j} - θ_{j}^{*})] .$

Fixed point = where the gradient is zero (steady state): $a_{j} (θ_{j} - c_{j}) + λ F_{j} (θ_{j} - θ_{j}^{*}) = 0.$ Solve for $θ_{j}$ : $θ_{j}^{eq} = \frac{a _{j} c _{j} + λ F _{j} θ _{j}^{*}}{a _{j} + λ F _{j}}$ Interpretation: the equilibrium is a weighted average of the new-task ideal $c_{j}$ and the old value $θ_{j}^{*}$ , with weights $a_{j}$ (new-task curvature) and $λ F_{j}$ (protection). See figure.

Limits (all cases):

$F_{j} \to 0$ (unimportant weight): $θ_{j}^{eq} \to c_{j}$ — free to fully adapt. ✔ plasticity.
$λ F_{j} ≫ a_{j}$ (very important weight, strong penalty): $θ_{j}^{eq} \to θ_{j}^{*}$ — pinned to old value. ✔ stability.
$λ = 0$ : reduces to plain SGD, $θ_{j}^{eq} = c_{j}$ — no protection (recovers L3 forgetting).

L5.2 — Numeric mastery check on the fixed point

Using the boxed formula, take $a_{j} = 4$ , $c_{j} = 5$ , $λ = 100$ , $F_{j} = 0.1$ , $θ_{j}^{*} = 1$ . Compute $θ_{j}^{eq}$ . Then double $λ$ to $200$ and recompute. Which way does the equilibrium move, and does that match "more $λ$ ⇒ more stability"?

Recall Solution

With $λ = 100$ : $λ F_{j} = 100 \times 0.1 = 10$ . $θ_{j}^{eq} = \frac{4 \cdot 5 + 10 \cdot 1}{4 + 10} = \frac{20 + 10}{14} = \frac{30}{14} \approx 2.142857.$

With $λ = 200$ : $λ F_{j} = 20$ . $θ_{j}^{eq} = \frac{20 + 20 \cdot 1}{4 + 20} = \frac{40}{24} = \frac{5}{3} \approx 1.6667.$

The equilibrium moved from $2.143$ toward $1.667$ — closer to the old value $θ_{j}^{*} = 1$ . Doubling $λ$ pulled the weight back toward its task-1 setting: more stability, less plasticity, exactly as claimed. ✔

Connections

Prerequisites and neighbours: Continual and lifelong learning · Catastrophic Forgetting · Regularization Techniques · Transfer Learning · Meta-Learning · Knowledge Distillation · Neural Architecture Search · Online Learning · Neuroscience - Memory Consolidation · Hinglish version