Continual and lifelong learning

Why this matters: Real-world AI must adapt continuously (new products in recommendation, new diseases in medical imaging, new languages in translation) without retraining from scratch every time. A self-driving car that forgets how to detect pedestrians after learning to recognize new road signs is catastrophic.

The Stability-Plasticity Dilemma

The challenge: standard SGD updates $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}_t(\theta)$ optimize only current task loss $\mathcal{L}_t$, overwriting weights useful for $\mathcal{T}_1, \ldots, \mathcal{T}_{t-1}$.

Lifelong Learning is used interchangeably, sometimes emphasizing transfer: new tasks benefit from old knowledge (forward transfer), old tasks improve from new knowledge (backward transfer).

Why Catastrophic Forgetting Happens (First Principles)

Let's derive why neural nets forget. Suppose we train task1 to convergence:

$\theta^* = \arg\min_\theta \mathcal{L}_1(\theta) = \arg\min_\theta \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}_1}[\ell(f(\mathbf{x};\theta), y)]$

At $\theta^*$, the gradient $\nabla_\theta \mathcal{L}_1(\theta^*) \approx 0$ (local minimum). Now task 2 arrives. We update:

$\theta \leftarrow \theta^* - \eta \nabla_\theta \mathcal{L}_2(\theta^*)$

Why this step? We're moving $\theta$ in the direction that reduces $\mathcal{L}_2$. But this gradient $\nabla_\theta \mathcal{L}_2$ is computed only on $\mathcal{D}_2$. If $\nabla_\theta \mathcal{L}_2$ points in a direction that increases $\mathcal{L}_1$, we're actively making task 1 worse.

Catastrophic forgetting occurs because:

1. Parameter overlap: The same weights $\theta$ must serve all tasks. Moving $\theta$ for task 2 disturbs the configuration optimized for task 1.
2. No explicit memory: Standard SGD has no mechanism to remember $\mathcal{L}_1$ once $\mathcal{D}_1$ is gone.

$\text{BWT}_i = \text{Acc}_{T,i} - \text{Acc}_{i,i}$

where $\text{Acc}_{j,i}$ = accuracy on task $i$ after training through task $j$. Negative BWT means forgetting.

Average forgetting across $T-1$ tasks:

$\text{Forgetting} = \frac{1}{T-1}\sum_{i=1}^{T-1} \max_{j \in \{i,\ldots,T-1\}} (\text{Acc}_{j,i} - \text{Acc}_{T,i})$

Why max? We compare final performance to the best performance achieved on task $i$ before learning later tasks (forgetting is relative to peak, not just initial).

Three Families of CL Solutions

1. Regularization-Based (Protect Important Weights)

Idea: Identify which weights $\theta_j$ are critical for old tasks, then penalize changes to them when learning new tasks.

$F_j = \mathbb{E}_{\mathcal{D}_{t-1}} \left[ \left( \frac{\partial \log p(y|\mathbf{x};\theta)}{\partial \theta_j} \right)^2 \right]$

Why Fisher? It measures how much the log-likelihood changes if we perturb $\theta_j$. High $F_j$ → parameter $j$ is "important" for the previous task's prediction.

For task $t$, we minimize:

$\mathcal{L}_t^{\text{EWC}}(\theta) = \mathcal{L}_t(\theta) + \frac{\lambda}{2} \sum_j F_j (\theta_j - \theta^*_{t-1,j})^2$

Why this form? The quadratic penalty is a second-order Taylor approximation of $\mathcal{L}_{t-1}$ around $\theta^*_{t-1}$:

$\mathcal{L}_{t-1}(\theta) \approx \mathcal{L}_{t-1}(\theta^*_{t-1}) + \frac{1}{2}(\theta - \theta^*_{t-1})^\top H (\theta - \theta^*_{t-1})$

where $H$ is the Hessian. Fisher $F$ approximates the diagonal of $H$ for classification (under mild assumptions). So we're saying: "move $\theta$ to minimize $\mathcal{L}_t$, but stay near $\theta^*_{t-1}$ in dimensions important for $\mathcal{L}_{t-1}$."

Hyperparameter $\lambda$: Controls stability-plasticity trade-off. Large $\lambda$ → strong penalty → more stability, less plasticity.

Compute Fisher on $\mathcal{D}_1$: For logistic regression, $F_j = \sum_{i} p_i(1-p_i) x_{ij}^2$ where $p_i = \sigma(\theta^\top \mathbf{x}_i)$.

Say $\mathcal{D}_1$ has 100 examples, average $p(1-p) = 0.25$, $\bar{x}_1^2 = 4$, $\bar{x}_2^2 = 0.01$:

• $F_1 \approx 100 \times 0.25 \times 4 = 100$ (important!)
• $F_2 \approx 100 \times 0.25 \times 0.01 = 0.25$ (less important)

Why this matters? When learning task 2, deviating $w_1$ by 0.1 costs $\frac{\lambda}{2} \times 100 \times (0.1)^2 = 0.5\lambda$. Deviating $w_2$ by 0.1 costs only $0.0125\lambda$. EWC automatically focuses rigidity where it's needed.

Task 2 loss with $\lambda=1000$:

$\mathcal{L}_2^{\text{EWC}} = \mathcal{L}_2(\theta) + 500(w_1 - 2.0)^2 + 0.125(w_2 - 0.1)^2 + 500(b + 0.5)^2$

2. Replay-Based (Store and Rehearse Old Data)

Idea: Keep a small memory buffer of old examples and interleave them with new task training. This directly prevents forgetting by continually "reminding" the model of old tasks.

$\mathcal{L}_t^{\text{replay}}(\theta) = \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}_t}[\ell(f(\mathbf{x};\theta), y)] + \beta \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{M}}[\ell(f(\mathbf{x};\theta), y)]$

Why this works? Gradients now include $\nabla_\theta \mathcal{L}_1, \nabla_\theta \mathcal{L}_2, \ldots$ from replayed data. Moving $\theta$ to reduce $\mathcal{L}_t$ while also satisfying old losses forces a compromise—stability.

Key challenge: Memory size. If $|\mathcal{M}|$ is fixed (say 5000 examples), how do we select which examples to store?

Reservoir Sampling: For a stream of $N$ total examples, select $k$ uniformly. When example $n$ arrives:

• If $n \leq k$: store it.
• If $n > k$: replace a random existing example with probability $k/n$.

Why? This ensures each example has equal probability $k/N$ of being in $\mathcal{M}$, regardless of when tasks arrived.

Generative Replay: Instead of storing real data (privacy issues, memory), train a generative model $p_\theta(\mathbf{x}, y)$ on old tasks and sample "pseudo-examples" from it.

$\mathcal{M}_{\text{gen}} = \{(\tilde{\mathbf{x}}, \tilde{y}) \sim p_{\theta_{-1}}(\mathbf{x}, y)\}$

Equal allocation: Keep 200 from each task → 600 total. When sampling a mini-batch (size 32) for task 3, include ~10 task-1 examples, ~10 task-2, ~12 task-3 (weighted by buffer size or uniform).

Why this step? Ensures all tasks get "rehearsal time" proportional to their representation. Without this, if we only sampled task 3, we'd still forget1 and 2.

Alternative: Class-balanced buffer—if tasks have different class distributions, store equal examples per class.

3. Architecture-Based (Dedicate Parameters to Tasks)

Idea: Allocate separate sub-networks or parameters for different tasks, so learning task2 doesn't touch task 1's parameters at all.

$h_i^{(t)} = \sigma\left( W_i^{(t)} h_{i-1}^{(t)} + \sum_{j=1}^{t-1} U_i^{(t \leftarrow j)} h_{i-1}^{(j)} \right)$

Why? $h^{(j)}$ are frozen features from old tasks. $U^{(t \leftarrow j)}$ learns to reuse them. Old parameters $\theta^{(1)}, \ldots, \theta^{(t-1)}$ are never updated → zero forgetting by construction.

Downside: Model grows linearly with tasks. 10 tasks → 10× parameters.

PackNet: Prune the network after each task, "freezing" the winning lottery ticket for that task. New tasks use the remaining capacity (pruned weights).

Dynamic Expandable Network: Start small, add new neurons/layers only when new task needs more capacity. Freeze neurons important for old tasks.

Task 2: Classify cats (10 classes). Add a parallel 3-layer CNN column (new random init). Lateral connections from dog-column layer1 → cat-column layer 1, etc.

Why this helps? Cat column can reuse edge detectors and texture features learned for dogs (via $U$ connections), but has freedom to learn cat-specific features in $W$ without destroying dog features.

After task 2: Dog accuracy still 90% (frozen), cat accuracy 88%. Total params: 2× original.

Task 3: Birds (10 classes). Add bird column, connect to dog + cat. Now3× params, but dog still 90%, cat 88%, bird 85%.

Common Scenarios & Metrics

• Class-Incremental Learning (Class-IL): At test time, task ID is unknown. Model must correctly classify over all seen classes.
  • Example: "Classify this image (could be dog, cat, or bird, but I won't tell you which task)."
  • Harder, requires learning task-agnostic features and output heads.

$\text{ACC} = \frac{1}{T} \sum_{i=1}^T \text{Acc}_{T,i}$

Forward Transfer (does task $t$ help future task $t+1$?):

$\text{FWT}_i = \text{Acc}_{i-1, i} - \text{Acc}_{\text{rand}, i}$

where $\text{Acc}_{\text{rand}, i}$ is random-init accuracy on task $i$. Positive FWT means knowledge from $1, \ldots, i-1$ helped.

Ideal CL system: High ACC (good at all tasks), low Forgetting (remembers old tasks), positive FWT (knowledge transfer).

Key Algorithms Detail

Memory-Aware Synapses (MAS)

Similar to EWC, but computes importance without Fisher. For each parameter $\theta_j$:

$\Omega_j = \frac{1}{|\mathcal{D}_{t-1}|} \sum_{\mathbf{x} \in \mathcal{D}_{t-1}} \left\| \frac{\partial f(\mathbf{x}; \theta)}{\partial \theta_j} \right\|_2^2$

Why output gradient? If changing $\theta_j$ drastically changes the output $f(\mathbf{x})$, then $\theta_j$ is important for the learned function, regardless of the loss.

Loss for task $t$:

$\mathcal{L}_t^{\text{MAS}}(\theta) = \mathcal{L}_t(\theta) + \frac{\lambda}{2} \sum_j \Omega_j (\theta_j - \theta^*_{t-1,j})^2$

Learning without Forgetting (LwF)

Knowledge distillation for CL. When learning task $t$, use the old model $\theta^*_{t-1}$ as a "teacher":

$\mathcal{L}_t^{\text{LwF}}(\theta) = \mathcal{L}_t(\theta) + \alpha \sum_{i=1}^{t-1} \text{KL}\left( p_{old}(y|\mathbf{x}; \theta^*_{t-1}) \| p_{new}(y|\mathbf{x}; \theta) \right)$

Why this step? KL divergence penalizes the new model if its predictions on old tasks' outputs diverge from the old model. We're saying "on data from task $t$, match your new predictions for old task heads to what the old model would have predicted."

No replay data needed, but requires multi-head output (separate heads for each task).

Task 2: 10-class CIFAR-100 (first 10 classes). Add a new output head $\mathbf{p}_2 \in \mathbb{R}^{10}$.

Training on task 2 data $\mathbf{x}$:

1. Forward pass with new $\theta$: Get $\mathbf{p}_1^{\text{new}}(\mathbf{x})$ and $\mathbf{p}_2^{\text{new}}(\mathbf{x})$.
2. Forward pass with old $\theta^*_1$: Get $\mathbf{p}_1^{\text{old}}(\mathbf{x})$ (frozen).
3. Loss: $\mathcal{L}_2(\mathbf{p}_2^{\text{new}}, y_2) + \alpha \cdot \text{KL}(\mathbf{p}_1^{\text{old}} \| \mathbf{p}_1^{\text{new}})$.

Why this matters? Even though $\mathbf{x}$ is from task 2, the KL term forces the task-1 head to maintain the same predictions the old model had. This implicitly preserves task-1 knowledge without seeing task-1 data.

Advanced: Gradient Projection Methods

$\langle \nabla_\theta \mathcal{L}_t(\theta), \nabla_\theta \mathcal{L}_i(\theta) \rangle \geq 0 \quad \forall i < t$

If violated, project $\nabla_\theta \mathcal{L}_t$ to the feasible region:

$\mathbf{g}_t^{\text{proj}} = \arg\min_{\mathbf{g}} \|\mathbf{g} - \nabla_\theta \mathcal{L}_t\|^2 \quad \text{s.t.} \quad \langle \mathbf{g}, \nabla_\theta \mathcal{L}_i \rangle \geq 0 \quad \forall i$

Why? This is a convex quadratic program. We're finding the gradient closest to $\nabla_\theta \mathcal{L}_t$ that doesn't hurt old tasks (non-negative dot product means loss doesn't increase in that direction).

Intuition: Think of each task's gradient as defining a "don't go this way" constraint. GEM navigates the intersection of all constraints.

Mistake 1: "Continual learning is the same as transfer learning."

Why it feels right: Both involve learning multiple tasks.

The fix: Transfer learning: train on source task, then fine-tune on target. Source data remains available. Continual learning: sequential tasks, old data discarded, must prevent forgetting. Transfer doesn't address catastrophic forgetting.

Mistake 2: "Just replay all old data—problem solved."

Why it feels right: Rehearsing old examples prevents forgetting.

The fix: In real continual learning, storing all data is often impossible (privacy, memory, streaming). Replay is a compromise with a small buffer. Also, true lifelong learning (millions of tasks) would require massive storage. Replay is a tool, not the complete solution.

Mistake 3: "EWC's Fisher matrix is the Hessian, so it's exact."

Why it feels right: Fisher is often called a Hessian approximation.

The fix: Fisher measures curvature under the model's current predictions, Hessian measures curvature of the true loss surface. They coincide only for correctly specified log-likelihood models. For deep nets, Fisher is a practical approximation (diagonal, easier to compute) but not exact. EWC's quadratic penalty is only locally valid around $\theta^*_{t-1}$.

Mistake 4: "In Class-IL, just concatenate all task outputs into one big softmax."

Why it feels right: Unified output seems natural.

The fix: Softmax normalization causes task recency bias: new task classes get higher probability because old task heads are frozen or under-trained. Need cross-entropy balancing, separate task outputs with gating, or specialized architectures (e.g., NCM classifiers).

Connections

• Catastrophic Forgetting: The core problem CL solves.
• Transfer Learning: CL extends transfer to multiple sequential tasks with forgetting prevention.
• Meta-Learning: Overlaps in learning to learn; MAML can adapt quickly with low forgetting.
• Regularization Techniques: EWC is L2 regularization weighted by importance.
• Knowledge Distillation: LwF uses distillation to preserve old task knowledge.
• Neural Architecture Search: Dynamic architecture methods (PackNet, DEN) relate to NAS for capacity allocation.
• Online Learning: CL is a form of online learning with non-stationary distributions.
• Neuroscience - Memory Consolidation: Biological inspiration; complementary learning systems (hippocampus for rapid learning, cortex for slow consolidation).

Why does this happen? The robot's brain (its "weights") is like a set of knobs. Turning the knobs one way makes it fetch. Turning them another way makes it roll over. When you train it to roll over, you twist the knobs, but that messes up the fetch settings.

Continual learning is about making the robot remember all its tricks. Three ways:

1. Protect important knobs (EWC): We mark which knobs are super important for fetching (like the "see the ball" knob). When teaching roll-over, we say "don't twist those knobs much."

2. Practice old tricks (Replay): Every day, we randomly make the robot fetch a few times while learning roll-over. It never fully forgets because it keeps rehearsing.

3. Separate controls (Architecture): Give the robot a "fetch brain" and a "roll-over brain." Each trick uses its own knobs, so they never interfere.

The goal? A robot that can learn10, 100, 1000 tricks and remember them all!

For EWC vs MAS: "Fish Eat Food" → Fisher uses Erors/Loss, MAS uses Function (output gradient).

#flashcards/ai-ml

What is catastrophic forgetting? :: When a neural network trained on Task 1, then Task 2, loses performance on Task 1 because parameters optimized for Task 1 are overwritten during Task 2 training. Standard SGD has no memory of old tasks.

EWC formula :: $\mathcal{L}_t^{\text{EWC}}(\theta) = \mathcal{L}_t(\theta) + \frac{\lambda}{2} \sum_j F_j (\theta_j - \theta^*_{t-1,j})^2$, where $F_j$ is the Fisher information (importance) of parameter $j$ for previous tasks. Penalizes changing important weights.

Concept Map

Hinglish (regional understanding)