Continual and lifelong learning

6.5.7 · AI-ML › Research Frontiers & Practice

Ye kyun zaroori hai: Real-world AI ko continuously adapt karna padta hai (recommendation mein naye products, medical imaging mein nayi bimaariyaan, translation mein nayi bhashayein) bina har baar scratch se retrain kiye. Ek self-driving car jo naaye road signs seekhne ke baad pedestrians detect karna bhool jaaye—woh catastrophic hai.

The Stability-Plasticity Dilemma

Challenge yeh hai: standard SGD update $\theta \leftarrow \theta -\eta {\nabla }_{\theta }{\mathcal{L}}_t(\theta )$ sirf current task ka loss ${\mathcal{L}}_t$ optimize karta hai, ${\mathcal{T}}_1,\dots ,{\mathcal{T}}_{t-1}$ ke liye useful weights ko overwrite karta hai.

Lifelong Learning isko interchangeably use kiya jaata hai, kabhi kabhi transfer par emphasis de ke: nayi tasks purani knowledge se benefit leti hain (forward transfer), purani tasks nayi knowledge se improve hoti hain (backward transfer).

Catastrophic Forgetting Kyun Hota Hai (First Principles)

Chaliye derive karte hain neural nets kyun bhoolte hain. Maano hum task 1 ko convergence tak train karte hain:

${\theta }^{\ast }=\arg {\min }_{\theta }{\mathcal{L}}_1(\theta )=\arg {\min }_{\theta }{\mathbb{E}}_{(\mathbf{x},y)\sim {\mathcal{D}}_1}[\ell (f(\mathbf{x};\theta ),y)]$

${\theta }^{\ast }$ par, gradient ${\nabla }_{\theta }{\mathcal{L}}_1({\theta }^{\ast })\approx 0$ (local minimum). Ab task 2 aata hai. Hum update karte hain:

$\theta \leftarrow {\theta }^{\ast }-\eta {\nabla }_{\theta }{\mathcal{L}}_2({\theta }^{\ast })$

Yeh step kyun? Hum $\theta$ ko us direction mein move kar rahe hain jo ${\mathcal{L}}_2$ ko reduce kare. Lekin yeh gradient ${\nabla }_{\theta }{\mathcal{L}}_2$ sirf ${\mathcal{D}}_2$ par compute hota hai. Agar ${\nabla }_{\theta }{\mathcal{L}}_2$ us direction mein point kare jo ${\mathcal{L}}_1$ ko increase kare, toh hum actively task 1 ko kharaab kar rahe hain.

Catastrophic forgetting isliye hota hai kyunki:

1. Parameter overlap: Wahi weights $\theta$ sabhi tasks ke liye kaam karni padti hain. Task 2 ke liye $\theta$ ko move karna task 1 ke liye optimize configuration ko disturb karta hai.
2. No explicit memory: Standard SGD mein ${\mathcal{D}}_1$ chale jaane ke baad ${\mathcal{L}}_1$ yaad rakhne ka koi mechanism nahi hai.

${\text{BWT}}_i={\text{Acc}}_{T,i}-{\text{Acc}}_{i,i}$

jahan ${\text{Acc}}_{j,i}$ = task $j$ tak training ke baad task $i$ par accuracy. Negative BWT ka matlab forgetting hai.

Average forgetting $T-1$ tasks mein:

$\text{Forgetting}=\frac{1}{T-1}{\sum }_{i=1}^{T-1}{\max }_{j\in \{i,\dots ,T-1\}}({\text{Acc}}_{j,i}-{\text{Acc}}_{T,i})$

Max kyun? Hum final performance ko task $i$ par baad ki tasks seekhne se pehle ki best performance se compare karte hain (forgetting peak se relative hai, sirf initial se nahi).

Teen Families of CL Solutions

1. Regularization-Based (Important Weights ko Protect Karo)

Idea: Identify karo ki kaun se weights ${\theta }_j$ purani tasks ke liye critical hain, phir nayi tasks seekhte waqt unhe badalne par penalty lagao.

$F_j={\mathbb{E}}_{{\mathcal{D}}_{t-1}}\left[{\left(\frac{\partial \log p(y|\mathbf{x};\theta )}{\partial {\theta }_j}\right)}^2\right]$

Fisher kyun? Yeh measure karta hai ki agar hum ${\theta }_j$ ko perturb karein toh log-likelihood kitna change hoti hai. High $F_j$ → parameter $j$ purani task ki prediction ke liye "important" hai.

Task $t$ ke liye, hum minimize karte hain:

${\mathcal{L}}_t^{\text{EWC}}(\theta )={\mathcal{L}}_t(\theta )+\frac{\lambda }{2}{\sum }_j{F}_j({\theta }_j-{\theta }_{t-1,j}^{\ast }{)}^2$

Yeh form kyun? Quadratic penalty ${\theta }_{t-1}^{\ast }$ ke aas paas ${\mathcal{L}}_{t-1}$ ka second-order Taylor approximation hai:

${\mathcal{L}}_{t-1}(\theta )\approx {\mathcal{L}}_{t-1}({\theta }_{t-1}^{\ast })+\frac{1}{2}(\theta -{\theta }_{t-1}^{\ast }{)}^{\top }H(\theta -{\theta }_{t-1}^{\ast })$

jahan $H$ Hessian hai. Fisher $F$, classification ke liye $H$ ke diagonal ko approximate karta hai (mild assumptions ke under). Toh hum keh rahe hain: "$\theta$ ko move karo ${\mathcal{L}}_t$ minimize karne ke liye, lekin ${\mathcal{L}}_{t-1}$ ke liye important dimensions mein ${\theta }_{t-1}^{\ast }$ ke paas raho."

Hyperparameter $\lambda$: Stability-plasticity trade-off control karta hai. Bada $\lambda$ → strong penalty → zyada stability, kam plasticity.

${\mathcal{D}}_1$ par Fisher compute karo: Logistic regression ke liye, $F_j={\sum }_i{p}_i(1-p_i)x_{ij}^2$ jahan $p_i=\sigma ({\theta }^{\top }{\mathbf{x}}_i)$.

Maano ${\mathcal{D}}_1$ mein 100 examples hain, average $p(1-p)=0.25$, ${\bar{x}}_1^2=4$, ${\bar{x}}_2^2=0.01$:

• $F_1\approx 100\times 0.25\times 4=100$ (important!)
• $F_2\approx 100\times 0.25\times 0.01=0.25$ (kam important)

Yeh kyun matter karta hai? Task 2 seekhte waqt, $w_1$ ko 0.1 se deviate karne par cost aati hai $\frac{\lambda }{2}\times 100\times (0.1{)}^2=0.5\lambda$. $w_2$ ko 0.1 se deviate karne par sirf $0.0125\lambda$ cost aati hai. EWC automatically wahan rigidity focus karta hai jahan zaroori hai.

Task 2 loss with $\lambda =1000$:

${\mathcal{L}}_2^{\text{EWC}}={\mathcal{L}}_2(\theta )+500(w_1-2.0{)}^2+0.125(w_2-0.1{)}^2+500(b+0.5{)}^2$

2. Replay-Based (Purana Data Store Karo aur Rehearse Karo)

Idea: Purane examples ka ek chota memory buffer rakho aur unhe nayi task training ke saath interleave karo. Yeh directly forgetting rokta hai model ko purani tasks continuously "remind" karke.

${\mathcal{L}}_t^{\text{replay}}(\theta )={\mathbb{E}}_{(\mathbf{x},y)\sim {\mathcal{D}}_t}[\ell (f(\mathbf{x};\theta ),y)]+\beta {\mathbb{E}}_{(\mathbf{x},y)\sim \mathcal{M}}[\ell (f(\mathbf{x};\theta ),y)]$

Yeh kyun kaam karta hai? Gradients ab replayed data se ${\nabla }_{\theta }{\mathcal{L}}_1,{\nabla }_{\theta }{\mathcal{L}}_2,\dots$ include karte hain. $\theta$ ko ${\mathcal{L}}_t$ reduce karne ke liye move karna jabki purani losses bhi satisfy karna ek compromise force karta hai—stability.

Key challenge: Memory size. Agar $|\mathcal{M}|$ fixed hai (maano 5000 examples), toh hum decide kaise karein ki kaun se examples store karein?

Reservoir Sampling: $N$ total examples ki stream ke liye, $k$ uniformly select karo. Jab example $n$ aata hai:

• Agar $n\le k$: store karo.
• Agar $n>k$: probability $k/n$ se ek random existing example replace karo.

Kyun? Yeh ensure karta hai ki har example ka $\mathcal{M}$ mein hone ka probability $k/N$ barabar hai, chahe tasks kab bhi aayen.

Generative Replay: Real data store karne ki jagah (privacy issues, memory), purani tasks par ek generative model $p_{\theta }(\mathbf{x},y)$ train karo aur isse "pseudo-examples" sample karo.

${\mathcal{M}}_{\text{gen}}=\{(\tilde{\mathbf{x}},\tilde{y})\sim p_{{\theta }_{-1}}(\mathbf{x},y)\}$

Equal allocation: Har task se 200 rakho → 600 total. Task 3 ke liye mini-batch (size 32) sample karte waqt, ~10 task-1 examples, ~10 task-2, ~12 task-3 include karo (buffer size ke proportion mein ya uniform).

Yeh step kyun? Yeh ensure karta hai ki sabhi tasks ko "rehearsal time" milti hai unki representation ke proportion mein. Iske bina, agar hum sirf task 3 sample karte, toh 1 aur 2 bhool jaate.

Alternative: Class-balanced buffer—agar tasks ki alag class distributions hain, toh har class se barabar examples store karo.

3. Architecture-Based (Tasks ko Parameters Dedicate Karo)

Idea: Alag tasks ke liye alag sub-networks ya parameters allocate karo, taaki task 2 seekhna task 1 ke parameters ko bilkul touch na kare.

$h_i^{(t)}=\sigma \left(W_i^{(t)}{h}_{i-1}^{(t)}+{\sum }_{j=1}^{t-1}{U}_i^{(t\leftarrow j)}{h}_{i-1}^{(j)}\right)$

Kyun? $h^{(j)}$ purani tasks ke frozen features hain. $U^{(t\leftarrow j)}$ unhe reuse karna seekhta hai. Purane parameters ${\theta }^{(1)},\dots ,{\theta }^{(t-1)}$ kabhi update nahi hote → construction se zero forgetting.

Downside: Model tasks ke saath linearly grow karta hai. 10 tasks → 10× parameters.

PackNet: Har task ke baad network prune karo, us task ke liye winning lottery ticket "freeze" karo. Nayi tasks remaining capacity (pruned weights) use karti hain.

Dynamic Expandable Network: Chota shuru karo, nayi neurons/layers sirf tab add karo jab nayi task ko zyada capacity chahiye. Purani tasks ke liye important neurons freeze karo.

Task 2: Cats classify karo (10 classes). Ek parallel 3-layer CNN column add karo (nayi random init). Dog-column layer 1 se cat-column layer 1 tak lateral connections, wagera.

Yeh kyun help karta hai? Cat column, dogs ke liye seekhe gaye edge detectors aur texture features reuse kar sakta hai ($U$ connections ke through), lekin $W$ mein cat-specific features seekhne ki azaadi hai bina dog features destroy kiye.

Task 2 ke baad: Dog accuracy abhi bhi 90% (frozen), cat accuracy 88%. Total params: 2× original.

Task 3: Birds (10 classes). Bird column add karo, dog + cat se connect karo. Ab 3× params, lekin dog abhi bhi 90%, cat 88%, bird 85%.

Common Scenarios & Metrics

• Class-Incremental Learning (Class-IL): Test time par, task ID unknown hoti hai. Model ko sabhi dekhi gayi classes mein se sahi classify karna padta hai.
  • Example: "Is image ko classify karo (dog, cat, ya bird ho sakta hai, lekin main nahi bataunga kaun sa task)."
  • Mushkil, task-agnostic features aur output heads seekhne padte hain.

$\text{ACC}=\frac{1}{T}{\sum }_{i=1}^T{\text{Acc}}_{T,i}$

Forward Transfer (kya task $t$ future task $t+1$ ko help karta hai?):

${\text{FWT}}_i={\text{Acc}}_{i-1,i}-{\text{Acc}}_{\text{rand},i}$

jahan ${\text{Acc}}_{\text{rand},i}$ task $i$ par random-init accuracy hai. Positive FWT ka matlab hai $1,\dots ,i-1$ ki knowledge ne help ki.

Ideal CL system: High ACC (sabhi tasks mein accha), low Forgetting (purani tasks yaad hain), positive FWT (knowledge transfer).

Key Algorithms Detail

Memory-Aware Synapses (MAS)

EWC jaisa, lekin importance Fisher ke bina compute karta hai. Har parameter ${\theta }_j$ ke liye:

${\Omega }_j=\frac{1}{|{\mathcal{D}}_{t-1}|}{\sum }_{\mathbf{x}\in {\mathcal{D}}_{t-1}}{\Vert \frac{\partial f(\mathbf{x};\theta )}{\partial {\theta }_j}\Vert }_2^2$

Output gradient kyun? Agar ${\theta }_j$ badalne se output $f(\mathbf{x})$ drastically change hota hai, toh ${\theta }_j$ learned function ke liye important hai, loss ki parwah kiye bina.

Task $t$ ke liye loss:

${\mathcal{L}}_t^{\text{MAS}}(\theta )={\mathcal{L}}_t(\theta )+\frac{\lambda }{2}{\sum }_j{\Omega }_j({\theta }_j-{\theta }_{t-1,j}^{\ast }{)}^2$

Learning without Forgetting (LwF)

CL ke liye knowledge distillation. Task $t$ seekhte waqt, purane model ${\theta }_{t-1}^{\ast }$ ko "teacher" ki tarah use karo:

${\mathcal{L}}_t^{\text{LwF}}(\theta )={\mathcal{L}}_t(\theta )+\alpha {\sum }_{i=1}^{t-1}\text{KL}\left(p_{old}(y|\mathbf{x};{\theta }_{t-1}^{\ast })\Vert p_{new}(y|\mathbf{x};\theta )\right)$

Yeh step kyun? KL divergence naaye model ko penalize karta hai agar purani tasks ke outputs ke predictions purane model se diverge karein. Hum keh rahe hain: "task $t$ ke data par, apni nayi predictions purane task heads ke liye waise hi rakho jaise purana model predict karta."

Koi replay data nahi chahiye, lekin multi-head output zaroori hai (har task ke liye alag heads).

Task 2: 10-class CIFAR-100 (pehli 10 classes). Ek naya output head ${\mathbf{p}}_2\in {\mathbb{R}}^{10}$ add karo.

Task 2 data $\mathbf{x}$ par training:

1. Naaye $\theta$ ke saath forward pass: ${\mathbf{p}}_1^{\text{new}}(\mathbf{x})$ aur ${\mathbf{p}}_2^{\text{new}}(\mathbf{x})$ lo.
2. Purane ${\theta }_1^{\ast }$ ke saath forward pass: ${\mathbf{p}}_1^{\text{old}}(\mathbf{x})$ lo (frozen).
3. Loss: ${\mathcal{L}}_2({\mathbf{p}}_2^{\text{new}},y_2)+\alpha \cdot \text{KL}({\mathbf{p}}_1^{\text{old}}\Vert {\mathbf{p}}_1^{\text{new}})$.

Yeh kyun matter karta hai? Bhale hi $\mathbf{x}$ task 2 se ho, KL term task-1 head ko force karta hai ki wahi predictions rakhe jo purana model rakhta tha. Yeh implicitly task-1 knowledge preserve karta hai bina task-1 data dekhe.

Advanced: Gradient Projection Methods

$⟨{\nabla }_{\theta }{\mathcal{L}}_t(\theta ),{\nabla }_{\theta }{\mathcal{L}}_i(\theta )⟩\ge 0\forall i<t$

Agar violate ho, toh ${\nabla }_{\theta }{\mathcal{L}}_t$ ko feasible region mein project karo:

${\mathbf{g}}_t^{\text{proj}}=\arg {\min }_{\mathbf{g}}\Vert \mathbf{g}-{\nabla }_{\theta }{\mathcal{L}}_t{\Vert }^2\text{s.t.}⟨\mathbf{g},{\nabla }_{\theta }{\mathcal{L}}_i⟩\ge 0\forall i$

Kyun? Yeh ek convex quadratic program hai. Hum ${\nabla }_{\theta }{\mathcal{L}}_t$ ke sabse paas ka gradient dhundh rahe hain jo purani tasks ko hurt na kare (non-negative dot product ka matlab hai loss us direction mein increase nahi hoti).

Intuition: Har task ke gradient ko "is direction mein mat jao" constraint ki tarah socho. GEM sabhi constraints ke intersection mein navigate karta hai.

Mistake 1: "Continual learning aur transfer learning same hai."

Kyun sahi lagta hai: Dono mein multiple tasks seekhna hota hai.

The fix: Transfer learning: source task par train karo, phir target par fine-tune karo. Source data available rehta hai. Continual learning: sequential tasks, purana data discard hota hai, forgetting rokna zaroori hai. Transfer catastrophic forgetting address nahi karta.

Mistake 2: "Bas sabhi purana data replay karo—problem solve."

Kyun sahi lagta hai: Purane examples rehearse karna forgetting rokta hai.

The fix: Real continual learning mein, sabhi data store karna often impossible hai (privacy, memory, streaming). Replay ek chote buffer ke saath compromise hai. Aur bhi, true lifelong learning (millions of tasks) mein bahut bada storage chahiye. Replay ek tool hai, complete solution nahi.

Mistake 3: "EWC ka Fisher matrix Hessian hai, toh yeh exact hai."

Kyun sahi lagta hai: Fisher ko often Hessian approximation kaha jaata hai.

The fix: Fisher, model ki current predictions ke under curvature measure karta hai, Hessian true loss surface ki curvature measure karta hai. Dono sirf correctly specified log-likelihood models ke liye coincide karte hain. Deep nets ke liye, Fisher ek practical approximation hai (diagonal, compute karna aasaan) lekin exact nahi. EWC ki quadratic penalty sirf ${\theta }_{t-1}^{\ast }$ ke aas paas locally valid hai.

Mistake 4: "Class-IL mein, bas sabhi task outputs ko ek bade softmax mein concatenate karo."

Kyun sahi lagta hai: Unified output natural lagta hai.

The fix: Softmax normalization task recency bias create karta hai: nayi task classes ko zyada probability milti hai kyunki purane task heads frozen ya under-trained hote hain. Cross-entropy balancing, gating ke saath separate task outputs, ya specialized architectures chahiye (jaise NCM classifiers).

Connections

• Catastrophic Forgetting: Woh core problem jo CL solve karta hai.
• Transfer Learning: CL, transfer ko multiple sequential tasks tak extend karta hai forgetting prevention ke saath.
• Meta-Learning: Learning to learn mein overlap; MAML kam forgetting ke saath quickly adapt kar sakta hai.
• Regularization Techniques: EWC importance se weighted L2 regularization hai.
• Knowledge Distillation: LwF, purani task knowledge preserve karne ke liye distillation use karta hai.
• Neural Architecture Search: Dynamic architecture methods (PackNet, DEN) capacity allocation ke liye NAS se relate karte hain.
• Online Learning: CL, non-stationary distributions ke saath online learning ka ek form hai.
• Neuroscience - Memory Consolidation: Biological inspiration; complementary learning systems (rapid learning ke liye hippocampus, slow consolidation ke liye cortex).

Yeh kyun hota hai? Robot ka brain (uski "weights") knobs ke ek set jaisi hai. Knobs ek taraf ghumana use fetch karata hai. Doosri taraf ghumana use roll over karata hai. Jab tum use roll over train karte ho, tum knobs ghuma dete ho, lekin isse fetch settings kharaab ho jaati hain.

Continual learning is baare mein hai ki robot ko saari tricks yaad rahen. Teen tarike:

1. Important knobs protect karo (EWC): Hum mark karte hain ki kaun se knobs fetch ke liye super important hain (jaise "ball dekho" knob). Roll-over sikhate waqt hum kehte hain "woh knobs zyada mat ghuma."

2. Purani tricks practice karo (Replay): Har roz, hum randomly robot se roll-over seekhte waqt kuch baar fetch karate hain. Woh poora nahi bhoolti kyunki rehearsal chalti rehti hai.

3. Alag controls (Architecture): Robot ko ek "fetch brain" aur ek "roll-over brain" do. Har trick apne khud ke knobs use karti hai, toh woh kabhi interfere nahi karte.

Goal? Ek robot jo 10, 100, 1000 tricks seekh sake aur sabhi yaad rakhe!

EWC vs MAS ke liye: "Fish Eat Food" → Fisher Errors/Loss use karta hai, MAS Function (output gradient) use karta hai.

#flashcards/ai-ml

Catastrophic forgetting kya hai? :: Jab ek neural network Task 1 par, phir Task 2 par train hota hai, toh Task 1 par performance lose kar deta hai kyunki Task 1 ke liye optimize parameters Task 2 training ke dauran overwrite ho jaate hain. Standard SGD ko purani tasks ki koi memory nahi hoti.

EWC formula :: ${\mathcal{L}}_t^{\text{EWC}}(\theta )={\mathcal{L}}_t(\theta )+\frac{\lambda }{2}{\sum }_j{F}_j({\theta }_j-{\theta }_{t-1,j}^{\ast }{)}^2$, jahan $F_j$ parameter $j$ ki purani tasks ke liye Fisher information (importance) hai. Important weights badalne par penalty lagaata hai.

Concept Map