Exercises — Open problems and future directions

• (a) Superposition — a network stores more concepts than it has neurons by letting features overlap in activation space (e.g. 1000 concepts packed into 100 neurons).
• (b) Catastrophic forgetting — training on Task B overwrites the weights that encoded Task A, so Task A performance collapses.
• (c) Distribution shift — the deployment data $P_{\text{test}}$ differs from the training data $P_{\text{train}}$, so learned patterns misfire.
• (d) Confounder — a hidden variable $Z$ that causes both $X$ and $Y$, creating a correlation between them that is not a cause.

False. Passive observation can only ever refine $P(Y\mid X)$; it never becomes $P(Y\mid do(X))$ when a confounder exists, because observation cannot cut the causal edge $Z\to X$ the way a real intervention does. You need either a randomized experiment or explicit causal assumptions.

By definition the penalty is the extra risk from the shift: $\text{penalty}={\mathcal{R}}_{\text{test}}-{\mathcal{R}}_{\text{train}}=0.32-0.05=\mathrm{0.27.}$ What this means: 27 percentage points of the model's failure come purely from the world changing, not from an intrinsically hard task.

(a) Observational — weight by $P(Z\mid \text{ice})$, which leans hot because ice cream co-occurs with heat: $P(\text{drown}\mid \text{ice})=0.10\cdot 0.9+0.02\cdot 0.1=0.090+0.002=\mathrm{0.092.}$ (b) Interventional — the $do$ operator cuts the edge $Z\to \text{ice}$, so we weight by the natural $P(Z)=0.5$ each: $P(\text{drown}\mid do(\text{ice}))=0.10\cdot 0.5+0.02\cdot 0.5=0.050+0.010=\mathrm{0.060.}$ Interpretation: observation says $9.2\%$, intervention says $6.0\%$. The extra $3.2$ points is pure confounding — forcing everyone to buy ice cream does not drown people; hot weather did the work. The figure below shows the "graph surgery": on the left the arrow from temperature $Z$ into ice cream $X$ is kept (observation); on the right it is cut with the red scissors ($do$). Notice that only the $Z\to X$ arrow disappears — the $Z\to Y$ arrow driving drownings stays, which is exactly why the interventional number is still nonzero but smaller.

Ratio $=\frac{10^9}{1}=10^9$. Taking ${\log }_{10}$ gives $9$. So the gap is indeed 9 orders of magnitude. The claim checks out. The lesson: adding data linearly cannot close a nine-decade gap; the fix must be structural (better inductive biases), not just bigger datasets.

When $\delta$ aligns with the eigenvector, ${\delta }^{\top }H\delta ={\lambda }_{\max }\Vert \delta {\Vert }^2$. So $\Delta \mathcal{R}\approx \frac{1}{2}{\lambda }_{\max }\Vert \delta {\Vert }^2.$ Sharp model: $\frac{1}{2}\cdot 50\cdot 0.1^2=\frac{1}{2}\cdot 50\cdot 0.01=0.25$ — a $0.25$ risk jump from a tiny $0.1$ shift. Flat model: $\frac{1}{2}\cdot 2\cdot 0.01=0.01$ — a $25\times$ smaller jump for the same shift. Why: the eigenvalue is the curvature of the risk surface. Big curvature = steep valley walls = a small sideways step costs a lot. In the figure below, look at the two parabolas: both start at the same point, but at the same horizontal shift $\Vert \delta \Vert =0.1$ (dotted line) the orange (sharp) curve has climbed to $0.25$ while the teal (flat) curve has barely risen to $0.01$. The only difference between them is the curvature ${\lambda }_{\max }$.

Degenerate solution: set $\varphi (x)=c$, a constant, for every input. Then $P_{\text{src}}(\varphi )$ and $P_{\text{tgt}}(\varphi )$ are both a spike at $c$, so $D_{\text{KL}}=0$ — perfect. But a constant feature carries no information about $x$, so the classifier cannot separate classes: source risk is terrible. Repair: optimize the joint objective with the trade-off knob $\beta$ from above, ${\min }_{\varphi ,f}{\mathcal{R}}_{\text{src}}(f\circ \varphi )+\beta D_{\text{KL}}(P_{\text{src}}(\varphi )\Vert P_{\text{tgt}}(\varphi )).$ The first term punishes throwing away label-relevant information (kills the constant collapse); the second term, weighted by $\beta$, punishes source/target mismatch. The tension between them selects features that are both discriminative and shift-invariant. Choosing $\beta$ too large re-invites the collapse; too small ignores the shift — hence the sweep.

Differentiate and set to zero — the standard tool for a smooth minimum, because at a minimum the slope is flat: $\frac{dJ}{d\theta }=6(\theta -6)+F(\theta -2)=6(\theta -6)+4(\theta -2)=0.$ Expand: $6\theta -36+4\theta -8=0\Rightarrow 10\theta =44\Rightarrow {\theta }^{\ast }=4.4$. Interpretation: ${\theta }^{\ast }=4.4$ sits between A's optimum $2$ and B's optimum $6$, pulled toward B (weight $6$) but held back by A's anchor (weight $4$). It is exactly the precision-weighted average $\frac{6\cdot 6+4\cdot 2}{6+4}=\frac{44}{10}=4.4$. Larger $F$ (A more important) would drag ${\theta }^{\ast }$ back toward $2$; $F=0$ would give pure B at $6$ (total forgetting). This is the mathematical heart of continual learning. In the figure below, the orange combined curve $J$ has its lowest point at ${\theta }^{\ast }=4.4$ (marked dot), sitting between the dotted lines at A's optimum $2$ and B's optimum $6$ — the anchor pulls the pure-B minimum leftward toward A.

Same derivative with symbolic $F$: $6(\theta -6)+F(\theta -2)=0\Rightarrow (6+F)\theta =36+2F\Rightarrow {\theta }^{\ast }(F)=\frac{36+2F}{6+F}.$

• $F\to 0$: ${\theta }^{\ast }\to \frac{36}{6}=6$ — pure Task B, total catastrophic forgetting of A.
• $F\to \infty$: ${\theta }^{\ast }\to \frac{2F}{F}=2$ — pinned at Task A, frozen, unable to learn B.
• $F=4$: ${\theta }^{\ast }=\frac{44}{10}=4.4$, matching 5.1. Why this matters: real continual learning tunes $F$ per-weight to sit in the useful middle — retain what mattered, still adapt.

Each ==...== marks a term the vault hides until you tap/reveal it. Cover the line, answer from memory, then reveal.

The observational quantity ML learns is ==$P(Y\mid X)$ but interventions require $P(Y\mid do(X))$==. The $do$-operator changes the weighting from $P(Z\mid X)$ to ==$P(Z)$ because it cuts the incoming edge to $X$==. A large Hessian eigenvalue ${\lambda }_{\max }$ in distribution space means high curvature, so small shifts cause large risk increases. In EWC-style continual learning, the anchor strength $F$ trades forgetting against new-task learning; $F\to 0$ gives total forgetting, $F\to \infty$ gives a frozen model.