Worked examples — Open problems and future directions

This page takes the five open problems from the parent note and turns each one into a computable scenario you can solve by hand. The parent told you these problems exist. Here we grind through every case class where they bite — so that when you meet one in the wild, you have already seen its shape.

We will lean on ideas built elsewhere in the vault: Statistical Learning Theory for the "how much data" questions, Causal Inference for the "correlation vs. cause" questions, and Transfer Learning for the "train here, deploy there" questions. Every symbol used below is defined the moment it appears.

---

The scenario matrix

Before working examples, we lay out every case class these five open problems can throw at you. Think of it as a checklist: by the end, every cell must have a worked example next to it.

Cell	Case class	The degenerate/limiting twist to watch
A	Sample efficiency — counting how sparse $k$ points are in high dimensions	limit as dimension $d\to \infty$
B	Compositional generalization — combining known parts	zero examples of the combination
C	Distribution shift — error jump when $P_{\text{test}}\ne P_{\text{train}}$	shift $=0$ (no shift) as the sanity anchor
D	Confounded correlation — $P(Y\mid X)$ vs $P(Y\mid do(X))$	confounder strength $\to 0$ makes them agree
E	Catastrophic forgetting — weights overwritten by a new task	learning rate $\to 0$ (no forgetting) as the limit
F	Real-world word problem — the ice-cream/drowning trap	sign of the intervention effect
G	Exam-style twist — combine two cells (shift and few-shot)	which term dominates the error

Every numeric answer below is re-checked in the machine-verification block. Let us define one recurring symbol now.

---

Cell A — How empty is high-dimensional space?

Forecast: Guess now — for $d=6$, do the 1000 points fill most cells, or a tiny sliver?

1. Count the cells. Chopping each of $d$ axes into $10$ bins gives $10^d$ cells. Why this step? This is exactly what "densely cover the input space" from the parent means — one representative point per cell.
2. Plug in: $d=1\Rightarrow 10$ cells, $d=2\Rightarrow 100$, $d=6\Rightarrow 10^6=1,000,000$. Why this step? Each extra dimension multiplies the volume by $10$; the growth is exponential, not additive. This is why we needed the exponent, not a sum.
3. Fraction covered $=\frac{\min (k,10^d)}{10^d}$. For $d=6$: $\frac{1000}{1,000,000}=0.001=0.1\%$. Why this step? At most one point lands per cell, so $1000$ points touch at most $1000$ cells.

Look at the figure: the same $1000$ dots that flood the 1-D line barely stipple the 2-D grid — and would vanish entirely in 6-D.

Verify: $10^6$ cells, $1000/10^6=0.001$. Units: both counts are dimensionless, ratio is a pure fraction — consistent. ✓

---

Cell B — Composition with zero examples of the combination

Forecast: Guess the number of unseen-but-understood pairs.

1. Total pairs $=(\text{colors})\times (\text{shapes})=2\times 3=6$. Why this step? Compositionality means treating color and shape as independent factors; the count of combinations is their product.
2. Seen $=4$ (given). Generalized $=6-4=2$. Why this step? A compositional model handles the leftover pairs "red hat", "blue cup" (say) that it never saw — the zero-example cell of our matrix.
3. Contrast a memorizer: it generalizes to $6-4=0$ unseen pairs, because it stored whole images, not factors. Why this step? This is precisely the gap the parent calls "memorization vs understanding."

Verify: $2\times 3=6$; $6-4=2$ generalized (compositional), $0$ (memorizer). ✓

---

Cell C & the no-shift anchor — the distribution-shift penalty

Forecast: Which deployment has the biggest penalty — and does the same-distribution one give ~0?

1. Anchor (Cell C degenerate case, shift $=0$). For same-distribution test, penalty $=0.03-0.01=0.02\approx 0$. Why this step? If $P_{\text{test}}=P_{\text{train}}$ the penalty should vanish; $0.02$ is the small residual from finite sampling. This sanity-checks our formula.
2. Corrupted. Penalty $=0.40-0.01=0.39$.
3. Sketches. Penalty $=0.70-0.01=0.69$. Why these steps? Bigger visual distance from training photos → bigger penalty, matching the parent's "sharp Hessian in distribution space."

The bars in the figure show the penalty stacking on top of the constant training error.

Verify: penalties $0.02,0.39,0.69$; ordering sketches > corrupted > same-dist. Units: all are error fractions in $[0,1]$. ✓ See Transfer Learning for how domain adaptation attacks this.

---

Cell D & F — Ice cream, drowning, and the $do$-operator

Forecast: Guess: will $P(Y\mid X=1)$ come out near $0.8$ or near $0.45$?

We use the two formulas from the parent, with the causal fact $P(Y\mid X,Z)=P(Y\mid Z)$ (ice cream is irrelevant once we know the weather).

1. Observational $P(Y=1\mid X=1)$. First get $P(X=1)=0.9(0.5)+0.1(0.5)=0.5$. Then Bayes with weighting by how often each weather produces high ice cream: $P(Z=1\mid X=1)=\frac{0.9(0.5)}{0.5}=0.9,P(Z=0\mid X=1)=\mathrm{0.1.}$ $P(Y=1\mid X=1)=0.8(0.9)+0.1(0.1)=\mathrm{0.73.}$ Why this step? Seeing high ice cream is evidence it is hot, and hot days drown people. The correlation flows through $Z$.
2. Interventional $P(Y=1\mid do(X=1))$. The $do$ cuts the arrow $Z\to X$, so $Z$ keeps its original frequencies $P(Z=1)=0.5$: $P(Y=1\mid do(X=1))={\sum }_{Z}P(Y=1\mid Z)P(Z)=0.8(0.5)+0.1(0.5)=\mathrm{0.45.}$ Why this step? Forcing everyone to buy ice cream tells you nothing about the weather, so we use the marginal $P(Z)$, not the ice-cream-informed one.
3. True causal effect $=P(Y=1\mid do(X=1))-P(Y=1\mid do(X=0))$. By identical algebra $P(Y=1\mid do(X=0))=0.45$ too, so the effect is $0.45-0.45=0$. Why this step? The sign is exactly zero — banning ice cream changes nothing, matching the parent's punchline.

The figure shows the graph before and after the $do$-scissors cut the $Z\to X$ edge.

Verify: $P(Y\mid X=1)=0.73$, $P(Y\mid do)=0.45$, causal effect $=0$, confounding gap $=0.28$. ✓

---

Cell E & its no-forget limit — catastrophic forgetting

Forecast: Guess how much loss Task A picks up after one selfish Task-B step.

1. Take the step. ${\theta }_{\text{new}}=4-0.4(5)=4-2=2$. Why this step? This is one plain Optimization Landscape update — the very move that overwrites old knowledge.
2. Measure forgetting. ${\mathcal{L}}_A(2)=\frac{1}{2}(2-4{)}^2=\frac{1}{2}(4)=2$. Why this step? Task A was at loss $0$; it is now at loss $2$ — that jump is catastrophic forgetting, quantified.
3. No-forget limit. As $\eta \to 0$: ${\theta }_{\text{new}}\to 4$ and ${\mathcal{L}}_A\to \frac{1}{2}(4-4{)}^2=0$. Why this step? Tiny steps don't move you off Task A's optimum — but they also barely learn Task B. This is the stability–plasticity trade-off in one line.

The figure plots Task A's loss bowl and marks the drift from $\theta =4$ to $\theta =2$.

Verify: ${\theta }_{\text{new}}=2$, forgetting ${\mathcal{L}}_A=2$, limit $\eta \to 0$ gives $0$. ✓

---

Cell G — Exam twist: few-shot and shift together

Forecast: Guess whether few-shot or shift is the bigger villain here.

1. Few-shot term. $\frac{c}{k}=\frac{0.8}{8}=0.10$. Why this step? The $1/k$ shape encodes Cell A: fewer samples, bigger error. With only $8$ scans it is non-trivial.
2. Sum the pieces. ${\mathcal{R}}_{\text{deploy}}=0.05+0.10+0.30=0.45$. Why this step? The parent's decomposition is additive; we simply stack base error, sparsity, and shift.
3. Dominant term. Compare $0.10$ (few-shot) vs $0.30$ (shift): shift dominates, contributing $0.30/0.45\approx 0.667$ of the total. Why this step? An exam wants the diagnosis: fixing the scanner mismatch (see Transfer Learning) buys more than collecting a few more scans.

Verify: $0.8/8=0.10$; total $=0.45$; shift share $\approx 0.667$; shift $>$ few-shot. ✓

---

Recall Self-test (reveal after answering)

Ice-cream observational vs interventional probability, and why they differ ::: $P(Y=1\mid X=1)=0.73$ but $P(Y=1\mid do(X=1))=0.45$; the $do$ cuts the $Z\to X$ edge so we use the marginal $P(Z)$ instead of the ice-cream-informed one. Fraction of a 6-D cube touched by 1000 points on a 10-bin grid ::: $10^{3-6}=0.001=0.1\%$. The single change that turns forgetting to zero in Cell E ::: send the learning rate $\eta \to 0$ (at the cost of learning nothing new).

---

Where these live in the wider map

Follow the branches: Statistical Learning Theory, Transfer Learning, Causal Inference, and Optimization Landscape each pick up exactly one cell of the matrix and go deeper.