Open problems and future directions: Question bank

This is a D5 question bank for the open-problems topic. Every item is a concept trap: a place where the obvious answer is wrong, or where a boundary case breaks a rule you thought was safe. Read the prompt, say your answer out loud, then reveal.

Before you start, one shared vocabulary reminder so no symbol is unearned. Each item below is anchored to a picture so the notation never arrives naked.

Definition

$E_{x \sim P} [g (x)]$ ::: the expectation ("average value") of a quantity $g (x)$ when inputs $x$ are drawn from distribution $P$ — imagine sampling millions of $x$ , computing $g$ for each, and averaging. The subscript says which distribution you sample from.
$L$ ::: the loss — a single number saying "how wrong was this prediction", small when right, large when wrong. So $E_{P} [L]$ is the average wrongness over a whole distribution.
$P (Y ∣ X)$ ::: the probability of outcome $Y$ given that we passively observed $X$ — the world was left alone.
$P (Y ∣ d o (X = x))$ ::: the probability of $Y$ after we reach in and force $X$ to value $x$ — an intervention that cuts the arrows that normally decide $X$ .
$Z$ (a confounder) ::: a hidden variable that is a common cause of both $X$ and $Y$ — it points an arrow into $X$ and an arrow into $Y$ , so it fakes a link between them (see the DAG below).
$P_{train}$ vs $P_{test}$ ::: the distribution of examples the model learned from vs the distribution it is used on. When these differ we call it distribution shift.
$D_{KL} (P ∥ Q)$ ::: the Kullback–Leibler divergence, a number measuring how far distribution $P$ sits from distribution $Q$ . It is $\geq 0$ , equals $0$ only when $P = Q$ , and is not symmetric ( $D_{KL} (P ∥ Q) \neq = D_{KL} (Q ∥ P)$ ).
Inductive bias ::: the structural assumptions baked into a model before it sees data (e.g. "nearby pixels relate"). It's what lets a model generalise from few examples.

The single figure below is your visual scaffold for the whole page — it draws the four recurring pictures these traps rely on: the confounding DAG, the high-dimensional volume shrinkage, LIME's local-vs-global fit, and curvature (the Hessian) making a shift catastrophic.

Figure — Open problems and future directions

True or false — justify

TF1. "With enough observational data, a model can eventually answer any intervention question like what if we banned X?"

False. Infinite passive data still only reveals

P (Y ∣ X)

; it never reveals

P (Y ∣ d o (X))

because you never observed the arrows cut (panel 1). You need a randomised experiment or explicit causal assumptions.

TF2. "Data augmentation can turn 3 examples into genuine few-shot understanding of a category."

False. Augmentation adds variety (rotations, crops, noise) but no new semantic content; you cannot rotate your way from three chairs into the concept "chair". You need better inductive biases, not more pixels.

TF3. "A model with 99% training accuracy and 97% test accuracy is robust."

False. Those numbers only measure the same distribution. Robustness is about

P_{test} \neq = P_{train}

— the same model can drop to 30% on sketches while looking excellent in-distribution.

TF4. "Neural nets are inherently uninterpretable — it's a law of physics."

False. "Uninterpretable" describes today's tooling, not a fundamental limit. We reverse-engineered the Linux kernel and the genome; interpretability is a young science, not an impossibility.

TF5. "Ice-cream sales predict drowning, so a high

P (drowning ∣ ice cream)

means selling ice cream causes drowning."

False. A confounder

Z

(summer temperature) raises both (panel 1). The correlation

P (Y ∣ X)

is real; the causal claim

P (Y ∣ d o (X))

is not.

TF6. "Catastrophic forgetting happens because the network runs out of memory capacity."

False. It happens because gradient descent on Task B overwrites the exact weights that encoded Task A — there's no partition protecting old knowledge, not a lack of room.

TF7. "If a single neuron fires strongly for cats, that neuron is the cat detector."

False. Neurons are polysemantic — one unit can respond to cats, car wheels, and text about fur. Concepts live in superposition across many neurons, not one-per-concept.

TF8. "A model that scores well on a benchmark understands the task."

False. High score can come from exploiting spurious shortcuts (backgrounds, watermarks, dataset artefacts) that co-occur with the label — that's memorising the distribution, not understanding the concept.

Spot the error

SE1. "We'll fix distribution shift by simply collecting a bigger, more diverse training set."

The error: you can never enumerate all future deployment conditions. The real fix is learning invariant features

ϕ (x)

whose distribution matches across domains, so unseen shifts don't move the input the model actually uses.

SE2. "LIME gave the explanation pointy ears → +40% cat, so that's the true reason the model decided cat."

The error: LIME is a local linear approximation valid only in a tiny neighbourhood (panel 3 — the straight line hugs the curve only near one point); a different perturbation gives a different slope. An adversarial tweak to the ears that stays 96% cat proves the explanation was a spurious local artefact, not the mechanism.

SE3. "To reduce drowning, ban ice cream — the data clearly links them."

The error: banning ice cream is an intervention

d o (ice cream = 0)

, which cuts the arrow temperature→ice cream but leaves temperature→drowning untouched. Drownings don't change; the model confused

P (Y ∣ X)

with

P (Y ∣ d o (X))

SE4. "The model learned fluffy + 4 legs → dog, so it will recognise a dog sketch."

The error: it actually learned natural-lighting + fluffy + 4 legs → dog because lighting and texture always co-occurred in training. A sketch lacks the lighting cue, the learned weights don't activate, and it fails — it never had incentive to disentangle lighting from identity.

SE5. "Mechanistic interpretability is basically solved — we can decompose

f (x)

into circuits."

The error: the decomposition idea exists, but in practice only roughly 1% of parameters are understood even in small models. Claiming it's solved skips the fact that superposition makes clean circuit extraction extremely hard.

SE6. "Domain adaptation minimises

D_{KL} (P_{source} (ϕ (x)) ∥ P_{target} (ϕ (x)))

, so any two domains can be aligned."

The error: matching feature distributions can destroy the label information if you push

ϕ

toward invariance too hard, and current methods only work for small shifts. Since

D_{KL} = 0

only when the two distributions are identical, forcing it to

0

can be too aggressive — alignment plus low source risk is a delicate trade-off, not a guarantee.

SE7. "GPT-4 reads about smoking and cancer, so it can answer whether banning cigarettes cuts cancer."

The error: it has absorbed the correlational statement, but the interventional query

d o (smoking = 0)

requires a causal model it doesn't hold. Unless that exact intervention appears in training, it's guessing from correlation.

Why questions

WHY1. Why does high dimensionality make few-shot learning hard?

In high-dimensional input space,

k

examples cover a negligible fraction of the volume (panel 2 — the covered ball shrinks toward zero as dimension grows), so density-based generalisation (

p_{data}

densely covering inputs) fails; nearby-in-value points are effectively unseen.

WHY2. Why can't we just read off a model's reasoning from its weights like reading source code?

There are no clean abstraction layers: unlike software with APIs between modules, every layer nonlinearly mixes all previous ones, and features sit in superposition — there's no boundary at which to pause and inspect a "variable".

WHY3. Why do small distribution changes sometimes cause catastrophic (not small) accuracy drops?

Because the risk surface in distribution space can have a sharp Hessian (panel 4) — large eigenvalues mean a tiny move

(P_{test} - P_{train})

climbs a steep valley wall, so the average loss

E_{P_{test}} [L]

jumps; the failure is non-linear, not gentle.

WHY4. Why is

P (Y ∣ d o (X))

generally not equal to

P (Y ∣ X)

Because

P (Y ∣ X) = \sum_{Z} P (Y ∣ X, Z) P (Z ∣ X)

carries the confounder term

P (Z ∣ X)

(where

Z

is the common cause from panel 1), while intervening replaces it with the un-conditioned

P (Z)

: forcing

X

cuts the arrows into

X

, so

Z

no longer bends toward the observed

X

WHY5. Why does training on Task B erase Task A even though the network could store both?

Because the loss for Task B has no term protecting Task-A performance; gradient descent freely moves shared weights toward B's minimum, and those same weights were doing A's job — nothing penalises the collateral damage.

WHY6. Why is "just add more parameters" not a cure for interpretability?

More parameters deepen superposition and polysemanticity, packing more overlapping concepts into the same activation space — you make the object you're trying to read more entangled, not less.

WHY7. Why do humans need ~3 examples where models need millions for the same category?

Humans bring strong compositional inductive biases (learn "red" and "chair" separately, combine them), letting them reach unseen combinations without examples; standard neural nets lack built-in compositionality and must see the combination directly.

Edge cases

EC1. What happens to a domain-adaptation objective when source and target distributions are already identical?

The alignment term

D_{KL} (P_{source} (ϕ) ∥ P_{target} (ϕ))

is already

0

(its minimum, reached only at equality), so the method collapses to ordinary training — it neither helps nor hurts; adaptation only earns its keep under genuine shift.

EC2. What if there are no confounders between

X

and

Y

(no shared cause)?

Then

P (Z ∣ X) = P (Z)

, so

P (Y ∣ X) = P (Y ∣ d o (X))

— the boundary case where correlation does equal causation, and passive data suffices.

EC3. What does few-shot learning need when

k = 0

(zero-shot)?

With no examples at all, only the inductive bias / prior knowledge carries the load — e.g. a text description or a shared embedding space; performance is entirely a test of the model's built-in structure, not its data.

EC4. What is the limiting behaviour of the distribution-shift penalty

E_{P_{test}} [L] - E_{P_{train}} [L]

P_{test} \to P_{train}

It goes to

0

: with no shift, the two averages coincide, and the whole robustness problem vanishes — confirming the penalty is purely a shift effect, not a property of the model alone.

EC5. What if you have unlimited interventional data (endless randomised experiments on

X

Then you directly observe

P (Y ∣ d o (X))

and the causal problem dissolves — the difficulty of causality is precisely that interventions are usually scarce, unethical, or impossible, not that the target quantity is undefined.

EC6. What happens to continual learning when Task A and Task B are identical?

There is nothing to forget: the weights that solve A already solve B, so B's gradients reinforce rather than overwrite them — forgetting is a symptom of conflicting objectives, not of sequential training per se.

EC7. What if a "confounder"

Z

actually sits on the causal path

X \to Z \to Y

(a mediator, not a common cause)?

Then you must not adjust for

Z

: conditioning on a mediator blocks part of the true causal effect you're trying to measure, giving a wrong

d o

-estimate — same-looking variable, opposite handling.

Recall Self-test before you move on

Which quantity does infinite passive data give you, and which does it never give you? ::: It gives $P (Y ∣ X)$ (observational) forever, but never $P (Y ∣ d o (X))$ (interventional) without experiments or assumptions. Name the one edge case where correlation equals causation. ::: When there are no confounders, so $P (Z ∣ X) = P (Z)$ and the two conditionals coincide. What does a large Hessian eigenvalue mean geometrically? ::: A sharp, narrow valley — a tiny sideways step raises the loss steeply, so a small distribution shift becomes catastrophic.

Related deep dives to connect these traps to theory: Causal Inference, Statistical Learning Theory, Transfer Learning, Meta-Learning, Optimization Landscape, Bayesian Methods, Reinforcement Learning, Cognitive Science, AI Ethics.