In-context learning mechanisms: Question bank

Before we start, three words this page reuses, defined in plain language:

Demonstrations (demos): the input→output example pairs you paste into the prompt.
Latent task / concept $c$ : the hidden "game" the model thinks you are asking it to play (translate? classify sentiment? sort?).
Posterior $p (c ∣ D_{k})$ : after seeing $k$ demos $D_{k}$ , the model's belief about which game is being played. "Posterior" just means "belief updated by evidence" — see Bayesian Inference.

True or false — justify

Weights change a little during in-context learning.

False — the parameters

θ

are frozen; only transient activations inside the forward pass carry the "learning," and they vanish when the prompt ends.

Zero-shot (

k = 0

) is not in-context learning because there are no demos.

False — zero-shot is still ICL; the instruction and query themselves condition the model and locate a task, they just do it without example pairs.

If flipping half the demo labels barely hurts accuracy, then labels never matter.

False — that finding holds for tasks the pretraining prior already knows; for genuinely novel or hard mappings the model has no prior to fall back on, so correct labels matter.

Adding a 30th demo to a 10-demo prompt reliably beats the 10-demo version.

False — the log-posterior-ratio grows about

k \cdot KL

, so belief in the true task saturates near 1; extra demos mostly add cost, context pressure, and recency noise.

Reordering the demos should never change the prediction, since the same information is present.

False — ICL is conditioning, not clean set-based learning; models are biased toward recent and majority-label demos, so order can flip answers.

The Bayesian "task-location" story and the "attention ≈ one gradient step" story contradict each other.

False — they are two lenses on the same fact: the algorithm is already baked into the weights, and the context selects and steers it.

More demos teach the model a brand-new function it never knew.

False — demos do not add a new function; they concentrate the posterior onto a latent task the pretrained mixture already contained.

ICL and fine-tuning both leave a permanent change in the model.

False — fine-tuning permanently edits

θ

; ICL is per-prompt and reversible, so a fresh prompt starts from the same frozen weights (Fine-tuning vs Prompting).

Spot the error

"The demos are the training data and the forward pass is the training loop, so weights get updated by tiny steps."

The first clause is a fine analogy, but the conclusion is wrong: the "gradient step" is simulated inside activations (the residual stream), not applied to

θ

; nothing external is optimized.

"Since

p (c^{*} ∣ D_{k}) \to 1

exponentially, two demos are already enough for any task."

The rate depends on the per-demo KL gap between the true and wrong concepts; for tasks whose concepts look nearly identical the KL is tiny, so convergence is slow and two demos may not suffice.

"The attention-as-gradient-step derivation proves real transformers literally run gradient descent."

It proves an equivalence of algebraic form for a linear-attention toy under specific value/key/query settings; real multi-head softmax attention only approximates this, so it is a mechanistic sketch, not a literal claim.

"Because the model marginalizes over concepts, its answer is always an average and never a confident single label."

Once the posterior collapses onto one concept, the weighted average is dominated by that concept, so the output is effectively a confident committed prediction.

"Randomly flipped labels help because wrong labels teach the model what not to say."

No — flipped labels neither teach nor harm much because what the demos mainly convey is the input distribution, label space, and format; the label values are largely ignored when a prior exists.

"If a demo's format is unusual, the model just ignores format and reads the meaning."

Format is one of the strongest signals demos carry — it tells the model the output space and shape; a broken or inconsistent format often hurts more than a few wrong labels.

Why questions

Why does ICL exist at all — what during pretraining forces it?

To minimize next-token loss on documents where an earlier pattern predicts later tokens (lists, Q&A threads, translations), the model is forced to become good at "spot the pattern in the context and continue it."

Why is each per-demo term in the log-posterior-ratio non-negative on average?

Because the expected log-likelihood gap between the true and a wrong concept is exactly a KL divergence, and KL is always

\geq 0

(strictly positive for distinguishable tasks).

Why does accuracy climb fast then plateau as

k

grows?

The log-ratio grows roughly linearly (

k \cdot KL

), so the posterior probability of the true task approaches 1 exponentially; once it is near 1 the "evidence budget" is spent and extra demos add little.

Why can the demo label space matter even when the label values are wrong?

The set of allowed outputs ("Positive/Negative") constrains what the model may generate, so it channels the answer into the right vocabulary even if individual examples are mislabeled.

Why is ICL called "reversible" while fine-tuning is not?

ICL's effect lives in activations tied to one specific prompt, so deleting the prompt removes the effect; fine-tuning writes the change into

θ

, which persists across all future prompts.

Why does the "one gradient step" picture start from

W_{0} = 0

Starting from zero makes the gradient of

\frac{1}{2} ∥ y_{i} - W_{0} x_{i} ∥^{2}

collapse to

- y_{i} x_{i}^{⊤}

, which cleanly matches the attention sum

\sum_{i} y_{i} (x_{i}^{⊤} x_{k + 1})

— it isolates the analogy without extra terms.

Why does Chain-of-Thought sit on top of ICL rather than replacing it?

CoT demos still condition a frozen model; they locate a "reason-step-by-step" task and format, so it is ICL applied to a reasoning trace (Chain-of-Thought Prompting).

Edge cases

k = 0

with only an instruction and no examples — is any task located?

Yes; the instruction words and query form evidence that shifts the posterior toward a task, which is why clear phrasing (Prompt Engineering) matters even with zero demos.

All demos share one label (e.g. every example is "Positive") — what happens?

Majority-label bias kicks in and the model tends to over-predict that label regardless of the query, because ordering/frequency of demos skews conditioning.

Two candidate tasks are nearly indistinguishable (tiny KL) — how does ICL behave?

The posterior separates very slowly, so many demos are needed and the model may stay uncertain or drift between the two interpretations.

The prompt is longer than the context window, so early demos are truncated — effect?

Only the surviving (usually recent) demos condition the model, amplifying recency bias and possibly discarding the evidence that would have located the task.

A truly novel task with no pretraining prior is given with few demos — outcome?

Accuracy is poor and here correct labels genuinely matter, because the model cannot fall back on a learned prior and must extract the mapping from the few examples alone.

Demos are internally contradictory (same input mapped to two different outputs) — what does the model do?

The likelihood term cannot favor a single concept cleanly, so the posterior stays diffuse and predictions become unstable or default to the pretraining prior.

Retrieved passages are pasted in as "context" for RAG — is that the same mechanism as demo-based ICL?

It uses the same conditioning machinery, but retrieved facts locate content/evidence rather than a task pattern, so ICL and Retrieval-Augmented Generation (RAG) compose rather than being identical.

Recall One-line self-test

If I paste examples and the model "improves," which of my numbers moved: $θ$ or the activations? ::: The activations only — $θ$ never moves in ICL.