Zero-shot and few-shot prompting: Question bank

Notation & terms this page uses (read first)

Before the traps, we earn every symbol and term. Nothing below appears in an answer until it is defined here.

The two figures below make the abstract "conditioning" and "truncation" ideas visual before you meet them in the traps.

Figure — Zero-shot and few-shot prompting

The left bars show a broad next-token distribution (zero-shot: the model is unsure of format); the right bars show the same distribution sharpened onto the correct answer token after two demos are placed into $x_{< t}$ . No weight ( $θ$ ) changed — only the condition did.

This shows the context window as a fixed-width box. When demos + query exceed it, the earliest tokens (often your instruction) slide out the left edge and are silently dropped — they no longer condition the answer.

True or false — justify

True or false: "Few-shot prompting teaches the model something new that it keeps for later chats."

False. The demos only enter the current prompt as context tokens

x_{< t}

; they condition one forward pass and vanish when the request ends. Nothing persists — that would require fine-tuning to change

θ

True or false: "Zero-shot means the model was given no information at all about the task."

False. Zero-shot still gives a natural-language instruction — "no examples" is not "no information." The model leans on task patterns baked into

θ

during pretraining plus that instruction.

True or false: "If zero-shot already works well, adding few-shot examples will always make it even better."

False. On common, clearly-instructed tasks few-shot often only matches zero-shot while adding token cost and possible demo-bias. Extra examples give diminishing returns and can even hurt.

True or false: "One-shot is a special case of few-shot with k=1."

True. "Few-shot" is the umbrella for k≥1 demonstrations; k=1 is just named "one-shot," and k=0 is zero-shot.

True or false: "Because it's called in-context learning, a gradient update is happening quietly inside the model."

False. No gradients, no change to

θ

— the "learning" is simulated inside a single frozen forward pass by conditioning on the demo tokens. The name is metaphorical.

True or false: "The labels in your demonstrations barely matter; the model mostly reads the inputs."

False. The demos' format and label distribution strongly steer the conditional

P_{θ} (y ∣ demos, x)

. Inconsistent or wrong labels degrade results even when the example inputs are perfect.

True or false: "Doubling the number of shots roughly doubles accuracy."

False. Accuracy typically plateaus after a small k, while token cost keeps rising and you risk overflowing the Context window (truncation).

True or false: "Few-shot prompting is a cheaper substitute for fine-tuning that gives identical, permanent behaviour."

False. It is cheaper and needs no training, but it is not permanent (it lives only in

x_{< t}

) and competes for context space every single call.

True or false: "Balanced demos matter only for classification tasks, not for extraction or formatting tasks."

False. Label balance matters wherever the model can copy a majority pattern; for extraction, consistent schema/format across demos plays the analogous role of "balance."

True or false: "A prompt that mixes prose text and a code snippet confuses the model into a separate 'mode'."

False. There is no mode switch — text and code are just tokens in the same

x_{< t}

. The model conditions on all of them together; well-delimited code can actually sharpen the distribution by signalling the wanted output shape.

Spot the error

Spot the error: "I'll add 20 diverse examples so the model definitely learns the task permanently."

Two errors. Twenty examples burns tokens for plateaued gains, and nothing is "learned permanently" — it's per-request conditioning of a frozen

θ

, not training.

Spot the error: "My few-shot demos are all positive labels because positive reviews were easy to find; the model should still handle both classes."

The demo set is label-imbalanced. The model estimates the answer's prior from what it sees in

x_{< t}

; seeing only positive shifts

P_{θ} (y ∣ demos, x)

toward positive, so it copies the majority label regardless of the real input.

Spot the error: "I mixed the delimiters — one demo uses Input:/Output: and another uses Q:/A: — the model will figure it out."

Inconsistent delimiters give the model two competing continuation patterns, so no single format dominates

P_{θ}

; the most-likely continuation becomes ambiguous and outputs turn unpredictable and hard to parse.

Spot the error: "For a made-up label neutral-but-sarcastic, I'll just describe it in the instruction and skip examples to save tokens."

Pretraining never named that label, so

θ

has no strong prior for it and instruction alone is ambiguous. This is exactly the case where even one demo anchors the novel label in

x_{< t}

and removes the ambiguity — skipping demos is the wrong economy.

Spot the error: "I ended my prompt with a friendly 'Thanks, please help!' instead of a trailing Output:."

Without a trailing cue, the highest-probability continuation given

x_{< t}

may be more prose (a polite reply), not the answer. A trailing Output: makes the answer tokens the likeliest next tokens.

Spot the error: "My extraction demos show age: '30 years' but I expect an integer back."

The demos are the schema the model conditions on; showing a quoted string makes a quoted string the likeliest continuation. To get an integer, the demonstrations must themselves use "age": 30.

Spot the error: "Zero-shot gave prose instead of JSON, so the model must be broken."

The model isn't broken — nothing in

x_{< t}

pinned the output schema, so prose was simply the more probable continuation. Add demos showing the exact JSON shape and the distribution concentrates on structured output.

Why questions

Why do a few examples "select" a task rather than "teach" it?

A giant pretrained model already can do the task; its capability sits in the frozen

θ

. It just doesn't know which game you're playing. The demos disambiguate by conditioning

P_{θ}

toward the intended task, not by adding new capability.

Why does conditioning on demos "sharpen" the model's output distribution?

The demos become part of

x_{< t}

, and

P_{θ} (y ∣ demos, x)

is computed given them. Concretely: seeing positive/negative twice makes those two tokens carry almost all the probability mass, so the smooth "many possible words" distribution collapses onto the label set — like the broad→sharp bars in figure s01.

Why does keeping the output format identical across all shots help?

Each demo adds evidence for the same continuation pattern, so

P_{θ}

compounds that evidence into one dominant, predictable shape. Mixed formats split the probability mass across rival patterns, leaving the "most likely continuation" undefined and unparseable.

Why is a trailing Output: (or →) so effective?

After that token, the conditional

P_{θ} (next ∣ x_{< t})

has its mass on answer tokens (the demos always put the answer right after that cue), so preamble and chit-chat become low-probability continuations.

Why can adding more shots eventually reduce quality?

Once the distribution is already sharp, extra demos add token cost, can inject redundancy or imbalance that skews

P_{θ}

, and push earlier tokens toward the edge of the Context window where they may be truncated — losing the very instruction you added.

Why is few-shot especially powerful for a niche or invented label?

θ

holds no strong prior for a label pretraining never saw, so instructions alone leave

P_{θ}

diffuse. A single demo places a concrete anchor into

x_{< t}

, letting In-context learning infer the label's meaning from that example.

Why does the chain-rule factorization

P_{θ} (x_{1 : T}) = \prod_{t} P_{θ} (x_{t} ∣ x_{< t})

explain how prompting works at all?

Every factor conditions on

x_{< t}

= all prior tokens, and your prompt (instruction + demos + query) is part of that prior. So your words appear inside every factor of the product and thereby reshape every subsequent prediction — without any change to

θ

Why start with zero-shot before reaching for few-shot?

For common tasks with clear instructions it is cheaper, avoids demo-induced bias, and is often equally accurate; escalate to demos only when zero-shot's format, label, or ambiguity actually fails.

Why does Retrieval-Augmented Generation (RAG) resemble few-shot even though it isn't classic demonstration?

RAG dynamically inserts retrieved passages into

x_{< t}

, conditioning the same forward pass on fresh grounding — mechanically identical to how demos condition

P_{θ}

, just with facts instead of input→output pairs.

Edge cases

Edge case: What is the "few-shot" prompt when k=0?

It degenerates exactly into zero-shot — the demonstration block is empty and only the instruction plus query remain in

x_{< t}

Edge case: Two demos contradict each other (same input mapped to different labels). What happens?

The model receives conflicting evidence in

x_{< t}

, so probability mass splits between the two labels and

P_{θ}

becomes unstable — contradictory demos are worse than none for that label.

Edge case: All k demos share one label and the real query belongs to a different class. Likely outcome?

The model reads a one-sided prior from

x_{< t}

and copies the only-seen label, mispredicting the query — the classic demo-imbalance symptom.

Edge case: Your demos plus query exceed the model's Context window. What happens?

The earliest tokens (often your instruction or first demos) are truncated from the front and drop out of

x_{< t}

silently — you lose that conditioning with no error message, as figure s02 shows.

Edge case: Extremely long demonstrations force multiple rounds of truncation across a long chat. What breaks first?

The oldest content goes first each round, so your instruction and earliest demos vanish while recent tokens survive; the model can drift off-task mid-conversation because the parts that defined the task are no longer in

x_{< t}

. Fix: keep the instruction and a couple of compact demos near the end of the prompt so they survive.

Edge case: A prompt mixes modalities — prose plus a code snippet plus a JSON schema. Does the model juggle "three formats"?

No — it is one token stream in one

x_{< t}

; the model conditions on all three jointly. The risk is conflicting format signals (prose demos vs. JSON demos steering different continuations), not the mixing itself. Keep the demos that define the wanted output format consistent, and let the other modalities serve only as input context.

Edge case: You restart a fresh chat after a successful few-shot session. Does the model still "know" the task?

No. Since

θ

was never written to, the new session starts blank; you must re-supply the demos to recreate the conditioning.

Edge case: The instruction says "classify positive/negative" but the demos only ever show positive. Which wins?

The demos usually dominate

P_{θ}

over the instruction, so the model skews positive — a reminder that the label distribution inside

x_{< t}

steers harder than words alone.

Edge case: A task is entirely novel and unseen in pretraining, with no way to describe it in words. Can few-shot rescue it?

Partially — demos can define the format and label space in

x_{< t}

, but if the underlying mapping was never learnable into

θ

during pretraining, In-context learning has limited signal and fine-tuning may be required.

Recall One-line self-test before you close

If someone says "few-shot made the model smarter," what's your one-sentence correction? It didn't get smarter — $θ$ is frozen; the examples only conditioned this one forward pass to pick the right task; restart the chat and it's gone.

Connections

Parent: Zero-shot and few-shot prompting — the mechanism these traps test.
In-context learning — why demos condition rather than train.
Large Language Models — the next-token predictor underneath.
Context window — the truncation edge cases.
Fine-tuning vs prompting — the true weight-changing alternative.
Retrieval-Augmented Generation (RAG) — few-shot-like grounding by retrieval.
Chain-of-thought prompting — demos that carry reasoning steps.
Prompt engineering — the broader craft these habits belong to.