Zero-shot and few-shot prompting

WHY does conditioning on examples help? (from first principles)

An autoregressive LM defines a distribution over the next token given all previous tokens:

$P_\theta(x_t \mid x_{1},x_{2},\dots,x_{t-1})$

The full sequence probability factorizes by the chain rule of probability:

$P_\theta(x_{1:T}) = \prod_{t=1}^{T} P_\theta(x_t \mid x_{<t})$

Why this matters: everything you type becomes part of $x_{<t}$ . So the answer the model generates is:

$\text{answer} = \arg\max_{y}\; P_\theta\big(y \mid \underbrace{\text{demos}}_{\text{your shots}},\; \underbrace{\text{query}}_{\text{your question}}\big)$

Zero-shot: the "demos" are empty. The model must rely entirely on task-descriptions it saw during pretraining. Works when the task is common in training data.
Few-shot: the demos $D = \{(x_i, y_i)\}_{i=1}^{k}$ sharpen the conditional. They tell the model (a) the output format, (b) the label space, (c) the style/granularity. So $P_\theta(y\mid D, x)$ concentrates on the correct kind of answer.

HOW to build a few-shot prompt (recipe)

<Task instruction>

Input: <example 1 input>
Output: <example 1 label>

Input: <example 2 input>
Output: <example 2 label>

Input: <REAL query>
Output:

Why each part?

Instruction: pins the task even before examples (helps generalization).
Consistent delimiters ("Input:/Output:"): the model copies the format → predictable, parseable outputs.
Trailing "Output:": forces the model's very next tokens to be the answer, not chit-chat.

Figure — Zero-shot and few-shot prompting

Worked examples

Worked example Example 1 — Sentiment (zero-shot vs few-shot)

Zero-shot: "Sentiment (positive/negative): 'The battery dies fast.' →" Risk: model may reply "The battery life is poor." (wrong format, right idea). Few-shot fix:

Review: "Amazing camera!" Sentiment: positive
Review: "Screen cracked day one." Sentiment: negative
Review: "The battery dies fast." Sentiment:

Model outputs negative. Why this step? The two demos define the exact label set {positive, negative} and the one-word output shape, so the distribution collapses onto a single token class.

Worked example Example 2 — Format control (structured extraction)

Task: extract name + age as JSON.

Text: "Ravi is 30." -> {"name":"Ravi","age":30}
Text: "Meera is 22." -> {"name":"Meera","age":22}
Text: "Sam is 45." ->

Why this step? Zero-shot might return prose. The demos teach the schema (age is an int, not "30 years"), which no instruction alone reliably enforces.

Trade-offs (the 80/20 core)

More shots ⇒ more tokens ⇒ higher cost + risk of hitting the context window limit.
Beyond a small $k$ , accuracy usually plateaus (diminishing returns).

Common mistakes (steel-manned)

Flashcards

Zero-shot prompting is

giving a task by instruction only, with no worked examples.

Few-shot prompting is

prepending k input→output demonstrations so the model infers the task from context.

The name for learning-from-prompt-examples without weight updates

in-context learning (ICL).

Do few-shot examples change the model's weights?

No — they only condition the forward-pass probability distribution.

The chain-rule factorization an LM uses

P(x_{1:T})=\prod_t P(x_t\mid x_{<t})

Main reason few-shot helps beyond content

it fixes output format, label space, and style.

One-shot means

exactly k=1 demonstration.

Biggest cost of adding many shots

more tokens → higher cost and context-window pressure.

Why keep demo formats identical

the model copies the pattern, giving parseable, consistent outputs.

A demo-set failure mode from imbalance

model biases toward the majority label.

When is zero-shot preferable

common task + clear instruction; cheaper, no demo bias.

Trailing "Output:" in a prompt does what

forces the next tokens to be the answer, not commentary.

Recall Feynman: explain to a 12-year-old

Imagine a super-smart friend who has read every book but is a bit spacey. If you just yell "sort these!" they might sort by color instead of size. That's zero-shot — you only told them in words. If instead you first show them: "🍎 small, 🍉 big" a couple of times, they instantly get which kind of sorting you mean. That's few-shot — showing examples. The friend's brain didn't get smarter; you just helped them figure out which game you're playing right now. Once the chat ends, they forget the game — nothing got permanently learned.

Connections

Large Language Models — the next-token predictor being conditioned.
In-context learning — the underlying mechanism.
Chain-of-thought prompting — adding reasoning steps as demos.
Prompt engineering — the broader craft.
Retrieval-Augmented Generation (RAG) — dynamically inserting retrieved context as few-shot-like grounding.
Fine-tuning vs prompting — the true "learning" alternative that does change weights.
Context window — the token budget limiting how many shots fit.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek LLM basically ek "next word predictor" hai — usne internet ka bohot saara text padha hai aur wo bas socht raha hai "iske baad sabse likely agla shabd kya hoga?". Jab tum prompt likhte ho, tum model ko naya kuch sikha nahi rahe (weights change nahi hote) — tum sirf uski probability ko condition kar rahe ho. Zero-shot matlab: bas seedha instruction do, koi example mat do. Few-shot matlab: pehle 2-3 example (input→output) dikha do, taaki model pattern samajh jaaye — is cheez ko in-context learning kehte hain.

Ab WHY: model ke paas har task ka gyaan already hai, problem sirf yeh hai ki wo confuse ho jaata hai ki abhi konsa game khelna hai. Zero-shot me tum shabdon se batate ho. Few-shot me examples dikha ke tum uska confusion khatam karte ho — output ka format, labels ka set (jaise sirf positive/negative), aur style sab fix ho jaata hai. Isiliye jab zero-shot galat format ya galat label deta hai, tab thode se examples add karna kaafi powerful hota hai.

Practical baat (80/20): pehle zero-shot try karo — sasta hai aur common tasks pe theek chalta hai. Jab fail ho (format galat, rare label, ambiguity), tab hi few-shot examples daalo. Examples clean, balanced aur same format ke hone chahiye. Zyada examples matlab zyada tokens, zyada paisa, aur context window bhar jaana — aur accuracy ek point ke baad plateau ho jaati hai, toh bekaar me 20 examples mat thoko.

Ek important galatfehmi: few-shot se model "train" nahi hota. Chat band karo, sab "seekha hua" gayab. Yeh permanent learning fine-tuning me hota hai, prompting me nahi. Isliye yaad rakho: SHOW, don't RETRAIN — examples sirf hint dete hain, model ko dobara train nahi karte.

WHAT are these things?

WHY does conditioning on examples help? (from first principles)

HOW to build a few-shot prompt (recipe)

Worked examples

Trade-offs (the 80/20 core)

Common mistakes (steel-manned)

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Alignment, Prompting & RAG

Connections