Alignment, Prompting & RAG
Difficulty Level: 1 (Recognition — MCQ, Matching, True/False with justification) Time Limit: 20 minutes Total Marks: 30
Section A — Multiple Choice (1 mark each) [10 marks]
Choose the single best answer.
Q1. In RLHF, the reward model is trained primarily on: a) Raw internet text b) Human preference comparisons between outputs c) Gradient penalties d) Cosine similarity scores
Q2. The core objective of Direct Preference Optimization (DPO) is to: a) Train a separate reward model then run PPO b) Optimize the policy directly from preference data without an explicit reward model c) Fine-tune only the embedding layer d) Maximize retrieval recall
Q3. In PPO for LLMs, the KL-divergence term against a reference policy is used to: a) Increase output length b) Prevent the policy from drifting too far from the reference model c) Improve retrieval speed d) Compute embeddings
Q4. "Few-shot prompting" differs from "zero-shot prompting" because it: a) Uses fewer model parameters b) Includes example input–output demonstrations in the prompt c) Requires gradient updates d) Skips the system prompt
Q5. Chain-of-thought prompting improves performance mainly by: a) Compressing the prompt b) Eliciting intermediate reasoning steps before the answer c) Reducing vocabulary size d) Increasing temperature to 0
Q6. Self-consistency decoding works by: a) Taking a single greedy answer b) Sampling multiple reasoning paths and taking the majority answer c) Reranking documents d) Reducing the KL penalty
Q7. In a RAG pipeline, the retriever's role is to: a) Generate the final answer tokens b) Fetch relevant context passages given a query c) Train the reward model d) Compute the PPO advantage
Q8. A vector database primarily enables: a) Exact keyword matching only b) Approximate nearest-neighbor search over embeddings c) SQL joins on text d) Token-by-token generation
Q9. Constitutional AI reduces reliance on human labels by: a) Using a set of principles to let the model critique/revise its own outputs b) Removing the reward model entirely and using rules only for retrieval c) Increasing chunk overlap d) Using BM25 scoring
Q10. Hybrid search typically combines: a) Two different LLMs b) Sparse (lexical) and dense (semantic) retrieval c) PPO and DPO d) Two vector databases of the same type
Section B — Matching (1 mark each) [8 marks]
Match each term in Column X to its best description in Column Y.
| # | Column X | Column Y | |
|---|---|---|---|
| Q11 | Reward modeling | A | Splitting documents into retrievable units |
| Q12 | Chunking | B | Model that scores outputs to approximate human preference |
| Q13 | Reranking | C | Using a strong LLM to score/grade other models' outputs |
| Q14 | LLM-as-judge | D | Re-ordering retrieved candidates by relevance before generation |
| Q15 | Tree-of-thought | E | Exploring/branching multiple reasoning states with search |
| Q16 | In-context learning | F | Learning a task from prompt examples without weight updates |
| Q17 | Hallucination mitigation | G | Grounding answers in retrieved evidence / citations |
| Q18 | Embeddings | H | Dense numeric vectors capturing semantic meaning |
(Write pairs, e.g., Q11 → B)
Section C — True/False WITH Justification (2 marks each) [12 marks]
State True or False (1 mark) and give a one-line justification (1 mark).
Q19. DPO requires running a separate PPO reinforcement-learning loop at training time.
Q20. Increasing chunk overlap can reduce the risk of splitting a relevant fact across two chunks, at the cost of more storage/retrieval redundancy.
Q21. In RLHF, the reward model outputs a probability distribution over the vocabulary.
Q22. Few-shot prompting always requires updating the model's weights with the provided examples.
Q23. Chain-of-thought and self-consistency can be combined to improve reasoning accuracy.
Q24. Retrieval-Augmented Generation can help reduce hallucinations by providing factual context to the generator.
Answer keyMark scheme & solutions
Section A (1 mark each)
Q1 → b. RLHF reward models learn from ranked/pairwise human preference comparisons; that is the training signal. (1)
Q2 → b. DPO reparameterizes the RLHF objective so the policy is optimized directly from preference pairs, skipping the explicit reward model + RL loop. (1)
Q3 → b. The KL term keeps the fine-tuned policy close to the reference/SFT model, preventing reward hacking and degeneration. (1)
Q4 → b. Few-shot supplies in-prompt demonstrations; zero-shot gives none. (1)
Q5 → b. CoT elicits intermediate reasoning tokens, improving multi-step accuracy. (1)
Q6 → b. Self-consistency samples many CoT paths and marginalizes via majority vote. (1)
Q7 → b. The retriever fetches relevant passages; the generator produces the answer. (1)
Q8 → b. Vector DBs do approximate nearest-neighbor (ANN) search over embedding vectors. (1)
Q9 → a. Constitutional AI uses written principles for self-critique and revision (RLAIF-style), reducing human labeling. (1)
Q10 → b. Hybrid search fuses sparse lexical (e.g., BM25) with dense semantic retrieval. (1)
Section B (1 mark each)
| Q | Answer |
|---|---|
| Q11 | B |
| Q12 | A |
| Q13 | D |
| Q14 | C |
| Q15 | E |
| Q16 | F |
| Q17 | G |
| Q18 | H |
Each correct pair (1 mark). Reward model→scores preferences (B); chunking→splitting docs (A); reranking→reorder candidates (D); LLM-as-judge→grade with an LLM (C); ToT→branching search (E); ICL→learn from examples no weight update (F); hallucination mitigation→grounding in evidence (G); embeddings→semantic vectors (H).
Section C (1 mark T/F + 1 mark justification)
Q19. False. DPO's key advantage is that it avoids a separate RL/PPO loop, using a closed-form preference loss on the policy directly. (2)
Q20. True. Overlapping windows keep facts that straddle boundaries intact within at least one chunk, but duplicate content increases index size and redundant retrievals. (2)
Q21. False. The reward model outputs a scalar reward/score (or preference logit), not a vocabulary distribution — that is the language model's output. (2)
Q22. False. Few-shot prompting is in-context; no weight updates occur, the examples only condition inference. (2)
Q23. True. Self-consistency samples multiple CoT chains and votes; the two are complementary and commonly combined. (2)
Q24. True. Grounding generation in retrieved factual context reduces unsupported fabrications, a core motivation of RAG. (2)
Mark Distribution
- Section A: 10 × 1 = 10
- Section B: 8 × 1 = 8
- Section C: 6 × 2 = 12
- Total = 30
[
{"claim":"Section A has 10 one-mark questions totaling 10 marks","code":"result = (10*1 == 10)"},
{"claim":"Section B has 8 matching items totaling 8 marks","code":"result = (8*1 == 8)"},
{"claim":"Section C has 6 questions at 2 marks totaling 12 marks","code":"result = (6*2 == 12)"},
{"claim":"Paper total is 30 marks","code":"result = (10 + 8 + 6*2 == 30)"},
{"claim":"Total question count is 24","code":"result = (10 + 8 + 6 == 24)"}
]