Question bank — Retrieval-Augmented Generation (RAG) architecture

RAG makes the language model's stored weights forget less over time. ::: False — RAG never touches the weights. It leaves the model's frozen memory alone and hands it fresh text in the prompt; the "knowledge" lives in the external store, not the parameters.

If you add a new document to the corpus, you must re-train or fine-tune the LLM. ::: False — that is the whole selling point over fine-tuning. You embed the new chunk and drop it in the vector store; the model reads it at query time with no retraining.

A perfect retriever guarantees a hallucination-free answer. ::: False — even with perfect context the generator can ignore it, over-summarise, or add unsupported claims. Grounding lowers hallucination but does not delete it; you still need "answer only from context / say not-found" instructions.

Cosine similarity and raw dot product always rank chunks in the same order. ::: False — only when every vector is normalised to unit length. On raw embeddings, dot product also rewards large magnitude, so a long high-norm chunk can outrank a shorter, more relevant one.

A cosine score of $0$ means the two texts are opposites. ::: False — $0$ means orthogonal, i.e. unrelated / no shared direction. The opposite end is $-1$ (pointing the other way). See Cosine Similarity & Nearest Neighbour Search.

Increasing $k$ (retrieving more chunks) can only improve the answer. ::: False — beyond a small $k$ you inject distractor passages and trigger the lost-in-the-middle effect, so accuracy can drop even though recall rises. Small $k$ (≈3–8) plus re-ranking usually wins.

The marginalisation $p(y\mid x)={\sum }_{z}p(z\mid x)p(y\mid x,z)$ requires summing over the entire corpus in practice. ::: False — the top-$k$ approximation sums only over the best few $z$, because every other passage has near-zero $p(z\mid x)$ and contributes negligibly to the sum.

Softmax over similarity scores gives each retrieved chunk roughly equal weight. ::: False — softmax is sharp: a score gap of $2$ can become a probability ratio near $7\times$. One dominant chunk typically carries most of $p(z\mid x)$, which is exactly why top-$k$ is a fine approximation.

"We embed the documents with one encoder and the query with a totally different unrelated model, that's fine because they both output vectors." ::: Error — query and document encoders must live in the same vector space (trained jointly or identical model). Mismatched spaces make cosine scores meaningless; nearest-neighbour search returns noise.

"To save space we normalise document vectors but leave the query vector un-normalised — cosine handles the rest." ::: Error — cosine divides by both norms, so leaving $\Vert \mathbf{q}\Vert$ in only rescales all scores by the same constant. Ranking is unaffected, but if you switched to raw dot-product search (assuming normalisation), the query's magnitude would now silently distort scores.

"Our chunks are single sentences so each embedding is super precise." ::: Error — chunks that are too small split an answer across several vectors and lose surrounding context, so no single chunk is retrievable as complete. Chunk size must balance precision against coherence, usually with overlap.

"The LLM cited a source, so the answer is definitely supported by that source." ::: Error — models can fabricate or mis-attribute citations. A citation is a claim, not a proof; you must verify the cited chunk actually contains the fact (grounding ≠ correct attribution).

"Cosine ranges $[0,1]$, so we never worry about negatives." ::: Error — cosine ranges $[-1,1]$. Embeddings can point in opposing directions, giving negative scores; assuming a $[0,1]$ range can break thresholds and softmax intuition.

"Retrieval and generation quality are one number, so we tune them together." ::: Error — they must be measured separately. Retrieval is judged by recall@k / precision; generation by faithfulness. Bundling them hides which stage is failing (bad chunks vs. bad reading).

"We concatenated the chunks but forgot the instruction, the model will still know to use only the context." ::: Error — without an explicit "use ONLY the provided context" instruction, the model freely blends in its parametric memory, reintroducing stale or hallucinated facts. See Prompt Engineering.

Why divide the dot product by the norms instead of just using the dot product? ::: Because the raw dot product $=\Vert \mathbf{q}\Vert \Vert \mathbf{d}\Vert \cos \theta$ mixes topic (direction) with length (magnitude). Dividing out the norms isolates $\cos \theta$, so ranking reflects "same topic", not "longer document".

Why do production systems normalise vectors once at index time and then use plain dot product? ::: Once $\Vert \mathbf{d}\Vert =1$, cosine equals the dot product, so you get cosine's fairness at dot-product speed — no per-query division, which matters at billion-vector scale.

Why treat the retrieved passage $z$ as a latent variable rather than picking one "true" passage? ::: Because we don't know which passage is correct, and several may partly help. Marginalising (${\sum }_z$) lets probability mass spread across candidates via the law of total probability, giving a principled objective instead of a hard guess.

Why can RAG cite sources while a plain fine-tuned model cannot? ::: RAG keeps facts as data with identity (this chunk, from this document), so it can point at the exact passage used. Fine-tuning dissolves facts into weights, leaving no addressable location to cite.

Why is "retrieval quality dominates" the 80/20 rule of RAG? ::: The generator can only reason over what it's shown. A great LLM on wrong chunks answers wrongly; an average LLM on the right chunks answers well. Fix the input before polishing the reader.

Why exponentiate scores in softmax instead of just normalising them directly? ::: Exponentiation forces all values positive (so they form a valid probability distribution) and amplifies gaps, cleanly concentrating weight on the top chunk — matching how one strong retrieval should dominate.

Corpus is empty (or retrieval returns nothing). What should the pipeline do? ::: With no chunks, the prompt has no grounding, so the model should be instructed to answer "not found / insufficient context" rather than fall back to its parametric memory and hallucinate.

The query asks about an event after the documents were written. ::: RAG cannot invent missing facts — retrieval will surface nothing relevant. This is a corpus staleness problem, not an LLM problem; the fix is updating the document store, not the model.

Two chunks are near-duplicates and both land in the top-$k$. ::: They waste a retrieval slot and skew $p(z\mid x)$ toward that repeated content, crowding out diverse evidence. De-duplication or MMR-style diversity in retrieval fixes it.

A chunk vector is the zero vector (empty or all-stopword chunk). ::: Cosine is undefined — dividing by $\Vert \mathbf{d}\Vert =0$. Such chunks must be filtered at index time or they crash / poison nearest-neighbour search.

The correct answer spans two chunks that were split apart during chunking. ::: Neither chunk alone contains the full answer, so top-$k$ may retrieve one and miss the other, giving a partial or wrong answer. Overlapping chunks or a larger chunk size mitigates this — a chunking decision.

Every retrieved chunk has a low similarity score (best cosine ≈ 0.2). ::: Low top score signals the corpus likely has no good match. Rather than force an answer from weak context, threshold on similarity and return "not found", preventing confident nonsense.