Vector databases and embeddings
WHY do we need embeddings at all?
WHAT an embedding is: a function that maps any piece of text to a fixed-length vector (e.g. ). It is trained so that texts with similar meaning get vectors pointing in similar directions.
HOW it's learned: models (e.g. sentence-transformers, OpenAI text-embedding-3) are trained on huge corpora so that related sentences are pulled together and unrelated ones pushed apart (contrastive learning). We don't hand-design the numbers — the model discovers them.
Measuring closeness: from first principles
We need a number saying "how similar are vectors and ?"
Derivation of cosine similarity. We want a measure that ignores length and cares only about direction (a long document about dogs and a short one about dogs should still match). Start from the dot product identity above and solve for :
This is cosine similarity. Why divide by the norms? To strip out magnitude so only the angle (meaning-direction) remains. Range: ; = identical direction, = orthogonal (unrelated), = opposite.
WHAT is a vector database?
Why not just a normal database? A SQL WHERE matches exact values. Here we need approximate geometric proximity over high-dimensional vectors, which needs specialized indexes.
The curse of exact search
Exact k-NN scans all vectors: cost . For this is far too slow per query.
HOW ANN works — HNSW (the popular one): build a Hierarchical Navigable Small World graph. Nodes = vectors; edges connect nearby ones across multiple layers (like a skip-list). A query starts at a sparse top layer, greedily hops toward closer neighbours, then descends layer by layer to refine. This gives logarithmic-ish search.
The full pipeline (this is the 80/20 core)
- Chunk documents into passages (so each vector represents a focused idea).
- Embed each chunk → store vector + original text (metadata).
- Embed the user query with the same model.
- Search the DB for top- nearest chunks.
- Feed those chunks to an LLM (this step = RAG, see Retrieval-Augmented Generation).
Worked examples
Recall Feynman: explain to a 12-year-old
Imagine every sentence is a dot placed on a giant star-map. Sentences that mean the same thing are placed as neighbouring stars. To answer your question, the computer turns your question into a dot too, then just looks around it for the nearest stars and reads what they say. The "vector database" is the telescope that finds nearby stars super fast without checking every star in the sky.
Common mistakes
Flashcards
What is an embedding?
Why use cosine similarity instead of raw dot product?
Formula for cosine similarity
What does a vector database do?
What is ANN and why use it?
How does HNSW search work?
Why must query and documents use the same embedding model?
If vectors are unit-normalized, dot product equals
Relation between Euclidean and cosine for unit vectors
The 5-step retrieval pipeline
Connections
- Retrieval-Augmented Generation — vector search is the "retrieval" engine.
- Prompt Engineering — retrieved chunks are injected into the prompt context.
- Cosine Similarity · Dot Product · Vector Norms
- Contrastive Learning — how embedding models are trained.
- HNSW and ANN Indexes · Chunking Strategies
- Curse of Dimensionality — why exact search gets hard in high .
Concept Map
Hinglish (regional understanding)
Intuition Hinglish mein samjho
Dekho, computer ke liye "dog" aur "puppy" bas alag-alag letters hain — usse pata nahi ki matlab same hai. Isliye hum har text ko ek embedding yaani numbers ki list (vector) mein badalte hain. Model aise train hota hai ki jo cheezein meaning mein similar hain, unke vectors space mein paas-paas aa jate hain. Matlab meaning ban jaati hai geometry — ab "similar dhundo" ka kaam sirf "paas ka point dhundo" ban gaya.
Similarity naapne ke liye hum cosine similarity use karte hain: dot product ko dono vectors ki length se divide kar do, taaki sirf direction (angle) matter kare, size nahi. Chota document ya bada document, agar dono dog ke baare mein hain to unki direction same hogi, similarity . Ye reason hai ki hum norms se divide karte hain — magnitude ka distraction hata dete hain.
Ab crores vectors ko store karke turant "nearest" nikalne ke liye chahiye ek vector database (FAISS, Pinecone, Chroma, pgvector). Har vector ko exact check karna slow hai (), isliye ye ANN (approximate nearest neighbour) use karte hain, jaise HNSW graph — thoda accuracy chhod ke bahut speed. Pipeline simple hai: document ko chunk karo, embed karo, store karo; phir user ka question bhi usi model se embed karo, DB se top- nearest chunks nikaalo, aur LLM ko do (ye RAG hai).
Sabse important trap: query aur documents dono ko same embedding model se embed karna, warna vectors alag "coordinate world" mein honge aur similarity bekaar aayegi. Isko yaad rakho aur 80% kaam ho gaya.