Vector Databases and RAG, Explained Simply
A plain-language walkthrough of how embeddings, similarity search, and retrieval-augmented generation combine to make language models less prone to making things up.
Why this matters
Large language models are fluent, but fluency and correctness are not the same thing. A model trained on a fixed snapshot of text has no way of knowing about your company's latest pricing sheet, a paper published last week, or the contents of a private document store. Ask it anyway, and it will often answer with total confidence, drawing on patterns rather than facts. This is the core problem that retrieval-augmented generation, usually shortened to RAG, is designed to solve.
The idea is simple to state: instead of relying purely on what the model memorised during training, you fetch relevant passages from a trusted source at the moment someone asks a question, and hand those passages to the model as extra context. The model then generates its answer grounded in that retrieved text, ideally citing it, rather than inventing an answer from scratch. The trick that makes this fast enough to use in practice is the vector database, so it is worth understanding what it actually does before treating it as a black box.
I find it useful to separate the system into two honest jobs: finding the right text, and writing a good answer from it. Most of the interesting engineering, and most of the ways things go wrong, live in the first job. A brilliant language model given the wrong three paragraphs will still produce a confident, wrong answer. So the retrieval half deserves as much scrutiny as the generation half, even though it gets far less attention in casual conversation.
Embeddings and similarity search, worked through
An embedding is a way of turning a piece of text into a list of numbers, a vector, such that texts with similar meaning end up close together in that numerical space. A sentence about refunds and a sentence about returns policy might end up near each other even if they share almost no words in common, because the embedding model has learned to represent meaning rather than surface spelling. This is the property that makes semantic search possible: you are no longer matching keywords, you are measuring distance between meanings.
Concretely, imagine you have a support knowledge base of ten thousand articles. You run each article through an embedding model once, offline, and store the resulting vector, say 768 numbers long, alongside a pointer back to the original text. That is the vector database: a specialised index built to store millions of these vectors and answer one recurring question extremely quickly, namely, given a new vector, which stored vectors are closest to it?
Now a user asks, 'why was I charged twice for one order?' You embed that question with the same model, producing another 768-number vector, and ask the database for the nearest neighbours by a distance measure such as cosine similarity. Suppose it returns the three articles on duplicate billing, payment retries, and refund timing. Those three passages, perhaps two thousand words in total, get inserted into the prompt sent to the language model, along with an instruction to answer using only that material. The model now has the specific, current facts it needs, rather than a vague statistical impression of billing policy from its training data.
The reason this scales is the index structure underneath, things like approximate nearest neighbour graphs, which trade a small amount of accuracy for enormous speed gains, letting you search millions of vectors in single-digit milliseconds instead of comparing against every one linearly. That trade-off is usually fine in practice, but it is a real trade-off, and it is worth knowing it exists rather than assuming retrieval is perfectly exhaustive.
Where it quietly breaks
The failure mode I would watch most closely is chunking. Documents get split into passages before embedding, because a whole fifty-page manual is too large and too unfocused to embed as one vector. Split too coarsely and a chunk mixes two unrelated topics, diluting its embedding until it matches nothing well. Split too finely and you lose context: a chunk that says 'this fee applies in that case' with no surrounding sentence to say what 'that case' means is retrieved perfectly and is still useless. Getting chunk size and overlap right is unglamorous, but it often has more effect on answer quality than swapping to a fancier embedding model.
A second trap is evaluating the system only by whether the final answer sounds plausible. That tells you almost nothing about whether retrieval actually worked. A more honest evaluation checks, separately, whether the retrieved passages contain the information needed to answer correctly, and only then checks whether the generated answer used them faithfully. I would build a small labelled set of questions with known correct source passages, held out from anything used to tune the pipeline, and measure retrieval accuracy on that set before ever judging the generated text. Without that separation, you cannot tell whether a wrong answer is a retrieval failure or a generation failure, and you will end up tuning the wrong component.
Leakage is the other quiet danger. If your test questions were written by looking at the documents, or if near-duplicate passages exist across your training and evaluation splits, retrieval will look artificially strong because the system is effectively matching on things it has seen framed before. This mirrors the same leakage problems that show up in any machine learning evaluation: the fix is the same discipline of keeping evaluation data genuinely unseen and structurally separated from anything used during development.
The practical takeaway
RAG is not magic that eliminates hallucination, it is a way of narrowing the model's job from 'recall everything you might know' to 'summarise this specific handful of passages faithfully', which is a much easier and more checkable task. The vector database is simply the fast lookup mechanism that finds those passages by meaning rather than exact wording, using embeddings and approximate nearest neighbour search under the hood.
If you are building or assessing one of these systems, treat retrieval as the thing to measure first and separately: check that the right passages come back before worrying about how eloquently the model writes about them. Pay attention to chunking choices, keep a genuinely held-out evaluation set free of leakage, and remember that a strong baseline, even a simple keyword search compared side by side with your embedding search, will tell you honestly whether the added complexity is earning its keep.
