Article
A language model only knows what it was trained on, and it was not trained on your customer contracts, your internal wiki, or last week's support tickets. Retrieval-augmented generation, RAG, is the fix: before the model answers, the system finds the most relevant pieces of your own documents and hands them to the model along with the question. The model does not need to know about your business in advance; it just needs the right excerpt at the right moment.
The four steps, in order
Chunk. Documents are split into pieces small enough to be individually relevant, typically a few hundred words, because handing the model an entire manual for every question wastes context and buries the answer in noise.
Embed. Each chunk is converted into a vector, a list of numbers that represents its meaning, using an embedding model. Chunks about similar topics end up as vectors that are numerically close to each other.
Retrieve. When a question arrives, it is embedded the same way, and the system finds the chunks whose vectors are closest to it — the ones most likely to be relevant, without ever matching on exact keywords.
Generate. The retrieved chunks and the original question go to a language model, which writes an answer grounded in what it was just shown, rather than in whatever it happened to memorize during training.
What problem this actually solves
Two problems, and it is worth naming them separately. First, knowledge the model was never trained on — your documents did not exist when the model was trained, so retrieval is the only way for the model to see them. Second, hallucination reduction — a model asked to answer from a specific excerpt, with instructions to say when the excerpt does not contain the answer, is far less likely to invent one than a model answering from memory alone.
What good retrieval looks like
Retrieval quality is not really about the model; it is about whether the right chunk was found at all. A model given the wrong three chunks will write a fluent, confident, wrong answer, because it is doing exactly what it was asked: answer from what you were shown. Most RAG systems that disappoint in production have a retrieval problem, not a generation problem, and the fix is in the architecture of the pipeline — chunking strategy, metadata filtering, and reranking — not in switching to a bigger model.
Where RAG is the wrong tool
RAG is not the default answer to "we want AI on our documents." Structured questions belong in a database query, small document sets fit directly in a model's context window, and contradictory document versions need curation before retrieval, not a better retriever. We wrote the full list of when RAG is the wrong answer because it is recommended far more often than it should be.
RAG versus the other way to add your data
The alternative most people ask about is fine-tuning — training the model itself on your data instead of retrieving it at question time. They solve different problems and are not really substitutes; our comparison covers which one your situation actually calls for, and it is usually RAG, for reasons that have as much to do with maintenance as with accuracy.
Where it goes from a working demo
A RAG pipeline that answers a handful of test questions well is a prototype. An enterprise RAG system has to handle permissions per document, contradictory versions, and a corpus that changes daily — the gap between the two is most of the engineering work, and it is where our AI/ML Development engagements spend most of their time.