Article
Retrieval-augmented generation has become the default answer to a common request: we have a lot of documents, and we want people to ask questions about them. Chunk the documents, embed the chunks, retrieve the nearest ones for each question, and hand them to a model with the question. It works, and it is often the right call. It is also recommended in plenty of situations where it is the wrong one. Here are the four we see most.
The question has a structured answer
"How many orders did we ship to Germany in March?" is not a document question. The answer lives in a table, and the right tool is a query, not a retrieval. A RAG system asked this will find a monthly report that mentions Germany, quote a number from it, and be wrong whenever the report predates a correction.
What to build instead: text-to-SQL over a small, well-described set of views, with the model writing the query and the database answering it. Show the query. If the question mixes structured facts and prose, route it: classify first, then send it to the query path or the retrieval path.
The documents contradict each other
Policy manuals, contracts, and specifications accumulate versions. A retriever does not know that the 2023 travel policy supersedes the 2021 one; it knows which chunk is closest to the question. The model then gets both and picks one, and the answer depends on phrasing.
What to build instead: a curation step before retrieval. Decide what is current, mark the rest as superseded, and keep the metadata in the index so filters run before similarity does. This is unglamorous work and it is the whole difference between a demo and a system people trust.
The corpus is small enough to fit in the prompt
Retrieval exists because a model cannot read everything. If "everything" is forty pages, it can. Modern context windows hold the whole employee handbook with room to spare, and a model reading the full document answers better than one reading five chunks of it, because it sees the structure.
What to build instead: nothing clever. Put the documents in the prompt, cache the prefix, and spend the saved engineering time on evaluation. Retrieval becomes worth its complexity somewhere past a few hundred pages, and the threshold moves every year.
The real need is a workflow, not an answer
"Ask a question, get an answer" is the shape of a search box. Many requests that arrive dressed as search are actually tasks: draft the reply, fill in the form, check the submission against the rules. A RAG system can find the rules; it cannot check the submission, because it was built to answer, not to act.
What to build instead: a workflow with a model inside it. Retrieval may still be one step, but the system is defined by its steps and its checks, not by a chat interface. This is usually cheaper to build than the chat interface people asked for, and it is the thing they wanted.
How to tell in advance
Before building retrieval, write down twenty real questions with their correct answers. Then ask where each answer lives. If most live in tables, build queries. If most live in one short document, use the context window. If most are really requests to do something, build the workflow. If most live across a large, curated body of text, build the RAG system, and you will have a test set ready for it.