Article
Every RAG tutorial shows the same four boxes: chunk, embed, retrieve, generate. A production system has six stages, and the two the tutorials skip — ingestion and reranking — are usually where the quality gap between a demo and a system people trust actually lives.
Ingestion: the stage nobody demos
Documents arrive as PDFs with broken text extraction, Word files with tracked changes still in them, wiki pages with stale sections nobody deleted. Ingestion normalizes all of this into clean text with structure preserved — headings, tables, and lists intact, not flattened into a paragraph that loses which row belonged to which column. A pipeline that skips this step is chunking noise, and no amount of retrieval tuning downstream fixes text that was garbled on the way in.
Chunking: the decision with the most leverage
Fixed-size chunking, cutting every 500 words regardless of content, is the simplest option and the one most likely to cut a table in half or separate a heading from the paragraph it introduces. Structure-aware chunking — splitting at headings, keeping tables whole, attaching a section's title to every chunk within it — costs more engineering time and consistently retrieves better, because the chunk that comes back to the model is a coherent unit of meaning, not an arbitrary slice.
Embedding: pick for your domain, not the leaderboard
General-purpose embedding models work well on general text and noticeably worse on domain-specific language — legal clauses, medical terminology, internal product names. When retrieval quality plateaus below what the business needs, a domain-adapted or fine-tuned embedding model is often the fix, and it is cheaper to fine-tune an embedding model than to fine-tune the generation model, because the target — a good similarity space — is a narrower problem than good writing.
Retrieval: similarity is a starting point, not an answer
Pure vector similarity retrieves what is topically close, not necessarily what answers the question, and it has no concept of "this document is current, this one is superseded." Metadata filtering — by date, by document type, by access permission — should run before or alongside similarity search, not after, so the candidate set is already the right candidate set before ranking begins.
Reranking: the step that catches what retrieval missed
The first retrieval pass optimizes for speed across a large corpus and returns a candidate set, typically twenty to fifty chunks. A reranker, a smaller model built specifically to score relevance, re-orders that candidate set with far more precision than the vector search alone, because it can actually read each candidate against the question rather than comparing pre-computed vectors. This second pass is inexpensive relative to the generation call and is the single highest-leverage addition to a retrieval pipeline that is retrieving plausible-but-wrong chunks.
Generation: constrained, not creative
The model's job here is narrower than open-ended writing: answer from what you were given, cite where possible, and say so explicitly when the retrieved chunks do not contain the answer. This instruction, enforced and tested rather than assumed, is what separates a RAG system that admits uncertainty from one that hallucinates confidently when retrieval comes up short.
Where this connects to the rest of the system
A RAG pipeline is one component inside a larger LLM application, and it needs the same observability as everything else — which chunks were retrieved, which were used, and which answer resulted, logged per request, so a wrong answer is diagnosable instead of mysterious. This is the architecture we build into every retrieval system delivered through AI/ML Development, and it is also the foundation an enterprise RAG deployment has to extend rather than bolt permissions onto after the fact.