Article
A RAG demo usually points at a tidy folder of a few hundred PDFs, and it works well, because the corpus was curated for the demo. An enterprise deployment points at a company's actual document store — millions of files, inconsistent permissions, versions nobody reconciled, and hundreds of new or changed documents a day. Almost none of the engineering effort in that gap is about the model; it is about the corpus.
Permissions have to travel with the document, not the query
The hardest problem in enterprise RAG is not retrieval quality, it is retrieval correctness: making sure a chunk from a document the requesting user cannot see is never eligible for retrieval, regardless of how well it matches the query. This means every chunk in the index carries the source document's access control list as metadata, and every query filters on it before similarity search runs, not after. Get this wrong and the system's failure mode is not a bad answer, it is a data leak — the same architectural discipline as tenant isolation in an agent platform, applied to documents instead of database rows.
Freshness at the scale of daily change
A demo corpus is static; an enterprise one is not. New documents, edited documents, and deleted or superseded documents arrive continuously, and the index has to reflect that within a bounded time, not whenever someone remembers to re-run ingestion. This means an incremental ingestion pipeline — detecting what changed, re-chunking and re-embedding only that, and removing stale chunks from the index — rather than a batch job that reprocesses everything nightly and leaves the system hours behind reality.
Version control for meaning, not just files
Enterprises accumulate contradictory versions of the same policy, and a retriever with no concept of "current" will confidently surface the 2022 travel policy alongside the 2024 one, with no signal to the model about which supersedes which. This needs explicit curation — a superseded flag, an effective-date field, a source-of-truth designation — carried as metadata and checked before or during retrieval. It is unglamorous, manual-feeling work, and it is the difference between a system that gets cited in a compliance audit favorably and one that does not.
Scale changes the retrieval math
At a few hundred documents, brute-force vector search over everything is fast enough not to matter. At millions of chunks, retrieval latency and infrastructure cost become real design constraints, which is where approximate nearest-neighbor indexes, sharding by document category, and pre-filtering by metadata before the vector search even runs stop being optimizations and start being requirements. This is the same architecture pipeline as a smaller system, under load characteristics that expose every shortcut the smaller system could get away with.
Auditability, because someone will ask "why did it say that"
Every answer needs a traceable path back to the specific chunks and documents it was built from, retained and queryable, not just logged to a text file. This is the same observability requirement as any enterprise LLM application, and in RAG specifically it doubles as the mechanism that lets a curation team find and fix the chunk that produced a wrong answer, rather than guessing.
What this actually costs
Enterprise RAG is rarely a "same system, more documents" scaling exercise; the permission model, incremental ingestion, and curation workflow are separate engineering efforts layered onto the retrieval pipeline, often exceeding the cost of the pipeline itself. This is what we scope explicitly in AI/ML Development engagements at this scale, because pricing only the pipeline and discovering the rest during rollout is how enterprise RAG projects run over budget and past deadline.