Article
The mistake we see most in LLM application design is architecture organized around the model, so that a provider change or a version upgrade ripples through the whole codebase. A production LLM application has four layers, and the point of separating them is that three of the four should not care which model is behind the fourth.
The model layer: thin, swappable, and boring on purpose
This layer's only job is turning a request into a call to a provider's API and returning a normalized response. It should be the only code in the system that knows which provider or model version is in use. When this layer is done right, a model upgrade — or a full provider switch — is a configuration change, tested against the evaluation set, not a rewrite. Applications where the model's SDK is imported in a dozen files find this out the hard way when the next better model ships.
The harness layer: where the actual engineering lives
Above the model sits the layer that decides what the model is allowed to see and do: tool definitions, permission checks, context assembly, and stop conditions. For an agentic application this is the agent harness; for a simpler application it is a narrower version of the same idea — deciding what goes into the prompt and what the output is validated against before it reaches a user. This layer is where correctness and safety live, and it is almost entirely independent of which model sits below it.
The data layer: retrieval, memory, and state
Whatever the model needs to know that was not in its training data — retrieved documents, conversation history, structured records — lives here, with its own architecture separate from the model call. If the application does retrieval, this is where the RAG pipeline sits. If it's an agent, this is where task state persists across steps so a restart does not lose progress. Conflating this layer with the model layer is how teams end up unable to change their retrieval strategy without touching the code that talks to the model provider.
The product surface: what the user actually sees
Chat interface, structured form, background job with a notification — this layer translates the harness's decisions and the data layer's results into something a user interacts with, and it should be swappable independent of the other three. A well-architected LLM application can go from a chat UI to a fully automated background process without touching the harness, because the harness never assumed a chat interface in the first place.
Where most rewrites actually come from
Teams rebuild LLM applications not because the model changed, but because a layer boundary was never drawn and a change to one concern required touching all of them. A cost optimization that should have been a routing change in the model layer instead required editing prompt templates scattered through the product surface, because nothing separated "what to ask" from "how to display it." Layered correctly, cost work, model upgrades, and UI redesigns each touch one layer and leave the other three alone.
Observability and security cut across all four
Logging and guardrails are not a fifth layer bolted on top; every layer needs to emit into one observability pipeline and respect one security boundary, so a problem in the data layer is diagnosable with the same tools as a problem in the harness. Architecture reviews that treat these as afterthoughts tend to produce systems where a security fix in one layer has no way to verify it did not break another.
What this buys, concretely
An application built on these four layers is the one that survives its second year, when the model has changed twice, the retrieval strategy has been rebuilt once, and the UI has been redesigned — all without a full rewrite, because none of those changes ever needed to cross a layer boundary. This is the architecture behind every custom AI application and AI/ML Development engagement we scope, whether the end product is a chat assistant, an analytics assistant, or an agent platform.