Article
"Let people ask our data questions in plain English" sounds like a chat feature and is architecturally a text-to-SQL system with a conversational front end. Building it well has almost nothing to do with prompting a model to "understand our data" and almost everything to do with constraining what it can ask the database and proving the answer is right before a user trusts it.
Ground the model in the schema, not the data
The model needs to know table names, column names, relationships, and business definitions — what "active customer" means, which of three date columns is the one that matters — but it does not need, and should not receive, the underlying rows as context. A schema description, kept current and specific, is what lets the model write a correct query; the data itself only needs to flow through after the query is generated and validated, not before.
Generate the query, show the query
The single highest-trust decision in this architecture is surfacing the generated SQL, not just the answer. A user who can see "this pulled orders where status equals shipped and date is within the last 30 days" can catch a misunderstanding immediately — "no, I meant fulfilled, not shipped" — where a bare chat answer gives them nothing to correct against. This is the same principle as RAG citing its sources: an answer with a visible basis is one a user can verify instead of one they have to trust blindly.
Constrain what the query can do
The generated query runs against a read-only connection, scoped to a defined set of views rather than the raw production schema, with row-level security enforcing what the requesting user is allowed to see. This is not a nice-to-have; it is what makes "the model wrote a wrong query" a wrong answer instead of a data breach, the same architectural principle as scoping a write-capable agent's tools to exactly what it's allowed to touch.
Route structured questions away from retrieval
A common mistake is bolting the analytics assistant onto an existing RAG system, retrieving a document that happens to mention a number, and citing it as the answer. "How many orders shipped last week" is not a document question, and treating it as one produces a plausible-sounding, frequently wrong answer. The decision between a database query and retrieval should be made explicitly, by classifying the question before deciding which path answers it.
Handle the questions the schema can't answer
Some questions are legitimate and structurally unanswerable — metrics that were never tracked, comparisons across systems that were never joined. The assistant needs to recognize this and say so, rather than generating a query against the nearest-sounding columns and returning a confidently wrong number. This is an evaluation-set problem as much as a prompting one: the test set needs unanswerable questions in it, not just answerable ones, or you will never know the assistant handles them badly until a user does.
Evaluate against real questions, not synthetic ones
Collect the actual questions people ask analysts today, run them through the assistant, and score the generated query for correctness against a person who knows the schema, not just whether the assistant produced some SQL. This is the same evaluation discipline any production AI feature needs, and for a text-to-SQL system it is non-negotiable, because a subtly wrong query is indistinguishable from a right one until someone checks the numbers against a source they trust.
What this becomes in practice
Built this way, an analytics assistant is a thin conversational layer over a governed query engine, not a black box guessing at numbers. It is one of the more common builds inside our Data Engineering and AI/ML Development work, precisely because the trust problem, not the language problem, is what determines whether people actually use it after the first week.