Production RAG Is Mostly a Plumbing Problem

Every RAG demo looks the same. Chunk some documents, embed them, drop them in a vector store, stuff the top-k results into a prompt. It works on the first try, which is exactly the problem: it teaches you that retrieval is the hard part.

It isn't. When we took a Slack-embedded coaching assistant from a working prototype to something a whole organisation leaned on daily, almost none of the work that mattered was retrieval work. It was plumbing.

The prototype lies to you

A prototype is evaluated by the person who built it, on documents they already know, with questions they already know the answer to. Every one of those conditions inverts in production.

Real users ask questions your corpus cannot answer. They ask follow-ups that only make sense given the previous three messages. They paste a wall of text and ask "thoughts?". They ask the same question two weeks apart and notice when the answers disagree. None of that shows up in a notebook.

The first honest signal we got was not an eval score. It was a Slack thread where someone asked a reasonable question, got a confident and wrong answer, and replied with a single question mark.

What actually took the time

Ranked roughly by how much engineering they consumed:

Context assembly. Deciding what goes into the prompt, in what order, and what gets dropped when the budget is tight. Retrieved chunks compete with conversation history, user profile, and system instructions for the same window.
Conversation state. Slack threads are the unit of conversation, not messages. Getting thread reconstruction right (including edits, deletions, and people joining mid-thread) was more code than the retrieval layer.
Failure behavior. What the assistant does when retrieval returns nothing relevant. Saying "I don't have anything on that" is a feature and it has to be engineered, because the default behavior of a language model is to answer anyway.
Cost and latency shaping. Streaming, caching, and knowing which requests deserve the expensive path.
Retrieval quality. Real, but a distant fifth.

Ground the model, then constrain it

The single highest-leverage change was not a better embedding model. It was making the prompt carry provenance, and making the instructions treat that provenance as the boundary of what may be said.

// Every retrieved chunk arrives with its source attached, and the system prompt
// is explicit that anything outside these blocks is off-limits.
const context = chunks
  .map((c, i) => `[${i + 1}] ${c.source}\n${c.text}`)
  .join("\n\n");

Once sources are in the prompt as structured, numbered blocks, two things get easier at once. The model cites them, so users can check the answer. And "answer only from the numbered sources, otherwise say you don't know" becomes an instruction the model can actually follow, because there is a concrete thing to point at.

The useful mental model: retrieval decides what the assistant is allowed to know. Everything after that is about making it behave as if that boundary is real.

Evaluate on the questions you actually get

We stopped writing eval sets by hand fairly early. The questions we invented were too clean and too close to the documents we had indexed.

What replaced them was boring and much better: sample real questions, label the answers, and keep the failures as fixtures. A regression suite built from ten genuinely bad production answers caught more than a synthetic set of a hundred.

The metric that survived was not similarity or faithfulness. It was the share of answers a reviewer would send to a colleague without editing.

Ship it where the work happens

The last thing worth saying is about surface. This assistant lives in Slack, not in a web app, and that was not a convenience decision.

A separate app is a place people have to remember to go. A bot in the channel where the conversation is already happening gets used because using it costs nothing. Adoption is a distribution problem long before it is a model problem, and the number that mattered most (daily active users) moved on distribution decisions, not on model upgrades.

If you are building something similar: budget your time for the plumbing. The retrieval will be fine.