RAG in Production: Building Knowledge Assistants Employees Actually Trust

The Demo-to-Production Gap

Retrieval-Augmented Generation is deceptively easy to demo: embed some documents, wire up a chat interface, ask a question the demo-giver already knows is answerable. It works, everyone applauds, a rollout is approved. Six months later usage has collapsed — because the tenth answer was confidently wrong, the eleventh cited a superseded policy, and the twelfth surfaced a document the asker shouldn't have seen. Trust in these systems is spent in single wrong answers and rebuilt in hundreds of right ones.

Production RAG is an information-retrieval engineering problem wearing an AI costume. The four disciplines below are where the projects that survive differ from the ones that don't.

1. The Corpus Is the Product

• Curate before you embed. Feeding the assistant every file share since 2009 guarantees it will retrieve the 2019 expenses policy alongside the current one. Define authoritative sources, exclude drafts and duplicates, and attach metadata — owner, effective date, supersedes — at ingestion.
• Chunk by structure, not character count. Split on headings, clauses and tables so retrieved passages are self-contained. A chunk that starts mid-sentence retrieves badly and reads worse in citations.
• Freshness is a pipeline, not a project. Documents change; the index must follow within hours. Stale-answer incidents are the fastest trust-killer in the category.

2. Retrieval Quality Is Measurable — So Measure It

Before tuning prompts, prove the right passages are being found. Build a golden set of 100-300 real questions with known source passages, and score retrieval (recall@k, precision) on every change. In our engagements, hybrid search — dense embeddings plus classic keyword/BM25, fused and re-ranked — reliably beats either alone, because employees ask questions full of product codes, form names and acronyms that embeddings alone fumble. Add query rewriting for follow-up questions ("what about contractors?" means nothing without the previous turn) and you've done more for answer quality than any prompt ever will.

3. Permissions Are Not Optional

The moment an assistant reads HR policies, board papers or client files, retrieval must be permission-aware at query time: filter candidate documents by the asker's entitlements before the model sees them, mapped from your identity provider. Post-hoc filtering of the answer is not a control — the leak already happened inside the context window. This is the first question your CISO will ask; have it answered in the architecture, not the roadmap.

An assistant that says "I don't know — here's who does" earns more trust than one that's right 95% of the time and confidently wrong the rest. Design for honest failure.

4. Answers That Show Their Working

• Citations on every claim, linking to the exact passage — not just the document — so verification is one click.
• Grounded-or-silent prompting: the model answers only from retrieved context, and explicitly declines when the context doesn't cover the question, routing to the owning team instead.
• Feedback loops: a thumbs-down flows to the knowledge owner with the question and retrieved passages attached. The corpus improves on a cadence; the golden set grows from real misses.

Where the ROI Actually Lands

The wins are unglamorous and large: policy and procedure questions answered in seconds instead of tickets to HR and IT; new starters productive in days; support agents and salespeople quoting the current version of the truth (the same retrieval layer powers the agents in our support automation guide). A typical production build — pipeline, hybrid retrieval, permissions, evaluation harness, chat surface in Teams/Slack — is a 6-8 week fixed-price engagement, benchmarked on your golden set so quality is a number, not a vibe.