Insights › AI
RAG beyond the demo: shipping retrieval that survives production
Most RAG systems work beautifully on three documents and fall apart on a million. Here is how we evaluate, ground and monitor retrieval pipelines that actually stay reliable.
By the KYNYX team · · Updated
A retrieval-augmented generation (RAG) demo answers questions about three PDFs. RAG in production answers questions for paying customers every day, at scale, without hallucinating in front of them. The distance between the two is not model choice — it is engineering discipline: how you chunk, how you measure retrieval, how you ground answers and how you watch quality over time.
Key takeaways
- Most RAG failures are retrieval failures, not generation failures — measure retrieval separately.
- Chunk by document structure (headings, sections, tables), not by a fixed character count.
- Build a labelled evaluation set before tuning anything, and re-run it on every change.
- Every answer should cite its sources, and the interface should show them.
- Treat the AI layer as a service with an SLA: monitoring, alerts and a rollback plan.
Why RAG demos fail in production
RAG, first described by Lewis et al. in Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, pairs a search step with a language model so answers come from your own documents instead of the model’s memory. On a small, clean corpus almost any setup looks impressive. Production corpora are different: thousands of documents, duplicated policies, outdated versions, scanned tables and questions phrased in ways nobody anticipated.
The three places retrieval pipelines quietly break are chunking that ignores document structure, embeddings and indexes that drift as the corpus grows, and evaluation that checks answer quality but never retrieval quality. We treat each as a first-class engineering problem rather than a prompt-tuning exercise.
Chunking that respects document structure
Fixed-size chunks cut tables in half and separate a heading from the paragraph that explains it. Structure-aware chunking splits on headings and sections, keeps tables intact, and attaches metadata — document title, section path, version, date and access level — to every chunk. That metadata powers filtering (“only the current policy”), better ranking and permission checks, so users never retrieve content they shouldn’t see.
Evaluate retrieval before you tune generation
We build an evaluation harness before tuning anything: a hand-labelled set of real questions mapped to the chunks that should answer them, refreshed monthly as the corpus changes. Retrieval recall and precision become tracked, deployable metrics, not a vibe. When recall dips, the harness tells us which chunking strategy, embedding model or index version caused it — and a change that lowers recall does not ship.
Hybrid search (keyword plus vector) and a re-ranking step usually lift recall more than swapping the language model. The harness is what lets you prove that instead of guessing.
Grounded answers with citations
Grounded generation matters as much as retrieval. Every answer should cite the passages it used, and the UI should show them. Citations are not a compliance checkbox — they are the trust layer that makes an AI feature safe enough to put in front of customers. Just as important, the system should be allowed to say “I don’t know” when retrieval returns nothing relevant, rather than improvising.
Monitoring RAG in production
Once live, log every query, the retrieved chunks, the answer and user feedback. Watch for queries with low retrieval scores, rising “no answer” rates and thumbs-down clusters — they reveal missing documents and gaps in coverage. Re-index on a schedule, version your prompts and indexes, and keep a rollback path. Security belongs here too: the OWASP Top 10 for LLM Applications is a useful checklist for prompt injection and data leakage.
FAQ
What is RAG in simple terms?
Retrieval-augmented generation finds the most relevant passages in your own documents and gives them to a language model, so the answer is based on your data and can cite where it came from.
How do you measure RAG quality?
Measure retrieval and generation separately: retrieval recall on a labelled question set, then answer faithfulness to the retrieved passages and user feedback once live.
Do we need a vector database?
Often yes, but many teams start with PostgreSQL and pgvector. The right choice depends on corpus size, filtering needs and latency targets — evaluation should decide, not trends.
Building RAG with KYNYX
We start narrow, instrument everything, and treat the AI layer like any other service with an SLA. That is what separates a demo from a product. See our generative AI development services, or read how we approach AI agents in production.