The RAG Playbook: From Demo to Production

Your RAG demo works because you loaded 40 PDFs, chunked them at 512 tokens, and asked the questions your test set was built to answer. Ship that to real users and watch the wheels come off. They ask compound questions your retriever can't decompose, upload a 200 page merger agreement your chunker slices mid clause, and complain the answers are "close but not quite" — the polite version of "your bot is wrong." The gap between a working notebook and production RAG isn't a bigger model. It's seven boring decisions — chunking strategy, embedding model, vector store, rerank, hybrid weighting, evaluation loop, latency budget — that nobody blogs about because none of them are sexy. This post is those decisions, in the order they matter, with numbers I've watched teams actually pay in 2026. If you internalize one thing: your LLM is the last thing to tune, not the first. Chunking is where 60% of your quality lives Fixed size 512 token chunks with 50 token overlap is the default. It's also why your bot returns half a sentence and hallucinates the rest. Two chunking strategies actually move the needle: Parent child (LlamaIndex calls it "small to big"): embed small chunks ( 200 tokens) for retrieval precision, but pass the parent chunk ( 1500 tokens) to the LLM. Retrieve on specificity, generate on context. This one change routinely improves faithfulness scores by 15 20 points on real corpora. Semantic chunking : split at points of low cosine similarity between adjacent sentences. Greg Kamradt popularized it; it's now in LangChain and LlamaIndex. Works well for essays, terribly for structured docs (contracts, code, tables). Don't blindly apply it. For structured content — contracts, SEC filings, technical docs — use hierarchical chunking that respects document structure. Split on H2/H3 first, then paragraphs, then…