RAG in Production: What Engineering Teams Get Wrong (and How to Fix It)

From the archive. Originally published 2025-02-20. Tool details, examples and offers reflect that publication date. See our current guides for newer material.

RAG is standard. Production-quality RAG is not.

Most engineering teams can ship a RAG proof-of-concept in a day. The hard part is everything that comes after: chunking strategies that don't destroy context, retrieval quality that doesn't degrade as the corpus grows, evals that catch regressions before users do.

The four most common RAG failures we see

1. Fixed-size chunking that breaks semantic coherence

Splitting documents every 500 tokens without regard for structure means your retrieval system routinely returns fragments that make no sense in isolation. Switch to structure-aware chunking (markdown headers, sentence boundaries, semantic sections) and retrieval quality improves immediately.

2. No reranking step

Embedding similarity is a good first filter, not a final answer. Adding a cross-encoder reranker (Cohere Rerank, BGE Reranker, or a custom fine-tune) between retrieval and generation cuts hallucination rates significantly on knowledge-intensive queries.

3. No evals pipeline

If you can't measure retrieval precision and answer faithfulness, you can't improve them. Teams that build evals early - even simple ones using Claude as a judge - catch regression before it reaches users. Teams that skip evals find out about degradation through support tickets.

4. Single-vector search with no hybrid fallback

Dense vector search alone misses exact-match queries (product SKUs, named entities, specific codes). Hybrid search (BM25 + vector + rerank) covers the full query distribution your users actually produce.

The production RAG stack we recommend

• Ingestion: LlamaIndex or custom pipeline with structure-aware chunking
• Vector store: Pinecone, pgvector, or Weaviate depending on your infra
• Hybrid search: BM25 + dense vector, merged with RRF
• Reranking: Cohere Rerank or BGE-M3
• Evals: Ragas or custom Claude-as-judge pipeline
• Observability: LangSmith or Langfuse for tracing and regression detection

Train your engineers on production AI systems

Our AI Engineering cohort covers RAG, agents, evals, and LLM infra - built around the systems your team ships. Book a discovery call →