RAG Systems

A language model knows what was in its training data and nothing else. Ask it about your company's refund policy, your codebase, or a filing published last month, and it will either decline or improvise. Retrieval-augmented generation, named in a 2020 paper by Lewis and colleagues at Facebook AI Research, University College London and NYU, solves this the unglamorous way: before the model answers, you search a corpus you control, place the best passages into the prompt, and instruct the model to answer from them.

Described like that, RAG sounds like a weekend project, and the first version genuinely is. Load documents, split them into chunks, embed the chunks, store the vectors, embed the question, take the nearest neighbours, paste them in. A short script, if you lean on a library. It will work beautifully on the handful of questions you thought of while building it.

The trouble arrives when other people use it. Real queries are not the clean paraphrases you tested with. They are two questions stapled together. They use an internal acronym that appears nowhere in the source document. They reference a policy version that was superseded eighteen months ago and is still sitting in the index beside the current one. Sometimes they ask something the corpus does not answer at all, and the system dutifully assembles a confident reply out of three loosely related paragraphs.

Nearly all of those failures happen before the model generates a single token. Generation is the well-behaved end of the pipeline. Retrieval is a search problem, and search has been a serious engineering discipline since long before transformers existed. Most teams who describe their RAG system as "not accurate enough" have a recall problem: the passage containing the answer was never in the top results, so no amount of prompt tuning could have saved the response.

This is also the honest answer to the question people ask first, which is whether they should fine-tune instead. Fine-tuning changes how a model writes and what conventions it follows. It is a poor and expensive way to install facts, and a terrible way to install facts that change weekly. Retrieval keeps the knowledge outside the weights where you can update it, delete it, permission it, and cite it. The two techniques solve different problems and are frequently combined, but if your requirement is "answer from these documents, and show me where the answer came from", retrieval is the mechanism.

Treating the whole thing as an information retrieval system rather than a prompt engineering exercise is the shift that makes the rest of the work tractable.

Chunking is where most retrieval quality is won or lost, and it gets almost no attention because it looks like a formatting detail. It is not. A chunk is the unit you retrieve, so its boundaries decide what can ever be found together.

The starting point in most libraries is fixed-size splitting with overlap: cut at a set length and repeat a little text at each seam so a sentence spanning a boundary survives. It is a reasonable baseline and a poor finishing point. Fixed-size splitting cheerfully cuts a table away from its header, separates a numbered clause from the sentence that scopes it, and strips a heading off the section it introduces.

The alternatives are worth knowing:

• Recursive splitting tries a hierarchy of separators, breaking at paragraphs first, sentences next, characters only as a last resort. Cheap and noticeably better than fixed windows.
• Structural splitting uses the document's own skeleton: Markdown headings, HTML sections, legal clause numbers, function definitions in source code. When your corpus has real structure, this usually beats everything else.
• Semantic splitting embeds sentences and cuts where consecutive sentences diverge in meaning. Expensive to compute and inconsistent in practice, but useful for prose with no formatting.
• Small-to-big, or parent document retrieval decouples the two jobs a chunk is doing. You search over small, precise chunks and then hand the model the larger parent section they came from. Retrieval precision and generation context have different optimal sizes, and this pattern stops you compromising between them.

Whatever you choose, attach metadata: source, title, section heading, date, version, author, permission scope. Metadata is what later lets you filter, expire, cite and audit. Prepending the document title and section heading into the chunk text itself is a small change that reliably helps, because an isolated paragraph often loses the subject it was about. Anthropic published a variation of this idea as contextual retrieval, where a model writes a short situating sentence for each chunk before it is embedded.

Embedding model choice matters less than people expect and in a different way than they expect. The Massive Text Embedding Benchmark leaderboard hosted by Hugging Face is a reasonable starting filter, but leaderboard position is measured on public academic datasets, and your corpus is not one of those. A model that ranks below another in aggregate may handle your domain vocabulary far better. Test on your own data with your own questions.

Three practical constraints usually decide it. Dimensionality drives storage and memory. Sequence length caps how large a chunk can be before it is silently truncated, which is a common and invisible bug. And the model is a hard dependency: changing it means re-embedding the entire corpus, so treat the choice as a migration you will have to pay for later rather than a config value.

Vector search is not a superset of keyword search. It is a different tool with a different failure mode, and understanding the difference resolves a large fraction of mysterious retrieval bugs.

Dense retrieval embeds text into a vector space where semantically related passages land near each other. It handles paraphrase and vocabulary mismatch well: a query about "cancelling my plan" can find a passage titled "subscription termination" with no shared words. That is the whole point.

Sparse retrieval, of which Okapi BM25 is the standard implementation, scores documents on term overlap with weighting for term frequency and document rarity. It is decades old, cheap, interpretable, and it wins outright on exactly the queries dense retrieval fumbles: part numbers, error codes, function names, acronyms, surnames, SKUs, and any rare token the embedding model never saw enough of to place meaningfully. If a user searches for a specific identifier and gets back thematically similar but wrong results, you are watching a dense-only system fail in its characteristic way.

Hybrid retrieval runs both and merges the ranked lists. Reciprocal rank fusion is the usual merge because it combines rankings rather than scores, which sidesteps the awkward problem that cosine similarity and BM25 scores are not on comparable scales. The BEIR benchmark suite from the UKP Lab at TU Darmstadt was influential precisely because it showed how differently retrieval methods behave across domains, which is the argument for not betting everything on one of them.

Then there is reranking, which is the highest-leverage addition most pipelines are missing. Retrieval is optimized for speed across millions of candidates, so it uses a bi-encoder: query and document are embedded separately and compared by distance. A cross-encoder instead reads the query and a candidate passage together and scores the pair directly. It is far more accurate and far too slow to run over a whole corpus. So you run it as a second stage: retrieve fifty candidates cheaply, rerank them properly, keep the best five. Late interaction models such as ColBERT, developed at Stanford, sit between the two approaches by keeping per-token representations.

The design question is your latency budget. Every stage costs milliseconds, and a chat interface that sits silent for several seconds before it starts streaming feels broken regardless of how good the answer eventually is. Retrieval depth, rerank depth and generation length all compete for the same budget, and deciding that trade-off deliberately is part of the engineering.

The vector database market is noisy, and the marketing implies the choice matters more than it usually does. What actually differs between options is the index, the filtering behaviour, and the operational story.

Exact search compares the query against every vector. It is trivially correct and perfectly fine at small scale. People underestimate how far this goes: a corpus of a few tens of thousands of chunks held in a NumPy array or a FAISS flat index, the library from Meta, is usually fast enough that search latency is not what you spend your time on, and it never surprises you with a missing result.

Approximate nearest neighbour indexes trade a little recall for a lot of speed. HNSW, from the paper by Malkov and Yashunin, builds a navigable multi-layer graph and is the default in most modern stores. IVF partitions the space into clusters and searches only the closest few. Product quantization compresses vectors so more of the index fits in memory. Each has knobs that trade recall against latency and memory, and the important thing to internalize is that approximate means approximate: your index can silently return the second-best passage instead of the best one, and no error will be raised.

Filtering is where the real differences show up, and it is the question to ask when evaluating a store. Almost every production system needs to restrict results by tenant, permission, date or document type. Filtering after the vector search can return too few results or none at all, because the top matches all got discarded. Filtering before it can defeat the index structure entirely. How a given engine handles constrained search is a genuine differentiator and is worth testing with your own filter cardinality rather than trusting a benchmark.

Operational questions decide the rest. Can you delete a document and have it actually disappear from results, which matters for both correctness and data protection requests. Can you update an index without a full rebuild. How does it behave on restart, what does backup look like, and can you run multiple tenants without cross-contamination. A dedicated engine such as Qdrant, Weaviate, Milvus or Chroma gives you purpose-built behaviour. Adding pgvector to a Postgres instance you already operate gives you transactions, joins, existing backups and one fewer system on the on-call rota, which for many teams is the better trade even at some cost in raw throughput.

The honest default: start with whatever you can run and inspect easily, get the retrieval quality right, and migrate when scale or filtering demands it. Choosing a vector database first is optimizing the component least likely to be your bottleneck.

The standard evaluation method is typing a few questions into the demo and being satisfied. It is how teams ship systems that are wrong far more often than they believe, because the questions a builder invents are drawn from a different distribution than the questions users ask.

The first discipline is separating the two failure modes. When an answer is bad, either the correct passage was not retrieved, or it was retrieved and the model still answered badly. These have completely different fixes, and conflating them leads to endless prompt tinkering on what was really a chunking problem. Log the retrieved chunks for every query. Without that trace you are guessing.

Retrieval is evaluated with metrics the IR community settled long ago. Build a small golden set of realistic questions, each labelled with the passages that genuinely answer it, then measure recall at k, which asks whether the right passage made it into the candidates at all, and a rank-sensitive metric like mean reciprocal rank or nDCG, which asks whether it came back near the top. Recall at your retrieval depth is the ceiling on everything downstream. If it is low, nothing else you do matters.

The set does not need to be large to be useful; it needs to be small enough that a domain expert will actually sit down and produce it, and honest enough that it includes the questions you are afraid of. Sourcing them from real user logs rather than imagination is what makes the set representative. Formal evaluation of retrieval systems has been done this way for decades, notably in the TREC conferences run by NIST.

Generation is evaluated differently. The properties worth measuring are faithfulness, meaning every claim in the answer is supported by the retrieved context; answer relevance, meaning it addressed the question asked; and appropriate refusal, meaning it declines when the context does not contain the answer. Libraries such as RAGAS implement versions of these using a model as judge.

Model-as-judge deserves scepticism. Judges are sensitive to answer length and phrasing, they exhibit position bias when comparing candidates, and they tend to agree with confident writing. They are useful for catching regressions across a fixed suite, not for producing an absolute quality number you can put in a slide. Calibrate the judge against a few dozen human-labelled cases before trusting it, and re-check when you change the judge model.

Finally, make the evaluation a suite you run on every change, not a study you do once. Chunk size, embedding model, retrieval depth, rerank depth and prompt are all coupled. Tuning them one at a time without a regression check means every improvement risks silently undoing an earlier one.

Prototypes and production systems fail in different places. These are the ones that recur.

Permissions leak through retrieval. This is the most dangerous bug in the category and the easiest to ship. If the index does not carry per-document access control and every query does not filter on the requesting user's entitlements, your assistant will eventually quote a document to somebody who should never have seen it. Retrieval is a new read path into your data and needs the same authorization as every other read path. Post-filtering the final answer is not sufficient, because the content already entered the prompt.

The index goes stale. Documents change; embeddings do not update themselves. Without an ingestion pipeline that handles updates and deletions, the system answers from a snapshot and nobody notices until it confidently states a policy that was replaced last quarter. Superseded and current versions of the same document sitting side by side in the index is a particularly nasty variant, because both look equally relevant to a similarity search.

Ingestion is an ETL problem in disguise. Parsing is where the ugly work lives. PDFs with two-column layouts interleave text in reading order that makes no sense. Tables flatten into unusable strings. Scanned documents need OCR, which introduces its own errors. Your pipeline needs retries, idempotency, dead-letter handling and observability, because it will be re-run and it will encounter files that break it.

More context is not better context. Once retrieval works, the tempting move is to pass more of it. Research from Stanford on long-context behaviour, often cited as the lost-in-the-middle finding, showed models attend unevenly across a long prompt and can overlook material placed in the middle. Padding the prompt with marginal passages also dilutes the good ones and raises cost and latency. Fewer, better-ranked chunks generally beat more chunks.

The system will not say it does not know. A model handed irrelevant context will still try to be helpful. Explicit instruction to answer only from the provided material, a mechanism for the retrieval stage to return nothing when scores are poor, and required inline citations all reduce this. Citations do double duty: they let users verify claims themselves, which changes the failure from invisible to obvious.

Nobody looks at the queries. The single highest-value operational habit is reading real user queries weekly and grouping the failures by cause. Most teams discover the corpus is missing an entire category of information users assumed was in it, which is a content problem no amount of retrieval tuning will fix.

Beyond the baseline pipeline sits a large collection of techniques. All of them add moving parts, so the useful framing is what each one buys and what it costs.

Query transformation attacks the mismatch between how users write and how documents are written. Rewriting turns a terse or context-dependent question into a fuller standalone query, which matters enormously in multi-turn chat where "what about the second one" is meaningless in isolation. Decomposition splits a compound question into parts and retrieves for each, then merges. Multi-query generates several phrasings and unions the results, trading latency for recall. HyDE, proposed by Gao and colleagues, has the model draft a hypothetical answer and embeds that instead of the question, on the theory that a fake answer sits closer in vector space to real answers than a question does.

Graph-based retrieval builds an entity and relationship graph over the corpus during ingestion, then traverses it at query time. Microsoft Research published a well-known implementation under the name GraphRAG. It targets a specific weakness: questions that require synthesizing across many documents, such as summarizing themes across a whole corpus, where chunk-level similarity search has no mechanism to connect the dots. The cost is a substantially heavier and more expensive ingestion stage, so it earns its place only when those global questions are actually being asked.

Agentic retrieval hands control of the search loop to the model. It decides what to search for, examines the results, judges sufficiency, and searches again or reformulates. This handles genuinely hard multi-hop questions that single-shot retrieval cannot, and it introduces unpredictable latency and cost along with a new class of failure where the loop wanders. Self-critique variants add a step where the model checks whether retrieved passages actually support an answer before generating one.

Caching and routing are the unglamorous techniques that matter commercially. Many production corpora see heavily repeated questions, so a semantic cache over query embeddings can serve a meaningful share of traffic without touching the pipeline. Routing sends different query types down different paths: a metadata lookup for structured questions, full retrieval for open ones, a direct refusal for out-of-scope ones.

The discipline is resisting all of it until measurement justifies it. Every technique here has a failure mode of its own, and a pipeline with six stages has six things to debug when quality drops. Get chunking, hybrid retrieval, reranking and evaluation right first. Most systems that reach for GraphRAG would have been fixed by better chunk boundaries and a cross-encoder.