RAG

Retrieval-augmented generation (RAG) combines search with generated answers: a system retrieves external information, then gives that information to a language model as context. It lets an application draw on a document collection, product catalog or database without relying entirely on knowledge stored in model weights. The term covers both the trained architecture described in the 2020 RAG paper and today's broader engineering pattern of connecting models to information sources.[2][10]

A straightforward RAG application follows a fixed sequence: receive a question, retrieve relevant material, select context and ask a model to answer from it. Retrieval chooses what the model gets to read. Generation turns that material into an explanation, comparison or response suited to the question.[10]

A vector index is one possible search mechanism, but RAG does not require a new vector database. LangChain's retrieval documentation describes querying existing SQL databases, CRMs and documentation systems directly or exposing them as tools.[10]

For document collections, the preparation step often divides files into smaller passages, or chunks. Two search approaches can help find those passages:[4][6]

• Semantic search represents passages and questions as vectors to find related meanings.[4]
• Lexical search, such as BM25, matches words.[4]

Combining the two can help preserve both conceptual matches and exact identifiers. Retrieval then needs context selection: finding plausible passages is only the beginning of deciding which evidence should reach the generator.[4][6]

Chunking can also destroy information. In Anthropic's Contextual Retrieval explanation, a hypothetical financial-filing passage reports revenue growth but lacks the company and quarter needed to interpret it. The proposed remedy prepends a short explanation of the chunk's place in the document before embedding and lexical indexing. This makes the passage easier to find in its intended context.[4]

Anthropic reported improvements in its retrieval evaluation; those results measure whether relevant material was retrieved, rather than whether generated answers were correct.[4]

Retrieval-assisted language models have several research predecessors. REALM, submitted in February 2020, proposed a learned retriever used during pretraining, fine-tuning and inference. In May 2020, Patrick Lewis and eleven coauthors submitted Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, later accepted at NeurIPS 2020. That paper established a named generative architecture while explicitly situating it alongside earlier retrieval and memory research.[3][1][2]

The original implementation paired DPR, a neural passage retriever, with BART, a generative model. BART received the question together with a retrieved passage. The paper described two ways for retrieved documents to contribute to an answer:[2]

• RAG-Sequence treated the document choice as shared across an answer.[2]
• RAG-Token allowed the contribution of different documents to vary across tokens.[2]

The researchers trained the query encoder and generator while keeping the document encoder and index fixed—a useful qualification to the paper's description of end-to-end learning.[2]

One experiment made external memory's appeal concrete. The researchers exchanged Wikipedia indexes from December 2016 and December 2018, then asked about 82 world leaders whose identities had changed. Accuracy was 70% and 68% with the matching indexes, compared with 12% and 4% when the indexes were mismatched.[2]

Changing the information store changed the answers without retraining the generator. The incomplete accuracy also shows the limit: replacing an index can update available evidence, but does not guarantee that a system will retrieve and use it correctly.[2]

In his October 2023 LlamaIndex interview, Jerry Liu called RAG a “hack” because developers commonly keep a model fixed and manually assemble retrieval and memory settings around its API. Those choices are not jointly learned against the application's final objective. He also emphasized the approach's accessibility and practical usefulness. His argument identifies a real design tension in prompt-based pipelines; it does not describe every trained retrieval-augmented architecture.[5]

Bryan Bischof offered a complementary perspective in a November 2023 interview about Hex Magic: RAG resembles a recommender system with retrieval, ranking and serving stages. A good candidate set still needs good ranking, and the final selection must satisfy downstream constraints. The analogy encourages engineers to treat context selection as a substantive system component, with choices to evaluate rather than a single similarity-search call.[6]

Finding evidence, using evidence and answering well are separate achievements. The RAGAs evaluation paper distinguishes three dimensions:[7]

• Focused, relevant retrieval: whether the system finds suitable passages.[7]
• Faithful use of retrieved passages: whether the answer uses the available evidence faithfully.[7]
• Generation quality: how well the system answers.[7]

These distinctions help locate failures: missing evidence calls for retrieval work, while an answer that misrepresents available evidence calls for work on generation. Automated metrics can support that analysis, but the paper's framework does not by itself establish that automated evaluators replace human assessment.[7]

Retrieval need not happen once, or for every question. Self-RAG, introduced in October 2023, proposes training a model to retrieve on demand and produce reflection tokens that assess retrieved passages and its output.[8]

A different implementation route uses tool-calling control flow: the LangGraph RAG tutorial lets a model choose retrieval or a direct response, grades retrieved documents for relevance, and rewrites the question when another search is needed. Such loops offer additional decisions and retries, with variable execution and latency.[11][10]

Some questions require a different view of the collection. Asking “What are the top five themes in the data?” calls for aggregation across documents, which a few semantically similar passages may not provide. Microsoft's February 2024 GraphRAG explanation describes constructing a knowledge graph, organizing semantic clusters and preparing summaries for this kind of question.[9]

It illustrates how the unit of retrieved evidence can grow from a passage to a summary of relationships across a dataset; Microsoft's demonstrations remain company-provided evidence.[9]

The useful design question is therefore what evidence this task needs, and how the system should obtain it. The designs offer different choices:[10][11][9][7]

• A fixed retrieval step offers predictable control flow, as described in LangChain's retrieval documentation.[10]
• An agent can decide when to search and retry, as illustrated by the LangGraph tutorial.[11]
• Graph-derived summaries can support collection-wide questions, as described in Microsoft's GraphRAG explanation.[9]

Across these designs, external information provides an opportunity for better-grounded answers, while evaluation must still establish whether the system found suitable evidence and used it faithfully.[10][11][9][7]