RAG Is a Hack—and a System Worth Understanding

Finding information and earning trust

A product comparison lists five alternatives, then recommends the one its author sells. Early in the Latent Space conversation with Jerry Liu, swyx uses this Quora example to distinguish finding information from trusting it. Liu’s recommendation work selected answers through embeddings—representations of users and items whose similarity the models learned to increase—and ranking optimized for preferences and engagement. That selection process did not itself explain how to detect disguised advertising.

Liu joined Quora full-time in 2017 after interning there in 2015, drawn to its mission of making otherwise inaccessible knowledge available. During a prolific three-month stretch, he read deep-learning papers and answered questions to learn and explain their ideas. His later educational Twitter posts continued that practice.

His career connected several ingredients of large language model (LLM) applications. After roughly a year at Quora, he worked on deep-learning training for self-driving and computer vision at Uber, then worked closely with Harrison at Robust Intelligence. Retrieval, model architectures, and software engineering would all matter to LlamaIndex, though Liu qualifies the suggestion that his earlier ranking work was already RAG.

From a knowledge tree to a toolkit

LlamaIndex began with a concrete constraint: Liu wanted to feed information into GPT-3, but ran into its context window—the amount of information it could accept in a prompt. Following an October GPT-3 hackathon and a short vacation, he returned to find Harrison building LangChain. His own experiments asked how a language model could navigate more data than fit in one prompt.

The first GPT Tree Index used summarization prompts to build a knowledge tree from the bottom up. The model would organize information, then traverse its own structure. Embeddings were deliberately excluded to test the language model’s capabilities. The system did not work particularly well; its creator considered pre-GPT-4 models unreliable for complex reasoning, while setting aside possible improvements from fine-tuning.

Developer demand shifted the project toward a broader problem: applying GPT-3 to personal data. Around December, work began on a more modular library, embeddings, and practical concerns such as latency, cost, and performance. Data loaders followed around January. Separating stored state from query behavior made room for routing, more complex reasoning, and agent loops.

The company decision took shape around December too. Investor conversations helped displace an earlier video-data idea, and the departure from the previous job came in early March. The opportunity was larger than a particular index: build infrastructure that made language models useful over an organization’s data.

Bigger context does not make data movement free

The existential questions arrived early. Would larger context windows eliminate the need to structure data? Could training absorb the knowledge instead? Would OpenAI build retrieval and orchestration into its own offerings? Retrieval-augmented generation (RAG) addresses the problem by fetching information and placing it in the model’s prompt. Its value depends on latency and cost as well as answer quality.

Enterprise collections measured in gigabytes, terabytes, or petabytes still need infrastructure connecting them to a model. Even with a hypothetical infinite context window, uploading 50 gigabytes for one response would incur network-transfer costs. Selective retrieval lets developers control how much context they supply, adding choices along the cost–performance curve.

LlamaIndex chose Greylock and compressed fundraising to preserve time for building. By the September 2023 recording, the team numbered about six, with hiring focused on strong engineers and active contributors who could develop both the open-source library and a future managed service. Developer education and the move from prototypes to reliable production applications were central to the strategy.

Context capacity also limits the detail available in both input and output. The conversation compares it to Mario’s progression from blocky graphics to detailed 3D: more capacity permits finer informational resolution. But retrieval must supply the right material. The suggestion that 2,000–4,000 tokens could suffice applies to granular retrieval-and-synthesis tasks given relevant context; it is a qualitative judgment, not a universal minimum.

Why RAG is a useful hack

The critique behind calling RAG a hack is precise: keep the model fixed and engineer what goes into its prompt. Retrieval algorithms and question decomposition make existing model APIs more useful, but the assembled components are not jointly learned against the application’s final objective.

Engineers choose how to split documents into chunks, how many matches to retrieve with top-k lookup, and how to represent short- and long-term memory. Those choices may work without being optimal. A learned retrieval algorithm or a new architecture might improve the system as a whole. That remains an open possibility; the immediate case for RAG is its accessibility and useful 80/20 tradeoff compared with fine-tuning a model.

External retrieval also provides two practical points of control: - Source visibility: expose the documents supplied to the model. - Access control: restrict which information a requesting user can retrieve from external storage. Selectively restricting knowledge already encoded in model weights is a harder problem.

The speakers disagree about whether trustworthy citations fundamentally require retrieval. One argues that a direct source connection is necessary because generated citations can be fabricated. Another allows that a future model might memorize enough information to cite accurately, while acknowledging current accuracy problems. The broader design question is how much knowledge belongs inside the model and how much it should obtain through tools.

Can fine-tuning absorb the knowledge?

An informal experiment tested knowledge injection through OpenAI’s chat fine-tuning interface. One participant generated questions and answers from a passage, then trained on those pairs to encourage memorization. The result was disappointing, but the work was limited to an hour or two at night, with no quantitative results reported. It was a failed attempt, not a verdict on whether fine-tuning can add knowledge.

The distinction is between instruction fine-tuning, which teaches responses through question–answer examples, and training directly over a corpus to predict the next token. The latter was suggested as a potentially better route to memorization, though the OpenAI API available at the time did not expose that approach. A proposed future architecture would internalize high-level knowledge through training and use RAG to fill gaps.

The appeal is familiar to programmers: ChatGPT can explain how to define a Pydantic model or produce graph code using knowledge already in its weights. Code Interpreter plays a separate role by executing the code. Extending that experience to custom knowledge faced uncertain methods, potentially high costs, restrictive interfaces, and poorly understood evaluation practices.

RAG’s access-control advantage also needs an implementation qualification. At recording time, LlamaIndex was an orchestration framework without built-in access control. Banks and consulting firms wanted those controls, but initial applications often used public, nonsensitive data and assumed equal access for all users. Getting prototypes working took priority before wider organizational deployment.

Open the three-line abstraction

Calling models without training them lowers the barrier to building applications. Improving their accuracy brings familiar machine-learning practices back into view: evaluation benchmarks, appropriate metrics, parameter tuning, and experiment tracking. This is the reason to build RAG from scratch at least once. LlamaIndex’s three-line quick start is convenient, but conceals decisions developers need to understand when answers disappoint.

A universal configuration is unlikely because useful settings depend on both data and task. PDFs, Markdown, financial filings, and Slack conversations can need different parsers. Summarization and narrowly targeted answers also demand different granularity. Optimizing chunk size alone cannot settle all those choices.

The framework exposes a customizable sequence: 1. Load data. LlamaHub provides loaders for sources such as PDFs, Slack, and Notion. 2. Parse and transform. Split text, attach metadata, and load the results into vector or document storage. 3. Retrieve. A built-in or custom retriever selects relevant material. 4. Synthesize. A response synthesizer supplies retrieved information to the language model to produce an answer. Developers can replace lower-level components while retaining higher-level abstractions. Response synthesis was also described as handling retrieved material larger than one context window, though the walkthrough did not explain that mechanism.

Data structure shapes retrieval. A flat index of embedded chunks supports top-k similarity lookup and potentially keyword search. A hierarchy with explicit relationships also permits traversal between nodes. The organization of the collection therefore determines which retrieval operations are available.

Data agents add reasoning to these workflows. At the simplest level, a classifier routes a query to an action or destination; it need not be an LLM. More complex workflows ask a model to create a query plan or use a loop to break a question into executable steps. OpenAI function-calling loops are one example. The criterion for choosing these mechanisms is how well they perform on the intended data task.

Tune retrieval without rebuilding everything

The embedding model is only one retrieval parameter. Chunking and metadata affect the representations being searched, while retrieval strategies determine how they are used. Fine-tuning the embedding model can change its embedding space and require re-indexing every document—a potentially infeasible operation in a production pipeline.

Query-side adaptation offers another route: freeze the document embeddings and train a linear or other transformation on query embeddings. The speaker credits the idea to Joe from Vespa in a webinar. Keeping stored vectors fixed avoids document re-indexing. The approach was available in LlamaIndex and reportedly produced small gains, though no dataset or metric was supplied.

Some retrieval failures originate in the collection itself. Duplicate and outdated documents require decisions about removal or time weighting, and a parameter change can affect retrieval broadly. LlamaIndex’s stated strategy was to make established ranking and evaluation techniques easier to use, alongside methods tailored to generation, such as decomposing a question before retrieval.

More techniques created a new burden: developers needed to know which ones helped on which tasks. Benchmarking and guidance on technique selection became a near-term priority.

A complete application reveals different bugs

SEC Insights, a full-stack chatbot for analyzing SEC 10-K and 10-Q filings, tested the library through a complete application. Building it exposed bugs and prompted improvements in callbacks, streaming, reliable subresponses, and source propagation. Carrying citations through the system mattered because the interface displayed highlights from supporting sources beside the answer.

The application also demonstrated transparent answers and complex questions comparing multiple documents. After a quiet launch to gather feedback, the team open-sourced it as a reusable template for chatbots over financial or other documents. Selling financial software was outside its intended scope. Detailed component guides and a reusable PDF viewer with citations were still planned.

A managed LlamaIndex service was also still in design. The proposal was a one-line toggle from the library into a service with a pipeline-oriented interface and unspecified enterprise guarantees. An interviewer raised the commercial tension: an easy-to-host open-source project can become its paid offering’s strongest competitor. The response emphasized complementary developer value, with revenue a lower priority that year.

The connector ecosystem showed why customization mattered. Most LlamaHub contributions came from the community. A loader sufficient for getting started could omit API coverage or metadata needed later, leading developers to contribute improvements or write their own. Custom loading meant defining a subclass and populating the flexible document abstraction with the desired content.

Models move; query requirements grow

OpenAI was described as accounting for most model usage, alongside Anthropic and experimentation with Llama 2 and other open models. A recent Ollama integration reflected the growing availability of self-hosting tools. The discussion balanced optimism about competition improving open alternatives against the pull of brands and benchmark performance; it stopped short of predicting fully open models with permissive licenses.

A specialized RAG architecture could optimize retrieval and generation together, but would face a moving baseline. Beating an older general-purpose model would not suffice if developers could assemble a better application around its successor. Modular components offered ease of use, while specialized models promised tighter optimization.

LlamaIndex aimed to remain neutral among storage providers, using standard interfaces while accommodating differences such as metadata-filter support. One promising direction was structured constraints combined with semantic search: a condition analogous to SQL’s WHERE clause narrows the collection, while semantic search finds material by meaning. That combination fits data containing both structured annotations and unstructured text.

Querying a separate SQL database and vector store, then combining their results, was an existing workaround. An integrated system was proposed as more efficient, with pgvector cited as progress in that direction. The desired interface would express structured filters, time conditions, semantic search, and relationships in the data. No preferred query language was established, and a new language might require training the model to use it.

Evaluate the answer, then isolate the failure

The proposed evaluation workflow begins with what the application actually produces: 1. Inspect whether the final answer makes sense for its query, then run basic end-to-end evaluations. 2. Diagnose the failure and select component-specific benchmarks. 3. Improve the implicated component while continuing to assess final answers. Retrieval failures were a frequent cause of poor responses. But good retrieval does not guarantee good generation; it helps isolate one component while the rest is optimized.

Retrieval evaluation is a ranking problem. Given a query and a known supporting context, reward the system for placing that context near the top of the results; penalize low placement or absence. One proposed way to build the test dataset was to have GPT-4 generate questions from supplied contexts, producing question–context pairs.

Using GPT-4 to judge final answers was more contested. Run-to-run variation made the evaluations unreliable, prompting an objection that an unstable judge makes a bad test. The optimistic response looked ahead: better models and judge-specific fine-tuning might provide a scalable alternative to extensive human annotation.

Agents add more failure points. Evaluation can inspect individual model calls or the final response; multiple agents introduce communication and delegation too. The objective matters: evaluating whether a simulation resembles human behavior is different from testing task completion. For an agent asked to send an email, the decisive question is whether it actually sent the requested email.

Memory remains open; build the pipeline yourself

The closing discussion returns to the unexpectedly fast arrival of believable generated text and images. One participant recalls struggling to train image-generation models that produced 32-by-32 images, then contrasts that experience with DALL-E and Midjourney. Personalized memory remains an important unsolved problem: how should an assistant learn information and behavior from ongoing conversations?

One position predicts that memory will become part of the model’s architecture instead of relying on a fixed vector store. The objection is that model weights are unreliable storage. The response emphasizes trainability: a parameterized neural system can be optimized. The specific architecture remains unknown, leaving reliable storage and learnable behavior as competing requirements.

The immediate recommendations are concrete. SEC Insights offers an open-source application template, with more modular components planned. Service integrations for retrieval, evaluations, and traces were under exploration. And the learning exercise is to build RAG from scratch: parse data, split it into chunks, implement top-k embedding retrieval, and examine the prompts and model choices behind the answer. Doing that work exposes the decisions a high-level query interface hides.