Context Engineering for Agents: What to Keep, Retrieve, and Rebuild

The tool loop that consumed half a million tokens

Lance Martin’s early deep research agent followed a simple loop: call a tool, append its result, ask the model what to do next, and repeat. Research tools returned large amounts of text. He reports roughly 500,000 tokens and $1–$2 per run for that implementation, without specifying the model, workload, or token accounting method.

That experience motivates Martin’s distinction in the Latent Space conversation: prompt engineering is a subset of context engineering. An agent must manage system and user instructions alongside the tool results arriving throughout a run. Martin credits Karpathy with popularizing the framing: supply the model with the right context for its next step.

Accumulation creates two problems. One is exhausting the context window, the amount of input the model can accept at once. The other is deteriorating performance before that window is full—what Martin connects to Chroma’s work on context rot. More capacity alone does not settle which results should remain in the agent’s working context.

Offload the evidence, preserve the clues

Context offloading separates stored evidence from the message history sent to the model. Instead of returning a complete tool result on every subsequent turn, store it elsewhere and return a summary or reference that lets the agent retrieve the original. Martin credits Manus’s filesystem-as-memory approach as an influence. A filesystem or an agent runtime’s state object, including LangGraph state, can hold the material.

The difficult part is deciding what stays visible. In [Open Deep Research](https://github.com/langchain-ai/open_deep_research), Martin prompts a summarization model to preserve a comprehensive set of key points from each document. The summary must help the agent recognize when to load the full text. That makes high recall—retaining the relevant points—more important than brevity alone. A summary that omits the reason to reopen a document weakens retrieval.

Martin reports that careful prompting has worked effectively for him, but summarization can require substantial investment. He cites Cognition’s use of a fine-tuned model to compress message history at agent boundaries. Both document compression and agent handoffs must preserve the details that later decisions depend on.

Parallelize reading; coordinate writing

Swyx connects compression to context isolation: give different agents the information relevant to their roles. Martin agrees, while noting the challenge of passing sufficient context across agent boundaries. A sub-agent needs enough information to work; its parent needs enough of the result to continue coherently.

A test-writing sub-agent exposes the coordination problem. How much implementation history should it receive? If it changes application code while writing tests, what must it report back? Cognition’s objection concerns these shared dependencies: independently working agents make implicit decisions that can conflict when their code is assembled.

The useful distinction is between two kinds of work: - Parallel reading: sub-agents gather information, then a central writer synthesizes it. - Shared writing: sub-agents produce interdependent parts of a solution and must coordinate their choices. Anthropic’s multi-agent researcher is described as gathering research in parallel and writing once at the end. This task structure helps explain why successful multi-agent research can coexist with skepticism about multi-agent coding.

This remains a practical distinction, not a permanent boundary. Claude Code already supported sub-agents at the time of the conversation, though tightly coordinated coding changes remained a concern. The participants leave open whether improvements in models will change that tradeoff.

Retrieval can begin with a document catalog

Retrieval determines what enters context in the first place. The coding tools discussed take different approaches. Windsurf is described as dividing code at meaningful boundaries, embedding those chunks for similarity search, combining the results with grep and knowledge graphs, and ranking the candidates. Claude Code and Cline are described as using agentic search: the model explores files with basic tools such as grep, without a code index.

One experiment compared Claude Code and Cursor on 20 LangGraph coding questions, using roughly three million tokens of documentation. It tested three ways to expose that collection: - Search a vector store containing indexed documentation. - Consult a Markdown catalog of document URLs and descriptions, then fetch selected pages with a loader tool. - Attempt to put all the documentation directly into context. The catalog reportedly worked very well: the agent used the descriptions to choose which pages to read.

The experiment’s author subsequently favored the catalog for its effectiveness and simpler maintenance. Claude Code reportedly outperformed Cursor in that April test, but no scores, model versions, or evaluation criteria were given. The result supports trying this simple design on a workload; it does not establish a general ranking of agents or retrieval methods.

A small utility automates catalog creation: crawl documentation, send each page to an inexpensive language model for a description, and compile those descriptions into a file. Its author reports better results with model-written descriptions, without quantifying the improvement. Description quality is a retrieval dependency: the agent relies on those descriptions to decide what to load.

MCP, the tool-access protocol discussed here, provides a way to package such an integration. MCP Doc exposes documentation catalogs and a file-search tool to coding agents. Its developer found that users struggled without suitable prompting. Instructions initially placed in the README should, the developer argues, live in the server and be available to the agent. Effective tool use depends on supplying the instructions along with the tool.

Project-specific servers can also contain their own prompts and resources. A further proposal moves extraction into the server, returning only the useful material to the model’s context. That is an early architectural possibility, rather than an established implementation recipe.

Compact at tool boundaries, keep recovery possible

Compaction reduces the material in working context, often by summarizing it. It need not wait until the context window is nearly full: tool-call boundaries offer an earlier opportunity. Hugging Face’s Open Deep Research implementation is described as expressing tool calls as executable code blocks instead of JSON. A coding environment runs the code, retains bulky raw material, and returns limited or compacted information to the model.

The safeguard is to preserve the originals. Manus and Cognition are cited for warnings about information loss, especially irreversible pruning. The recommended sequence is: 1. Save raw tool observations to storage. 2. Return a useful summary to the model. 3. Keep the original material retrievable. The summary can still lose information; retaining the source makes omissions recoverable.

Failed paths are more contentious. Manus is described as favoring retention so the agent can learn from mistakes. Another view favors removing misleading history and explicitly instructing the agent not to repeat the error. Context poisoning captures the risk: a hallucination remains in history and steers later actions off course.

For tool-call errors, one speaker prefers retention, citing experience with Claude Code using printed failures to correct its behavior. Selective pruning also adds logic to the agent harness, the code managing the agent’s execution and history: developers must decide what to remove and when. These are practical reasons to retain errors, rather than measured evidence that retention always wins.

These recurring problems help explain why context engineering became a useful name. It describes work agent builders were already doing. The conversation draws a parallel with AI engineer: a label connecting employers seeking software engineers fluent in AI with engineers applying prompting and conventional software engineering to AI systems.

Caching lowers cost; memory selects what returns

Agent loops repeatedly submit earlier tool calls as part of their history, making old context a recurring input cost. Manus is credited with emphasizing caching to reduce latency and cost. The participants leave provider-specific behavior unresolved, including when caching is automatic and when explicit controls are required.

Caching does not shorten context. A cached 100,000-token history still supplies that material to the model. The expectation expressed is that long-context quality problems remain, though no cache-versus-no-cache experiment is presented. Provider-specific caching also creates portability concerns; running open models is proposed as a way to control caching directly.

Alessio identifies a harder compaction target: decision rationale. Research can finish with a report, but coding often continues through many changes. In his experience, knowing why earlier choices were made helps the model. Extracting that history efficiently remains unresolved. Devin’s reported use of a fine-tuned summarization model illustrates the specialized effort compaction can require.

Persistent memory introduces two decisions: when to write information for future use, and when to read it back. Either can be more or less automated. Claude Code is presented as a simple design that loads its instruction files each session and saves memories when the user asks. ChatGPT is presented as deciding when to save and retrieve memories behind the scenes.

Remembering something does not make it relevant now. An image-generation anecdote describes a remembered location entering an image without the user requesting it. At scale, reading memories becomes a retrieval problem over past interactions, though ChatGPT’s internal retrieval implementation is left unknown. For long-running agents, the discussion also points toward background memory processing as work extending beyond immediate context selection.

An email assistant offers a narrow, concrete memory-writing mechanism. Before an email is sent, the user can pause the agent, correct its tone, or edit the tool call through an interface. A language model compares those edits with existing memory instructions and revises the instructions. Human review becomes a memory-writing event, turning a specific correction into guidance for later actions.

The workflow that became a bottleneck

The Bitter Lesson brings these implementation choices back to architecture: general algorithms with fewer built-in assumptions can benefit more from increasing data and compute. Drawing on Hyung Won Chung’s argument, the discussion makes a practical distinction: structure can improve performance at today’s compute budget and become tomorrow’s bottleneck. Add what makes the system work, then reassess whether it still belongs.

Open Deep Research’s evolution supplies the example. Its 2024 implementation avoided tool calling because its builder considered it unreliable. A fixed workflow instead planned report sections, wrote them in parallel, and assembled the result. The builder reports that this initially worked more reliably than an agent.

As models improved, that workflow blocked adoption of better tool calling and MCP. Moving to an agent let the model choose its research path. One inherited constraint remained: isolated sub-agents still wrote separate report sections, producing a disjointed document. Removing independent section writing and using one final writing pass addressed the coordination problem encountered earlier.

The builder reports rebuilding the system twice to capture model improvements. The lesson is to reassess assumptions encoded in the harness. A structured approach can perform better for a substantial period and still become the wrong architecture later. Claude Code’s deliberately simple, general design is offered as another approach to exposing improving model capabilities without extensive scaffolding.

The product implications are debated more strongly. One participant argues that incumbents can improve workflows with AI yet struggle to remove them when general approaches become competitive, because the structure is the product. The stronger position is that models are already capable enough to justify starting with little structure. A related argument favors building ahead of current capabilities: Cursor’s improvement with Claude 3.5 is offered as product lore about a model catching up to an application’s needs, rather than a measured causal result.

Choose building blocks you can recombine

That argument creates a test for frameworks: can their abstractions be unwound? Martin distinguishes low-level orchestration—composable nodes, edges, and state—from agent abstractions that hide implementation choices. He used LangGraph to build Open Deep Research’s workflow and then rebuild it as an agent. In his account, those low-level components made the architectural change easier.

Martin values LangGraph’s checkpointing and state management while remaining wary of opaque agent abstractions. Understanding what happens underneath matters: a developer must be able to dismantle and rebuild the system when its assumptions stop serving the model.

Shared components also solve an organizational problem. Martin describes Shopify’s Roast as serving a similar orchestration role: independently built agents and workflows become difficult to coordinate and review. He draws an analogy to MCP’s emergence from proliferating tool integrations at Anthropic around mid-2024. In that account, a common protocol made authentication and review easier while reducing cognitive load.

Martin closes by pairing inspectable software with an educational on-ramp. His free Open Deep Research class walks through notebooks so learners can understand and recreate the system. He also previews forthcoming GPT-5 results, without giving measurements. The invitation is to build the project, understand its choices, and learn how to change them.