Retrieval After RAG: Search Built Around Storage, Concurrency, and Cost

A serious attempt, with a deadline

Simon Eskildsen’s commitment to Turbopuffer came with a stopping condition: if the company had not found product-market fit—evidence that customers needed what it was building—by year-end, he would return the funding. He recalls telling investor Lachy that he and Justine wanted to hire, commit fully, and be candid about whether the business was working.

Turbopuffer’s immediate job is search over unstructured data: full-text search for words and phrases, and vector search for similarity between numerical representations of content. Eskildsen frames its broader purpose as connecting models to information outside their weights. Models compress knowledge and reasoning capabilities; retrieval gives them access to the underlying material. At the time of the conversation, search remained the primary reason to choose Turbopuffer.

His thesis for building a large database company has three parts: a new workload with broad demand, a storage architecture that existing databases cannot easily retrofit, and room to support more query plans—the ways a database executes requests. He sees connecting company data to AI as the new workload. The architectural opportunity is to build around fast local storage and cloud object storage, then expand what customers can ask of the data.

The feature that cost more than the application

Eskildsen moved from Denmark to Canada at 18 to work at Shopify. He describes Danish pragmatism and attention to aesthetics as lasting influences. Over almost a decade on Shopify’s infrastructure team, beginning around 2013, his task was to keep the site available while preparing for the following year’s scale. That meant sharding databases and moving from company-operated data centers to the cloud. He recalls celebrity merchandise events reaching a million requests per second in site traffic.

Self-hosted Elasticsearch was the database he found hardest to scale and most frustrating to operate on call. Projects could be held back by difficulty getting the required queries or underlying Lucene search capabilities to work. His criticism concerns the version his team used around 2015; he explicitly allows that it has probably improved at many of those tasks.

After Shopify came “angel engineering”: roughly three-month stints solving infrastructure problems at friends’ companies in exchange for some equity. These included Readwise, Replicate, and Causal. At Readwise, preparations for the Reader launch initially involved Postgres performance work, principally autovacuum tuning. Around ChatGPT’s arrival, the work expanded into recommendations and experiments with large language models.

The recommendation prototype converted articles into embeddings—vectors used to find related content—and recommended reading based on recently read articles. Its developer recalls recommendations about having a child that suggested a Readwise cofounder was expecting one. That striking impression of relevance ran into a simpler obstacle: putting the article embeddings and vector index into production was estimated to cost about $30,000 per month. Readwise’s existing infrastructure, including Heroku and Postgres, cost roughly $5,000 per month in total.

The bootstrapped company deferred the feature. The belief that it would have shipped at one-tenth the estimated cost became Turbopuffer’s founding hypothesis, based on this one experience rather than additional customer interviews. In early 2023, work began on understanding vector indexes and using the Napkin Math repository to reason from hardware bandwidth and storage costs toward a cheaper architecture.

Cheap storage, few sequential waits

The design keeps all data in [object storage](https://turbopuffer.com/docs/architecture), which holds files as individually addressable objects. Accessed data moves onto local solid-state drives using NVMe, or Non-Volatile Memory Express; frequently queried data moves into dynamic random-access memory, or DRAM. The cost model assumes that only a small working set needs these faster tiers. In the proposed one-terabyte example, perhaps 5–10% would reside on local drives and a smaller fraction in memory.

The tradeoff is latency when writing or accessing cold data—data that has not yet reached a faster tier. The initial architectural sketch allows a couple hundred milliseconds for writes and roughly half a second for a first query. These are rough expectations, without a specified benchmark configuration.

To make those waits productive, the database minimizes dependent round trips and maximizes concurrent requests within each round. An illustrative pattern sends about 1,000 requests to Amazon’s S3 object-storage service together, uses the results to decide what to fetch next, and aims to finish in roughly three rounds. The governing constraint is how often one result must arrive before the next request can begin.

A simplified vector index shows how this works. 1. Group nearby vectors into clusters, store a description of those clusters in one object, and store each cluster’s contents separately. 2. A query fetches the description, selects nearby clusters, then downloads their contents concurrently. 3. It can search those vectors locally for the nearest matches. That requires two dependent storage rounds and is described as close to Turbopuffer’s first implementation.

The design still anticipated substantial write traffic. Incoming articles and updates to searchable content can outnumber human-entered queries; the recollection of Shopify’s workload supports that intuition without establishing a precise read/write ratio. Nor was the architecture presented as a claim of historical priority. Neon’s earlier work was acknowledged, with the distinction that Turbopuffer was a new database built around object storage rather than an adaptation of Postgres.

Keeping metadata in the same storage system

Object storage supplies durability as well as capacity. Eskildsen says all Turbopuffer servers could be turned off without losing data. Turbopuffer also avoids a separate consensus service of its own, relying on object-storage consistency and conditional writes to coordinate changes.

Consider a shared metadata file listing the database’s files. Several servers may finish writing data and need to update that list. Compare-and-swap lets each server read the metadata, modify it, and write the replacement only if the original has not changed in the meantime. A conflicting writer rereads and retries. With 100 nodes competing to update the same file, the example anticipates slow progress, though the updates can eventually converge.

Turbopuffer initially ran on Google Cloud Platform because of familiarity and relationships from Shopify. Discovering conditional object writes there allowed the team to defer a separate coordination service such as ZooKeeper. The enabling changes arrived at different times: cloud NVMe availability around 2017, S3 strong consistency in December 2020, and S3 compare-and-swap in late 2024. Google Cloud already offered the conditional-write primitive when Turbopuffer began using it.

That choice became harder to sustain while pursuing Notion in mid-2024. Notion ran on Amazon Web Services, while Turbopuffer wanted to retain its Google Cloud architecture. The team bought a dark-fiber connection between the cloud environments and tuned the network instead of introducing another metadata system to run on S3. Years of on-call experience shaped that preference: outages involving state that no longer agrees across systems were particularly painful.

The Oregon connection was described as routing through Seattle, adding a detour and producing roughly 14 milliseconds of latency. The team moved the exchange to Portland, tuned the Transmission Control Protocol’s windows to send more data per round trip, and prewarmed connections. Small reductions could compound: halving a round trip from 14 to seven milliseconds would allow another exchange in the same time budget. No achieved end-to-end latency was specified.

Turbopuffer absorbed the networking costs. It could afford only one fiber line, so it tested smooth fallback to the public internet. At the time of the conversation, Notion’s workload remained on Google Cloud because the arrangement had proved reliable. The customer’s requirement was latency; Turbopuffer took responsibility for making the deployment meet it.

Cursor makes the economics concrete

Before Notion, the first major customer test was Cursor. Turbopuffer’s initial deployment was a Rust binary running inside the terminal session manager tmux on one eight-core machine. Deployments meant watching the request log, stopping the process during an apparent gap, and replacing the binary. This deliberately minimal setup tested whether anyone needed the product.

Cursor cofounder Arvid reached out, and early exchanges focused on queries per second, current spending, and expected growth. In late 2023, a call with Swaleh led to a visit to the team in San Francisco. Cursor reportedly migrated in roughly one to two weeks and cut the relevant costs by 95%. Justine joined as cofounder around the migration, followed by another month or two of work to make the service dependable for Cursor.

Cursor’s retrieval pipeline divides the entire codebase into chunks and embeds them using its own model. Semantic search finds code with similar behavior or code that performs a requested function, while grep supplements it with text matching. A roughly 25% improvement was reported for one particular evaluation, but without its metric, baseline, or workload; it does not establish a general advantage over text search.

The deployment also used obfuscated file paths and data encrypted with Cursor-controlled keys in Turbopuffer’s bucket. The proprietary embedding model was described as making the stored representation harder to interpret. These measures accompanied a practical retrieval strategy: semantic search helps with particular questions, while text and regular-expression search remain useful.

A tentative analogy, credited to Swaleh at Cursor, treats embeddings as cached computation over a chunk of context. The representation preserves useful work performed before the query arrives. The analogy offers a reason to keep using embeddings alongside other search methods, without predicting how their value will change as models improve.

Agents turn one search into a burst

Early retrieval-augmented generation, or RAG, used search to supply useful material before a model generated an answer. The conversation recalls the constraint as making an 8,000-token context window count. In agent workflows, search increasingly becomes a tool the model calls while reasoning through a task. One user’s agent can issue many searches against a warm dataset in as few turns as possible.

Concurrency alone is insufficient if the requests repeat the same search. An interviewer raises that problem using Cognition’s eight parallel searches as an example. Hybrid retrieval—using different search methods—is suggested as one way to diversify the results. The pattern resembles the storage architecture: do more useful work during each sequential wait.

More searches per task also put pressure on query prices. Turbopuffer announced a fivefold reduction during the conversation to accommodate these workloads, with further reductions an aspiration. Writes still substantially outnumbered reads, though wider adoption of concurrent agent searches could change that balance.

Low pricing initially ran ahead of the implementation. The first prices estimated what an efficient system should cost, including storage amplification—the extra storage required by the database—and a small margin. After Cursor joined, the Google Cloud bill substantially exceeded what Cursor paid. With no funding at that stage, the founders financed the gap and optimized aggressively, initially aiming for even a roughly 5% margin. That pressure was credited with making Turbopuffer profitable by the time of the conversation.

Storage, writes, and queries remained the basic billing dimensions. The business constraint runs through the stack: Turbopuffer must earn a return above cloud costs, and customers must earn a return above Turbopuffer’s charges. Notion’s internal champions evaluated per-user economics as part of a demanding proof of concept. The proposed build-versus-buy test was whether a vendor could save the customer time and work like an extension of its team.

Deployment choices addressed different requirements. - Cursor used the hosted software-as-a-service offering; Notion used a single-tenant cluster dedicated to it. - Anthropic used bring-your-own-cloud deployment, with everything inside its own virtual private cloud, a private cloud network. The enterprise plan included a base fee while detailed pricing for its capabilities was still being worked out.

From a project to a company

Turbopuffer developed from a blog post into a project and then a company. The motivation was to make a technical idea happen with enjoyable collaborators. Accepting investment made that commitment explicit: bringing employees and investors aboard created a responsibility to give the venture full effort.

Fundraising began amid concern about a peer launching a similar product. The investor-selection criterion was unusually personal: someone the founders could call without preparation and speak to honestly. That led back to the promise to return the money if product-market fit had not arrived by year-end. Lachy accepted the framing and helped with candidates and customers, complementing the team’s existing database expertise. He was already involved by the time Notion became a customer.

Operational support arrived early too. Around the twelfth hire, Turbopuffer brought in a full-time finance leader whose work extended into business operations, spanning responsibilities associated with both a chief financial officer and a chief operating officer.

Engineers who can improve—and explain—the limits

“P99 engineer” is Turbopuffer’s internal language for its hiring ideal, rather than a measured percentile. A traits document is reviewed after interviews and during hiring recaps. One described practice is to propose rejecting the candidate regardless of the preceding discussion, testing whether anyone will strongly advocate for hiring them. One committed advocate can be enough; unanimous enthusiasm and excellence on every interview dimension are unnecessary.

The desired traits include a history of making difficult software capabilities work, obsessive interest in a subject, and an ability to articulate tradeoffs. A digression about engineers who enjoy browsing maps is explicitly a joke. A more useful map analogy comes from another participant: developer relations should show a tool’s boundaries, its main landmarks, and a practical route through them.

Turbopuffer’s third version of approximate nearest-neighbor search, ANN V3, supplies a concrete engineering example. It reportedly searched 100 billion vectors at roughly 40 milliseconds median latency and 200 milliseconds at the 99th percentile. Nathan, identified as chief architect, was credited with enabling this particular workload in six to eight weeks with substantial team help. Recall—the share of true nearest matches found—vector dimensions, hardware, concurrency, and benchmark methodology were unspecified, so the figures should be read as a workload-specific result.

The performance method begins with a rough capacity estimate: use machine count, disk capacity, and bandwidth to estimate achievable queries per second, then compare that with the real system. An order-of-magnitude gap points toward software worth investigating. Further optimizations may also reveal mistaken assumptions in the estimate.

That process still has external constraints. Improving software toward a hardware-based estimate differs from reducing the storage provider’s own latency. Turbopuffer’s roughly 100-millisecond write latency is presented as a deliberate tradeoff that makes it unsuitable for high-transaction workloads. Good engineering includes making that boundary clear.

Broader search before a broader database

Vector search was Turbopuffer’s first act; full-text search was its second. The company reported outperforming Lucene on some benchmarks involving very long queries generated or augmented by language models over web-scale datasets such as Common Crawl. No numerical comparison or benchmark configuration accompanied that claim. Expanding the features users expect from full-text search remained a priority.

A hypothetical command-palette query makes the need concrete. Someone typing “si” may be starting to type “Simon,” while an embedding search might associate the input with agreement. Prefix matching and semantic similarity answer different questions about the same characters. Search needs to support the interaction the user intends.

The stated roadmap paired incremental full-text features with cheaper, faster searches across roughly 100 billion vectors or documents. - ANN V3 had been released, V4 was in development, and V5 was planned. - Features described as full-text search V3 would arrive incrementally. - A more capable database console was another priority, with phpMyAdmin offered as inspiration for close integration between the database and its interface.

Possible later directions included simpler analytical queries, traces and logging, time series, and graph capabilities. These were exploratory candidates informed by customer behavior. Turbopuffer was described as a key-value store with additional capabilities exposed above it, and some customers were already using it for graph workloads.

Cursor reportedly moved about 20 terabytes of Postgres data into Turbopuffer to defer sharding, relying on particular query plans already known to work well. That example did not change the adoption rule: choose Turbopuffer primarily when search matters. Auxiliary queries can complement search, while broader aggregations and joins require careful sequencing. Customers will want more from stored data; the risk for the company is trying to deliver too much at once.

An Airtable full of tea

The closing conversation turns to a smaller database: an Airtable tracking roughly 200 teas tried over about 15 years. Its keeper’s favorite is Yabukita Kamairicha from The Green Tea Shop, especially a fresh spring harvest. A travel kit includes a thermometer to control brewing temperature.

The collection also supplies an incentive for colleagues to bring demos to team meetings. When there are too few, unused time becomes a deliberately unrelated presentation. One recent meeting included a 20-minute tour of the tea Airtable and travel kit.

More informal conversation may follow elsewhere: a recent hour on X Live with Sam Lambert of PlanetScale was enjoyable enough to suggest a repeat. The possible show names remained undecided.