The Vector Database Category Gives Way to Search

When an example becomes an architecture

After ChatGPT’s launch, developers looking to connect it to their own data encountered cookbook examples built around embeddings. The guest in this Latent Space conversation, published May 1, 2025, argues that those examples encouraged an unnecessary leap: retrieving context required vector embeddings, and building an AI application required a vector database. The host had helped write that playbook, contributing the Chroma example to the OpenAI cookbook after investing in Chroma before it became a vector database.

Retrieval-augmented generation (RAG) connects a model to information retrieved for its task. Embedding similarity is one way to select that information. The companion article traces the specialized vector database category to a real infrastructure need: storing, indexing, and searching high-dimensional vectors at scale. It identifies Pinecone among the companies leading that category in 2022–2023, while arguing that growth after ChatGPT’s late-2022 launch partly reflected confusion between a retrieval method and retrieval itself.

Vector search spreads beyond vector databases

The guest’s central claim is that feature convergence weakens the separate category, without implying that its vendors will disappear. PostgreSQL offers vector search through pgvector; established search engines including Elasticsearch, Solr, and Vespa offer it too. When an existing system already provides vector search, another database needs an application-specific justification.

Competition continues within that changing market. At the time of the discussion, the guest interprets Pinecone’s messaging as a return toward developers and describes TurboPuffer as an emerging competitor with a similar software-as-a-service model, different pricing, and developer-oriented communication. Their concern is that vector operations alone are too narrow to sustain a durable category identity.

The broader interface is search as a tool for a reasoning model. The guest points to Windsurf’s Cascade mode: alongside file editing, it exposes codebase search, web search, and grep. A model can formulate queries and choose an appropriate search operation, leaving keywords, semantic similarity, and graph exploration as implementation choices beneath that interface. The host likewise describes Chroma’s ambition as information retrieval beyond the narrow RAG use case.

Similarity is one signal in search

The guest remains enthusiastic about embeddings: neural representations of data as vectors. They can represent multimodal inputs and be adapted to a particular domain. Post-ChatGPT adoption and OpenAI’s embedding APIs helped bring techniques already used at large technology companies into mainstream developer practice.

A similarity score, however, leaves out other reasons a result might be useful. For web search, the guest names freshness and authority: ranking an embedded web solely by semantic similarity would omit both. The host favors hybrid queries incorporating metadata, then asks where that richer search should live—in the primary database or in a separate index containing copied documents?

The guest revises their earlier criticism of pgvector, acknowledging its subsequent additions to indexing and support for half-precision and binary vectors. Their infrastructure recommendation depends on the workload: - Keep search in Postgres when its capabilities fit. If the data already lives there, scale is moderate, and the database serves other workloads, keeping vector search alongside them can make sense. - Consider a dedicated search engine when search quality drives the business. That requirement warrants evaluating a broader retrieval system and representing the data there.

Large-scale recommendation offers a useful parallel: a retrieval-and-reranking cascade. Embedding-based retrieval selects candidates from a large pool, then successive ranking stages narrow the selection presented to the user. The guest gives roughly 100 final candidates as an illustration, not a prescribed size. As search adopts embeddings, it shares more of this machinery with recommendation.

Build from clean data to a retrieval baseline

Asked where to start, the guest offers a conditional sequence for developing a RAG application rather than a mandatory progression through database products: 1. Inspect and clean the source data. Getting usable content out of PDFs and other inputs comes first. 2. Establish a BM25 baseline. This classical keyword-matching method provides a starting point for comparison. 3. Try embeddings and hybrid search. Use an off-the-shelf embedding model and explore combining keyword and embedding retrieval. 4. Add reranking if the budget permits. Evaluate the additional stage against both latency and cost constraints.

Deployment choices also depend on which work must happen while a request waits. The host recommends batching work outside the request’s critical path, clarifying that this is what they mean by offline. The guest illustrates the inference tradeoff with an external embedding API returning JSON floats: - At thousands of queries per second, they prefer faster local inference without the external request dependency. - At lower scale, an API service can make the application much easier to build.

Keeping vector search in a database does not settle where inference should run. Inference and database workloads have different scaling properties, the guest argues. A large SQL statement that transforms data, embeds it, and writes it back can obscure external calls and make cost and performance harder to control. They prefer connecting separate infrastructure. The host sees a less clear-cut boundary, citing a cron scheduling service in the Supabase ecosystem as functionality gathered around the database.

Long context changes the small-corpus decision

The guest expects retrieval to remain useful even as the vector database category weakens. But long context can eliminate retrieval for some small collections by letting the model receive the material directly. Two examples establish the conditions: - One PDF with visual information. Passing the document into context may be sufficient, although high query throughput could change the choice. - A particular collection of about 300 articles. The guest recounts advising someone that their collection would fit in a Gemini model’s context window without a vector database. With no article lengths or specific model given, this is a case-specific judgment, not an article-count threshold.

Some RAG demos still embody early-2023 designs built around 4K or 8K context windows, the guest argues. Larger windows can make parts of those designs unnecessary. Yet corpus size and repeated query workload still matter: the guest describes a COVID-related corpus as roughly 170,000 documents and 36 million tokens, illustrating why loading an entire collection for every query can remain unattractive.

Graph retrieval starts with building the graph

Graph RAG retrieves information using a knowledge graph of entities and relationships. Graph databases can accelerate traversal—following edges between connected data—but the guest identifies a harder prerequisite: constructing the entities and relationships in the first place. Choosing a storage engine does not solve that problem.

A search engine can also support graph exploration, so using graphs does not inherently require a dedicated graph database. The guest allows that graph retrieval or a hybrid approach may outperform vector retrieval in some cases; this is a possibility rather than a measured comparison.

The host remains interested in validating knowledge graphs’ value for language-model applications. Both see promise in using large language models (LLMs) to generate entity–relationship triplets—the relationships that populate the graph. Generating those triplets has been a construction bottleneck. Making it easier could unlock more experimentation, while the application-level payoff remains unresolved.

Richer embeddings need a sustainable business

The closing technical discussion returns to better representations. The guest praises Voyage’s domain-specific embeddings and Jina’s work, particularly in European languages, while identifying two directions for further improvement: - Represent documents for their domain. Models could produce one or multiple embeddings for PDFs tailored to legal, finance, or health applications. - Embed page screenshots directly. Using vision-language models as the foundation for embeddings could bypass OCR and simplify complex processing pipelines. The guest expects richer page representations from this approach, without presenting benchmark results.

The economics are less certain. An embedding API provider needs efficient batching and enough revenue to cover compute, while customers’ willingness to pay remains unclear. The guest sees pressure to move toward enterprise search or other higher-value offerings. Useful embedding technology can persist while the products built around it broaden their scope.

The guest closes by inviting listeners to connect on X. That prompts a smaller disagreement about where ideas circulate: the host calls X an echo chamber and describes trying to grow on LinkedIn and YouTube. The guest values X’s AI community and points out that their connection there made this conversation possible.