Jeff Huber

Jeff Huber is a founder of Chroma, which builds retrieval infrastructure for AI applications. In his August 2025 Latent Space interview, he spoke as its CEO about the team's decision to prioritize developer experience over a quick hosted release, and advocated context engineering and small, carefully labeled datasets as practical foundations for reliable AI systems.[1]

Huber traced Chroma's origins back to conversations in approximately 2021–2022, after years of work in applied machine learning. Demos were easy to build, he recalled, but making them reliable in production was difficult: improving a system could feel like stirring a pot and hoping it worked better. The team's objective was to make that transition feel more like engineering. Search became its immediate focus because it was a key workload for AI applications.[1]

Reflecting on earlier startups, Huber said he had made trade-offs involving colleagues, customers and pride in the technology. By 2025, he emphasized doing work he loved with people he enjoyed working with, serving customers he wanted to serve, and getting that work widely used. For him, impact required adoption as well as invention.[1]

Huber said the team could have quickly offered hosted single-node Chroma, but chose a longer development path because that service would not meet its standard for developer experience. The benchmark was the simplicity of local onboarding: install the package and begin using it. For the cloud, he wanted developers to avoid configuring node sizes, sharding, backups and data tiering—“zero config, zero knobs to tune”—alongside usage-based billing. Those criteria motivated the design of Chroma Distributed, the infrastructure behind Chroma Cloud.[1]

That emphasis on craft also shaped Huber's leadership. He described deliberately slow, selective hiring and treated consistency across the API, website, office and interview process as something a founder must maintain. He credited talented designers for Chroma's design work rather than claiming it as his own. He also said he had started the SF Systems Group the previous year to give people interested in systems a meeting place; it was still running at the time of the interview.[1]

Huber criticized “RAG” as an abstraction that bundled retrieval, augmentation and generation together and had become associated with a narrow recipe of dense vector search. He preferred context engineering: deciding what information belongs in an LLM's context window at each generation step. His definition distinguished an inner loop—constructing the context for the current step—from an outer loop that improves the relevance of that information over time.[1]

He said Chroma's Context Rot research began while the team was investigating whether access to prior successes and failures could help agents learn. Growing multi-turn conversations exposed instructions that were present but ignored. Huber cautioned that the report's comparative findings applied to its particular task, rather than establishing a universal ranking of models. His own preference was a hypothetical 60,000-token window with reliable attention and reasoning over a nominal five-million-token window.[1]

Huber described memory as a benefit that context engineering could provide: selecting the right information allows a system to use prior interactions. He argued that feedback about retrieval quality could also guide what the system should remember. He allowed for other mechanisms, including improving the model through reinforcement learning, rather than treating context changes as the only route to memory.[1]

Huber explained indexing as a trade-off: slower ingestion buys faster queries. For some code-retrieval applications, he suggested generating natural-language descriptions of code before embedding it, and extracting metadata during ingestion to simplify later searches. He stressed that the right approach depends on the application; regex or vector search alone may be sufficient.[1]

To choose among those approaches, Huber advocated small golden datasets pairing queries with the chunks that should be returned. He described Chroma's Generative Benchmarking work as teaching an LLM to produce useful queries from existing chunks, supplying the missing half of those evaluation pairs. Retrieval quality should then be weighed alongside cost, speed and API reliability. His concrete advice to customers was to order pizza and spend an evening labeling data together: even a few hundred high-quality examples, he argued, could be valuable.[1]

Huber compared Silicon Valley's expectations of AGI to a religion, pointing to promises that abundant, free intelligence would solve humanity's problems, including death. He expressed skepticism toward such recently emerged belief systems: “I'm always skeptical of religions that haven't been around for more than five years.”[1]