Matei Zaharia

Matei Zaharia is a Databricks cofounder and CTO and an associate professor of EECS at UC Berkeley. His AI work includes coauthoring DSPy and GEPA research and leading the Databricks research team that builds Genie, a data-science agent. In 2026, he described a consequential choice for that work: prioritize systems that make strong models useful through data access and customized components, while continuing specialized model training.[6][1]

Zaharia helped bring coding agents and custom enterprise agents into a shared infrastructure project, Omnigent. In his June 2026 Latent Space interview, he explained how two efforts converged: the developer-infrastructure team had built Isaac, a wrapper around Claude Code and Codex, while his research team was building Genie. Both encountered the need to switch models and harnesses, retain session history, share work, and control access to data. He argued for treating these as the same infrastructure problems and tested prototypes before the team built the system.[1]

Omnigent’s core is a common API across agent harnesses: a session accepts messages and files, returns streams of text and tool-call events, and permits cancellation of a turn. Zaharia explained that maintaining this interface lets developers build their own applications above it without continually adapting to each harness’s API changes. The interface also needed to support hands-on work: he recalled that his first prototype offered only agent chats, and that Reynold Xin asked for a shell to inspect files and tail logs. Shell access was subsequently added.[1]

Zaharia’s control design uses contextual policies that track earlier actions in a session when deciding whether to allow, block, or require approval for later ones. His example separates individually useful permissions from a dangerous combination: an agent might need to read confidential documents or publish to a website, but publishing after sensitive access or risky package installation could expose data. Policy libraries translate low-level events into higher-level operations so rules can address those combinations.[1]

Xin described Zaharia as having spent much of the preceding five years architecting Unity Catalog, Databricks’ data-governance layer; Zaharia confirmed the account. Session tracking also informed spending controls. Zaharia recalled an agent spending $500 while debugging by reading extensive logs, and described setting a $5 session budget that would require permission to continue beyond the limit.[1]

Zaharia favors open source for layers whose value grows through integrations and collaboration. He cited Spark’s composable machine-learning and graph libraries, along with community-built data connectors, as an earlier example of that ecosystem benefit. He distinguished such software from infrastructure services that need an operational team to keep workloads running reliably. For Omnigent, openness also gives teams a working agent application they can customize and lets contributors supply reusable policy libraries.[1]

After releasing DBRX, Databricks collectively chose to focus less on scaling a general frontier model and more on making capable models useful, Zaharia explained in June 2026. Genie exemplified the goal: an agent able to query company data using relevant data and machine-learning libraries. The approach combines external models with fine-tuned or customized components. Training continued, including a specialized document-vision model that turns pages into structured JSON.[1]

Zaharia expects model customization to become easier as stronger base models produce better reinforcement-learning traces and synthetic training data. He described an unresolved practical threshold: when techniques that currently require specialist researchers become simple enough for users to describe a task and supply their own material.[1]

His 2026 emphasis on combining models and infrastructure had an earlier public expression in the compound-AI work he coauthored in 2023–2024.[4][3][1]

Zaharia coauthored the DSPy paper, submitted in October 2023, with Omar Khattab, Christopher Potts, and other collaborators. DSPy represents language-model pipelines as computational graphs of declarative, parameterized modules. A compiler optimizes the pipeline against a chosen metric, replacing reliance on manually assembled prompt templates with a systematic way to tune demonstrations and other module parameters.[4]

In the February 2024 essay “The Shift from Models to Compound AI Systems”, Zaharia and ten coauthors argued that interacting models, retrieval systems, and tools would remain a leading way to improve AI quality and reliability as models advanced. Their reasons included faster iteration on system designs, retrieval of timely information, access controls, and output checks. They also identified an optimization challenge: search engines and code interpreters are not differentiable, so optimizing the whole application requires methods beyond conventional end-to-end neural-network training. DSPy offered one approach by tuning instructions, examples, and other parameters against an application metric.[3]

Before this language-model research, Zaharia’s work addressed distributed computation, cloud storage, and machine-learning workflows. Those earlier projects supply context for his experience building the infrastructure on which AI applications depend.[6][5]

Zaharia started Apache Spark during his Berkeley PhD in 2009. In the 2012 resilient distributed datasets paper, he and eight Berkeley collaborators presented the abstraction implemented in Spark. RDDs addressed iterative algorithms and interactive data mining by enabling fault-tolerant, in-memory computation across clusters. The technical choice was restricted shared memory based on coarse-grained transformations rather than fine-grained updates to shared state. The paper received an NSDI Best Paper award; his Spark dissertation received the ACM Doctoral Dissertation Award in 2014.[6][5]

At Databricks, Zaharia co-developed Delta Lake, bringing transactional guarantees and structured data management to cloud object stores. He also started MLflow, which addressed fragmented machine-learning workflows through experiment tracking, model versioning, and deployment across tools and environments. ACM announced him as the recipient of the ACM Prize in Computing on April 8, 2026, recognizing his contributions to distributed data systems and computing infrastructure.[6]