Databricks

Databricks builds a data and AI platform where teams prepare and query data, train and customize models, and deploy AI applications under shared governance. Its AI work includes Dolly instruction-tuned models, the acquisition of MosaicML, DBRX, and tools for evaluating and operating agents. The Data + AI Platform combines SQL, Python, and Scala workflows with model services, Unity Catalog governance, and Lakebase, an integrated managed Postgres database.[2][7][10][11][15]

Founded in 2013 and headquartered in San Francisco, Databricks grew out of academic and open-source work. That earlier history supplies context for its combination of distributed processing, analytics, and machine learning.[1][5]

Matei Zaharia started Apache Spark during his Berkeley Ph.D. in 2009. Spark supported distributed data processing, with SQL, machine learning, and graph-processing capabilities becoming parts of the surrounding Berkeley Data Analytics Stack. AMPLab launched in 2011, with roots in the earlier RAD Lab. In a June 2014 account, Michael Franklin described how working open-source systems connected research with users: SparkR began as a class project and expanded after people found and used its code online. Community training and Spark Summit also formed part of this research setting.[4][5]

Co-founder Ali Ghodsi led engineering and product management before becoming CEO in January 2016. Co-founder Zaharia is CTO and an associate professor at UC Berkeley; his work includes Spark, Delta Lake, MLflow, and Dolly.[3][4]

Databricks began developing Delta Lake in 2016. Its transactional metadata layer adds table management and versioning over data-lake files. In their 2021 CIDR paper, Michael Armbrust, Ali Ghodsi, Reynold Xin, and Matei Zaharia proposed the lakehouse as an architecture combining open storage formats, database management, SQL performance, and direct access for machine learning. They argued that keeping a separate lake and warehouse adds copying pipelines, delays, and opportunities for inconsistency. A lakehouse retains open files while adding transactions, indexing, caching, and query optimization; teams still have to build the logic that cleans and curates their data.[6]

A training job might need only three columns and records from a particular period. DataFrame selections and projections can construct a lazy query plan, describing the required data before loading it. At execution, Spark's data-source integration can use caching, data skipping, and storage-layout optimizations to reduce reads. This brings database-style optimization to the data preparation feeding machine-learning libraries.[6]

MLflow connects experiment tracking with this storage layer by recording the Delta Lake table versions used in an experiment. An engineer can recover that data version after the table changes. This preserves the experiment's input data; it does not by itself reproduce the complete training environment.[6]

Instruction tuning trains a language model on example requests and responses. Databricks initially used the Stanford Alpaca dataset for Dolly 1.0 and reported a $30 fine-tuning run. Its launch team subsequently identified the dataset's OpenAI-generated responses as an obstacle to commercial use. For Dolly 2.0, released April 12, 2023, Databricks instead fine-tuned EleutherAI's Pythia-12B on employee-written demonstrations and released the dataset, training code, and model weights for commercial use. The dataset uses the Creative Commons Attribution-ShareAlike 3.0 license.[7][8]

Databricks organized collection around seven task categories, including question answering, summarization, extraction, brainstorming, classification, and creative writing. A contest offered awards to the top 20 contributors and nightly leaderboards. The company reports collecting more than 15,000 examples within a week, then closing the contest over productivity concerns. Demonstrations included a detailed answer about building a campfire; some other tasks used Wikipedia passages as reference contexts.[7][8]

Dolly's repository describes it as “not a state-of-the-art generative language model” and records weaknesses in programming, mathematics, complex prompts, factual accuracy, and hallucination. Its dataset may contain typos and factual errors, reflect Wikipedia biases, and reflect the interests of an employee population that is not representative of the global population. Employee-written responses and Wikipedia reference passages have distinct provenance.[8]

Databricks completed its acquisition of MosaicML on July 19, 2023. MosaicML provided technology for building and deploying large models and had developed the MPT language-model family. Databricks described the acquisition's purpose as enabling customers to train and deploy bespoke models using their proprietary lakehouse data. MosaicML's product remained available for direct purchase at completion.[10]

DBRX, released March 27, 2024, was led by the Mosaic team that had built MPT, with contributors across Databricks. Its mixture-of-experts architecture has 132 billion total parameters and 36 billion active parameters, selecting four of sixteen experts for an input. Routing through selected experts lets the model draw on a larger stored parameter set while activating only part of it. DBRX was pretrained on 12 trillion tokens of text and code; Databricks used Spark and notebooks for processing, Unity Catalog for governance, and MLflow for experiment tracking.[11]

Databricks released base and instruction-tuned DBRX weights under its Open Model License. The license permits conditional use, modification, and distribution, but prohibits using DBRX, its derivatives, or outputs to improve another LLM, except DBRX or its derivatives. It also imposes separate permission requirements on certain very large licensees and incorporates an acceptable-use policy, including disclosure requirements for machine-generated material placed publicly.[11][12][13]ⁿ¹

Agent Bricks launched in beta on June 11, 2025. Users describe a task and connect data; the system creates task-specific evaluation datasets and judges through MLflow 3, then searches optimization techniques such as prompt engineering, model fine-tuning, and test-adaptive optimization. Users can choose cost-optimized or quality-optimized models. Supported task categories at launch included structured extraction, knowledge assistance, text transformation, and multi-agent orchestration.[14]

The launch also introduced Agent Learning from Human Feedback: natural-language guidance can drive changes to retrieval, prompts, vector-database filtering, or the agent's structure. For example, guidance to ignore data before a specified date can become a technical adjustment to the system. This gives domain experts a way to steer behavior beyond a thumbs-up or thumbs-down rating.[14]

By September 2026, Databricks supported third-party model serving, retrieval-augmented generation, tool-calling agents, and multi-agent systems. A supervisor can coordinate Genie Agents, agent endpoints, Unity Catalog functions, MCP servers, and custom agents. MLflow tracing and evaluation track behavior, quality, cost, and latency. External agents can be registered as Agent Services in Unity Catalog, a beta feature that makes them discoverable and controls access through grants.[15]

In a June 2026 interview, Zaharia described a decision to emphasize Genie and specialized systems while continuing substantial model training. Rather than concentrating primarily on scaling a general frontier model, the team was building systems using external models and customized components. He cited document parsing as a specialized workload, with a vision model turning pages into structured JSON.[16]

Zaharia recalled that Databricks engineers were building their own agent workflows and interfaces, while teams building business agents faced similar needs for portability, history, sharing, and security. Teams wanted to reuse one another's applications but lacked a common collaboration server. Omnigent supplied a shared layer across coding agents and custom agents, rather than requiring each team to build those facilities separately.[16]

The open-source Omnigent project combines agent harnesses such as Claude Code, Codex, Cursor, and Pi—the software running an agent's model calls and tool execution. It synchronizes sessions across interfaces, supports shared and forked conversations, runs agents in sandboxes, and applies approval, spending, and tool-use policies. Zaharia described contextual policies that consider session history: an agent allowed to publish to a website might lose that permission after reading confidential documents or taking another risky action. Session state also supports spending caps.[18][16]

Databricks also operates a managed Omnigent server integrated with workspace identity and model gateways. In September 2026 it required workspace preview enablement and a supported region, with separate preview and region requirements for Databricks Sandbox sessions. Its built-in policy handlers and deployment constraints differ from the open project's broader credential and platform options.[17][18]ⁿ²

Zaharia linked open-source decisions to the value of composable libraries and community connectors: Spark could support integrations that Databricks lacked the time to write itself, and an agent layer could benefit similarly from shared integrations and policy libraries. He distinguished that collaborative software from managed infrastructure operations, such as keeping streaming jobs and databases reliable, which require an operating team.[16]

At the MosaicML acquisition's completion in July 2023, Databricks reported serving more than 10,000 organizations and over 50% of the Fortune 500. Its company page, updated in August 2026, reported more than 20,000 organizations, 70% of the Fortune 500, and over 1,200 cloud, software, and consulting partners worldwide.[10][1]