UC Berkeley

UC Berkeley—the University of California, Berkeley—is a public research university whose AI work includes vLLM, an open-source inference and serving library developed at Berkeley, and DocETL, built by its EPIC Data Lab and Data Systems and Foundations group for LLM-powered data processing. These projects address two practical parts of AI applications: serving models efficiently and organizing model calls into data transformations.[6][4][2]

vLLM launched on June 20, 2023, after two months of deployment at Chatbot Arena and the Vicuna demo. Its central mechanism, PagedAttention, addresses the growing key-value cache that stores attention tensors during token generation. Inefficient allocation wastes GPU memory and limits how many requests a server can batch together.[4]

PagedAttention divides each sequence’s KV cache into blocks holding a fixed number of tokens. A block table maps logically contiguous blocks to physical blocks that can sit anywhere in memory; new blocks are allocated as tokens are generated. Sequences producing different answers from the same prompt can share prompt blocks, with reference counts and copy-on-write protecting shared data. More efficient memory use permits larger batches without changing the model architecture.[4]

The launch authors reported up to 24× the throughput of HuggingFace Transformers and up to 3.5× that of HuggingFace Text Generation Inference. Their experiments used LLaMA-7B on an NVIDIA A10G and LLaMA-13B on an NVIDIA A100 with 40GB of memory, with request lengths sampled from ShareGPT. The Sky Computing Lab’s project record separately reports 2–4× throughput at the same latency against systems such as FasterTransformer and Orca, and lists the PagedAttention paper at SOSP 2023.[4][3]

LMSYS and the vLLM team integrated vLLM with FastChat to relieve the serving bottleneck in the public demos. FastChat supplied the multi-model chat frontend and vLLM the inference backend; Vicuna, Koala and LLaMA had used the integration since mid-April 2023. In the June launch account, the team reported that LMSYS cut the GPUs needed for the described traffic by 50%, while vLLM handled an average of 30,000 requests daily, with a peak of 60,000.[4]

The launch credited Woosuk Kwon and Zhuohan Li with equal contribution, alongside Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Yu, Joey Gonzalez, Hao Zhang and Ion Stoica. The collaboration extended beyond Berkeley: Yu was an independent researcher, and Zhang held Berkeley and UC San Diego affiliations. Zhang received specific credit for the FastChat integration and its accompanying explanation.[4]

Berkeley’s EPIC Data Lab and Data Systems and Foundations group built DocETL to process large structured and unstructured collections with LLMs. Its documentation, as of September 18, 2026, describes natural-language map, reduce and filter operations, parallel orchestration and tabular results. Pipelines can be written in Python or YAML. One example maps over customer interviews to extract themes, then reduces those results into recurring themes across the collection.[2]

DocETL’s documented optimizer can swap models, rewrite prompts, decompose operations and replace subtasks with code. These transformations aim to improve accuracy and reduce cost, automating work that otherwise requires developers to wire together individual LLM calls and tune the pipeline by hand.[2]

Earlier EPIC research supplies context for Berkeley’s work on AI data pipelines. In March 2023, Berkeley databases PhD researcher Shreya Shankar described an unfinished system intended to unify ML development and production. Her proposed execution engine would treat models as regularly recomputed data transformations, monitor changing data distributions and incorporate retraining into pipeline testing. She expressed the idea as: “Models are views that are frequently recomputed over your data to use database speak”.[1]

A February 18, 2024 essay on BAIR’s blog advanced a related systems argument: AI tasks can be handled by interacting model calls, retrievers and external tools. Its authors—Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao and Ali Ghodsi—argued that such compound AI systems would remain important for quality and reliability. They identified a technical challenge in optimizing components such as search engines and code interpreters that are not differentiable, while allocating cost and latency budgets across the system.[10]

These projects sit within an older Berkeley computer science tradition of five-year collaborative research labs. AMPLab closed in fall 2016; RISELab’s January 2017 kickoff announced a five-year focus on Real-time Intelligence with Secure Execution. Its mission was to enable applications to interact intelligently and securely with their environment in real time, combining basic research and software development with open publication and open-source software.[5]

RISELab had 11 founding sponsors: Amazon Web Services, Ant Financial, Capital One, Ericsson, GE Digital, Google, Huawei, IBM, Intel, Microsoft Research and VMWare.[5]

Berkeley formally launched the Sky Computing Lab in the week of May 24, 2022, with Ion Stoica leading a team of more than 60 faculty, staff and students. Named colleagues included Natacha Crooks, Joseph Gonzalez, Raluca Ada Popa, Koushik Sen, Scott Shenker and Michael Jordan. Founding sponsors included Google, IBM, Intel, Samsung SDS and VMware.[6]

Sky proposed intercloud brokers to help users combine compatible services across cloud providers. The design paired a catalog of services, compatibility, pricing and performance with workload intake and an optimizer that selected clouds for a user’s needs. Proposed tools would move or replicate data, allocate resources and execute the plan across clouds. At launch, the team was testing an early broker prototype for machine-learning workloads.[6]

Berkeley’s institutional history reaches back to the original University of California, founded March 23, 1868. It merged the private College of California in Oakland with the state land-grant Agricultural, Mining and Mechanical Arts College, opening with 10 faculty members and 40 students. Classes began at the College of California site in 1869. The university moved to Berkeley in 1873, with close to 200 students, and instituted the PhD degree in 1885.[8][9]

Berkeley’s earlier computing work included Donald O. Pederson’s team developing the SPICE circuit-simulation program in 1972, Bill Joy releasing Berkeley UNIX under the Berkeley Software Distribution name in 1977, and David Patterson directing the RISC CPU-design project in 1981. These were distinct contributions to circuit design, software distribution and processor architecture, preceding the later AI-serving and data-system projects.[9]

UC Berkeley reported $965 million in total FY25 research awards, calculated on a project-period basis, for the fiscal year ending June 30, 2025. This is an institution-wide award total rather than an AI budget or annual expenditure. The university reported that federal funding supplied about half of the total, state and other governments 10%, and nonprofit partners 25%; NASA, the National Institutes of Health and the National Science Foundation were the largest federal contributors.[7]