LMSys

LMSYS (Large Model Systems) develops open models, evaluation tools and systems for training and running AI models. Its projects include Vicuna, FastChat, SGLang and Chatbot Arena, which graduated from the organization. LMSYS Corp. describes itself as a 501(c)(3) nonprofit that incubates open-source projects and research.[2]

Arena’s separation was a consequential organizational decision. In a November 2024 interview, Arena lead Anastasios Angelopoulos described LMSYS as initially a student club and credited Lianmin Zheng and Ying Sheng with creating it. He explained that they had moved on to SGLang, while Arena had become its own project. Decoupling the two would let LMSYS support new projects without its name becoming synonymous with Arena.[1][3]

The work preceding that separation began in 2023. LMSYS dates its origins to a collaboration involving UC Berkeley, Stanford, UC San Diego, Carnegie Mellon and MBZUAI, and its nonprofit incorporation to September 2024. Its early Vicuna and Arena projects supply the context for its model-building and evaluation work.[2][3][4]

The March 30, 2023 Vicuna announcement introduced Vicuna-13B, a LLaMA fine-tune using approximately 70,000 user-shared ChatGPT conversations from ShareGPT. Building on Stanford Alpaca’s recipe, the team computed training loss on chatbot outputs in multi-turn conversations and expanded context length from 512 to 2,048 tokens. Gradient checkpointing and FlashAttention addressed memory pressure; SkyPilot managed spot instances reduced training expense. The team reported one day of training on eight A100 GPUs with PyTorch FSDP, at approximately $300 in training cost.[3]

Vicuna was a joint effort across the five universities, with contributors including Wei-Lin Chiang, Ying Sheng, Lianmin Zheng and Hao Zhang, and advisors Joseph E. Gonzalez, Ion Stoica and Eric P. Xing. Its headline comparison with ChatGPT came from a preliminary GPT-4 judging exercise that the team called “a fun and non-scientific evaluation.” The announcement acknowledged reasoning, mathematics, factual-accuracy and safety limitations. Training, serving and evaluation code was released through FastChat under Apache 2.0; model and demo use carried non-commercial restrictions and additional underlying terms.[3]

Chiang later recalled that evaluating the chatbot was a central difficulty: how could the team establish whether it was better than another model? The demonstration site developed into side-by-side comparisons and then anonymous community voting. The May 3, 2023 Arena announcement described a platform served by FastChat, where users conversed with two anonymous models and chose their preferred answer. Model identities appeared after voting, and the analysis used only votes cast while identities were hidden. Elo ratings translated pairwise outcomes into relative model rankings.[1][4]

Arena’s first published leaderboard covered nine models and 4,700 valid anonymous votes collected during April 24–May 1, 2023. Initially, the team favored matchups it expected to be strong based on prior benchmarks, then switched to uniform sampling for broader ranking coverage. It released voting results without conversation histories, citing privacy and toxicity concerns, and credited MBZUAI for donated compute. Growth was uneven: Angelopoulos recalled declining participation after launch and a period when the project’s prospects were uncertain. He joined a couple of months into the project to work on its statistics.[4][1]

By the June 22, 2023 update, Arena’s leaderboard used 42,000 anonymous votes collected through June 19. The update also introduced Vicuna-v1.3 models ranging from 7B to 33B parameters and added MT-Bench, a curated multi-turn benchmark graded by GPT-4, alongside Arena Elo and MMLU. MT-Bench provided a controlled complement to live user evaluation. The associated study reported over 80% agreement with controlled and crowdsourced human preferences, while identifying position, verbosity and self-enhancement biases and weaknesses in judging mathematics and reasoning.[5]

The January 17, 2024 SGLang announcement introduced a Python-embedded language and runtime designed together for programs with multiple model calls, control flow and constrained generation. Its RadixAttention mechanism retains key-value caches for prompts and generated results in a radix tree. Later requests with shared prefixes can reuse those intermediate computations—for example, common few-shot examples or earlier chat turns. Least-recently-used eviction manages limited GPU memory, while cache-aware scheduling improves reuse.[6]

The SGLang team reported up to five times higher throughput across its tested workloads, using Llama-7B on one NVIDIA A10G and Mixtral-8x7B on eight A10Gs in FP16, against vLLM 0.2.5, Guidance 0.1.8 and Hugging Face TGI 1.3.0. The announcement also reported serving the LLaVA online demo and integration as a DSPy backend. It credited designs and reused code from Guidance, vLLM, LightLLM, FlashInfer, Outlines and LMQL.[6]

In April 2024, Tianle Li, Wei-Lin Chiang and collaborators introduced Arena-Hard, a pipeline that selected difficult benchmark prompts from live Arena data. It processed 200,000 user queries, grouped them by topic and selected 500 prompts from 250 high-quality clusters. Automated pairwise judging swapped answer positions to reduce position bias, producing 1,000 judgments per model evaluation. In a specified top-20 model comparison against English-only Arena rankings, Arena-Hard-Auto-v0.1 reported 89.1% confidence-aware ranking agreement and 87.4% separability, at $25 per model evaluation. The benchmark omitted multi-turn interactions and retained risks from judge bias, judgment variance and prompt selection.[7]

RouteLLM, announced in July 2024 by Isaac Ong, Amjad Almahairi and collaborators, used preference data to choose between a stronger, more expensive model and a cheaper model for each query. It explored similarity-weighted ranking, matrix factorization, BERT classification and causal-LLM classification. Experiments routing between GPT-4 Turbo and Mixtral 8x7B reported cost reductions exceeding 85% on MT-Bench, 45% on MMLU and 35% on GSM8K while retaining 95% of GPT-4’s benchmark performance. Training distribution mattered: Arena-only routers performed near randomly on MMLU, while adding roughly 1,500 MMLU validation examples substantially improved results. The project released code, routers, datasets and a serving-and-evaluation framework.[8]

The August 29, 2024 Style Control analysis by Tianle Li, Anastasios Angelopoulos and Wei-Lin Chiang addressed how presentation affects votes. It added normalized differences in token length and counts of Markdown headers, bold elements and lists to Arena’s Bradley–Terry logistic regression. The authors found length was the dominant measured style factor, and adjusted rankings changed: GPT-4o-mini fell from sixth to eleventh in the overall table, while Claude 3 Opus rose from sixteenth to tenth. The analysis was observational; unmeasured factors could affect both length and substantive quality. Style Control remained an optional leaderboard view in the November interview.[9][1]

Arena also collaborated with frontier labs on private model testing. Asked in November 2024 whether selecting the best of several private variants inflated rankings, Angelopoulos argued that the resulting selection bias was small relative to voter variability and would diminish as fresh live votes accumulated. He acknowledged that Arena’s confidence intervals were not adjusted for multiple comparisons; Bonferroni correction was an available method he described, rather than an implemented policy.[1]

The July 2025 slime announcement extended the SGLang ecosystem into reinforcement-learning post-training. The framework combines SGLang inference, Megatron-LM training, customizable rollouts and Ray resource management, supporting training and inference on shared or separate GPUs and asynchronous execution. The slime Team credited AReal with collaboration on SGLang’s /abort_request endpoint, which terminates ongoing requests and recovers partially generated content for partial rollouts, and AMD’s GenAI Foundation Model Team with day-one hardware support.[10]

By February 2026, SGLang-Diffusion work included token-level sequence sharding, separate text-encoder and denoiser parallelism, distributed VAE operations and fused kernels for video generation. A concrete serving repair addressed Wan2.2’s two-transformer architecture: both transformers had used the total inference-step count for caching, and cache contexts persisted across requests. Different video shapes could therefore produce buffer mismatches and server crashes. The repair configured each transformer with its own step count and refreshed cache contexts for every request. The team credited Skywork.ai, Song Rui and the SGLang-Diffusion Team for these optimizations.[12]

LMSYS welcomes donations of cash, GPUs and cloud credits; its website, retrieved September 18, 2026, lists supporters including NVIDIA, Google Cloud, AMD, Voltage Park and MBZUAI. Financing for the graduated Arena company is separate: on May 27, 2025, the Arena Team announced $100 million raised in seed funding, led by a16z and UC Investments, with participation from Lightspeed, Laude Ventures, Felicis, Kleiner Perkins, The House Fund and others.[2][11]