Haotian Liu

Haotian Liu is a multimodal AI researcher and coauthor of LLaVA, which connects a vision encoder to a language model and trains it to follow instructions about images. He and Chunyuan Li received equal-contribution credit on the original work, alongside Qingyang Wu and Yong Jae Lee. Liu now leads the Omni team at xAI.[3][2]ⁿ¹

On his personal website, Liu describes himself as a core contributor to Grok-1.5V and Grok-2, leader of the vision effort for Grok-3 and Grok-3 Reasoning, and leader of development for Grok Imagine v0.9 and v1. He describes his recent research focus as “building steerable large models.”[2]ⁿ¹

During his doctoral research, Liu developed LLaVA. He completed his Ph.D. at the University of Wisconsin–Madison in May 2024 under Yong Jae Lee, working with Chunyuan Li at Microsoft Research during his doctorate. His computer science doctorate included a minor in quantitative biology. Earlier, he earned an honors bachelor’s degree at Zhejiang University, where he worked with Xiaogang Jin and Fei Wu.[2][8]

Liu and his collaborators introduced LLaVA in Visual Instruction Tuning, first submitted April 17, 2023 and presented as a NeurIPS 2023 oral. They used language-only GPT-4 to turn COCO captions and bounding-box information into image-related conversations, detailed descriptions and reasoning examples. GPT-4 did not directly view the images in this process. The team collected 158,000 instruction-following samples and publicly released the generated data, model and code.[4][3][6]

The original LLaVA joined a pretrained CLIP ViT-L/14 vision encoder to Vicuna through a simple projection matrix. Training first aligned visual features with the language model by updating only that matrix on a subset of CC3M. Instruction fine-tuning then updated both the projection and language model, with separate training for visual chat and ScienceQA.[3]

The team reported an 85.1% score relative to GPT-4 on a synthetic evaluation containing 90 instructions for 30 unseen images, with GPT-4 rating the responses. This was a specific evaluation rather than a claim of general GPT-4 equivalence. On ScienceQA, they reported 90.92% accuracy for LLaVA alone and 92.53% for a scheme in which text-only GPT-4 combined its own answers with LLaVA’s. Liu also participated in the LLaVA poster-session segment of Latent Space’s December 2023 NeurIPS recap.[3][1]

For LLaVA-1.5, first submitted in October 2023 and later published as a CVPR 2024 highlight, Liu worked with Chunyuan Li, Yuheng Li and Yong Jae Lee. They replaced the linear connector with a two-layer MLP, used a 336-pixel CLIP encoder and added academic visual-question-answering data. The authors reported state-of-the-art results across 11 benchmarks, using approximately 1.2 million public training samples and roughly one day of training for the final 13B checkpoint on eight A100 GPUs.[5][6]

A consequential design choice was to make response format explicit. Liu and his coauthors diagnosed ambiguous prompts as one contributor to models overfitting to short answers, even when users requested explanations. They appended “Answer the question using a single word or phrase.” to relevant training questions and fine-tuned the language model. Their qualitative examples showed the resulting model answering a shirt-color question with a single word while providing brief or detailed explanations of a man ironing on a moving car when asked.[6]

The improved baseline retained limitations: LLaVA-1.5 could not process multiple images, higher-resolution inputs prolonged training, and the model still produced hallucinations. The authors also found that correctly answering an object-attribute question did not guarantee an accurate description of that attribute in a longer account of the whole image.[6]

In January 2024, Liu coauthored LLaVA-NeXT with Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen and Yong Jae Lee. The team increased image inputs to four times as many pixels while reusing LLaVA-1.5’s pretrained connector. Its dynamic-resolution approach encoded image grids to preserve local details, alongside a downsampled image supplying global context.[9][6]

The collaborators also changed their OCR training mixture after recognizing that TextCaps shared training images with TextVQA. They removed TextCaps and replaced it with DocVQA and SynDog-EN to better understand zero-shot OCR performance during development. To broaden visual conversations, they collected 15,000 instruction examples from actual LLaVA demo requests, filtered samples with potential privacy or harm concerns, and generated responses with GPT-4V.[9]