Ethan He

Ethan He is an AI engineer who worked on NVIDIA’s Cosmos world model, helped build Grok Imagine at xAI, and led a small team pursuing real-time, long-horizon video generation. His central public argument is that language models and agents increasingly supply the intelligence behind video generation—a view he connected to his decision to leave xAI and pursue language-model research in an interview published June 1, 2026.[1]

He moved from NVIDIA to xAI in mid-2025. He said building Cosmos had convinced him that video models followed scaling laws similar to language models and needed more compute. At xAI, a few engineers built and released Grok Imagine 0.9 in three months, according to his account. The new video effort began without its own infrastructure, data or model, but he credited xAI’s existing data and model inference foundations, close collaboration among engineers, and his previous experience building Cosmos with enabling the pace.[1]

He emphasized iteration speed: acquiring data, trying an algorithm, training a model and evaluating whether it improved on the previous version. Compute and infrastructure shortened that cycle and made more mistakes recoverable. In his experience, many of the largest quality gains came from finding small bugs in data and training pipelines. He also argued that better coding models could make compute a bottleneck again, because researchers could implement ideas in hours rather than weeks.[1]

He shifted from video-model training toward post-training, including video extension and reference-to-video. From his video-domain perspective, a world model means real-time, interactive, long-horizon video. These requirements create competing demands: compressing several frames together reduces sequence length by exploiting temporal redundancy, while processing frames individually permits more immediate responses to user input at the cost of longer context.[1]

Under his small world-model team, video extension became an intermediate step toward longer interactions. He described Grok Imagine’s extension as retaining historical context from all previously generated clips, including speakers and objects. That addresses a weakness of extending a video from its last frame alone: earlier events and voices can fall outside the conditioning context. Keeping the entire history, however, rapidly increases the number of video tokens.[1]

He also helped build reference-to-video, which he described as accepting up to seven conditioning images of characters, objects or scenes at the time of the interview. References offered an intermediate way to supply relevant information without carrying every intervening frame. His longer-term proposal was for a model to select its own references—for example, retrieving a character’s earlier appearance when that character returns later in a movie.[1]

He argues that much visual intelligence comes from language. A prompt rewriter turns a short user request into the detailed scene description a video generator expects, supplying decisions about backgrounds, motion and composition. He recalled generating a “happy sheep” with Cosmos: rewriting the prompt markedly improved the result without joint training of the language and video models.[1]

His vision of a video agent extends this process into iterative creation, combining generative models with Photoshop, video editors and FFmpeg. Deterministic tools can handle precise operations, such as placing text at a particular timestamp. As he put it in the interview, “you don’t have to put all of the capabilities into the generation model itself.”[1]

He said he left xAI to pursue research he could not undertake there, particularly on language models, while company priorities could change quickly. He identified language and agents as the bottleneck he wanted to address. His proposed directions included models that understand and manage their own context—adding, removing and compacting information—and models that choose or modify their agent harness at test time. These were research proposals, rather than capabilities he claimed to have shipped.[1]

Before Grok Imagine, He’s work included both video generation and language-model scaling at NVIDIA. That earlier work provides context for his move between the two domains.[1]

Looking back in the 2026 interview, He recalled researching image recognition, object detection, tracking and neural-network compression with ResNet authors Xiangyu Zhang and Jian Sun roughly ten years earlier. He had wanted to become a professor, but said rejection from the top PhD programs he applied to led him into industry. At Facebook AI Research, which he described as led by Yann LeCun, he moved into self-supervised learning.[1]

At NVIDIA, he worked on Cosmos and mixture-of-experts models. Explaining Cosmos, he described detailed human annotations that bootstrap video captioning, followed by synthetic text–visual pairs. Image models provide a cheaper foundation for learning text–visual relationships before video training; VAEs compress images and videos into latent tokens, and diffusion transformers learn to remove noise from those tokens. He also noted a small unlabeled-data component. He explicitly declined to describe xAI’s specific training process, so this explanation concerns Cosmos and general methods.[1]

He is first-listed among ten authors of Upcycling Large Language Models into Mixture of Experts, first submitted October 10, 2024 and revised June 16, 2025. With Abhinav Khattar, Ryan Prenger and seven other coauthors, he studied converting pretrained dense models into sparse MoEs. The team proposed “virtual group” initialization and weight scaling for fine-grained experts, and reported benefits from applying softmax before top-K routing. In their comparison using the same one trillion training tokens, upcycled Nemotron-4 15B achieved 67.6% MMLU, versus 65.3% for continued dense training.[4]