Jim Fan

Linxi “Jim” Fan is an AI researcher at NVIDIA who co-leads Generalist Embodied Agent Research (GEAR) with Yuke Zhu. His work concerns agents that perceive their surroundings and take actions in games, simulations and the physical world. He shares equal-advising credit with Anima Anandkumar on Voyager, which accumulates executable Minecraft skills, and Eureka, which uses language models to design rewards for robot learning.[2][3][4][5]

Fan is a coauthor of SONIC, led in the author list by Zhengyi Luo and first submitted in November 2025. The work uses motion tracking as dense supervision for humanoid whole-body control, learning human motion priors without manual reward engineering. Its August 2026 revision describes scaling controllers from 1.2 million to 42 million parameters using more than 100 million frames from 700 hours of motion capture. A unified interface supports both virtual-reality teleoperation and vision-language-action models.[8]

In February 2026, Fan joined Seonghyeon Ye, Yuke Zhu, Joel Jang and other collaborators on DreamZero. Its World Action Model jointly predicts video and actions using a pretrained video-diffusion backbone, allowing heterogeneous robot data to teach physical skills. The team reports more than twice the generalization performance of its tested vision-language-action baselines in real-robot experiments, and closed-loop control at 7 Hz with an optimized 14-billion-parameter model.[7]

Fan is also an NVIDIA-affiliated coauthor of T-Rex, a 2026 collaboration whose equal-contribution authors are Dantong Niu, Zhuoyang Liu and Zekai Wang. T-Rex combines slower vision-and-language planning with faster tactile corrections: a lightweight touch expert refines a cached action plan without rerunning the vision tower. Across 12 real-world manipulation tasks, the team reports a 65% macro-average success rate against EgoScale’s 35%, a 30-percentage-point difference, with 16 randomized trials per task.[9]

These robot-control collaborations followed Fan’s earlier work on browser agents, visual reinforcement learning and Minecraft. That history supplies concrete examples of the agent interfaces and learning mechanisms he worked on before the recent robotics papers.[2][4][6][8][7][9]

Fan recalls joining OpenAI in summer 2016 after friends recommended a new startup, during the interval before his Stanford Ph.D. He worked with mentors Andrej Karpathy and Ilya Sutskever on World of Bits, an agent interface mapping screen pixels to keyboard and mouse control. OpenAI’s December 2016 Universe announcement credits the project to Karpathy, Tianlin (Tim) Shi, Fan, Jonathan Hernandez and Percy Liang. Fan describes himself as OpenAI’s first intern.[10][11][2]

Fan studied at Stanford Vision Lab under Fei-Fei Li from 2016 to 2021. His thesis, Training and Deploying Visual Agents at Scale, is listed in September 2021. His work during those years included the SURREAL distributed reinforcement-learning framework and SECANT, a visual-policy learning algorithm for generalization to new environments. He later credited Li with teaching him “how to develop good research taste”—especially how to identify problems worth solving. He joined NVIDIA in December 2021.[2][12][10]

At NVIDIA, Fan led the MineDojo effort. The collaborative MineDojo paper won a NeurIPS 2022 Outstanding Paper Award. It combined a Minecraft task suite with internet knowledge from narrated videos, wiki pages and Reddit discussions. Its MineCLIP model learns associations between video and language; an agent’s gameplay video can then be scored against a text prompt, providing a signal for whether its behavior matches the requested activity.[2][6]

The 2023 Voyager project took a different approach. Fan and Anandkumar advised a team including Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao and Yuke Zhu. Fan recalls that GPT-4’s coding ability prompted the question, “what if we use coding as action?” The team chose programs over low-level motor commands because code can express extended actions and combine simpler behaviors into more complex skills.[4][10]

Voyager combines an automatic exploration curriculum, a growing library of executable skills and a feedback loop that revises failed programs. Concrete corrections include recognizing that crafting sticks requires two more planks and replacing a nonexistent acacia axe with a wooden axe. Successful skills are stored for retrieval and composition. This accumulates capabilities through black-box GPT-4 prompting and in-context learning, without fine-tuning the language model’s parameters.[4]

Fan and Anandkumar also share equal-advising credit on Eureka, with Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman and Yuke Zhu. Eureka gives a coding language model the environment’s source code and a task description, then asks it to generate reward functions. GPU-accelerated reinforcement learning evaluates candidates; summaries of training statistics guide subsequent revisions. The language model designs the learning signal, while reinforcement learning trains the robot policy.[5]

Its simulated Shadow Hand demonstration first trains a policy to reorient a pen, then fine-tunes that policy toward sequences of spinning configurations. Human feedback can also guide reward revisions without updating the language model’s weights. Across a simulator suite of 29 tasks and 10 robot morphologies, the team reports that Eureka rewards outperform expert-written rewards on 83% of tasks, with a 52% average normalized improvement.[5]

In the November 2023 Latent Space DevDay interview, Fan advocated programming through natural language and refining behavior through dialogue, comparing it to onboarding an employee through conversation. He distinguished this from fine-tuning with a dataset: “It’s programming by data, right?” His optimism about agent interfaces came with a qualification: agent robustness and capabilities were ultimately bottlenecked by the underlying model.[1]

In a Sequoia interview situated in 2024, Fan explained GEAR’s humanoid focus through two considerations: environments and tools are built for human bodies, and human video offers abundant training material. He favored building generalist models first, then prompting, distilling or fine-tuning them for specialized tasks, while acknowledging that narrow specialists are easier to demonstrate in the short term.[10]

Fan’s robotics data strategy combines internet data, simulation and real-robot recordings. In his account, internet video supplies diverse knowledge of objects and activities but lacks motor-control labels; simulation supplies actions and their consequences at scale but introduces physical and visual gaps; real-robot data provides direct grounding but is expensive and slow to collect. He argues that the three sources have complementary strengths.[10]

He also distinguished fast motor control from slower reasoning and planning. His example was the verb “open”: opening a door, bottle or phone requires different motions despite sharing an abstract instruction. In the 2024 discussion, he left open whether a robot should integrate these capabilities in one model or in communicating, cascaded models. His hoped-for robotics breakthrough remained a forecast, with affordability, hardware safety, privacy and regulation additional barriers to everyday deployment.[10]