From Video Models to Video Agents

Three months to a first video model

When Ethan He joined xAI in mid-2025, its video effort had no dedicated infrastructure, dataset or model. A few engineers released Grok Imagine 0.9 three months later. He had already spent about a year building Cosmos at NVIDIA, a video foundation model intended to simulate the world for robotics development. Seeing video models improve with scale motivated his search for more compute.

The new effort could draw on xAI's broader data and inference infrastructure. He describes strong engineers working closely toward one goal, with roughly a daily sync and little other meeting overhead. The resource to maximize was completed research cycles: acquire data or design an algorithm, train a model—perhaps at smaller scale—and evaluate it against the previous version. Faster cycles allowed more failed attempts and more opportunities to find bugs.

Some of the largest quality gains, He says, came from fixing small bugs in data and training pipelines. Coding models became especially useful to him by December 2025, although quick implementations could leave thousands of lines that neither he nor the model could maintain. As those tools improve, he expects compute to become a bottleneck again: shortening implementation from weeks to hours, in his illustrative comparison, leaves more experiments ready to train. Sustaining that pace requires automation, good health and a regular schedule.

Teach the model what the camera sees

The training explanation uses Cosmos examples; the guest declines to disclose xAI's recipe. The first problem is supervision. A mountain video might have a title about its uploader's happiness rather than the visible scene. A vision-language model, or VLM, supplies the missing description by captioning the media. These text–visual pairs are synthetic because their captions are generated, while the underlying videos already exist.

Bootstrapping the captioner requires detailed human descriptions. Cosmos labelers were asked to cover objects, characters, interactions and dialogue, with enough detail for someone unable to see the scene to reconstruct it mentally. Learning from interspersed images and text provides another source of capability; the respondent considers both useful. Generative training also commonly includes a small proportion of unlabeled media, teaching generation without text conditioning to help generalization.

The next constraint is representation. A 1,000-by-1,000 image contains a million pixels. A visual tokenizer, or compressor, maps pixels into a smaller continuous representation called a latent space and decodes that representation back into an image. The described encoders operate on patches and can use convolutional networks or transformers. Their purpose is to give the generator a compact representation it can learn efficiently.

A diffusion transformer learns to remove noise from those visual latents while using language tokens as conditioning. Training adds random noise and asks the model to recover clean visual tokens. Inference starts with noise and repeatedly denoises it before decoding the result into pixels. Image pretraining supplies broad language–visual associations relatively cheaply, then provides a foundation for video training. Each video example contains more tokens, so learning comparable associations through video alone costs more.

Compression trades context length for responsiveness

Storage compression does not guarantee a learnable representation. Attempts to train directly on MP4-compressed representations are described as difficult for transformers to learn. Learned visual compressors produce more continuous latent spaces, although their designs also vary in learnability.

Video adds a temporal compression opportunity because adjacent frames share much of their content. Holding spatial compression fixed, the discussion compares two choices: - Group frames across time: an 8×8×4 scheme combines four temporal positions into one, shortening the sequence but introducing lag. - Compress each frame separately: an 8×8×1 scheme needs four times as much context in this example, but supports responding as individual frames are produced. Removing temporal grouping improves the opportunity for responsiveness while moving the computational burden into longer sequences.

Interfaces generated directly as pixels

[Flipbook](https://www.flipbook.page/n/43e8c7b08ab14571810fee265c331cb3) generates browser-like pages with an image model. In the demonstration, participants explore how the Great Pyramid was built and select topics such as levering techniques. Each click requests another generated page within an imagined environment rather than retrieving an existing website. The live demonstration appears to skip frames; heavy usage is suggested as a cause, but not established.

The proposed destination is user intention directly to pixels. Language and coding models would work behind the scenes while a diffusion model generated the interface. Email could become a swipeable feed; an easily misclicked Like button could disappear. A participant's shorthand of a diffusion front end and deterministic back end needs a qualification: code generation itself is not deterministic. The intended distinction is generating code once and then executing it.

That vision has several conditions: - Serving cost: assuming an H100 costs $1 an hour, eight hours of daily use for 30 days costs $240 a month. This illustrative calculation motivates the prediction that cheaper compute could make such interfaces viable within roughly two years. - Cost at a fixed capability: smaller and better models can reduce inference expense beyond hardware savings alone; the two trends should be distinguished. - Accessibility and preference: screen-reader support remains unresolved. Spoken requests with visual responses are proposed as a high-bandwidth interaction pattern, but participants also emphasize that some users prefer text.

[Neural OS](https://neural-os.com/) simulates an operating system through generated video, including experiences resembling Firefox and Doom. One participant finds it faster and more consistent than an earlier image-model shooter demo. Another sees a useful glimpse of a possible future, with limited visual quality and a model characterized as overfit to existing operating systems. Broader training on screen recordings is proposed as a route toward new interfaces; the demonstration does not establish that generalization.

Training costs include data movement; inference benefits from fewer steps

One billion videos averaging five megabytes require five petabytes before derived features. Stored features from the visual compressor can occupy a comparable amount of space. Video training therefore needs a budget for storage and repeated transfers alongside GPU hours. The cloud bills quoted in the conversation are rough estimates without a complete workload or pricing configuration.

Owning storage near the GPUs could reduce cloud costs, but GPU facilities may not have been designed for video-scale storage. Data loading can leave accelerators waiting. Cosmos reportedly required substantial caching and input/output optimization to avoid that bottleneck. The broader training cost comparison is with medium-sized language models: large parameter counts, tens of trillions of visual tokens in the recalled Cosmos account, and potentially less efficient infrastructure.

Inference offers a more direct saving: step distillation teaches a student to generate in fewer denoising steps. In the illustrative comparison, a teacher takes 100 steps and a student learns to generate in ten. The speaker's intuition is that imitating a fixed teacher is easier than modeling the full complexity of internet media. Cosmos is recalled as having four- and eight-step variants, while Cosmos Transfer supports one-step generation for simpler image-to-image tasks. Step counts alone do not establish equal quality or wall-clock speedups.

Consistency models, distribution matching and adversarial objectives can be combined for few-step generation. In the adversarial setup, a generator produces an image and a discriminator judges whether it looks real. The speaker argues that producing a plausible image can be easier than reconstructing a particular training image—an intuition for the objective, rather than evidence that one method dominates.

Audio and video must agree in time

Grok Imagine 0.9 is described as generating audio and video jointly. That requires cross-modal alignment: learning which sounds belong with which visual events. The speaker argues that strong visual understanding does not automatically supply detailed audio understanding. Audio captioning needs both human and synthetic data, describing beat, tone, music and dialogue precisely enough to convey how they sound.

An overall caption is insufficient for synchronization. Text can describe a video's contents without placing every event at a particular second, whereas audio and video tokens need time-based correspondence. The discussion also challenges reducing speech to text plus a few characteristics: disfluencies carry information that such a simplification misses.

This leads to a broader question about models' sense of time. Implausible task-duration estimates could reflect weak temporal awareness, or the reproduction of human estimates learned from text despite a model's different execution speed. A separate conceptual proposal makes world models recursive: they should represent that their own understanding can be wrong and needs updating.

A world model must respond and remember

The respondent offers a personal, video-centered definition of a world model, with three requirements: - Interactive: respond reasonably to keyboard, mouse and potentially voice input. - Real-time: meet the application's latency needs. Competitive gaming illustrates sub-10-millisecond demands; a digital human receives a more forgiving target of roughly 200 milliseconds. - Long-horizon: sustain minutes or hours of interaction rather than a few seconds. These are proposed requirements, not measured capabilities. Even the more forgiving latency target is difficult because removing temporal compression expands the sequence the model must process.

Video extension provides a useful intermediate product. Feeding the last frame of one clip into the next can work briefly, but omits preceding motion and events. A short video window preserves more information yet can still lose important history. The respondent recalls quality degrading over repeated extensions and voices changing when earlier dialogue falls outside the conditioning window.

[Grok Imagine](https://grok.com/imagine) extension retains context from previously generated clips, according to the account, including speakers and objects. How that history is represented is not disclosed. Keeping every visual token would quickly become expensive: the respondent recalls roughly 50,000–60,000 tokens for five seconds of Cosmos video, implying hundreds of thousands for tens of seconds and potentially millions for longer histories.

Another participant proposes solving the functional problem first, then optimizing serving through techniques such as distillation. The response identifies savings beyond pixel compression: history need not remain active when it is irrelevant. A character who disappears after the opening scene need not occupy context throughout the intervening video; its appearance becomes useful again when the character returns.

Several approaches supply that history more economically: - Manual references: a separate reference-to-video feature accepts up to seven images of characters, objects or scenes. - Retrieved references: the envisioned model selects its own references, retrieving a character's first appearance when needed later. - Compressed older history: Frame Pack is cited as retaining recent history in detail while shrinking older representations to keep a fixed maximum sequence length. Manual references and compression heuristics are intermediate approaches. Automatic selection is the larger research goal.

Language agents face a related problem. A long tool result can remain in context after the subject changes; the surrounding software, or agent harness, compensates by pruning outputs or returning only part of a file. Selecting context differs from attending over existing context. Setting sparse attention aside, ordinary dense attention does not itself decide what history to discard or retrieve. The discussion treats that higher-level decision as a shared research problem for language and video models.

Scheduling research and judging generated media

The account of xAI's culture combines speed, ambitious goals and first-principles reasoning. Build the schedule from limiting operations: video acquisition speed, end-to-end training time, the benefit of more GPUs and human-data turnaround. The respondent also describes hands-on interaction with Elon, while clarifying that voice mode belonged to a different team.

On generated-media safety, a speaker reports watermarks in certain countries and rapid takedowns, without specifying countries or timing measurements. Participants describe their own detection cues shifting from obvious anatomical errors toward mismatched or repetitive audio, implausibly polished lighting and skin, and failures of logical coherence. These are personal cues, not validated detectors. Adversarial training helps explain their fragility: a generator trained against a realism judge is being optimized to remove visible tells.

Language turns a sparse request into a scene

The central thesis is that much of further video improvement may come from language intelligence. Prompt rewriting provides the concrete mechanism. In the Cosmos setup described, a separate language model expands a short request into a detailed video description. That language model is recalled as larger than the 7B video generator.

This follows from the captioning pipeline: training descriptions are detailed, while user requests are sparse. A prompt upsampler fills that instruction gap. The illustrative failure is a request for a cat producing a stationary animal against a white background because neither action nor setting was specified. A later recollection describes a happy-sheep prompt becoming much more appealing after rewriting, even without joint training—a subjective example of language-side improvement.

Prompt preparation can become an agent workflow: an image about today's news might require fetching stories, digesting them and planning a layout before generation. The discussion distinguishes three architectures: - Separate language and diffusion models: one prepares instructions for the other. - A language model with a diffusion head: language reasoning feeds a component that generates images. - Discrete image-token generation: images are represented and generated as discrete tokens. A prompt rewriter followed by a separate generator is architecturally distinct from a language model with a diffusion head. Suggestions about specific commercial implementations remain speculative or unconfirmed.

A proposed refinement loop generates an image, passes it through a vision-transformer encoder and uses the resulting representation for further reasoning. Generation becomes something the model can inspect and revise. Participants disagree about whether the encoder and generator require joint training to avoid representation mismatch, or whether a component can remain frozen at some training stages.

A video agent owns the production task

A video agent is proposed as a language model that calls generative models and editing tools, then iteratively refines the result. Professional creators already take generated clips into post-production. An agent could perform more of that workflow, combining diffusion with conventional image and video editing rather than treating a generated clip as finished work.

Grok Imagine Agent beta is presented as an early attempt. A motivating request is to make an existing video one minute long. That goes beyond the removal, addition, replacement and style-transfer operations described for the editing model, but makes sense as an agent's production assignment. During the demonstration, a participant reports frame generation followed by code and FFmpeg to stitch frames together. Manual intervention remains available; full automation is a forecast.

How much requires new training rather than a better harness? One participant proposes attaching an off-the-shelf reasoning model to Grok as a tool. The response identifies model-specific prompting as a possible benefit of joint training, without quantifying it. A clearer division of responsibility emerges around precise edits: use deterministic tools for exact placement in time. An editor can add text at a specified timestamp directly, where a video generator may interpret the request imprecisely.

The demonstrated output is reported as 36 seconds despite an expectation of minutes, and participants disagree about its quality. The prediction is that video agents become a major success by the end of 2026, once their output is suitable for presentations and advertising. Iteration raises cost as well as quality potential: agents produce many variations and revisions. The commercial bet is that finished work crossing a usability threshold will justify larger enterprise budgets.

The next bottleneck: models managing their own context

The closing discussion extends the argument toward physical AI: a language model with strong real-time video understanding and computer-use capabilities might eventually control robots as tools. This remains a proposed path to embodiment, rather than a demonstrated transfer from screen recordings to physical control.

Asked why they left xAI, the respondent cites research they could not pursue there and rapidly changing company priorities. The desired move toward language research follows the view that the language or agent component is now video's bottleneck. Another participant challenges the strongest interpretation: video generation still needs language and diffusion research. The respondent agrees.

The most concrete proposal is context management under model control. Longer windows are one route; choosing what to retain is another. A model could learn how close it is to its context limit before automatic compression interrupts its work. Roughly 80% occupancy is an illustrative warning point, not a universal threshold. OpenClaw's attachment of local time to messages and harness pruning of intermediate tool results illustrate external mechanisms that supply or remove context.

Training inside coding harnesses may already teach implicit expectations about compaction and sub-agents, one participant argues. A more explicit proposal gives a model its harness code, if short enough to fit, and lets it modify future executions. It could choose how to process a long document: read it all in chunks and combine summaries, or inspect only an opening portion. The proposed adaptation happens through code at test time. Another participant cites self-modifying harnesses in OpenClaw and Pi, while emphasizing how much work remains.

A career following transferable training principles

The final career reflection begins roughly a decade earlier, with image recognition, object detection, tracking and neural-network compression research alongside ResNet authors Xiangyu Zhang and Jian Sun. The respondent had wanted to become a professor, but was rejected by all the top PhD programs they applied to despite several first-author conference papers. Facebook AI Research followed, along with a shift to self-supervised learning.

NVIDIA brought work on both Cosmos video diffusion and language-model mixtures of experts, followed by the move to xAI for larger compute scale. The lesson drawn from those transitions is that core large-model training principles carry across modalities. Moving from computer vision to video and language can be less daunting than a researcher's specialization suggests; the new direction follows the perceived bottleneck.

The conversation closes with a call for xAI to explain more of the technical depth behind its releases through podcast appearances. The respondent promises more about the next chapter soon, but leaves those plans undisclosed.