From Persistent Pixels to Playable Worlds — ICML 2024, Part 1 of 6

A blue sign survives the cut

In a Sora-generated Tokyo street scene, a blue sign persists through an obstruction and a shot change. That small detail poses a substantial problem for video generation: an object must remain part of a scene even when temporarily invisible. In his ICML 2024 presentation, Bill Peebles interprets the example as object permanence: persistence through occlusion, and reports similar consistency across many samples.

Sora generates video from prompts, with the demonstrated model described as capable of 1080p output up to a minute long. Peebles introduces it as joint work with Tim Brooks and other OpenAI colleagues. His presentation in Latent Space’s ICML recording begins with visual consistency and develops toward a larger question: what knowledge of the world can video generation learn?

The opening examples require several kinds of coordination: - Multiple shots in one generation: a paper-craft coral reef includes different views of sea life without manually stitching clips together. - Simultaneous interactions: snowy Tokyo combines camera movement, crowds, a couple holding hands, vendors, and airborne petals. - Character consistency across cuts: a spaceman trailer retains its protagonist across settings, a behavior Peebles attributes to training on video at scale.

A shared representation makes diverse video trainable

Sora borrows an organizing idea from language models: stories, code, and mathematics can share a token vocabulary. Visual training needs an equivalent way to accommodate different resolutions, durations, and aspect ratios. Sora’s visual encoder compresses space and time into a shared latent representation—a compressed sequence on which diffusion transformers train. This lets the training system accommodate diverse images and videos without forcing every example into the same frame shape.

Peebles illustrates the effect of increasing training compute with a qualitative comparison: - Base compute: some consistency under camera movement, but poor textures. - 4× compute: more recognizable humans and dogs, though weak visual fidelity. - 32× compute: finer hand–dog interactions and snow textures. The multipliers are relative to the base model. No absolute compute budget or numerical quality measure accompanies the comparison.

At generation time, Peebles describes flexible aspect ratios and resolutions, illustrated by sea turtles rendered in different formats. He proposes generating directly for different devices to avoid cropping or black bars. Preserving native sizes during training has another benefit: cropping can discard composition information. A baseline trained with hard center crops sometimes frames a scuba diver awkwardly, whereas native-size training better preserves composition patterns in the original data.

The same model generates still images. Peebles shows a stylized tiger and a photorealistic face at 2K × 2K resolution, describing photorealism as a particular strength.

Conditioning controls what stays fixed

Sora accepts visual inputs as conditioning: supplied content that constrains a generation. An image from DALL-E 3 seeds a video that extends forward in time. A Shiba Inu image placed in the middle of a sequence anchors generation both backward and forward. Visual conditioning can occupy any temporal index; another demonstration fixes a San Francisco logo at the ending while resampling the preceding events.

SD Edit, a diffusion editing technique, re-renders an input video from text prompts without additional Sora training. Its noise level controls how much scene structure survives. In the car example, different styles preserve a passage through a cave or overhang; a medieval theme replaces the car with a red horse carriage. The input was Sora-generated, though the technique is also described as applicable to real video.

Fixing two endpoint videos leaves Sora to invent the transition. A chameleon becomes a bird; a drone in the Colosseum becomes an underwater butterfly. In the latter, the environment changes gradually but the creature transformation is sudden. Between a photorealistic town and a gingerbread village, the model instead places the village off to one side and zooms into it. Endpoint constraints do not specify the connecting events.

Language control involves two complementary operations: - Training-time recaptioning: detailed synthetic captions describe visual content more closely than coarse or unrelated alt text. Sora adapts this approach from DALL-E 3, where it improved controllability and intricate scene generation. - Inference-time prompt expansion: GPT expands a user’s request into a detailed video description before Sora receives it. Reliable expansion required substantial prompt engineering; the system-prompt example discussed comes from DALL-E 3. Control therefore depends on both the video model and the system translating a request into conditioning text.

World simulation requires consequences that persist

The Sora team’s long-term hypothesis is that sufficiently faithful generation will require richer models of behavior. A realistic dinner-table conversation would need plausible human interactions as well as convincing appearances. World simulation remains a scaling hypothesis: the presentation explicitly acknowledges that Sora has not reached the envisioned capability.

The demonstrations distinguish several forms of consistency: - Geometry under motion: people and scenery appear spatially consistent as the camera moves, without hard-coded 3D rules. A geometric sanity check by Ben Mildenhall supports some generated scenes, not all. - Coherence across environments: a fictional jewelry-store-and-zoo generation moves from outdoors to indoors without manual stitching. - Identity across interruptions: a cyberpunk robot recurs across cuts, while a Dalmatian remains present as people pass in front of it.

A harder requirement is persistent state change: an interaction must alter what happens afterward. Brush strokes should remain on a canvas; biting a burger should leave marks. The burger demonstration retains bite marks, but persistent consequences remain unreliable. Remembering an event and allowing it to affect generation minutes later is a major unresolved challenge.

The proposed scope includes digital environments such as operating systems and games. Minecraft-like samples generate player movement, scenery, and animals together. The presenter describes the player’s behavior as intelligible but slightly boring and treats the result as a crude proof of concept for eventually extracting information about policies—how an agent behaves. These examples generate behavior within a clip; they do not demonstrate a user choosing the player’s next action.

Visual plausibility does not ensure reliable physics. Failure cases include malformed hands, candles blowing in inappropriate directions, a cup spontaneously jumping and cracking, and an archaeological discovery where a plastic chair starts flying. Even basic interactions such as glass shattering remain unreliable.

Creative usefulness precedes precise control

At the time of the ICML talk, Sora was a research system rather than a product. The team worked with red teamers and artists to explore deployment risks and useful workflows. Shy Kids emphasized surreal imagery: its film features a fictional narrator filled with hot air and vulnerable to wind and punctures. The film combines model output with conventional editing, as did some Sora-assisted films shown at Tribeca.

Asked about making an entire movie without actors, the presenter forecasts that long-term character consistency is technically achievable. Directors already found simple background scenes, such as crowds, useful. For intricate close-ups and emotional connection, the presenter expects human actors to retain an advantage in the near term, with workflows mixing human and digital performances.

The Q&A establishes the research system’s practical boundaries: - Camera control was indirect: users could describe motion or extend a video already exhibiting it, but had no finer trajectory interface. - Joint audio-video generation was absent: development had prioritized visual quality. - Training composition remained undisclosed: the presenter declined to answer about synthetic training data.

Video training might also improve still images by clarifying ambiguous observations. Fingers hidden behind an object do not imply a hand with fewer fingers; motion can reveal the difference. The respondent predicts that video models will eventually supersede image-only generators. At the time, however, Sora’s image generation was not productized, and ChatGPT used DALL-E 3. Further questions about model size and capacity went unanswered; a future ecosystem of differently sized open and closed models was offered as a guess.

Sora also struggled with directors’ shot and camera vocabulary, making captioning one possible route to better control. Asked about squashing a hamburger, the respondent was unsure whether the model could do it. Even biting had taken time and substantial compute to emerge. The scale needed for a particular physical capability was unknown, as was whether the current model could reliably improve real-world robot policies.

The Sora discussion closes on inductive biases: built-in assumptions such as explicit physics rules. The team had prioritized scalability with as few such assumptions as possible and had not explored adding those rules. They might help smaller models or narrow applications, the respondent allows, while assuming they could hinder the largest general models. This is a design assumption, not an experimental finding that explicit physics is harmful.

Genie turns video into an action loop

Genie—Generative Interactive Environments—learns action-controllable worlds from unlabeled video. The podcast introduction also discusses the later Genie 2, describing 3D environments, jumping and swimming, and memory for regions outside the current view. Its reported consistency of up to a minute comes without an evaluation setup or success rate. Those later-model remarks are separate from the original Genie presentation that follows.

Presenting the Google DeepMind collaboration with Ashley, Jack begins with the missing environment for agent training. AlphaGo and AlphaZero have game simulators; broadly capable embodied agents lack an equivalent general environment. He defines world models by sequential action control. A consistent clip may contain world knowledge, but caption-level control alone does not let an agent choose actions and learn from their consequences.

Internet videos show events without recording the controls that caused them. Genie addresses this with a latent action space: discrete changes inferred from video rather than supplied as action labels. Jack describes the intuition as clustering possible outcomes from a frame. Ashley explains three components, typically trained on sequences of about 16 frames: - Video tokenizer: converts patches across the sequence into discrete video tokens. - Latent action model: compresses changes between consecutive frames into discrete actions. - Dynamics model: predicts next-frame tokens from tokenized history and latent actions.

At inference, the user can intervene between frames: 1. Supply an initial prompt image. 2. Choose a discrete latent action using an integer value. 3. Generate the next frame from the history and action. 4. Feed the frame back into the model and continue choosing actions. This frame-by-frame feedback loop is the operational distinction Jack emphasizes between Genie and whole-clip generation.

Consistent controls and sharp frames need different objectives

Ashley describes roughly 300,000 hours of game footage, primarily 2D platformers, filtered to 30,000 hours. Scaling experiments led to an 11-billion-parameter model with a batch size of 512. Video quality alone could not establish controllability: the team also needed to enter the generated environments and play.

The demonstrations separate two tests: - Same image, different actions: varying action sequences produces different trajectories from the same starting frame. - Same actions, different images: one sequence produces similar trajectories across four prompts, testing whether controls retain their meaning. Ashley limits the consistency result to those environments. The actions were learned without ground-truth action labels, object detection, or segmentation.

Prompts outside the training distribution—unlike the platformer footage used for training—include sketches and photographs. A hand-drawn ladder becomes climbable, and a dog photograph becomes an interactive scene. A separate two-billion-parameter model trained on robotics data demonstrates similar behavior under the same action sequence across different prompt images, including examples involving deformable objects.

Training agents inside Genie had not yet been demonstrated. Instead, learned latent actions could label unseen videos so agents could imitate their behavior. Unlimited generated training environments remained a future direction. Jack also describes a smaller-model case study trained on a mid-range accelerator in under a week, with fairly consistent actions across prompts and student reproductions; detailed hardware specifications and reproduction measurements are not given.

The Q&A explains why Genie needs two prediction components. The latent action model already has a decoder that predicts the next frame, but consistent controls and sharp generations favored different objectives: - Pixel prediction helped learn actions: it produced more controllable, consistent latent actions, but its decoder would generate blurry frames. - The generation objective favored visual output: using it to learn the action space produced less consistent actions. The team retained separate predictors despite considering the arrangement inelegant.

Control also did not guarantee an interesting world. Especially with unfamiliar prompts, the model could move a character while repeatedly extending background patterns. Controllable movement and varied world behavior remained separate problems.

At the poster: inferring actions unlocks more video

In the poster conversation, Jack describes Genie as a meeting of research directions. Work on open-ended learning and environment generation needed a way to learn action-conditioned futures from internet video. Ashley Edwards’s work on behavior cloning—learning to imitate observed behavior—provided a complementary approach to inferring actions. Combining them suggested learning both controls and environments from video.

Jack gives a more specific collection size of 280,000 hours of public platform-game footage. A classifier trained on a small subset labeled by the team selected the 30,000-hour training set. The latent-action decoder learned eight discrete actions through pixel prediction, then a separate dynamics model learned to predict frames conditioned on those inferred actions. Removing the action-annotation requirement was central to using larger video collections.

He reports improvements at each tested model-size increase from tens of millions of parameters to roughly two billion, and with larger batches. These experiments motivated the 11-billion-parameter model, though the interview gives no metric or numerical gains. Playing with it revealed an unexpected use: photographs of a co-author’s children’s drawings became worlds with movable characters. Agent-environment research had also produced a way to interact with personally created images.

Jack identifies several possible applications: - Short interactive experiences: the paper demonstrates platformer-like environments generated from prompts. - Robotics: he considers this potentially more promising in the near term and cites 1X’s reported open-source Genie model for its world-model challenge as an encouraging signal. - Custom driving scenarios: interactive situations could potentially support autonomous-vehicle testing or training. The team remained focused on fundamental research. Roughly six months after submission, most were still working on Genie, but Jack made no commitment to a release version or date.

The closing distinction: building agents

Jack sees Sora’s visual progress as reinforcing the potential of scale while distinguishing whole-clip generation from Genie’s frame-by-frame control. Asked about VideoPoet, the respondent characterizes it as more oriented toward cinematic clips. Genie sits between a video model and an autoregressive image generator: it generates one frame at a time, using prior frames to continue.

Many Genie researchers came from reinforcement learning and focused on agents, while sharing architectural ideas, infrastructure, and compute needs with generative-media work. The closing answer frames agent building as the continuing goal, with changing methods. Earlier work trained agents through reinforcement learning from scratch; newer work starts with large language models and trains for additional capabilities. Part 1 ends during that answer, before a completed prediction about whether agent hype will persist.