How Runway Turns Video Generation Into an Interactive World

From a prompt to a world that responds

A generated clip usually asks the viewer to wait for the whole scene. An interactive world has a different problem: what happens after the viewer acts? Move the camera, ask a character to walk closer, or change an event, and the next frames must respond while the scene is still unfolding.

Runway’s GWM Worlds 2 is a research preview for that problem. WorldPrompt lets a creator specify an initial scene and a sequence of timed events, with actions also prompted during a session. The model produces video and audio as the interaction proceeds. CTO Kamil Sindi and principal research scientist Robin Kahlow describe it as a control layer for subjects, cameras and the environment.

The interface resembles a game in what a user can ask for, but the underlying control is prompting: it does not expose a game engine’s scripts or a structured world state. Kahlow says movement works relatively reliably in the preview, while more difficult instructions can fail.

How a video model becomes interactive

A video generator faces two separate changes when it becomes a real-time system. It must produce the next frame or small group of frames before the whole clip is known, and it must do that quickly enough for an interaction to feel continuous.

Runway describes a three-part path:

1. Teach the input format. Fine-tune a base audio-video model to follow the WorldPrompt representation. 2. Generate causally. Post-train it to extend a sequence frame by frame or in small groups, so later output can react to earlier output and new inputs. 3. Reduce inference work. Distillation makes generation fast enough to serve interactively.

GWM Worlds 2’s research preview is described as streaming 720p video at 24 frames per second with 48 kHz audio. Those are output characteristics, not a claim that all prompts are followed perfectly or that a session can continue indefinitely.

Runway co-CEO Anastasis Germanidis distinguishes two ways to distill diffusion models: make a smaller model, or make a model produce an output in fewer denoising steps. His example reduces roughly 50 steps to four, trading some quality for speed. A step reduction can therefore accelerate a large model without requiring every deployment to use a smaller one.

The error that comes back as input

In autoregressive generation, yesterday’s output becomes tomorrow’s context. A slight error in one frame can feed the next prediction, then the next, until the scene drifts. The model also has a finite context budget: the system must decide what history to retain and what to discard without exhausting GPU memory.

Long sessions are easier in some constrained applications than in open-ended worlds. Germanidis described generating up to 30 minutes for Runway’s character model, while its more open-ended world exploration was on the order of a few minutes. Those are different workloads; the longer figure should not be read as a general duration for every interactive world.

Memory remains a research problem. If a character leaves a room, a useful world should preserve what happened and what objects were there when the character returns. A convincing next frame alone does not ensure that continuity.

Why plausible video is not enough

Germanidis offers a football example: a model may have seen more videos of successful goals than misses, so it may render a goal more convincingly than a failed attempt. An interactive simulator needs both outcomes to be plausible when different actions lead to them. Otherwise it can make success look more likely than it is.

This is the counterfactual challenge: hold the situation fixed, change an action, and ask whether the simulated consequence remains credible. It matters especially when a robot policy is evaluated in simulation. The model must represent failure as well as success, or an evaluation loop can reward the wrong behavior.

Runway uses automated, verifiable tests, but the research-preview framing matters. Sindi notes that with multiple prompts, characters and scenes it gets harder to determine what caused an outcome. Kahlow recommends trying the model directly to find where it breaks.

From creative tools to a world model

Runway’s route to world models began with creative software. Early users worked with Green Screen, a tool for separating a subject from a video background—a job known as rotoscoping. The company’s later generative models changed the workflow from editing existing footage to synthesizing and steering new imagery.

For Gen-1, a user could film a rough scene made from boxes or books, convert that video into a depth map, and generate a new visual treatment. The input supplied structure; the model supplied appearance. Germanidis describes this video-to-video route as easier than generating an entire scene directly from text.

The first Gen-2 pipeline came from a weekend experiment: generate a depth map from text, then use Gen-1 to turn that depth into video. The two-stage shortcut brought text-to-video to users quickly, although Germanidis says the intermediate depth stage could leave visible structural errors.

Then came control. Motion Brush let creators draw where objects should move; camera controls let them specify movement through a scene. Germanidis traces Runway’s interest in world models partly to this experience of navigating generated video rather than simply watching it.

The Sora shock and the bet on scaling

Runway had spent much of fall 2023 building distributed-training infrastructure for larger diffusion transformers. Then OpenAI showed Sora in February 2024, with video quality far beyond Runway’s Gen-2. Germanidis recalls an existential jolt followed by a three-month push: the team worked out model parallelism and increased model size and training compute roughly tenfold. Runway released Gen-3 that summer.

Runway’s research bet is that predicting video at larger scale can improve a model’s grasp of physical dynamics. Germanidis points to internal results on Physics-IQ, including examples such as predicting how a hanging ball falls. He also warns that a polished clip can conceal errors: selective camera cuts or cherry-picked examples can make a model appear to understand more than it does.

There is an active disagreement behind that bet. Alternatives include learning more abstract representations or using explicitly 3D approaches. Germanidis favors directly predicting pixels, while acknowledging that the best long-term direction remains a research question.

Robotics: evaluate an action before moving the arm

A world model can also serve as a simulator. Give it an initial scene and an action from a robot policy; the model generates what might happen next, and the policy chooses another action from that result. Repeating the loop lets researchers test a policy without executing every attempt on physical hardware.

The hard part is whether results in the model correspond to results on a real robot. Germanidis describes a Runway evaluation that compared policy performance in its simulated scenarios with corresponding real-world benchmark tasks and reported correlation.

Runway’s data bet is to pretrain on plentiful third-person video, then adapt for particular robot bodies with robot data. Germanidis says the company post-trains for partners’ embodiments, such as single-arm, bimanual or humanoid robots. A separate action head can turn a video model into a policy that predicts robot actions; a convincing prediction of pixels alone is not the same thing as a tested physical action.

An interface made of pixels

The same idea can produce a software interface. In Runway’s interface world model, a click or drag is an input and the model generates the pixels that follow. A prompt can describe how an element should respond, including an audio effect. The example is a generated visual interaction rather than a conventional page assembled from HTML, CSS and React.

Germanidis suggests prototyping, visual exploration and educational interfaces as possible uses. The cost is an important open issue: a model that continually generates a moving scene requires more computation than rendering a static page, and Runway is still exploring how to make that tradeoff worthwhile.

For agent training, a generated interface could provide varied visual environments for computer-use tasks. Kahlow makes the corresponding point for worlds: an agent observes the generated video and audio, with no structured state exposed by this preview. Whether those simulated interactions transfer to real software or physical environments remains the central test.

Creative judgment still shapes the model

Runway’s creative team participates in captioning and evaluating training material, including camera direction and aesthetics. Germanidis says those decisions affect what a user can later ask the model to produce—and what the team considers usable output.

He expects the priorities of creative users to keep shifting: image quality remains important, but control and latency determine how quickly a person can explore alternatives. A model that generates ten options quickly changes the workflow because the creator can iterate while an idea is still taking shape.

The longer-term ambition is a model that learns across more than ordinary RGB video: audio, robotics observations, and even numerical simulations might share useful patterns. Runway describes early experiments adapting a video model to scientific simulation data. The claim is a research direction, with modality transfer and its limits still to be established.