Video as Tokens, Motion as Features — ICML 2024, Part 2 of 6

Learning a world agents can practice in

An agent acting over long horizons in the physical world needs somewhere to practice. The opening discussion of Genie argues that unrestricted real-world exploration is unsuitable, while hand-building a sufficiently rich simulator is impractical. A learned simulator could fill that gap. This ambition—embodied agents interacting with the world—is presented as complementary to language-model agents pursuing long-horizon tasks in text-based environments.

The ICML 2024 recording then turns to VideoPoet, a language model that generates video and audio one token at a time. Lijun Yu introduces it with a generated skeleton drinking soda. The example establishes the talk's central claim: video generation does not require diffusion.

VideoPoet: make the task a sequence

VideoPoet casts multimodal generation as a shared sequence-to-sequence problem. A conditioning sequence contains task indicators, text, visual or audio inputs, and output-format controls. The model then generates visual and audio tokens autoregressively: each output token depends on preceding tokens. Its decoder-only prefix language model uses bidirectional attention within the conditioning sequence, so those inputs can inform one another, and causal attention over the output.

Three pretrained components provide the transformer's inputs: - MAGVIT V2 converts pixels into discrete visual tokens, with a decoder that maps them back to pixels. It handles images, videos, depth, optical flow, and cropped or masked video. - SoundStream converts audio waveforms into tokens. Its causal one-dimensional convolutional network uses residual vector quantization to produce multiple token levels. - T5 supplies text features, reducing the burden of learning human language from scratch during video-model training.

The visual tokenizer uses a temporally causal three-dimensional convolutional architecture and a vocabulary exceeding 200,000 codes. Yu credits the causal design with enabling joint training on images and support for long videos. Reconstruction and adversarial objectives train the mapping between pixels and compressed tokens.

The sequence format lets one model combine variable-length inputs and outputs across modalities using standard large language model (LLM) infrastructure. Yu argues that causal training covers the full decoding trajectory in one training step. He also claims that inference caching can keep total decoding computation within one full forward pass, although he supplies no measured latency comparison with diffusion.

Task design determines what data the model can use. Pretraining mixes raw internet videos with audio, machine-captioned videos, human-captioned videos, and image–text pairs. A later adaptation phase can specialize the model using higher-quality task-specific data. The training objectives cover: - Generation and continuation: unconditional video or audio generation also supports continuing a supplied sequence and animating an input image. - Cross-modal generation: produce audio from video, video from audio, or visual and audio outputs from captions. - Editing: fill missing regions, extend the frame, interpolate video, or condition stylization on depth and optical flow.

After the language model generates video tokens, optional latent super-resolution increases resolution before pixel decoding. This stage uses a mask transformer with non-autoregressive decoding and windowed attention to handle long sequences at high resolution.

Yu reports favorable zero-shot text-to-video benchmark results and human preferences across text fidelity, video quality, motion interestingness, and motion realism. The human study favored VideoPoet over prior work and, in most cases, concurrent work; preference percentages, sample counts, and evaluation protocols are not given. Demonstrations include image animation, stylization, editing, and jointly generated video and audio.

The talk closes with possible extensions: learning new motions or characters from context, adding modalities at inference time, and real-time streaming generation for neural games and interfaces. These are research directions, rather than demonstrated capabilities.

The tokenizer enables generation—and limits fidelity

In the questions, the presenter confirms that VideoPoet uses classifier-free guidance in its autoregressive setting, while leaving the technique's mechanics unexplained. He considers quantitative evaluation of video instruction following an open problem. Comparing autoregression with diffusion also requires care: different latent representations and reconstruction quality confound architectural comparisons, alongside different training setups. A comparison of finished systems does not isolate those factors.

VideoPoet's tokenizer is an encoder–decoder pair trained to reconstruct video, rather than a second generative model such as a diffusion decoder. Lossy compression makes decoding ambiguous: multiple pixel-level reconstructions can correspond to the same compressed representation. The presenter credits the adversarial objective with reducing blur and producing sharper video, and temporal causality with making the tokenizer suitable for autoregressive decoding.

An audience member distinguishes easy implementation from difficult training. The presenter agrees: tokenizer hyperparameter searches are tricky and depend on dataset statistics. An earlier tokenizer implementation was available, but future release plans were uncertain. Despite his work on tokenizers, his longer-term aspiration is to eliminate them.

For VideoPoet, the presenter locates the main design work in the token space and sequence layout, saying the team reused an off-the-shelf LLM architecture and training recipe. Shared layouts encourage generalization: text-to-image can form a prefix of text-to-video, while video-to-audio can share sequence structure with unconditional video–audio generation.

At the poster: motion, scale, and pixel fidelity

In the poster interview, Dan Kondratyuk describes VideoPoet as a foundation model organized around video output. Its contribution is combining tasks and scaling an ordinary language model to competitive video generation. He reports training on hundreds of millions of videos and recalls using more than a billion images, with some paired video and audio data.

Task composition makes the design tangible. An image animation can feed a stylization task conditioned on depth and optical flow. In one example, the output becomes an oil painting of a snowman in a red hat yawning while retaining the original motion. Video outpainting extends content above and below the input frame. The interview also highlights prompt following and larger, more interesting motion as strengths in the team's comparisons.

Asked which uses favor which approach, the interviewee separates three considerations: - Pixel fidelity favors diffusion in his assessment. VideoPoet's strong compression forces generation at small resolutions followed by super-resolution; diffusion can operate on less compressed latent representations. - Training and scaling favor language models in his experience. He reports convergence after a few weeks and substantial improvement from one billion to eight billion parameters. No hardware- or compute-matched comparison is supplied; further improvement with scale is his expectation. - Shared sequences support multimodal generality. He argues that autoregressive text performance made language models the stronger basis for a model spanning modalities at the time, while leaving open future progress in text diffusion.

Kondratyuk closes by distinguishing this work from his subsequent move to Luma: VideoPoet was undertaken at Google and remained unreleased at the time of the conversation.

Controllable video starts with a representation

Tali Dekel opens with an ambitious filmmaking scenario: generate a script, turn it into a film, then add bullet-time effects without actors or cameras. The footage is actually from The Matrix. Her point is that convincing generated clips remain far from complete computational filmmaking. She cites failures of physical interaction and object persistence, including in Sora: a rigid chair floating implausibly, a treadmill following a person, and objects appearing or disappearing in scenes with multiple entities.

Filmmaking requires fine-grained controls that, Dekel argues, contemporary video foundation models did not fully provide: - Camera and scene: camera position and lighting. - Characters and action: identity, emotion, position, and movement. - Audio: sound and speech. The expense of developing these models also concentrates research among a few industry players. Dekel argues for progress beyond data and scale.

Her research began with single-video models: neural frameworks deliberately fitted to one input video, without additional training data. One example removes all moving people except a girl, including the trampoline deformations associated with the removed people. Work dated to 2021 stylizes a bicycle video's background and adds flower textures to a deforming dress, with edits following the original motion.

Layered neural atlases reduce video editing to editing canonical 2D images. A foreground atlas and a background atlas represent content across the video. Learned mappings take each video pixel into these shared atlas coordinates. An edit to the atlas can then be mapped back across the video, instead of being repeated independently in every frame.

Small neural networks represent the atlases implicitly. Separate foreground and background networks map video pixel positions to 2D coordinates between −1 and 1. Another predicts color at each atlas coordinate, while a visibility network determines the foreground and background contributions to each reconstructed pixel. Training is self-supervised: video reconstruction supplies the main loss, with additional objectives encouraging interpretable structure and preserved correspondences.

The two approaches supply complementary capabilities: - Foundation models supply prior world knowledge—learned patterns of appearance and change over time—but are expensive and offer limited control. - Single-video models supply flexible representations, requiring only a few GPUs in Dekel's account, but know nothing about the world beyond their input video. Her proposed direction combines explicit control with motion, generative, and semantic knowledge from pretrained models.

Text2LIVE, presented at ECCV 2022, makes this combination concrete. It keeps a pretrained neural atlas fixed as a video renderer and replaces manual atlas edits with text-driven edits guided by the pretrained CLIP model. Dekel reports localized semantic editing without a generative model—a capability she considers challenging even for powerful foundation models.

Subsequent projects extend pretrained text-to-image models while keeping them fixed: - Plug and Play preserves a reference image's semantic layout while conditioning generation on text. - Multi-Diffusion adds arbitrary resolutions and region-based text controls. - A scene-generation project builds an actual 3D mesh to support plausible walkthroughs. The underlying research problem is to understand what these models encode well enough to reuse their knowledge in new algorithms.

TokenFlow: propagate an edit through features

TokenFlow begins with a failed baseline: apply an image-editing method independently to each video frame. The resulting content changes from frame to frame, beyond high-frequency flicker. To investigate, the researchers invert the original frames into the model and extract intermediate features. They reduce these high-dimensional features to three dimensions for visualization and observe consistency corresponding to the original video's consistent content.

Nearest-neighbor matching turns that observation into a mechanism. For each feature location in one frame, the method finds the closest feature in another. Applied densely, this produces a cross-frame correspondence field. The team then warps source-frame tokens along those correspondences and substitutes them during target-frame generation. Their reported ability to reconstruct the target frame supports reusing corresponding features across frames.

Independent editing breaks consistency in both the output and its internal features. TokenFlow therefore enforces the original video's feature correspondences during generation: 1. Extract the original correspondence field. Invert the video, extract internal features, and compute their matches across frames. 2. Jointly edit sampled keyframes. Extended attention gives these frames rough global coherence. 3. Propagate edited features. Use the original correspondences to transfer keyframe features to the remaining frames, repeating the process during generation.

Consistency depends on correspondence quality. Matches are more reliable between nearby frames; greater temporal distances and complex motion can break the method. A further limitation motivates the next project: text-to-image models provide two-dimensional information without the motion and action knowledge needed to adapt movement to a substantially different object.

Transfer the action while allowing the body to change

Turning a dog jumping into a river into a dolphin performing a similar action requires a different body shape and plausible fine motion, including tail movement. The motion-transfer task therefore treats motion as the progression of semantic object parts, allowing substantial deviations from the original pixels. It uses Zeroscope, a publicly available text-to-video model that Dekel describes as behind leading systems but still containing useful knowledge of dynamics.

The researchers extract features with shape F × M × N × D: F frames, M × N spatial locations, and D channels. They probe this representation through feature inversion—optimizing a new input to reproduce a reference input's features while keeping the model fixed. Such optimization generally needs regularization to avoid adversarial solutions. Matching the full video features reconstructs both appearance and motion, making it too restrictive for an edit that must change identity and shape.

Spatial averaging reduces the representation to F × D, one feature vector per frame. In an inversion experiment, these pooled features retain a woman's pose and movement while permitting more variation in structure and appearance. Yet using them directly as editing guidance still preserves too much: an elephant-to-camel edit produces a camel squeezed into the elephant's shape.

The solution is to match relationships between frames rather than absolute feature values. The method constructs an F × F matrix describing pairwise differences between spatially averaged frame features, then guides generation to match the source video's matrix. This relaxes the source-object constraint. The kitten-to-bunnies example illustrates the intended balance: preserve the driving action while allowing the generated animals to move plausibly as bunnies.

Initialization combines low-frequency inverted noise with high-frequency random noise in the initial video latent. The presenter reports that this makes optimization more robust and less sensitive to the random seed.

Evaluation must allow the same structural freedom as the edit. Text fidelity uses CLIP similarity. Motion fidelity compares independently tracked, unaligned trajectories from the source and output videos: each trajectory finds a highly correlated trajectory in the other set, in both directions, and the correlations are summed. The exact correlation formula and normalization are unspecified.

The comparison exposes a tension between preserving motion and completing a substantial edit: - TokenFlow preserves motion strongly, but its lower text fidelity reflects difficulty completing large changes in content. - The low-noise editing baseline preserves source motion, while limiting changes to the video's content. - The high-noise version permits stronger edits, while losing motion fidelity. The proposed method is reported to offer the best balance in this comparison, without numerical scores or evaluation-set details.

The method remains bounded by the video model's motion knowledge. When plausible movement for the target object cannot fit the source action, the output can deviate from the source and move unnaturally.

Reusable video features, richer controls, better compression

Dekel closes by advocating foundation models combined with representations fitted to individual videos. Three open directions follow: - Reuse pretrained video features. Internal representations may support downstream tasks beyond generation, potentially without heavy adaptation training, but require better understanding. - Build controls beyond text. Dynamic content needs richer representations, tools, and intuitive interfaces for specifying and manipulating motion. - Compress information across time. A video compressor with a latent space suitable for generation could improve on straightforward extensions of image methods. This is a research expectation, not a measured cost reduction.

Diffusion begins with a directly sampled noisy state

The recording next introduces Sander Dieleman, with the host framing his interpretation of diffusion as autoregression in the frequency domain. Dieleman's opening proposes a geometric view of diffusion to connect its different formalisms and make guidance intuitive. He begins with the common mechanism: gradually corrupt data with noise, then learn to remove it.

Start with a training example x₀ at time zero and repeatedly add small increments of Gaussian noise. As corruption grows, the original image becomes visually unrecognizable. The computational shortcut is that many Gaussian increments combine into one larger increment. A noisy sample at a chosen time can therefore be constructed directly: xₜ = x₀ + σ(t)ε Here ε is standard normal noise and σ(t) is the noise schedule, which controls the amount added. Training can sample that corruption level without simulating every preceding step.

The practical formulation adds a time-dependent factor αₜ to rescale the input and stops corruption at a finite terminal time T. Part 2 ends as Dieleman begins explaining how that endpoint is chosen.