General Intuition: From Gameplay Highlights to Playable Worlds

Save the moment after it happens

An unexpected move or a recovery from a mistake is over before a player decides to record it. Medal’s retroactive capture solves that problem: the recorder runs in the background, and a button saves the preceding 30 seconds. At the time of this Latent Space conversation, Medal reportedly had 12 million users and had accumulated 3.8 billion clips.

General Intuition, Medal’s spin-out, learns from the perception–action loop captured in gameplay: a player sees a scene, acts, and sees the resulting change. Pairing frames with actions connects observations to the inputs that changed the environment. The lab’s independence followed a reported decision to decline OpenAI’s $500 million offer for Medal’s data and build with backing from Khosla Ventures.

Learn actions from human highlights

The preview begins with a vision-only policy, an agent that receives video frames and predicts actions. An early version, from roughly four months before the interview, reportedly navigates, opens the scoreboard, gets stuck, and recovers with four seconds of memory. The newer base model runs in real time against human players, according to the presenter, using pure imitation learning with no reinforcement learning, fine-tuning, or access to internal game state.

Imitation learning learns from recorded human behavior. That explains the presenter’s account of exceptional play: selected highlights let the policy inherit exceptional human actions. The comparison with Move 37 has a firm boundary—the agent is not described as discovering a new strategy through reinforcement learning. It also makes a mistake the presenter considers unlike a human’s, then recovers.

The preview’s visuals are unavailable to listeners, and its human-like or exceptional behavior is described qualitatively, without benchmark results. The presenter says the agent is not given a goal; the exchange does not establish that it independently invents one.

Action prediction also offers a way to label video whose actions are missing. The presenter describes training on less realistic games, transferring to a more realistic game, and then applying the model to real-world footage. For that footage, labels represent hypothetical keyboard-and-mouse control: the inputs one might use if the observed sequence were playable. This is the proposed bridge to using internet video for pretraining.

The evaluation separates two kinds of error: - Individual prediction error: Reset the comparison baseline for each prediction to isolate its error. - Accumulated sequence error: Retain the baseline across the sequence to expose how earlier errors compound. Recorded actions provide ground truth for these comparisons in games. Equivalent labels were not yet available for the real-world footage, so its action-labeling accuracy remained unverified.

Returning to gameplay, the presenter describes the same policy changing behavior with its held items and distinguishing cover from an exposed position. This later sequence uses bot opponents, whereas the preceding gameplay used humans. First-person perception is considered useful for the current agent; third-person views could become useful when controlling multiple objects.

Generate consequences while preserving the world

The preview then switches to world models: models that generate what follows an action. A policy chooses an action from observations; a world model generates the next state from the current state and an action. Pim de Weele describes both pretraining world models from scratch and fine-tuning open-source video models to investigate transfer of physical behavior.

The demonstrations probe continuity under changes in what the camera can see: - Rapid camera motion: One second of video provides context, then recorded actions drive a generated continuation. De Weele argues that skilled players’ fast movements supply useful training examples. - Partial observability: With the same one-second setup, smoke hides the scene. The model reportedly emerges at the expected location, which de Weele interprets as maintaining position despite the temporary loss of visibility. - Zoomed and unzoomed views: A sniping sequence is presented as maintaining depth and scene consistency when content leaves the scope’s view and later returns.

A roughly 20-second generation from a held-out clip introduces a revealing distinction: transferred behavior can differ from the source game. An explosion produces camera shake that de Weele says the game itself lacks. He interprets the effect as physical-world behavior inherited from video training.

The team also uses distillation to reduce model size. A smaller model’s long sequence includes less effective behavior, such as running into a wall. De Weele says the demonstrated models run in real time, but provides no parameter counts, hardware specifications, or latency measurements to quantify the tradeoff.

Games offer a more direct connection between perception and control than ordinary video, in de Weele’s view. With YouTube footage, a learner must recover a person’s pose, infer the actions behind the motion, and account for where the person was looking before acting. Inverse dynamics here means inferring action labels from observed behavior. Gameplay already pairs actions with frames, while hand-controlled camera motion records changes in the player’s view, avoiding some of that reconstruction.

The recorder became a training pipeline

De Weele’s route to modeling began with teenage coding and a large private RuneScape server, followed by satellite-based map generation for disaster response and then Medal. Writing Medal’s video transcoders gave him substantial GPU and video-infrastructure experience. Modeling felt less familiar, so his co-founders assembled coursework to strengthen his fundamentals. He connects that effort to his belief that technically grounded leadership improves a lab, and credits reverse engineering with shaping his deductive approach to problems.

The training pipeline grew alongside consumer features. A Flight Simulator controller overlay, for example, pairs the game scene with joystick activity. Medal developed action capture to support these overlays and navigation through clips by events. Its discussion of controller-overlay protections provides related product context.

Game actions replace literal keystrokes in the data design. Inputs are converted into actions rather than retained as raw keys such as W, A, S, and D. The guest presents this as a privacy choice that also removes training noise. Actions can be mapped back to generic computer inputs for a game, while the representation reportedly cannot reconstruct an individual user’s inputs. Building the labels involved thousands of people over roughly a year and a half; exact game coverage and label counts are unspecified.

Medal’s growth strategy was to make capture useful before building the consumption network. The recorder holds footage in memory, writes only the selected sequence to disk, and syncs it to the user’s phone, removing the work of managing a long recording and processing a multi-gigabyte file. The guest credits that utility, existing sharing behavior, Fortnite, COVID-era growth, and Discord’s connections between gamers with helping the network grow.

Retroactive selection lets players save natural behavior without planning to perform for a recording. It also creates a coverage question: highlights leave out routine activity, including ordinary travel between locations. An interviewer raises that gap, but the discussion does not explain how training handles it. The guest separately reports that context-window size substantially changes agent behavior, without giving sizes or measured comparisons.

Imitation first, reinforcement learning later

The training analogy is a large language model (LLM) corpus for interactivity: predict action tokens from visual input at scale. The guest recalls cross-game navigation research in which an agent trained on nine games performed roughly as well on a held-out tenth as an agent trained specifically on that game. That tentative recollection motivated the broader question: could a sufficiently diverse interaction corpus support a foundation agent?

GI deliberately starts with a small action space—the controls a model can express—limited to gaming-style inputs. Longer term, it hopes to learn action representations that transfer across interfaces. Diamond’s playable world model supplied a complementary inspiration: the guest recalls interactive execution on a consumer GPU with comparatively little training footage. Although the hardware and data totals are uncertain recollections, that combination prompted outreach to its researchers.

GI initially expected world models to be the dataset’s main use. Its strategy shifted to imitation-trained policies, with world models developed in parallel. Those generated environments would later support reinforcement learning (RL), allowing agents to learn through interaction. When an interviewer suggests eventually eliminating world models, the guest corrects the target: ideally, imitation learning would eventually give way. World models remain central to the proposed next stage.

This sequencing explains the independence bet. GI expects its recorded interaction data to help it reach customers before competitors generate equivalent experience through world models. Games and simulation are the initial focus, including industrial uses of game engines. Simulating human behavior in Omniverse to improve factory-floor training data is one proposed application.

Steerability adds a separate challenge: aligning action sequences with instructions. The guest anticipates a vision-language model managing multiple action policies, instructing instances that perform the actions. This is a proposed deployment architecture; the policies shown in the preview were not yet steerable through text.

Value data by the capabilities it enables

The guest describes Vinod Khosla’s investment evaluation as starting with a 2030 vision and working backward, challenging the technical assumptions connecting it to the present. The corresponding test for a founder holding proprietary data is to train a model and examine what the data enables, while recognizing that not every company can afford that investigation.

The guest’s advice separates three decisions: - Test durable value. Better labeling models and synthetic generation may reduce demand for some ground-truth data. Look for capabilities neither already available nor clearly on the way. - Understand the intended use. Know which model and application a buyer plans to build. GI wanted to avoid enabling products that competed with game developers. - Seek participation in the upside. For large data deals, consider equity where possible, recognizing that many buyers will refuse; alternatively, investigate the dataset’s potential directly.

Doing the technical work helped the guest assess early skepticism about Medal’s data and build the confidence to start the lab. Francois Fleuret’s deep-learning book and course supplied part of that foundation, connecting linear algebra, calculus, and the chain rule to constructing neural networks. The guest compares the resulting understanding to earning a driver’s license, with the co-founders still the Formula One drivers. A new field nevertheless leaves room for a newcomer to suggest experiments simply because many possibilities remain unexplored.

Interactivity demands more than plausible video

The later definition sharpens the world-model requirement: the next state must depend on the action and the world already generated. Producing plausible or entertaining frames is insufficient. A continuation must account for possible outcomes and remain consistent with the user’s actions.

To investigate why the data could not simply be simulated, the guest built a rudimentary PyTorch physics engine. The lesson was that interaction complexity grows with the number of agents, their degrees of freedom, and the information each action conveys. Speech makes the last factor concrete: saying “water” or “fire” can provoke very different responses. Predicting consequences therefore involves more than modeling motion. This is a conceptual argument, not a measured scaling law.

The guest sees different roles for alternative approaches: - Video transfer and generated environments can capture varied interactions that are difficult to specify in a simulator; text steerability adds another attraction. - Traditional simulation remains useful, particularly where video is scarce. - Explicit 3D representations, including the splat-based environments associated with Fei-Fei Li’s approach, offer reuse in game engines and a verifiable setting. The guest reserves judgment on their eventual interactive behavior.

GI’s preference for frames begins with its corpus: the training data is already video. The guest questions whether generating explicit 3D representations adds unnecessary output complexity, favoring smaller representations that can later be enriched through diffusion upscaling. Rendering 3D environments into frames is considered possible but nontrivial. This remains an architectural preference, not a measured advantage over splat-based systems.

The guest accepts that LLMs develop implicit world models and considers them useful, especially for orchestration. The proposed shift is to make spatial and temporal experience the basis for generalization, with language included within it. The conceptual objections to a text-centered system concern transfer to spatial tasks and the fit between sequential text prediction and a continuously changing environment. Speech and text generation could eventually become actions within a broader policy.

Fill the empty match, then transfer the foundation

The first commercial problem is concrete: a player logs in at 3 a.m., and too few humans are online to fill a match. The guest argues that capable bots can preserve engagement by filling gaps alongside people. GI says it can calibrate bot skill using information about the players represented in its data.

The proposed integration replaces scripted player controllers, including behavior trees, with a frames-in, actions-out API. GI reports working with major game developers and game engines. Text conditioning remains a goal. Its longer-term scientific ambition is to represent problems in three-dimensional space and combine spatial and temporal perception with LLM reasoning to work on them.

GTA V role-playing, Truck Simulator, and PowerWash Simulator illustrate the broader repertoire: players perform purposeful sequences resembling real-world activities. The transfer goal is less downstream data collection, with customers adapting a pretrained foundation. A future driving company needing only 1% or 10% of its usual data is an uncertain aspiration, not a measured reduction or a claim of zero-shot self-driving.

Robotics transfer is limited to gaming-style controls. GI reports working with robotics and manufacturing companies, but does not claim transfer to robots requiring more degrees of freedom than its interface supports. The aim is to let customers adapt a pretrained model through post-training.

The initial business model is an API. GI also plans customized policies using customers’ video and reports an existing arrangement in which it distills a model for a customer to turn into a product. The company says it does not sell the underlying data.

Make the crash playable

Near the close, an interviewer proposes generating viral gameplay clips that nobody played. The guest rejects their appeal because the human experience is central to games. The more attractive application is letting a player—or a friend—enter a recorded moment and play it. Making every Medal clip playable remains a research goal, connecting a consumer feature directly to the reinforcement-learning roadmap.

The guest describes Medal as episodic memory for simulation: hours of play reduced to a few memorable minutes, including exceptional successes and failures. Truck crashes illustrate the proposed training workflow: 1. Select a negative event. User-written titles help identify crash clips from games such as Euro Truck Simulator and American Truck Simulator. 2. Recover the lead-up. Recorded action labels preserve what the player did before the crash. 3. Reconstruct a playable scenario. A world model turns the recorded situation into an environment where an agent can act. 4. Reward performance within the scenario. The agent learns through interaction with the reconstructed situation. Scaling this process to billions of clips would move training from imitating recorded actions to trying actions within those situations.

GI also sees its proprietary corpus as a way to support more open research: publishing methods would still leave competitors without the data needed to reproduce its models. It reports a partnership with an open-science lab in Paris and welcomes university research access. One proposed project would use truck-crash data for negative-event prediction, potentially for trucks in India; transfer to real roads remains a research aspiration.

The closing invitation is for researchers and infrastructure and research engineers to join an ambitious scaling program. Runs involving thousands, tens of thousands, and eventually hundreds of thousands of GPUs describe the intended direction, rather than completed training runs.

The 2030 ambition extends from simulation to physical interaction. The guest wants GI to become a standard for intelligence and power 80% of AI-mediated “atoms-to-atoms” interactions—physical-world interactions driven by AI. That future assumes sufficiently capable models would encourage robotics supply chains to converge on simpler systems with gaming-style controls, while specialized stacks serve other needs.

Simulation is expected to lead because it has fewer constraints, including safety constraints, and can support faster adoption and scientific applications. The guest imagines roughly 100 times more activity there than in the targeted physical-world interactions. These are ambitions, with playable clips providing the concrete research path: learn from recorded actions, generate their consequences, and turn memorable situations into opportunities for further interaction.