From Image Captioning to Editable Worlds

Turn an image into a world

Give Marble a photograph, several images, or a text description, and it generates a matching 3D world. This is the starting task in the November 2025 Latent Space conversation with World Labs cofounders Fei-Fei Li and Justin Johnson. Marble serves two ambitions: make world generation useful for creative work now, and build toward [spatial intelligence](https://drfeifei.substack.com/p/from-words-to-worlds-spatial-intelligence?utm_source=substack&utm_campaign=post_embed&utm_medium=web)—the ability to reason, understand, move, and interact in space.

The partnership grew from a shared research trajectory. Justin joined his advisor’s Stanford lab in 2012, the quarter AlexNet appeared, watching the ImageNet excitement without participating in its announcement. After his PhD, his interests shifted toward 3D vision, graphics, and generative modeling: he expected computer vision’s next decade to bring AI into the physical world. When advisor and former student reconnected, they found their interests had converged on world models.

Justin’s explanation for the timing starts with data and compute. He estimates roughly a thousandfold improvement in performance per GPU since the AlexNet era, and a millionfold increase in compute available to a single model once larger clusters are included. These are broad historical estimates, without a specified workload or numerical precision. His argument is that visual and spatial data require much more processing than language, giving expanding compute capacity a demanding application.

Academia’s role beyond the largest models

Commercial products and open research need not displace one another. Asked whether funding and return on investment explain the shift from public challenges to products, a guest points out that open, closed, and proprietary approaches coexisted during the ImageNet period. The greater concern is academia’s resource imbalance. Stanford’s Behavior dataset and benchmark for robotic learning in simulated environments exemplifies continuing open science. Policy work on a National AI Research Resource, envisioned as a national compute cloud and data repository, addresses the resource problem.

Justin recalls approaching a Google team after undergraduate computer-vision research and being told to complete a PhD first. Scaling has since changed that industry–academia boundary. The proposed academic agenda emphasizes unconventional algorithms, new architectures, theoretical foundations, and interdisciplinary basic science. One speaker worries that universities are becoming preparation for industrial labs rather than places to pursue ideas that may fail or take years to establish.

A concrete research bet is to design neural networks for distributed computation from the outset. Matrix multiplication fits GPUs well, but practical training increasingly spans entire nodes and clusters. Could different computational building blocks better fit that hardware? An interlocutor raises the hardware lottery: NVIDIA’s established position may favor continued investment in its existing approach. The proposal remains a multiyear question about hardware ten or twenty years out, rather than a promised breakthrough on a startup’s three-month schedule.

From one caption to many regions

The conversation returns to an earlier problem beyond object recognition: describing an image in natural language. The captioning system combined a convolutional network’s image representation with a recurrent language model called LSTM to generate one sentence. The researchers identify it as a 2015 conference paper. One had expected scene storytelling to occupy an entire career; a working captioning system arrived far sooner. They recall discovering that Google had independently developed similar work, with New York Times reporter John Markoff covering both efforts together.

A related 2015 project removed the image component to study small recurrent language models in isolation. While predicting tokens in Linux source code, an LSTM unit activated at an opening parenthesis and deactivated at a closing one. This was an observed correlation between internal activity and text structure, not a causal demonstration of what the unit controlled.

Dense captioning addressed what a single sentence left out: the different parts of a scene. The 2016 system drew boxes around interesting regions and described each—for example, a water bottle on a table or a person wearing a black shirt. One joint network represented the whole image, proposed regions, represented each region, and generated its text. The researchers describe all these operations as running in a single forward pass, in an implementation that predated PyTorch.

The captioning work also became a browser demo: webcam images traveled from a laptop to a Stanford server, which ran the model and returned predictions. The researchers recall showing a captioning demo at a 2015 conference in Santiago while inference ran in California. It managed approximately one frame per second with substantial delay—working interaction, but far from immediate feedback.

Looking right and behaving correctly

Moving from captions to worlds changes what a model must represent. A guest argues that three-dimensional space and its changes over time have structure absent from a one-dimensional generative signal. Another advocates pixel maximalism: printed text belongs to the visual world, and reducing a page to text can discard useful font and layout information. DeepSeek-OCR enters as an example of processing text through images, with a possible efficiency tradeoff rather than a quantified advantage.

More visible information still leaves hidden forces unresolved. A host describes an unnamed orbital-modeling paper in which plausible trajectories accompanied incorrect force vectors, then asks a concrete version of the question: Marble generates attractive arches, but does it represent what keeps an arch standing? The reported appearance establishes visual plausibility, not structural competence.

Two learning approaches frame the problem: - Explicit physical supervision: obtain forces from a traditional simulator and train a model to predict them. - Implicit physical learning: train on a broader prediction task and hope useful physical structure develops internally. One guest challenges the leap from learned patterns to causal laws. Another suggests human learning might also fit patterns, but across broader experience, longer horizons, or different objectives. Whether predictive training yields the desired generalization remains unresolved.

The application determines the required correctness. A virtual film backdrop may succeed by looking plausible; a building intended for construction must model forces correctly. One respondent expects better models and more data to bridge that gap. Another favors a unified model while leaving implicit force learning an open question. Traditional physics engines can help by generating training data, effectively distilling their behavior into neural-network weights, even if they do not cover every situation a general world model should handle.

The same distinction limits what to infer from impressive behavior. People attribute understanding to others partly by analogy with their own thought processes. A guest argues that this analogy is unreliable for AI: a convincing output does not establish human-like cognition or additional, untested capabilities.

Gaussian splats make camera control concrete

Marble makes part of the larger agenda usable: generate a 3D scene from text or images, edit it, and export it in several formats. Example edits include making a bottle blue, removing a table, and rearranging microphones. These changes offer early hints of counterfactual modeling—representing how a world could change—while serving emerging uses in gaming, visual effects, and film.

The model natively outputs Gaussian splats: small semitransparent particles with positions and orientations in 3D space. Many splats together form the scene. Efficient rendering on client devices lets users place, orient, and move a camera through it, then record the result. The guests contrast this geometric control with verbal camera instructions to a video generator, where specifying a precise movement does not necessarily produce one.

Splats are Marble’s current representation, not a requirement for world models. World Labs’ RTFM model generates individual frames as users interact. Tokens representing chunks of a 3D world are another proposed direction. Each choice changes the unit the model produces and how interaction can work.

Adding motion and physical interaction opens two routes: - Predict properties, then simulate: assign splats mass or virtual springs coupling neighboring particles, then use a classical or learned simulator. - Regenerate after an action: have the model produce a revised scene, potentially expressing changes beyond explicitly modeled physical properties, at substantially greater computational cost. These are research directions for possible successors, rather than announced Marble capabilities.

Scene detail has a rendering budget. Mobile devices and virtual-reality headsets constrain the number of splats a scene can contain. High-resolution rendering at 30–60 frames per second on a four-year-old iPhone is offered as a design target, not a benchmark result. Newer hardware or relaxed frame-rate and resolution requirements permit more splats and finer scene detail.

From synthetic training worlds to kitchen remodels

Robotics supplies another reason to generate environments. High-fidelity real-world training data is scarce, while internet video lacks the controllability needed to train agents that act in an environment. Simulation offers a middle ground, but building it requires curating assets, composing situations, and providing varied states for interaction. Marble is proposed as a way to generate those synthetic worlds; improved robot-training outcomes remain a prospective benefit.

At the time of the conversation, Marble focused on creative industries, with the underlying technology intended to reach other sectors. The guests leave open whether virtual-world applications must precede embodied ones.

A kitchen remodel offers a more immediate workflow: reconstruct the space from two images, then edit countertops, floors, or cabinets to preview alternatives. A guest reports that this already worked well without features built specifically for the task. Another reports early beta users building interior-design applications through programmatic access.

What language leaves out

Grasping a mug makes spatial intelligence tangible. You must perceive the mug and its surroundings, locate your hand, match its opening to the mug’s geometry, and contact suitable points. Language can describe these operations, but narration cannot itself carry out the interaction. At another scale, a guest offers the deduction of DNA’s double helix as an explicitly simplified example of reasoning about molecules and chemical bonds in three dimensions.

The argument treats language as a lossy description of embodied experience. Large language models moved directly into abstract language and reasoning, potentially omitting part of the spatial basis humans take for granted. Perception’s apparent effortlessness can hide its difficulty, whereas learning grammar and writing makes effort conspicuous. That does not mean vision arrives fully developed: a guest qualifies the claim by noting that visual acuity, perception, and motor control develop over time.

An object resting on another without falling through it illustrates physical commonsense. A respondent suggests that people interpret such descriptions by mapping them into a familiar three-dimensional world. This motivates combining spatial and linguistic capabilities. Marble already accepts language; the guests anticipate cooperation between models, while a universal model remains speculative. Language also retains an immediate interface role: people want to talk to systems.

Predicting a trajectory, discovering a law

Could a world model reconstruct physical theory without being given existing laws? A guest proposes feeding celestial-motion data to a model and testing whether Newtonian laws emerge. Two voices predict they probably would not. They distinguish predicting a planet’s position on successive days from formulating an abstraction such as F = ma. Both the trajectory capability and the failure to discover laws are conjectures here, not experimental findings.

A host asks how visual observation would move a model beyond an apparently geocentric account. The response turns to learning through interaction: form an expectation, act, observe a mismatch, and update the belief. Searching for keys is the everyday example—expect them downstairs, look, fail to find them, and revise their presumed location. One speaker proposes that scientific theory-building scales up this process. The learning method is a separate question from whether inputs are linguistic or spatial.

Another participant describes experimental learning as maintaining possible worlds consistent with the evidence, then choosing actions that eliminate alternatives. They extend the idea to theory of mind: testing hypotheses about what another person is thinking. Observations support candidate explanations; interaction helps distinguish among them.

Keep attention; reconsider the sequence

The final technical exchange separates new modalities from new architectures. One guest favors retaining working components such as attention while exploring different architectures and learning methods. Another expects world models to move beyond sequence-to-sequence modeling. The response distinguishes this from abandoning transformers: recurrent networks are architecturally committed to sequences, while transformers can operate on sets of tokens.

The explanation follows the transformer’s operations. Feed-forward networks and per-token normalization act independently on each token. Attention connects tokens and is described as permutation equivariant: rearranging inputs produces the corresponding rearrangement of outputs. Positional embeddings supply sequence-order information. This is a useful account of the set-based machinery, with a scope limitation: the discussion does not address attention masks, so it does not establish the stronger claim that positional embeddings are the only possible source of order.

World Labs closes with a recruitment pitch spanning model research, training optimization, inference, product engineering, and business roles. Exposing the model through Marble is presented as an opportunity to advance both the research and the product.

Then an interface problem brings the conversation back to the user. The team says advanced editing can change bottle colors, floors, and trees, noting that users have had only about 24 hours to discover it. A participant reports selecting Create while trying to edit and getting a different world. The team acknowledges room to improve the interface, without completing the navigation fix. Alongside its endorsement of intellectual fearlessness, the closing exchange leaves a concrete product task: make the model’s capabilities easier to find and use.