From Denoising Images to Inferring 3D Worlds — Part 3 of 6

A blurry prediction can guide generation

Take a clean image x₀ and add Gaussian noise to produce xₜ. Once noise obscures the details, the observation no longer identifies one original image. Reversing this process requires a denoiser to use learned knowledge of the data distribution—the range of images it is modeling—to decide which clean images remain plausible.

With a mean squared error objective, the best clean-image prediction is the conditional expectation `E[x₀ | xₜ]`: an average of plausible clean images given the noisy observation. Shared structure survives; differing details blur together. Geometrically, this resembles a centroid toward which the sampler can move. That is an intuition, with limits: images occupy a high-dimensional pixel space, where conclusions drawn from two-dimensional geometry need not hold.

Sampling repeatedly revises a locally useful direction. The sampler takes a small step toward the prediction, then asks the model again. As noise decreases, fewer clean images remain plausible and the prediction becomes more specific. Some samplers add noise back between steps; the intuitive justification offered here is robustness to systematic prediction errors that could accumulate. The intended endpoint is a sample from the data distribution, not necessarily the image used to illustrate forward corruption.

Train one denoising step, reuse it many times

Predicting the clean image is one choice of target. A common alternative predicts the Gaussian noise ε. The linear corruption relationship lets a trained model’s prediction be converted between these representations given xₜ and the corruption coefficients. Other targets include linear combinations of x₀ and ε, such as v-prediction and the flow-matching target expressed here as ε − x₀. Predicting the previous step, xₜ₋₁, motivates the original denoising diffusion probabilistic models (DDPM) framing: reverse corruption one step at a time.

Convertible predictions do not imply equivalent training objectives. Choosing a target changes the relative weighting of noise levels in the aggregate loss, which can affect perceptual output quality.

Training avoids simulating the full corruption trajectory: 1. Take a clean example x₀ and sample a timestep t. 2. Construct xₜ directly at that noise level. 3. Predict the chosen target, such as x₀ or ε. 4. Minimize the squared prediction error. For clean-image prediction, the conditional expectation is precisely the minimizer of mean squared error.

This makes diffusion a deep recurrent computation trained without backpropagating through the sampling loop. A 100-layer denoiser reused across 100 timesteps gives an unrolled computation 10,000 layers deep. The score-matching training approach discussed here requires gradients through only one denoising application, even though generation repeatedly invokes the network.

Noise levels determine which visual details matter

The frequency content of images offers an explanation for diffusion’s effectiveness. Natural-image power falls approximately with frequency raised to a negative exponent, often around −2, while the Gaussian noise used here has a flat spectrum. Adding noise creates a hinge-shaped spectrum: weaker high-frequency details disappear beneath the noise floor before stronger low-frequency structure.

Reversing that process suggests spectral autoregression—generation progressing approximately from low frequencies toward high frequencies. The presenter credits Severi Rissanen and colleagues’ work on generative modeling with inverse heat dissipation as inspiration. This interpretation applies to images and video, with audio also described as following the relevant power law; it need not extend to language.

Reweighting noise levels implicitly emphasizes different spatial frequencies. A weighting that favors frequencies important to human vision can therefore act as a perceptual loss. The presenter proposes this as one reason diffusion advanced so rapidly in image generation.

Video generation trades sequence length against memory

Autoregression generates an ordered sequence one step at a time; diffusion learns to reverse a corruption process. Both divide generation into repeated network calls. For video, this creates three architectural choices: - Autoregress over video tokens. Serialize spatiotemporal patches into a sequence. The presenter favors this for integration with language models, while warning that long sequences can accumulate errors. - Diffuse over the whole video volume. This is presented as more robust to error accumulation and amenable to faster sampling through distillation, but generating a large volume at once creates substantial memory pressure. - Combine temporal autoregression with spatial diffusion. Each step resembles image-conditioned image generation, allowing reuse of image methods. Temporal dependence can reintroduce error accumulation, including during distillation.

A separate choice is what detail deserves modeling capacity. The presenter argues that likelihood measured directly over audiovisual inputs overemphasizes perceptually unimportant high frequencies, linking this to the difficulty of scaling PixelCNN and WaveNet to larger inputs. Learned latent spaces—compressed feature representations—can abstract away irrelevant variation. A convincing field needs a convincing grassy texture without separately modeling every possible arrangement of blades.

Consider the proposed example of shifting the grass texture in a dog-in-a-field image by one pixel. The presenter expects viewers not to notice, motivating different priorities at different scales: - Fine textures may need only a few convincing realizations. Adversarial models suit this goal because broad coverage of every tiny variation may be unnecessary. - Large-scale content benefits from broader coverage. Likelihood-based and diffusion models are proposed as better fits for capturing the range of possible configurations. Diffusion’s noise-level weighting already reduces emphasis on some fine detail; latent representations can amplify that effect.

Guidance strengthens conditioning by narrowing variety

Guidance trades diversity for sample quality. Classifier guidance uses a classifier that can handle noisy inputs to obtain an input-space direction favoring the requested class. Adding that direction to the denoiser’s direction steers sampling toward the condition. In the Bayesian interpretation, combining an unconditional image model with a classifier p(C | x) allows conditioning to be added after the base model is trained.

A guidance scale γ amplifies the classifier-derived direction. For a rabbit condition, stronger guidance emphasizes recognizable rabbit characteristics. In the probability-space account, this corresponds to raising the classifier probability to a power and renormalizing. The presenter interprets it as temperature tuning—sharpening a distribution—at the classifier’s level of abstraction.

Classifier-free guidance uses two predictions from the generative model itself. Occasional conditioning dropout during training lets the same model operate with or without the condition. During sampling, the difference between conditional and unconditional predictions provides a direction associated with the requested condition; scaling that difference strengthens conditioning. The Bayesian interpretation treats P(X | C) / P(X) as a classifier-like signal whose strength is adjusted by γ.

The presenter describes classifier-free guidance as less prone to adversarial input-space directions than classifier guidance. Qualitative comparisons from GLIDE illustrate better-looking but less varied guided samples, and motivate the claim that many leading image models depend heavily on guidance. Guidance also applies to autoregressive models, though the presenter believes it is more effective in diffusion. The Bayesian account remains an intuition rather than a complete explanation; alternative accounts are acknowledged.

Latent diffusion balances compression, quality, and learnability

The diffusion presentation closes with Imagen 3, a text-to-image latent diffusion model, and Veo, a text-to-video latent diffusion model, both announced at Google I/O that May. In this 2024 talk, Imagen 3 was expected to become available soon, with Veo taking longer. Imagen 3 marked a move to latent diffusion for the team’s model family. Veo encodes prompts and optionally input frames, generates in latent space, and decodes to pixels at resolutions up to 1080p. The sample assessments are qualitative; no quality benchmark or video duration is specified.

The presenter also points to a personal blog series on diffusion and generative models, emphasizing intuition over mathematical rigor. In the ensuing questions, the expectation is that image and video generation still have substantial gains ahead from scaling, with both judged earlier in that process than language modeling.

Latent diffusion usually uses two-stage training: learn a compressed representation and freeze it, then train diffusion on the extracted features. The representation therefore need not keep changing during diffusion training. Its design balances three requirements: - Compression: remove redundancy to reduce memory use for large images and videos. - Reconstruction quality: retain enough information to recover a good output. - Learnability: keep the representation tractable for a generative model to learn. More aggressive compression can hurt learnability; entropy coding the latents is offered as a hypothetical example.

Evaluation still depends heavily on human attention. Roughly 200 images arranged in a grid can reveal broad behavior at a glance. Video requires closer inspection of individual moving samples, and audio must be heard sequentially. Conventional metrics are treated as useful alarms for serious failures, rather than complete quality measures. Predicting human judgments with a learned evaluator looks promising, but optimizing that evaluator as a reward could undermine its usefulness as a metric.

The final questions address two different limits: - Near-copying and generalization: more data is proposed as a remedy for outputs close to training examples and as a way to improve combinatorial generalization—combining elements into configurations absent from training. This is an expectation, not a measured scaling result. - Physical consistency: more data and capacity are expected to help video models learn physical behavior. Data curation and physical priors are nearer-term options, tempered by the concern that hand-built constraints may eventually lose to learned behavior at scale.

Three photographs do not determine a 3D world

Ben Poole begins with the need for explicit 3D content. Mixed-reality headsets, games, and mobile experiences let people move around scenes and interact with objects. Better video alone does not solve asset modeling, lighting, rigging, or integration. He also connects 3D priors to robotics: locating and grasping an object requires spatial understanding.

A neural radiance field, or NeRF, represents a scene through a differentiable network. In Poole’s explanation, it maps 3D positions to density and color. Rays cast from known cameras query points along their paths; accumulating their densities and colors produces pixel predictions. Comparing rendered predictions with captured photographs trains the representation. ZipNeRF illustrates how this approach can reconstruct an entire navigable house.

The catch is viewpoint coverage. Poole characterizes these reconstructions as largely interpolating between observations, with little ability to infer unseen regions. A Lego bulldozer needs photographs from nearly all sides. With only three photographs, the demonstrated reconstruction matches the captured images but produces inaccurate novel views and geometry. Matching observations does not establish usable depth: that shape would be a poor basis for a robot’s grasp.

The shared task is inference over possible 3D worlds given partial observations. Those observations determine how much invention is necessary: - Dense-view reconstruction: many photographs strongly constrain the result. - Single-image completion: one visible view is constrained, while unseen content needs plausible completion. - Text-to-3D generation: a description supplies looser constraints on the desired object or scene.

Training directly on 3D data faces expensive acquisition, competing representations, and limited coverage, often centered on isolated objects. Ordinary photography rarely captures objects from every side. Poole instead proposes 2D priors for 3D inference: learned image distributions can assess rendered views, even though they do not directly define a distribution over 3D structure.

DreamFusion turns an image prior into a 3D optimization signal

A bear playing a guitar might look convincing from one angle and fail from another. The proposed criterion is to render a candidate 3D model from multiple viewpoints and assess each rendering with a 2D prior. This leaves several linked design choices: - Conditioning: what information should the image prior receive? - Objective: how should visual quality be measured and optimized? - Viewpoints: which cameras expose useful evidence without ending up inside an object? - Representation: which form of 3D model suits the task?

DreamFusion introduces score distillation sampling, a way to turn a diffusion prior into a differentiable optimization signal. The method assesses an image rendered from a parameterized representation, then propagates gradients through that image into the underlying 3D parameters.

The motivation is to seek a mode—a high-density region—of the learned image distribution, hoping it corresponds to a visually plausible result. Noisier versions of the distribution are smoother, and diffusion provides gradients of log density without explicitly exposing those densities. In the peacock-on-a-surfboard example, optimization starts from a random NeRF and gradually builds a model whose novel-view renderings look plausible. These text-to-3D generations require no 3D training data.

Independent novel views create conflicting worlds

Text-only control becomes frustrating when the goal is to reconstruct a photographed scene: enter a prompt, wait, inspect a poor result, and try again. The image-conditioned follow-up—introduced as “ReConfusion” and later called ReconFusion in the recording—grounds generation in photographs. In the three-photo bulldozer demonstration, the presenter reports improved novel views and decent geometry, without a quantitative accuracy measure.

ReconFusion takes input images and their camera poses, then predicts one novel view at a target pose. PixelNeRF supplies features rendered at that target pose to a latent diffusion model; image embeddings from CLIP replace its text features. Training therefore requires sets of photographs with camera poses. The described mix includes Real Estate 10K for scenes, CO3D and MVImageNet for contextual object views, and synthetic renders from 3D models.

Independently plausible views can disagree about the same scene. ReconFusion predicts novel images independently, leaving reconstruction to reconcile competing explanations. In uncertain regions, optimization can average those explanations into blur. Single-image hydrant and bench examples illustrate this failure.

Iterative optimization also costs time: roughly half an hour for a DreamFusion asset and an hour for ReconFusion, as reported without hardware or other timing conditions. The practical objection is concrete. A person could spend that hour taking more photographs; a robot cannot wait that long to reconstruct a scene before moving its hand.

CAT3D generates correlated views before reconstruction

CAT3D, Create Anything in 3D, learns correlations among views by drawing on video diffusion. It accepts observed images, their camera poses, and multiple target viewpoints. Images are encoded into latents, cameras use a ray representation, and a mask identifies observed and unobserved views. The goal is to generate a set of images that could all belong to one 3D scene.

The procedure separates generation from reconstruction: 1. Generate correlated views with the multi-view latent diffusion model. 2. Feed them into a 3D reconstruction pipeline, using a robust loss to reconcile remaining inconsistencies. The presenter reports about a minute for this procedure versus about an hour for ReconFusion, without specifying timing conditions. Qualitative comparisons using three input images emphasize better-looking completion of uncertain regions, especially backgrounds.

Single-image dog examples illustrate the payoff: reconstructed 3D models permit more camera movement than warping a color image with a depth map. A text-to-image model can also provide the input, creating a prompt-to-image-to-3D cascade. Factory robots, fantasy creatures, and small scenes illustrate the appeal of interactive viewing beyond a prerecorded object rotation.

The remaining limitations affect different parts of the pipeline: - Camera placement remains manual. A few chosen trajectory sets cover different scenes; learning placement from scene content remains an open goal. Camera conditioning also affects generated-view quality. - Posed data limits scaling. Camera-pose estimation struggles with dynamic scenes, restricting usable training data. - Plausible views do not guarantee accurate geometry. Attractive renderings can conceal incorrect recovered shape. - Static-scene assumptions are fragile. Even a camera operator’s moving shadow changes a scene across views.

Representing views as a set, rather than an ordered video, permits strategies that generate multiple frames in parallel. Sampling first and reconstructing second is flexible, but requires a more expensive prior to generate correlated views. The presenter contrasts this with score distillation and its variational variant: those optimization methods can express uncertainty about unseen views, but are judged slower, more complicated, and lower quality than direct sampling. Closing that gap remains an inference research problem.

A viewable reconstruction is not yet a usable game asset. The closing obstacles are inaccurate geometry, missing material properties, unsuitable mesh topology, and baked-in lighting. Attractive novel views leave substantial work before an asset can be integrated into a game.

Where should 3D consistency live?

The questions return to enforcing consistency during sampling. The response describes MultiDiffusion as resolving disagreements by averaging within diffusion. Extending that operation to 3D is harder because updating a 3D representation may require several optimization steps instead of an analytic average. Enforcing consistency during sampling remains a promising direction.

One result complicates the architectural choice: moving from ReconFusion to CAT3D removed 3D structure from the diffusion model and improved results. How much structure belongs in the learned prior, and how much should reconstruction extract afterward? The desired system would learn from images while producing 3D outputs that update interactively as someone captures a scene. View Set Diffusion and Render Diffusion are cited as attempts to embed 3D structure within diffusion, with scaling still an obstacle and hybrids a promising direction.

A final exchange considers oversaturated text-to-3D results. One proposed explanation is that guidance favors prompt-associated features while moving away from the unconditional prior: a frog prompt might encourage a green background because of learned associations, without improving the sample. Optimization losses are also implicated. The explanation remains incomplete: strong classifier-free guidance can produce excessive contrast and saturation, but score distillation is described as producing much more severe artifacts.

Flow matching starts with transport to one example

Ricky TQ Chen, introduced at the time as being with Meta AI/FAIR, frames flow matching as a recipe for learning how to transport samples into a data distribution. He presents its scope across Euclidean, Riemannian, and discrete domains, introducing the recent Discrete Flow Matching paper as an application to discrete data. Material, code, and text generation are mentioned as applications, without performance results here.

The starting point is simple: define a conditional velocity that transports particles toward one target example x₁. Then learn the expected conditional velocity. Following the resulting velocity field with the same transport rule can generate samples from the training data distribution. Chen connects this relationship to the continuity equation and the linearity of the divergence operator, leaving the fuller derivation for later.

In the opening Euclidean construction, conditional probability paths pₜ(x | x₁) converge to a delta distribution—all probability concentrated at the target x₁. Averaging those paths over targets drawn from the data distribution q(x₁) gives a path from noise to the data distribution at time one. Part 3 of the ICML 2024 recording ends at this construction, before the discrete-domain mechanics are developed.