ICLR 2024, Part 1 of 4: Learning Representations, Compression, and Control

Recover a clean digit from a noisy one

Take a clean MNIST digit, add noise, and feed the corrupted image into an encoder. Train the decoder to recover the clean original. This gives a denoising autoencoder its learning task: preserve the digit’s structure while removing corruption. The opening tutorial in Latent Space’s ICLR 2024 recording extends the approach to neural inpainting: blank out a rectangle and train the network to reconstruct the full image. Removing a parked car from a film scene is one suggested application.

A variational autoencoder (VAE) changes what the encoder produces. In the formulation introduced here, it maps an image to a distribution over latent codes, represented by vectors for its mean and standard deviation. The decoder receives a sample from that distribution. Training balances two demands: - Reconstruct the input: an expected reconstruction loss accounts for sampling the latent vector. - Keep the distribution near a reference: a KL-divergence term penalizes deviation from a Gaussian with zero mean and unit standard deviation.

Sampling raises a backpropagation problem: how can the decoder’s loss update the encoder’s distribution parameters? The reparameterization trick writes the sample as z = μ + σ × ε. Randomness lives in external noise ε; the learned mean μ and scale σ participate in differentiable multiplication and addition. Gradients reach those parameters through the transformation, without differentiating through the noise-sampling operation itself.

Make latent inference fast enough to train

ICLR’s inaugural Test of Time Award recognized Kingma and Welling’s Auto-Encoding Variational Bayes, which combined deep learning and probabilistic modeling through a jointly trained encoder and decoder. The award also acknowledged concurrent work by Danilo Rezende, Shakir Mohamed, and Daan Wierstra, published at ICML 2014 and discussed in the retrospective as Stochastic Backpropagation.

The historical overview begins with conventional autoencoders: deterministic mappings from a high-dimensional signal into a smaller code and back, where compression encourages the code to capture structure. It cites Mark Kramer’s 1991 paper as the earliest example the presenter could find. Denoising autoencoders, highlighted in 2008, introduced a different constraint: reconstructing corrupted inputs lets learning proceed without relying on a smaller bottleneck. Later work connected denoising autoencoders to diffusion through score matching, a connection the overview names but does not explain.

The Helmholtz machine was another precedent, already pairing recognition and generative networks. It differed in its multiple stochastic layers, binary units, mean-field methods, and wake-sleep training. In the recognition-network update, wake-sleep reverses the order of the KL-divergence arguments. Having two networks was only part of the solution; their training objective mattered.

The discovery account starts from supervised deep learning, where feedforward networks efficiently optimized models whose variables were all observed during training. Autoregressive language models also fit that description, but generate dimensions sequentially, each conditioned on preceding dimensions. The retrospective identifies this sequential sampling as costly for images and video; tokenization makes the dimensionality more manageable for language.

A latent-variable generative model first samples an unobserved variable Z, then maps it through a nonlinear function into observed data X. When the conditional distribution permits it, output dimensions can be generated in parallel on a GPU. The latent space may also support controllable generation or continuous representations of discrete observations. The difficulty is inference: both the probability of an observation, p(X), and the distribution over latent variables given that observation, p(Z|X), are generally intractable for the models discussed.

Three earlier approaches exposed the tradeoff between inference speed and model quality: - Optimize a latent value for each example: treating latent variables as optimization parameters requires per-example iteration and can overfit. - Perform Bayesian inference per example: this addresses the stated overfitting problem but retains the slow inference loop. - Use wake-sleep: inference is fast, but the researchers found weaker models in their experiments, which they attributed to a poor approximation of maximum likelihood.

The VAE replaces repeated per-example inference with amortized inference: a reusable model q(Z|X) learns to approximate p(Z|X). It trains that inference model and the generative model together using the evidence lower bound (ELBO), a bound on log-likelihood. Training samples a latent value from q, evaluates the generative model and objective, and updates both models. For distributions that admit reparameterization, the sampled value becomes a differentiable function of external noise and the inference-model parameters.

The retrospective names Dropout as an inspiration for reparameterization. It also recalls a fully Bayesian extension in the paper’s appendix: use the same technique to estimate a posterior over the generative neural network’s parameters. The associated experiments did not appear in the paper.

Learned latents offer control—and moving targets

The framework extends to hierarchical latent variables, conditional priors, class conditioning, and distributions beyond Gaussians. A class-conditional model trained on Street View house numbers illustrates separating digit identity from visual style: infer an input image’s style, then generate other digits with that style. The retrospective calls this early unsupervised disentanglement, though it does not specify which part was unsupervised within the class-conditional setup.

Learning the latent space also creates optimization difficulties: - The inference objective is difficult to optimize: ELBO training optimizes the inference model through a reverse-KL objective. - Hierarchical targets move during training: changing the inference model changes the generative model’s learning targets under a hierarchical prior. Together, these can lead to poor local optima and posterior collapse, where the latent space encodes no information.

Diffusion offers an alternative: keep the inference process fixed while learning generation. The retrospective presents diffusion models as a special case of VAEs that avoids the difficulties associated with learning q. It also reports that many contemporary diffusion objectives can be rewritten as an ELBO with data augmentation in observation space.

A separate connection is latent diffusion, where a VAE compresses the data before diffusion modeling begins. Stable Diffusion uses this arrangement: the VAE reduces spatial resolution while increasing channels, giving the generative model a smaller representation to work with.

The applications survey shows what continuous latent representations make possible: - User–item modeling: encode a binary interaction matrix into continuous latents and predict the interactions again. - Lossy compression: separately weight the two VAE loss terms to trade compressed size against reconstruction error. Video compression and beta-VAE illustrate this rate–distortion tradeoff. - Treatment-response prediction: Drug Response Variational Autoencoder incorporates pretreatment and post-treatment gene expression, using shifts in latent representations to model treatment effects. - Neural dynamics: a time-dependent VAE maps neuronal spikes into evolving continuous factors and predicts spike rates. - Molecular search: encode molecules into a structured continuous space, then search it for desired properties using Bayesian optimization.

Other scientific applications include predicting pathogenicity from protein sequence alignments, reconstructing galaxies from gravitationally lensed images, and rapidly selecting anomalous particle-collision events for further study.

The retrospective then returns to the 2015 paper Deep Unsupervised Learning using Non-Equilibrium Thermodynamics. Its thermodynamic interpretation treats forward noising as information destruction and increasing entropy, with learning used to reverse the process. The presenter proposes free energy as a shared mathematical framework: parameterize Q and P as Markov chains, with a target supplied by a known distribution, data, or a reward function. This motivates connections among variational inference, sampling, optimal control, transport, diffusion, and flows, rather than developing those connections in detail.

Durk closes with an open challenge: scale a model that learns its latent space to the largest language and image settings. Fast parallel sampling and eliminating the tokenizer are proposed benefits. In his view, diffusion remains incomplete because it does not learn the Z space in the formulation under discussion. This is a research ambition, not a demonstrated scaling result.

Würstchen: shrink the space where generation learns

Würstchen targets initial training cost as well as inference. Sampler improvements, scheduler changes, and distillation applied after training can accelerate generation, sometimes at a quality cost, while leaving the initial training expense intact. Würstchen instead pushes compression further so that the generative model learns in a much smaller space.

For a 1,024 × 1,024 image, the cited 8× spatial compression produces a 128 × 128 grid. Würstchen aims for roughly 42× per spatial dimension, producing 24 × 24 while preserving enough information for faithful reconstruction. In the presentation’s diffusion-transformer example, sequence length falls from 16,384 positions to 576. These are representation sizes, not measured speedup factors.

Three components divide the work: - Stage A preserves reconstruction quality: the image autoencoder, identified as a VQGAN, uses relatively low compression and decodes its representation into the final image. - Stage B reconstructs from highly compressed latents: an efficient encoder produces those latents; a diffusion decoder uses them to reconstruct Stage A’s representation. - Stage C generates from text: it learns to produce the highly compressed latents that condition Stage B. Generation runs C → B → A. The text-conditioned model works in the smallest space, while the later stages handle reconstruction.

The presenters report faster inference than Stable Diffusion 2.1 and XL, without supplying speedup values, hardware, or sampling conditions. They also argue that Fréchet Inception Distance emphasizes distribution reproduction without adequately capturing image fidelity, motivating two distinct preference evaluations:

- Human preference: 90 participants compared paired images generated from randomly sampled prompts. The presenters report majority preference for Würstchen over Stable Diffusion 2.1, despite using the same training dataset and a fraction of its GPU hours. Exact percentages and compute counts are not given. - Algorithmic preference: a separate estimator favored Würstchen over Stable Diffusion 1.4 and 2.1, but favored the larger XL over Würstchen. This comparison was not an additional human trial.

The presentation closes by announcing training and inference code through GitHub and Diffusers, with model weights on Hugging Face.

Explain a concept, then intervene on its features

Hila Chefer’s concept-decomposition work asks two questions: which features represent a concept such as a painter, and which contribute to one particular generated image? The method uses Stable Diffusion’s vocabulary as semantic feature prototypes. A small neural network learns a coefficient for each token, producing a weighted linear combination of the vocabulary as the concept’s representation.

Learning and testing the decomposition follows a concrete procedure: 1. Generate 100 images of the concept, then add noise at different diffusion steps. 2. Train the coefficients through denoising, encouraging features useful for reconstructing those images to receive higher weights. 3. Check reconstruction with a fixed seed, replacing the original concept with the learned decomposition. 4. Zero selected coefficients and regenerate with the same seed, treating a resulting change as evidence that the removed feature contributed to that image. Chefer reports fairly accurate reconstruction in the examples discussed, without a numerical reconstruction metric.

The associations can be visual rather than merely textual. Chefer connects sweet peppers to fingers through shape, a camel to cashmere through texture and color, and a snake to horse and gecko features. In a painter example, she reports that removing the Monet feature makes the painting within the scene disappear, interpreting this as evidence that a Monet-associated style contributes to that part of the image.

The coefficients also become controls. Adjusting the stork feature in a “crane” decomposition shifts generation between machinery and bird characteristics. Chefer describes a machine shaped like a bird’s head and, for another ambiguous concept, a bass guitar with a fish-shaped body. She also applies the method to a plush toy from the DreamBooth dataset, associating it with Elmo through color and shape and with shrimp through its limbs.

Chefer proposes reducing coefficients associated with stereotypes, including gender features for professors and nurses and an obesity feature for opera singers. In one reported intervention, a generated man becomes a woman while similar clothing, glasses, composition, library background, and desk materials remain. This illustrates editing a selected feature while preserving other scene properties—a qualitative result, not an aggregate fairness evaluation.

Chefer distinguishes three explanatory approaches: - Concept-based explanation investigates how a model represents a whole concept, as this method does. - Neuron-level investigation asks what activates an individual neuron and what it does. - Classifier heat maps identify input regions associated with a prediction; she argues that these have limited explanatory value for models generating content from scratch. Even the definition of a satisfactory generative explanation remains unsettled. She presents decomposition as one approach and considers vision interpretability less mature than language-model interpretability.

The interview turns to controllability, including visual prompts such as segmentation maps that specify object locations. A deeper connection is that an internal explanation can become a control loss. In a project described in the discussion, the model sometimes omitted subjects named in a prompt. Researchers used its attention maps directly as a loss to encourage different generation behavior. Understanding the representation supplied a place to intervene; no quantitative evaluation of that intervention is given.

Why should learning one objective help another?

Ilya Sutskever frames the puzzle of unsupervised learning as objective mismatch: minimizing reconstruction or denoising loss helps a model perform a different task. He regards the ambition of discovering useful structure from unlabeled images or text as empirically fulfilled, while remaining dissatisfied with its mathematical explanation. He emphasizes continuity with older noisy autoencoders, which he sees as closely related to later language and diffusion models; earlier small-scale systems produced less impressive results.

Saying that a model learns the input distribution’s structure does not settle the issue. Sutskever raises a uniform distribution as a case where unsupervised learning would fail, explicitly qualifying the argument as only approximately true. The question is what assumptions make unlabeled data useful for the task.

He first considers distribution matching: given unpaired data sources X and Y, find a function f whose outputs f(X) have a distribution resembling Y. English and French sentences, or text and speech, are candidate pairs. Requiring transformed long sentences to resemble another language imposes constraints on f; he suggests sufficiently high-dimensional data might nearly determine the mapping. Substitution ciphers offer another example. He presents this as a setting with a guarantee analogous to supervised learning, while acknowledging that it is artificial relative to typical unsupervised-learning setups.

Compression gives him a broader framing. Predictors and compressors can be converted into one another, but compression makes the shared-structure argument easier to express. Consider two files, X and Y, and a sufficiently good compressor. Compressing their concatenation should do at least as well as compressing them separately, because patterns in either file can help encode the other. Additional savings from joint compression reveal shared structure.

Sutskever identifies that gap with algorithmic mutual information. Let X be unlabeled data and Y the data for a supervised task: patterns in X can help encode, and therefore predict, Y. A better compressor should detect more of the relationship. If a simple function maps one distribution into the other, the compressor should exploit it and might internally recover the function.

The proposed criterion is low regret: leave little predictive value unexploited. An algorithm compresses Y while being allowed to consult X. Low regret means a better compressor could not extract much more benefit from X, whether that file contains the answer or is useless. This judges how well an algorithm uses the available data without assuming all unlabeled data must help. The discussion supplies a conceptual criterion, rather than a precise comparator or formal regret bound.

Compression across modalities; token mappings learned from context

Christian Szegedy’s criticism of Language Modeling is Compression concerns novelty. The connection, he argues, is long established: language models train with a compression objective, and arithmetic coding can turn their predictions into a compressor. He had used that conversion as an interview question at Google and xAI.

What surprised him was a text-only model compressing images despite never seeing images during training. He found this cross-modal generalization interesting but saw no immediate practical application. The interview supplies no quantitative compression result.

Szegedy then describes an unnamed paper by researchers working with Percy Liang at Stanford: randomize a transformer’s embedding table after every instance, yet retain useful next-token prediction. The model learns the token mapping from context. Prediction is initially poor—typically through the first half of a sequence, in his account—and improves afterward. Less aggressive perturbations, or changes restricted to frequent tokens, are variants he discusses.

He reports improved in-context learning relative to other models, without naming a benchmark or precise comparison setup. His interpretation is that relationships between embeddings matter more than their particular values. Unlike the cross-modal compression observation, he considers this training result practically useful.

The expo-floor conversation briefly turns to engineering needs. At the time of the interview, Szegedy says xAI’s small team was hiring for gaps in diffusion, image and video generation, infrastructure, and product. He describes systems expertise as broader than Kubernetes: building highly distributed products, working with databases and large-scale data pipelines, and handling reliability and authentication.

Adversarial examples and the cost of dismissing a result

The recording returns to the Test of Time awards with the runner-up paper by Christian Szegedy, Ilya Sutskever, Wojciech Zaremba, Joan Bruna, Dumitru Erhan, Rob Fergus, and Ian Goodfellow. The award introduction credits it with exposing neural-network vulnerability to small, almost imperceptible input changes, stimulating research on adversarial learning and defenses. Szegedy calls it his first machine-learning paper, the result he is proudest of, and the worst paper he has written.

Discovery preceded publication. Szegedy and Hartmut Neven intended to write up adversarial examples in 2011 but delayed. In 2013, Zaremba contacted him about publishing the existing result alongside other findings. Szegedy regrets that Neven was omitted from the authorship, even as he celebrates the work’s contribution to a subfield with conference tracks devoted to robustness and adversarial examples.

Szegedy attributes part of the delay to prominent Google researchers dismissing the result as trivial, already-known overfitting and questioning his standing in the field. His mentoring lesson is to offer criticism without dismissing a young researcher’s direction. He values criticism but avoids telling young researchers that their ideas are trivial, unoriginal, boring, or not worth pursuing.

The technical discussion resumes with a misclassification from a 2015 Inception presentation. Szegedy says the model predicted a camel and interprets its use of the object’s shape as nontrivial for the time. He asks whether the mistake is an adversarial example, then begins arguing that it is not. Part 1 ends before that distinction is explained; the image’s correct label and the rest of his reasoning are not available here.