Sander Dieleman

Sander Dieleman is a machine-learning researcher working on generative modelling and representation learning across audio, images and video. He coauthored WaveNet and Continuous diffusion for categorical data (CDCD), and contributed to Google DeepMind’s Lyria, Imagen 2, Imagen 3 and Veo models. His public technical writing connects diffusion with autoregression while examining where that connection holds—and where it breaks down.[3][6][7][4]

In his September 2, 2024 essay, “Diffusion is spectral autoregression”, Dieleman explains why image diffusion tends to generate large-scale structure before fine detail. Natural-image spectra approximately follow a declining power law: lower spatial frequencies generally contain more power than higher ones. Gaussian noise has flat spectral power in expectation, so adding it progressively obscures weaker high-frequency structure. Reversing that corruption reconstructs an image roughly from lower to higher frequencies—what he calls “a soft version of autoregression in frequency space”.[4]

The interpretation is approximate and holds in expectation across images. Individual spectra need not decline monotonically, and the transition between signal and noise spans a broad range rather than a sharp frequency boundary. Dieleman credits Rissanen and colleagues’ work on inverse heat dissipation as direct inspiration. He presented the essay as a reproducible Python notebook so readers could rerun and modify its plots and analyses.[4]

His audio experiments reveal a useful limit. The speech and music waveforms he analyzed lacked the monotonically declining spectrum required for the same frequency-order interpretation. Log-scaled mel-spectrograms, treated as images, showed a power-law-like spectrum instead. This prompted him to revisit his December 2022 objection that spectrograms’ time and frequency axes have different semantics. In the 2024 essay, he conceded that treating them as images “seems to work pretty well in practice”, suggesting their spectra as one possible explanation. He did not establish that this spectral property determines generation quality.[4]

Dieleman also presented these intuitions publicly. Latent.Space’s December 10, 2024 ICML compilation features his recorded talk, “Wading through the noise: an intuitive look at diffusion models”, beginning at 02:27:07.[1]

Dieleman was the first-listed author of Continuous diffusion for categorical data, submitted in November 2022, with Laurent Sartran, Arman Roshannai, Nikolay Savinov and ten other collaborators. CDCD tackles a mismatch between continuous diffusion and discrete data such as language, preserving diffusion that is continuous in both time and input space.[7]

A deliberate project goal was to make diffusion familiar to language-modelling practitioners. In his January 2023 explanation, Dieleman describes a standard Transformer and a training procedure resembling BERT: embed the input tokens, add noise to their embeddings, and predict the original tokens with cross-entropy through score interpolation. An active-learning strategy called time warping adapts the distribution of sampled noise levels during training, addressing how to weight the different denoising tasks.[5]

Despite working on CDCD, Dieleman judged in January 2023 that diffusion was still premature as a serious alternative to autoregression for language modelling at scale. His reasoning included training and sampling costs: autoregressive training supplies gradients for every sequence position in one forward-backward pass, while diffusion samples a noise level; autoregressive generation can also reuse cached activations, whereas diffusion generally processes the whole noisy input at each step. He cautioned that fewer sampling steps alone do not establish lower computational cost.[5]

By September 2024, he described the division between autoregression for language and diffusion for other modalities as an “unstable equilibrium”, anticipating models that natively combine language, images and sound. He considered either broader use of autoregression or multimodal diffusion plausible, without choosing a winner. His qualification was practical: a unified approach should preserve diffusion’s ability to change sampling-step counts at inference time, accelerate generation through distillation, and improve samples with classifier-free guidance.[4]

These generative-model contributions followed earlier work on learning representations from musical audio and exploiting image symmetry. At Ghent University, Dieleman’s doctoral research concerned hierarchical representations of music for classification and recommendation; he defended his thesis in January 2016.[2][3]

In the 2013 work Deep content-based music recommendation, Dieleman, Aäron van den Oord and Benjamin Schrauwen addressed the cold-start problem: songs without listening data cannot be served well by conventional collaborative filtering. Their model predicted recommendation factors from music audio using a convolutional neural network. Dieleman and Schrauwen’s 2014 End-to-end learning for music audio then investigated learning directly from raw waveforms, with networks discovering frequency decompositions and invariant features rather than relying on supplied spectrograms.[3]

During his PhD, Dieleman also competed on Kaggle. In his April 5, 2014 Galaxy Zoo account, he reported finishing first. The task was to predict the fractions of volunteers giving particular answers about galaxy morphology, preserving disagreement among observers rather than assigning each galaxy a single definitive class.[8][2]

He identified overfitting as his main obstacle and attacked it through augmentation, regularization and increased parameter sharing. His architecture exploited rotational symmetry: regular, rotated and mirrored views were divided into sixteen aligned, overlapping parts, processed by the same convolutional architecture, and combined at the top of the network. His final submission averaged seventeen models, each evaluated across sixty input transformations. He subsequently wrote the galaxy-morphology paper with Kyle W. Willett and Joni Dambre.[8][3]

In September 2016, Dieleman coauthored WaveNet with Aäron van den Oord, Heiga Zen, Karen Simonyan and five other researchers. WaveNet generates raw audio probabilistically and autoregressively, conditioning each sample’s predictive distribution on preceding samples. The team reported that human listeners rated its English and Mandarin text-to-speech output as more natural than the parametric and concatenative systems evaluated in the paper. Dieleman was also one of the lead developers of Lasagne, the Theano-based neural-network library.[6][2]