Patrick Esser

Patrick Esser is a generative-image researcher and a cofounder of Black Forest Labs alongside Robin Rombach and Andreas Blattmann. His research contributions include jointly credited work on VQGAN, coauthorship of latent diffusion models, and first-listed authorship of the 2024 ICML paper Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.[6][3][4][5]

In the 2024 paper, Esser and his collaborators studied rectified flow, a generative formulation that connects data and noise along a straight line. They changed the training noise sampling to emphasize perceptually relevant scales. The team reported that this approach performed better than established diffusion formulations in its large-scale study of high-resolution text-to-image synthesis.[5]

The team's transformer architecture uses separate weights for image and text while allowing information to flow in both directions between their tokens. The authors reported improvements in text comprehension, typography and human preference ratings. They also found predictable scaling trends and a relationship between lower validation loss and better generated images in their metrics and human evaluations.[5]

Esser appeared in Latent Space's ICML 2024 coverage in a segment about Stable Diffusion 3.[1]

Earlier work supplies context for these image-generation contributions: Esser collaborated on models that represent visual content through learned parts, and later coauthored latent diffusion research.[3][4]

Esser and Robin Rombach share equal-contribution credit on Taming Transformers for High-Resolution Image Synthesis, with Björn Ommer as their coauthor. Presented as an oral paper at CVPR 2021, the work combines a convolutional VQGAN that learns a codebook of context-rich visual parts with an autoregressive transformer that models how those parts are composed. This combines the efficiency of convolutional approaches with the expressivity of transformers.[3]

Esser also coauthored High-Resolution Image Synthesis with Latent Diffusion Models with Robin Rombach, Andreas Blattmann, Dominik Lorenz and Björn Ommer. Rombach and Blattmann receive the paper's equal-contribution attribution.[4]

At Runway, Esser contributed to Gen-1. The February 2023 research page credits him alongside Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog and Anastasis Germanidis, with Runway as their affiliation.[7]

His public GitHub profile is pesser, which identifies him as Patrick Esser and links the handle @pess_r.[2]