Noam Shazeer

Noam Shazeer is a computer scientist, Transformer coauthor and Character.AI cofounder. His contributions include attention mechanisms that combine information across a sequence, sparse expert layers that expand model capacity, and architectural changes that reduce the cost of generating text. His personal biography identifies him as working at OpenAI, with Google Gemini co-leadership and Google VP Engineering among his previous roles.[1][2][3][4][8]

His earlier work at Google provides context for these contributions: before developing neural architectures, Shazeer worked on language modeling for search spelling correction.[1][9]

Shazeer joined Google in 2000. In Paul Buchheit’s recollection, the early “did you mean” feature used an existing spelling corrector that struggled with words absent from ordinary dictionaries, such as TurboTax. Buchheit used spelling correction as an interview problem, then handed the newly hired Shazeer his code before leaving for the holidays. He recalls returning two weeks later to a substantially improved system. Shazeer later remembered the corrector as running in memory on one machine around 2000–2001.[1][10][9]

In his February 2025 interview with Dwarkesh Patel and Jeff Dean, Shazeer described the appeal of neural next-word prediction: “give me a probability distribution over the next word.” The task was simple to state and had abundant training material. Ordinary text supplies its own targets: a model can use the preceding words to predict the next one. He saw this as an ability to learn from observations of the world.[9]

The 2017 paper Attention Is All You Need introduced the Transformer, an encoder-decoder architecture for translation built around attention, dispensing with recurrence and convolution. Its contribution note credits Shazeer with scaled dot-product attention, multi-head attention and parameter-free position representation, and involvement in nearly every detail. Jakob Uszkoreit proposed replacing recurrent networks with self-attention; Ashish Vaswani and Illia Polosukhin designed and implemented the first models. Niki Parmar, Llion Jones, Łukasz Kaiser and Aidan Gomez contributed model experiments, implementation and infrastructure. The paper specifies equal contributions and random author order.[2]

Attention lets a token gather information from other positions. A query is compared with keys to produce weights, which determine how to combine the corresponding values. As an illustration, in “The animal crossed the street because it was tired,” attention at “it” could give greater weight to information from “animal” when building a representation of the pronoun. These are numerical representations learned during training.[2]

• Scaled dot-product attention divides query–key dot products by the square root of the key dimension before applying softmax. The scaling counteracts large scores that can leave softmax with very small gradients.[2]
• Multi-head attention uses different learned projections to perform several attention operations in parallel, then combines their outputs. This lets the model attend to different representation subspaces and positions.[2]
• Positional encodings supply sequence order. The original Transformer added fixed sinusoidal encodings to token embeddings; learned positional embeddings produced nearly identical results in its comparison.[2]

Self-attention connects positions without passing information through a recurrent chain, allowing substantially more parallel computation during training. Autoregressive generation remains sequential because each newly generated token affects the next step.[2][4]

In January 2017, Shazeer and six collaborators published Outrageously Large Neural Networks, introducing a sparsely gated mixture-of-experts layer. A learned gate selects a small combination of expert subnetworks for each input, so the model can hold many parameters while evaluating only a few experts at a time. These experiments placed expert layers between stacked LSTMs, recurrent networks that carry a memory state through a sequence. Building on earlier mixture-of-experts and conditional-computation research, the team addressed practical obstacles: uneven expert usage, small batches per expert and communication between devices.[3]

Shazeer later coauthored Switch Transformers with William Fedus and Barret Zoph. The work simplified sparse-model routing and addressed communication costs and training instability, extending sparse pretraining to models with up to 1.6 trillion parameters.[6]

Shazeer coauthored T5 with Colin Raffel and seven other collaborators. T5 expressed translation, summarization and classification through a common text-to-text interface: text goes in and text comes out, including a word representing the label for a classification task. This provided a shared framework for comparing training and transfer-learning methods.[7]

Shazeer’s sole-authored 2019 paper Fast Transformer Decoding: One Write-Head is All You Need addressed a bottleneck in generating text one token at a time. Attention repeatedly reads keys and values for earlier tokens, making memory bandwidth a limiting resource. Multi-query attention shares keys and values across heads while retaining separate queries, reducing the tensors that must be stored and read. The paper reported faster decoding with minor quality degradation in its experiments.[4]

His 2020 paper GLU Variants Improve Transformer examined Transformer feed-forward layers. In gated variants such as GEGLU and SwiGLU, two learned projections are multiplied, with an activation applied to one branch, allowing it to modulate the other before an output projection. Because this construction uses three weight matrices instead of two, Shazeer reduced the hidden width to two-thirds of the baseline to match parameter and operation counts.[5]

GEGLU and SwiGLU produced the best held-out log-perplexities in the reported T5-style experiments: they assigned higher probabilities, on average, to the correct tokens in text withheld from training. Shazeer declined to offer a causal explanation, ending the conclusion with “we attribute their success, as all else, to divine benevolence.”[5]

Shazeer and Daniel de Freitas carried their work on open-ended conversation into Character.AI. Their signed December 2022 introduction credits de Freitas with inventing and leading Meena and LaMDA, and describes thirteen months of company work and a September 2022 public beta. Users could create conversational agents and interact with them. The founders argued that conversation could support tasks requiring clarification, feedback and adaptation to individual needs.[8]

At that stage, the founders described owning the engineering stack from data, modeling and training through serving and the interface. Their stated rationale was to align research and product development, optimize the whole stack and let user feedback inform improvements.[8]

Shazeer and Google publicly confirmed his return to Google DeepMind on August 2, 2024. De Freitas and some other Character.AI employees also joined Google, which entered a non-exclusive technology licensing agreement with the company. Shazeer expressed pride in what they had built and confidence that the licensing funds would support Character.AI’s continued work. By the February 2025 Dwarkesh interview, he was introduced as one of Gemini’s co-leads; his personal biography now lists OpenAI as his affiliation.[12][9][1]