Yi Tay

Yi Tay is a language-model researcher at Google DeepMind working on Gemini reinforcement learning, post-training and reasoning. He co-led model training for the advanced Gemini Deep Think model that reached gold-medal standard at the 2025 International Mathematical Olympiad, and leads one of DeepMind Singapore’s teams advancing Gemini. Earlier, he helped develop Synthesizer, UL2 and Differentiable Search Index, worked on PaLM-2 modeling and architecture, and co-founded Reka as its chief scientist.[2][1][6][14][17][12][9][11]

In his December 2025 reflection, Tay described moving from a focus on pretraining, architectures and objectives toward reinforcement learning and reasoning. He still regarded himself as a modeling researcher, calling RL “the technical canvas which we innovate upon these days.” One highlight was co-leading model training for the IMO effort.[6]

Google’s July 2025 announcement reported that an advanced Gemini Deep Think model solved five of the six IMO problems perfectly and scored 35 out of 42 points, reaching gold-medal standard. It produced proofs directly in natural language within the 4.5-hour competition time limit. Google names Tay among the leads and key contributors, credits overall technical direction to Thang Luong, and credits overall coordination to Luong and Edward Lockhart.[14]ⁿ¹

Deep Think explores and combines multiple possible solutions in parallel, rather than following only one linear chain of thought. For the IMO version, the team added reinforcement learning using multi-step reasoning, problem-solving and theorem-proving data, access to a curated corpus of mathematics solutions, and general problem-solving instructions. The natural-language approach differed from the specialist formal-language systems used in DeepMind’s 2024 effort; Google states that research on AlphaGeometry and AlphaProof continues.[14]

Tay helped build Google DeepMind’s Singapore presence with Quoc Le in 2025. In his year-end reflection, he weighed the pleasure of hands-on experimentation against the responsibility of building and mentoring a team. Discussions with Le and encouragement from Mostafa Dehghani about mentoring helped shape that decision. Tay chose to build the team while intending to remain closely involved in his own experiments.[2][6][3]

Google announced its Singapore research lab on November 19, 2025. Tay leads one of its teams advancing Gemini, with work on post-training, reasoning and Deep Think. Singapore government records also identify him as a researcher who co-leads Gemini model training.[15][17][1][16]

Before this reasoning work, Tay investigated transformer components and pretraining objectives at Google Brain, then trained multimodal models at Reka. Those earlier projects provide context for his modeling work and his experience with the infrastructure needed to run large experiments.[1][4][6]

Tay earned his bachelor’s degree and PhD at Nanyang Technological University and joined Google in late 2019. He initially worked on model architecture and efficient transformers. In his March 2023 departure essay, he described approximately 3.3 years at Google Brain and Research as an education in taking research from ideas through publication, patents, production and service to users. He credited managers Quoc Le and Don Metzler, along with close collaborators including Mostafa Dehghani, Vinh Tran and Jason Wei.[1][16][7][3]

With Dara Bahri and Donald Metzler, Tay developed Synthesizer, first submitted in May 2020. Standard dot-product self-attention compares queries with keys to determine how strongly tokens draw information from one another. Synthesizer replaces those pairwise comparisons with synthetic alignment weights, including weights generated from individual tokens by a neural network and randomly initialized matrices that can be learned or kept fixed.[12]

The experiments showed that some synthetic alternatives could perform competitively, but results depended on the task and variant. Standalone synthetic variants struggled on language-understanding tasks involving sentence pairs, while combining synthetic weights with dot-product attention improved performance across many tested tasks. The work tested the role of particular self-attention components; it did not establish that attention could be universally removed.[12]

Tay and Mostafa Dehghani co-led UL2: Unifying Language Learning Paradigms, first submitted in May 2022. UL2 separates a model’s architecture from its learning objectives. Its mixture of denoisers combines ordinary span reconstruction, more extensive text corruption and continuation-style prediction. These tasks ask the same model to use surrounding context, rebuild missing material and generate what comes next.[9]

UL2 also introduces mode switching: special tokens identify the denoising task during pretraining and can signal the appropriate behavior during downstream use. The team released a roughly 20-billion-parameter encoder–decoder model and later its Flan instruction-tuned version. The approach sought to support both supervised fine-tuning and prompt-based use through a shared training framework.[9]

In his 2024 Latent Space interview, Tay recalled that UL2’s success and visibility inside Google led to his involvement in adding its approach to PaLM-2 and becoming one of the modeling workstream’s co-leads. He distinguished that role from his contributor role on PaLM-1. He also emphasized that he was a core contributor to Flan, rather than its lead, and cautioned that his public visibility could make his contribution appear larger than it was.[7][1]

Tay and Vinh Q. Tran co-led Transformer Memory as a Differentiable Search Index, first submitted in February 2022. With their collaborators, they explored whether retrieval could become a generation task, with the index encoded in a transformer’s parameters. The model learns two connected operations:[10][11]

• Indexing: generate a document’s identifier from its content.[11]
• Retrieval: generate relevant document identifiers from a query.[11]

The output is a document address, which identifies the document to retrieve. The researchers also constructed semantic identifiers through hierarchical clustering, so similar documents share identifier prefixes. Training had to preserve the association between content and identifiers while teaching query-based retrieval: joint training worked better than learning indexing first and retrieval afterward.[11]

The initial Natural Questions experiments covered approximately 10,000, 86,000 and 228,000 unique documents in NQ10K, NQ100K and NQ320K respectively, counted using each document’s first 4,000 UTF-8 characters. The dataset names denote query–document pairs. Scaling to larger corpora and updating the model as documents change remained open research directions.[11]

After leaving Google in 2023, Tay became a co-founder and chief scientist at Reka. With the Reka team, he contributed to Core, Flash and Edge, models trained from scratch to process text, images, video and audio. The team’s technical report describes dense 7-billion-parameter Edge and 21-billion-parameter Flash models.[1][3][5][13]

In his March 2024 engineering essay, Tay described a “hardware lottery” across rented clusters: unreliable nodes interrupted training, and slow checkpoint input/output consumed usable compute time. Reka responded with monitoring, efficient checkpointing, optimizations and a custom filesystem. He reported that these investments improved utilization and reduced downtime, while allowing that infrastructure and hardware-team competence could explain his reliability contrast with Google better than accelerator type alone.[4]

As a contrast, Tay recalled accidentally leaving the Google UL2 20B training job running for a month without a failure. His point concerned the reliability of Google’s training environment. At Reka, even clusters using the same accelerator model could require different fixes, making recovery and storage part of the practical training problem.[4]

The team chose PyTorch for accessibility and speed of development, despite Tay’s preference for Google’s T5X and Mesh TensorFlow tooling. Compute constraints also limited systematic hyperparameter sweeps. He described reaching the Flash and Edge recipes through a small number of shorter, smaller-scale experiments, changing multiple variables and relying on prior training experience to narrow the search.[4]

Tay announced his return to Google DeepMind on November 25, 2024, after approximately a year and a half in startups, reporting again to Quoc Le as a senior staff research scientist. He credited his fellow founders and valued learning GPU and PyTorch infrastructure alongside company building. His stated reason for returning was that he still identified more as a scientist and researcher, and wanted to concentrate on research again.[5]