Tri Dao

Tri Dao is a machine-learning researcher and Together AI co-founder and chief scientist whose work connects model algorithms to the hardware that runs them. His contributions include FlashAttention, which executes dense Transformer attention without storing its large intermediate matrices, and Mamba, developed with Albert Gu, which processes sequences through a selectively updated recurrent state. The projects address different questions: how to execute an existing computation more efficiently, and how to build a different sequence model around efficient computation. As of September 16, 2026, Together AI lists him as founder and chief scientist, and Princeton lists him as a visiting research scholar.[3][2][17][6][10]ⁿ¹

Dao completed his Stanford computer science PhD in 2023, co-advised by Christopher Ré and Stefano Ermon. In July that year, he announced his move to Together AI as chief scientist, stating a goal of making open-source AI more accessible and cost-competitive. In his July 2023 Latent Space interview, he said Together’s open model and dataset releases, colleagues he knew and trusted, and its focus on efficient training and inference attracted him to the company.[2][5][4]

He joined Princeton’s faculty in 2024. Dao described academia and industry as complementary: industry offered resources for scaling models, while academia could support evaluation, deeper understanding, and riskier research directions. In November 2025, he was named an AI2050 Early Career Fellow; his fellowship project was to focus on intelligent systems that learn through experimentation.[17][4]

Dao’s July 2023 views on openness emphasized practical access. He welcomed commercially usable Llama 2 weights and argued that businesses fine-tuning and serving their own models could reduce the concentration of decision-making power in a few companies. He also described legal barriers, annotation costs, and competitive incentives that made datasets harder to release than model weights.[4]

The work behind those 2023 career decisions had already begun at Stanford: the original FlashAttention appeared in 2022, before his PhD completion and Together move. His account of that earlier research explains why GPU memory traffic became central to the project.[6][4][5]

Dao recalled beginning his PhD by studying which matrices admit theoretically fast matrix-vector multiplication. He credited Stanford’s Hazy Research, led by Ré, with providing exchanges among theory, systems, and application researchers: implementations exposed practical failures, and applications such as seizure detection supplied demanding feedback. He also credited Ré’s encouragement to study textbooks and older literature alongside new research.[4]

Dao had worked on approximate attention methods that reduced arithmetic. He recalled learning from practitioners that these methods often failed to reduce elapsed time, despite performing fewer operations. Conversations with systems researchers drew his attention to memory reads and writes and parallelism. FlashAttention consequently targeted data movement between levels of GPU memory.[4][6]

Dao developed FlashAttention with Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Standard dense attention compares queries with keys, normalizes the scores using softmax, and combines values with the resulting weights. A straightforward implementation writes both the score and probability matrices to GPU high-bandwidth memory, or HBM. Each contains N² entries for a sequence of length N: doubling the sequence length quadruples these intermediates.[6]

FlashAttention loads blocks of queries, keys, and values into fast on-chip SRAM, computes attention for those blocks, and accumulates the output without storing the full matrices in HBM. Softmax ordinarily couples all scores in a row; running maxima and normalization sums allow successive blocks to be combined while rescaling earlier partial results. During the backward pass, the algorithm reconstructs attention intermediates from saved outputs and normalization statistics instead of retaining the quadratic matrices.[6]

Dense attention arithmetic remains quadratic; its memory footprint becomes linear in sequence length. Recomputation adds arithmetic but can still reduce runtime by avoiding slow memory accesses. Dao credited established tiling, kernel fusion, and online-softmax techniques, and the team used Apex FMHA code as an implementation starting point. “Exact” means the attention computation has no algorithmic approximation; floating-point outputs and gradients can still differ within numerical tolerance.[6][4][8]

Dao also prioritized usable code. He recalled emailing an advisor while writing FlashAttention that “the most important thing will be the artifact, which is the code.” He and his co-authors continued improving it after release. The public FlashAttention repository supplies implementations, hardware and feature documentation, and checks of outputs and gradients against reference implementations.[4][8]

Dao was the sole listed author of FlashAttention-2, released in July 2023. Its rewrite used NVIDIA’s CUTLASS 3.x and CuTe. It reduced non-matrix arithmetic and added parallelism along sequence length to keep more GPU resources occupied when batches were small. Within a thread block, splitting queries across warps while sharing keys and values gave each warp its own output slice, avoiding the shared-memory communication needed to combine partial outputs. The paper credits Phil Tillet’s Triton implementation with first suggesting and implementing the sequence-parallel scheduling ideas.[7][5]

The authors reported roughly 2× faster attention than FlashAttention in their A100 experiments. Whole-model gains were smaller: tests of 1.3B- and 2.7B-parameter GPT-style models at sequence lengths of 2K or 8K on eight A100 GPUs showed up to 1.3× faster end-to-end training than FlashAttention.[7]

For FlashAttention-3, released in July 2024, Dao collaborated with Jay Shah, Ganesh Bikshandi, Ying Zhang, Vijay Thakkar, and Pradeep Ramani. The team adapted attention to Hopper GPUs, overlapping data movement with computation and matrix multiplication with softmax. It also introduced FP8 processing techniques that spread out activation outliers to reduce quantization error.[9][1]

Dao co-authored the March 2026 FlashAttention-4 paper with Ted Zadouri, Markus Hoehnerbach, Jay Shah, Timmy Liu, and Vijay Thakkar. On Blackwell, tensor-core throughput had grown faster than shared-memory bandwidth and exponential-processing capacity. The team redesigned asynchronous pipelines, computed some exponentials in software, and used tensor memory and paired thread blocks to reduce backward-pass memory traffic. Its Python-embedded CuTe-DSL implementation also addressed development cost: the authors reported 20–30× shorter compile times than FlashAttention-3’s C++ template implementation.[12]

Albert Gu and Dao introduced Mamba in December 2023. It makes state-space model parameters depend on the current input, enabling the recurrent state to selectively propagate or forget information. Input dependence prevents the efficient convolution formulation used by earlier models, so they developed a hardware-aware parallel scan with fusion and recomputation. The original Mamba architecture used selective state-space blocks without attention or separate MLP blocks.[10]

During generation, Transformer attention keeps a growing key-value cache of past tokens. Mamba compresses history into a fixed-size recurrent state. This bounds state memory with respect to context length, but makes retention a modeling problem: the compressed state cannot provide an individually accessible record of every earlier token.[10][14][16]

In their May 2024 Mamba-2 paper, Dao and Gu developed state space duality, connecting state-space models and attention variants through semiseparable matrices. Their algorithm uses hardware-efficient matrix multiplication within chunks and a smaller recurrence between chunks. The authors reported a 2–8× faster core layer than Mamba’s optimized selective scan. The design trades some state-transition expressivity for hardware efficiency; its duality does not make Mamba equivalent to ordinary softmax attention.[11]

Dao’s Mamba-3 explanation contrasts Mamba-2’s emphasis on training efficiency with increasing inference demand from post-training and deployment. The eight-author March 2026 paper, co-authored by Dao and Gu, introduces a richer recurrence, complex-valued state transitions for state tracking, and multi-input, multi-output updates. These updates perform more computation without expanding the recurrent state, using otherwise idle compute during memory-bound decoding. The MIMO variant requires more training work, while the team’s benchmarks show similar decode latency to Mamba-2.[14][13]

The team retained the fixed-state tradeoff: pure Transformer models performed better on retrieval tasks in its evaluations. Dao’s explanation predicts that linear layers will predominantly be combined with global attention for language modeling, while acknowledging that the interactions within these hybrids remain incompletely understood.[14][13]

With Tanishq Kumar and Avner May, Dao co-authored Speculative Speculative Decoding. Ordinary speculative decoding alternates a small model’s drafts with a larger model’s verification. Their Saguaro algorithm runs the draft model on separate hardware, preparing continuations for likely verification outcomes while verification is underway. A matching outcome makes the next draft immediately available; misses use a fallback. In the May 2026 revision, the authors reported an average 30% speed improvement over their strongest speculative-decoding baselines, with experiments spanning Llama-3 and Qwen-3 model families.[15]

Dao’s June 2026 ReplaySSM explanation addresses the cost of moving recurrent state even when its size is fixed. ReplaySSM keeps a checkpoint state and caches recent inputs, writing the full updated state back only when the buffer flushes. For Mamba-2, it can compute outputs directly from the checkpoint and buffered inputs. Rejected speculative inputs can be discarded by moving a buffer pointer, avoiding a full state snapshot for each draft token. The method preserves the original computation up to floating-point error while reducing memory traffic and enabling parallel verification.[16][1]