Dan Fu

Dan Fu is an assistant professor of computer science and engineering at UC San Diego and VP of Kernels at Together AI. He leads UCSD’s SandyResearch Lab and works on making machine learning faster through hardware-aware algorithms and efficient architectures. His collaborative research, published under Daniel Y. Fu, includes FlashAttention, Monarch Mixer and FlashFFTConv.[2][5][6][7]

At Together AI, Fu builds a team focused on low-level performance engineering and GPU kernels. He described a recurring obstacle in developing new architectures: months spent writing efficient CUDA implementations, with even a small PyTorch change requiring substantial kernel work. ThunderKittens, developed collaboratively with researchers at Stanford, addresses that burden through matrix-based compute primitives suited to NVIDIA tensor cores. In his account, knowing the hardware’s preferred matrix operations can also inform model choices such as state sizes and update functions.[2][1][3]

Fu’s goal for ThunderKittens includes shortening the adaptation cycle when new GPU generations arrive. In an April 2026 Together AI profile, he explained: “We wanted to build tools to make it easier to quickly build kernels for the new hardware generation, and ThunderKittens was the answer.” The profile recounts the team adapting the library to Blackwell GPUs in March 2025, including new tensor cores, tensor memory and coordination between thread blocks.[3]

Fu’s earlier Stanford research supplies context for this work on kernels and model design. In March 2024, he was a computer science PhD student co-advised by Christopher Ré and Kayvon Fatahalian; his research already combined hardware-aware algorithms with architectures for longer-context applications.[4]

Fu coauthored FlashAttention with Tri Dao, Stefano Ermon, Atri Rudra and Christopher Ré. First submitted on May 27, 2022, it computes exact attention using tiling to reduce transfers between GPU high-bandwidth memory and on-chip SRAM. The authors reported a 15% end-to-end training speedup for BERT-large at sequence length 512 against the MLPerf 1.1 training record, and a 3× speedup for GPT-2 at sequence length 1K. The contribution made attention faster by accounting for memory movement rather than relying on an approximation.[5]

Fu later recalled that the team did not expect much attention to the release. They were preparing blogs and code for Tuesday morning when Andrej Karpathy tweeted about FlashAttention on the Monday evening of Memorial Day weekend: “so then we knew it was something people were paying attention to.”[3]

In October 2023, Fu was the first-listed author of Monarch Mixer, alongside Simran Arora and eight other collaborators. The architecture uses structured Monarch matrices to scale subquadratically in both sequence length and model dimension. Its causal construction uses polynomial evaluation and interpolation to avoid the quadratic bottleneck introduced by masking. In proof-of-concept evaluations, the authors reported matching BERT-base and BERT-large on downstream GLUE quality with up to 27% fewer parameters and up to 9.1× higher throughput at sequence length 4K. Separately, a 360-million-parameter causal model matched GPT-style Transformer pretraining perplexity on The PILE.[6]

Fu also collaborated with Hermann Kumbong, Eric Nguyen and Christopher Ré on FlashFFTConv, which computes long depthwise convolutions using the fast Fourier transform. Its implementation supports implicit padding and gating without extra memory I/O, supporting models such as Monarch Mixer, H3 and Hyena. The repository’s gated-convolution benchmark reports a 7.93× speedup and 6.40× memory savings against PyTorch at sequence length 1K, on one H100-SXM with results scaled to batch size 64 and hidden dimension 768.[7]

In his joint presentation with Eugene Cheah at Latent Space Live at NeurIPS 2024, published that December, Fu argued for hardware and kernel support from the beginning of architecture development. A theoretically efficient model that runs slower in practice, he warned, is “dead on arrival.” He also highlighted how efficient recurrent models could change prompting: discussing Simran Arora’s Just Read Twice work, he explained that rereading a document after seeing the question lets a fixed-state model know what information to retain on its second pass.[1]

Fu distinguished running a recurrent model indefinitely from its ability to use information across that context. Looking back to 2022, he said he would have released long-context benchmarks alongside efforts to extend context length. His proposed sequence was practical: define what users want the model to do, build benchmarks for it, measure performance, and then determine whether the architecture needs to change.[1]