Sasha Rush

Sasha Rush, also known as Alexander Rush and Alexander M. Rush, is a machine learning researcher at Cursor. His contributions include The Annotated Transformer, an implementation that connects a research paper to working code; The Annotated S4, developed with Sidd Karamcheti; and collaborative research on masked diffusion language models. He also served as organizational chair of the Conference on Language Modeling in 2024.[2][3][6][8][9][5]

At Cursor, Rush studies post-training AI systems for coding and related tasks. He describes his current interest as “improving model reasoning for long-horizon tasks” on his personal site. This work follows his academic career at Harvard and Cornell and his research involvement at Hugging Face.[4][3]

Before his current work on coding systems, Rush worked on natural-language understanding and generation. He earned a B.A. in computer science at Harvard and a Ph.D. in computer science at MIT, then held a postdoctoral fellowship at Facebook Artificial Intelligence Research in New York before joining Harvard. His early publications included A Neural Attention Model for Abstractive Sentence Summarization with Sumit Chopra and Jason Weston in 2015, and Sequence-Level Knowledge Distillation with Yoon Kim in 2016.[2][4]

Rush coauthored OpenNMT: Open-Source Toolkit for Neural Machine Translation with Guillaume Klein, Yoon Kim, Yuntian Deng, and Jean Senellart in 2017. OpenNMT subsequently supplied much of the code underlying his annotated implementation of the Transformer.[4][7]

In April 2018, Rush published The Annotated Transformer, crediting Vincent Nguyen and Guillaume Klein for their help. He reorganized sections of Attention Is All You Need, added commentary, and paired the text with a line-by-line implementation in a working PyTorch notebook. The resulting workshop paper framed reproduction as an open-source contribution: code should closely follow the original work while producing an efficient system that others can use and modify.[7][6]

The notebook makes implementation details concrete. It shows how decoder masking prevents a prediction from using future output tokens, starts with a task that copies an input sequence, and then builds a German-to-English translation example using IWSLT data. Rush also emphasizes practical training choices, including grouping sentences into batches with minimal padding. His contribution was the explanation and reproduction of the existing Transformer model.[7][6]

Rush joined Cornell Tech from Harvard as an associate professor in 2019. At the time, his research combined deep learning and structured prediction, concentrating on text generation and document-level understanding. He describes his professorships at Harvard and then Cornell as spanning 2016–2026, and says he also worked as a Hugging Face researcher from 2019–2024, helping on early open-source LLM projects.[2][4]

In March 2022, Rush and Sidd Karamcheti published The Annotated S4, a blog post and library explaining the S4 architecture developed by Albert Gu, Karan Goel, and Christopher Ré. The pair wrote that colleagues had found it difficult to build intuition for the model, so they linked explanations to concrete implementations. Although they mainly used Torch, they chose JAX with Flax because, in their judgment, its functional nature suited some of S4’s complexities.[8]

A central explanation is how a linear time-invariant state-space model can run in two forms. Recurrence updates a hidden state one step at a time; unrolling that recurrence yields a discrete convolution that supports efficient training. The implementation therefore uses convolution during training and converts to recurrence at test time. A spring-and-mass example introduces the state-space equations before the tutorial develops S4 layers and demonstrates image completion and raw-audio modeling. Rush and Karamcheti credited Gu and Goel for assistance and Conner Vercellino and Laurel Orr for feedback.[8]

Rush’s state-space-model teaching also appeared in Latent Space’s May 2024 ICLR recap, which featured his invited workshop-day talk on the development and applications of state-space models.[1]

Rush was one of eight coauthors of Simple and Effective Masked Diffusion Language Models, submitted in June 2024 and presented at NeurIPS 2024. The team combined an improved training recipe with a simplified Rao-Blackwellized objective—a mixture of classical masked-language-modeling losses. The approach trains encoder-only language models with samplers that can generate arbitrary lengths of text semi-autoregressively. The authors reported state-of-the-art results among diffusion models on language-modeling benchmarks and perplexity approaching that of autoregressive models.[9]

Rush says he helped start the Conference on Language Modeling in 2024. He served as organizational chair of COLM 2024, held at the University of Pennsylvania in Philadelphia on October 7–9. The conference established an academic venue devoted to understanding, improving, and critiquing language-model technology, bringing researchers from different disciplines into a shared forum.[4][5]