Quentin Anthony

Quentin Anthony is an AI systems engineer and, as of September 2026, VP of AI Engineering at Zyphra, where he leads performance optimization for training and inference on heterogeneous hardware. His work includes helping scale GPT-NeoX-20B at EleutherAI and leading a collaborative systems report on training ZAYA1 on AMD GPUs and networking.[3][1][5]

Anthony is first author of Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design, and a corresponding author alongside Beren Millidge. Written with collaborators at Zyphra, IBM, and AMD, the report describes ZAYA1-base, a mixture-of-experts model with 760 million active and 8.3 billion total parameters. Its systems contribution combines hardware measurements, model-sizing guidance, communication tuning, checkpointing, and fault recovery.[5]

The team's training choices reflected the hardware's memory and communication constraints. Each MI300X GPU's 192 GB of memory allowed initial pretraining with data parallelism and a ZeRO-1 distributed optimizer, which shards optimizer states. Context parallelism was added for longer sequences. The team also measured matrix-multiplication performance to choose efficient model dimensions, rather than assuming that similarly sized operations would achieve similar throughput.[5]

Two optimizations illustrate that approach. Gradient fusion buffers were sized to reach measured network-bandwidth saturation without growing so large that communication became harder to overlap with computation. For the distributed Muon optimizer, the team replaced a memory-intensive AllGather with exchanges between neighboring ranks: under its contiguous-shard layout, most parameter tensors were already complete locally, so only incomplete tensors at shard boundaries needed reconstruction.[5]

Recovery received similar attention. Each rank wrote its own optimizer checkpoint shard, disk writes ran asynchronously, and checkpoints were marked complete only after every artifact finished writing. The authors reported more than a tenfold reduction in checkpoint time compared with their naive-checkpointing baseline. Their Aegis service automated known failure responses while leaving unfamiliar failures to human intervention. In one operational account, the team traced recurring loss spikes to silent corruption on a faulty GPU; automatic restarts restored the pre-spike training trajectory.[5]

Anthony's earlier work at Ohio State and EleutherAI provides context for these systems concerns: distributed-training bottlenecks, checkpointing, and communication were already central to his research before the AMD report.[2][1]

At Ohio State's Network-Based Computing Laboratory, Anthony researched under D. K. Panda, focusing on checkpointing, model and optimizer compression, and deep-learning framework co-design. He contributed to MVAPICH2 and its GPU-cluster and deep-learning subprojects. His publications include a 2021 study of multi-level checkpointing with Donglai Dai and the 2023 collaborative paper MCR-DL: Mix-and-Match Communication Runtime for Deep Learning.[2]

In his August 2023 Latent Space interview, Anthony described himself as an Ohio State PhD student beginning his fifth year and EleutherAI's head engineer. He said he joined EleutherAI when GPT-NeoX-20B was encountering scaling problems, applying his research to communication and synchronization across hundreds of GPUs. He recalled a cluster assembled during COVID-era shortages with different network switches on different nodes, where tuning had to contend with the slowest switch.[1]

Anthony coauthored Transformers Math 101 to make practical training knowledge available beyond a single institution. He explained that EleutherAI engineers had accumulated internal notes that could help others get more performance from existing hardware. He favored useful approximations over spending another month tracing every small memory allocation. His description of his engineering role was direct: “someone else has done like some deep learning optimization and I need to make it run fast.”[1]

His explanations distinguish model weights, optimizer states, gradients, and activations rather than treating parameter count as total training memory. In his Adam accounting, the 12 bytes per parameter assigned to optimizer states include an FP32 master-weight copy, momentum, and variance; they are not the entire training footprint. He also described activation recomputation as a trade between memory and compute. NeoX, in his 2023 account, used a simple tensor-size threshold for discarding activations instead of implementing more elaborate selective-recomputation heuristics.[1]

Anthony's 2023 advice was to find the minimum GPU count that fits a model, estimate training time, and add GPUs when the schedule requires it. Hardware failures, software bugs, and synchronization problems qualify theoretical linear scaling. He described 3D parallelism—combining data, tensor, and pipeline parallelism—as offering “the most control and also the most ways to go wrong,” and argued that the choice between it and simpler sharding should reflect available engineers and GPUs. He identified multimodal and RLHF parallelism, along with compressing communication buffers before network transfer, as research priorities.[1]

Anthony also helped turn RWKV collaborators' experiments and discussions into a digestible paper, explicitly saying he was not a core contributor. His educational work includes the EleutherAI Model Training Cookbook, credited jointly to Anthony, Jacob Hatef, Hailey Schoelkopf, and Stella Biderman. The repository provides training and inference calculators, communication benchmarks, and transformer-sizing and matrix-multiplication benchmarks.[1][4]

Anthony coauthored BlackMamba, submitted in February 2024, with Yury Tokpanov, Paolo Glorioso, and Beren Millidge. The architecture combines Mamba state-space modeling with mixture-of-experts, pairing sequence-length-linear generation with sparse expert computation. The authors reported training two model configurations on 300 billion tokens and releasing their weights, checkpoints, and inference code.[6]

He also coauthored Zyda: A 1.3T Dataset for Open Language Modeling, initially submitted in June 2024 and revised that September, with Yury Tokpanov, Beren Millidge, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, and James Whittington. Zyda combines existing open datasets into a 1.3-trillion-token corpus, applying filtering and deduplication both within individual datasets and across their boundaries.[7]