Training LLMs: From Compute Estimates to Working Clusters

Getting hundreds of GPUs to cooperate

When EleutherAI began training [GPT-NeoX 20B](https://huggingface.co/EleutherAI/gpt-neox-20b), keeping hundreds of GPUs communicating and synchronized became a scaling problem. Quentin Anthony joined because those challenges overlapped with his PhD work. His task was to help the training system make useful progress across all those devices.

That gap between understanding a transformer and operating one motivated [Transformers Math 101](https://blog.eleuther.ai/transformer-math/). EleutherAI engineers had accumulated notes on practical problems, including splitting inference across two GPUs. Publishing them could help others improve performance on hardware they already owned, instead of rediscovering lessons hidden inside each organization’s software stack.

Anthony cites [BLOOM](https://huggingface.co/docs/transformers/model_doc/bloom) alongside OPT as examples of openly documented training practice. His own guidance deliberately uses experience-based approximations: tracing every small memory allocation might take another month, while a sufficiently accurate estimate is already useful for engineering decisions.

From computation to time and budget

The starting approximation is `C ≈ τt ≈ 6PD`: training computation C depends on parameter count P and training-token count D. The same work can be expressed as throughput τ multiplied by time t. FLOPs count floating-point operations; FLOPs per second measure their execution rate. To turn estimated work into a monetary budget, divide by achieved throughput and apply instance pricing.

Anthony makes that calculation operational: 1. Map the intended parameter count to a concrete transformer architecture. 2. Benchmark it on the intended hardware and software stack. 3. Measure how throughput scales with the chosen distribution scheme. 4. Divide estimated computation by that throughput to obtain time, then calculate cost. His allocation target is the smallest GPU deployment that meets the schedule. Start with enough GPUs to fit one model instance; add replicas if its training duration is unacceptable.

More GPUs introduce more opportunities for hardware failures, hung devices, numerical overflow and synchronization bugs. Doubling the GPU count need not halve elapsed time. Anthony’s example of a larger run taking three times longer because something breaks every day illustrates operational risk; it is not a scaling measurement.

Software complexity affects that risk. Anthony describes Mosaic’s approach as using ZeRO sharding, which divides stored training state across GPUs. EleutherAI’s Megatron–DeepSpeed-based stack can, in his assessment, achieve somewhat higher efficiency, but requires more tuning and creates more failure modes. The detailed distribution choices come later; the immediate budget question includes the engineering effort needed to keep the run working.

The compute approximation allocates 2PD to the forward pass and 4PD to the backward pass. These coefficients are used without a derivation; the intuition offered is that backpropagation does more work. The coefficient in 6PD also does not establish a training-token budget. The discussion’s token-to-parameter recommendations are inconsistent, so dataset sizing needs a separate justification.

A host proposes a GPT-3 sanity check using 175 billion parameters, 300 billion tokens and a reported training total of 3.14 × 10²³ FLOPs. Applying the approximation to those inputs gives 6 × (175 × 10⁹) × (300 × 10⁹) = 3.15 × 10²³ FLOPs. That checks the scale of the work estimate, without determining the run’s elapsed time or hardware utilization.

Throughput depends on waiting and software

GPUs spend part of training waiting for CPU–GPU data movement or synchronization with other devices. Achieved throughput includes that waiting. A model’s estimated operation count and a GPU’s theoretical processing rate therefore answer different questions: how much work must happen, and how quickly hardware might execute it.

For the deep-learning kernels discussed in the 2023 conversation, the expected rates were roughly 100–180 trillion operations per second on an A100 and 30–40 on a V100. The suggested A100 troubleshooting threshold of 115 TFLOPS came from tuning multiple frameworks and hardware configurations. These are empirical reference points, not universal device specifications: no particular model, precision or benchmark configuration accompanies them, and the frameworks used different FLOP counters.

Hardware selection also spends developer time. The argument for Nvidia and PyTorch was that familiar failures often had public GitHub issues and shared fixes. AMD’s ROCm software stack was described as less efficient and harder to debug at the time. FlashAttention illustrated the gap: support had spread across more Nvidia devices while AMD users still faced delays. Catching up within a few years was a prediction made in 2023.

EleutherAI’s supercomputing experiments made network layout another constraint. Summit had six V100 GPUs per node, with slower communication between nodes than within one. The next grant being pursued was for [Frontier at Oak Ridge](https://www.ornl.gov/news/frontier-supercomputer-debuts-worlds-fastest-breaking-exascale-barrier), described as having four AMD GPUs per node and similar challenges in overlapping communication with computation. Government centers such as Oak Ridge and Lawrence Livermore offered open researchers a route to apply for compute, but using an allocation efficiently still required systems work.

Precision changes speed and storage differently

FP32 stores each parameter as a 32-bit floating-point value: four bytes. Moving computation to 16-bit FP16 on V100 Tensor Cores helped motivate mixed-precision training software. Yet mixed precision can increase weight storage. In the scheme described, training keeps an FP16 copy for computation and an FP32 master copy for updates—six bytes per parameter together, compared with four for FP32 alone.

BF16 trades numerical precision for dynamic range. Compared with FP16, it devotes more bits to the exponent and fewer to the mantissa, allowing a wider range of magnitudes with less fine detail. That wider range is credited with reducing much of the instability encountered in large FP16 training runs. Hardware support, software implementations and training practice evolve together.

Post-training quantization converts trained weights into smaller representations, such as eight-bit INT8 or four-bit INT4, while trying to preserve accuracy. One argument in the conversation favors spending even abundant GPU memory on a larger quantized model: additional training examples and input variation can matter more than fine numerical precision. This is a capacity-allocation preference, not an accuracy guarantee.

For 70 billion parameters, the storage calculation is approximately 280 GB in FP32 versus 70 GB at eight bits. The latter fits within an 80 GB H100’s capacity for weights alone. Inference also needs memory for the context being processed and generated.

The KV cache retains work—and consumes memory

Autoregressive generation produces one next token at a time using the preceding context. The KV cache retains information computed from the prompt and earlier generated tokens, allowing each new step to reuse prior work. With sufficiently long context, that cache can occupy as much memory as the weights, or more. Fitting the weights therefore does not establish that the intended inference workload fits.

A usable cache estimate needs sequence length, batch size and storage precision. The tentative spoken formula leaves those unspecified and cannot serve as a complete sizing equation.

[RWKV](https://arxiv.org/abs/2305.13048) enters as a research direction seeking transformer-like quality without quadratic attention overhead. The recommendation is to read the paper; no architecture walkthrough or comparative results accompany it. The participating contributor describes their role as turning distributed experiments and discussions into an accessible paper, so readers would not need to reconstruct a year of Discord history. They distinguish that work from being a core technical contributor.

Training memory extends far beyond weights

For training, the optimizer—the method that calculates parameter updates—can dominate memory. Quentin frames his systems role as making an established optimization method run fast. Adam’s extra stored state matters because it can force training onto more GPUs, introducing communication that limits progress even when raw compute is plentiful.

The mixed-precision setup discussed has four main allocations: - Model weights: two bytes per parameter in FP16 or BF16. - Optimizer-side storage: 12 bytes per parameter—four each for FP32 master weights, momentum and the quantity described as variance. - Gradients: two bytes per parameter in the lower-precision format used here. - Activations: tensors created during execution. The first three sum to 16 bytes per parameter before activations and other allocations. The master weights are already included in the optimizer-side figure. Other optimizers may retain different states or representations.

That accounting changes where memory optimization pays off. Reducing FP16 weights to INT8 saves one byte per parameter. Reducing four-byte optimizer-state values to one byte can save more across a larger allocation. Eight-bit Adam in `bitsandbytes` is offered as an example: inspect optimizer storage before assuming weight quantization offers the largest training-memory reduction.

A model can load successfully and still run out of memory during an iteration. Model and optimizer states remain resident, while activation memory grows as computation proceeds. Loading persistent state is only the first memory test.

Trade recomputation for memory

Activation recomputation discards intermediate tensors and regenerates them when needed, exchanging extra computation for lower memory use. The discussion describes three policies: - Recompute everything when memory is overwhelmingly more constrained than compute—an uncommon choice, potentially useful with slow distributed communication. - Selectively retain or recompute tensors by weighing storage cost against regeneration cost. - Discard tensors above a fixed size threshold, X. Megatron is cited for selective recomputation. NeoX uses the simpler threshold approach, described as providing adequate savings without the engineering effort of more elaborate heuristics.

Precision choices also differ between training and inference. Training is presented as more sensitive to numerical precision, while an already-trained model can more readily use INT8. That motivates the prediction that BF16 will persist longer in training: an effective inference format does not automatically make a good training recipe.

Fine-tuning raises another set of practical questions: how much data is needed to transfer between domains, how many tokens are appropriate for a model, how new abilities are acquired, and how the learning-rate schedule should change after pretraining has decayed it. These are offered as research interests, with acknowledgment that fine-tuning scaling-law work already exists and the assessment may reflect gaps in the speaker’s knowledge.

ZeRO saves storage through sharding

ZeRO reduces per-GPU storage by dividing training state across devices. Its stages cover progressively more allocations: - Stage 1: optimizer state. - Stage 2: optimizer state and gradients. - Stage 3: optimizer state, gradients and model parameters. In the stage-three illustration, each of N GPUs owns approximately a 1/N share of these states. This describes state ownership, rather than a complete peak-memory budget.

The savings require repeated communication to distribute and assemble state for computation. Some movement can overlap with compute. But adding GPUs leaves each device with less work while retaining synchronization costs; on a slow interconnect, further sharding can reduce throughput.

A rough Summit example places a 20-billion-parameter model’s useful scale around 100–200 GPUs, beyond which communication increasingly limits progress. This is a hardware-dependent rule of thumb, without a benchmark configuration. The stopping point depends on network speed, device memory, work remaining per GPU and acceptable achieved throughput.

Three dimensions of parallelism

3D parallelism combines three ways of distributing work: - Data parallelism: replicate the model, run forward and backward passes, then synchronize and average gradients. In the combined scheme described, optimizer state is sharded along this dimension. - Tensor parallelism: partition model tensors, compute on each partition and synchronize when required. - Pipeline parallelism: place different layers on different GPUs and pass activations between them. Together, these dimensions specify where parameters, optimizer state and related training data reside.

A four-layer model split across four GPUs exposes the pipeline problem: if one batch moves through sequentially, most devices sit idle. Microbatches, smaller subdivisions of the batch, allow overlap. After GPU 1 sends the first microbatch to GPU 2, it starts the next while GPU 2 processes the first. Effective overlap requires enough batch work to subdivide.

Unequal hardware complicates synchronous execution. In the schemes discussed, smaller GPU memory can constrain allocation, and slower devices make faster ones wait. The memory claim is qualified by the speaker’s knowledge of available schemes. Network differences caused a concrete problem during EleutherAI’s 20-billion-parameter training: COVID-era switch shortages produced a cluster with different switches across nodes, and even consistently reaching the slowest switch’s speed required tuning.

Pure ZeRO sharding offers a simpler operating model. Equal state partitions avoid explicit pipeline-stage and tensor-placement decisions, easing debugging, checkpointing and training. The conversation describes a trend toward this approach in 2023, while crediting 3D parallelism with greater control. Choosing between them means weighing engineering capacity against GPU resources.

Mixed workloads and communication compression

Multimodal models make placement decisions harder. A vision component might occupy its own GPUs, share devices with text parameters through sharding, or become a pipeline stage. Each new component requires adapting the existing distribution scheme.

Bringing reinforcement learning into the training loop creates another allocation problem: some GPUs perform inference while others train. Distributing those roles—and the inference KV cache within the larger workflow—combines their distinct memory and execution requirements. The closing discussion identifies this as an area needing more systems research.

A second proposed direction is on-the-fly communication compression: compress an outgoing buffer with a GPU kernel, transmit it, then decompress it at the destination. Tensor-parallel communication across nodes can become prohibitively expensive; the aim is to shift spending from network fabrics toward GPUs. This is a research proposal, with no measured speedup offered.

Asked which development arrived sooner than expected, the answer is FlashAttention, praised for its kernels. The conversation ends with an invitation to send follow-up questions by email or, for faster replies, Discord—the same practical exchange of knowledge that motivated publishing the engineering notes.