The GPU Is Only as Useful as the System Feeding It

A $500 million machine is not a $500 million training run

A cluster of 20,000 A100 GPUs sounds like an enormous training bill. Early in the conversation with Dylan Patel, a host proposes roughly $500 million for the supercomputer behind GPT-4. Patel accepts the approximation, while putting the capital cost slightly higher. But buying the machine and consuming its compute are different costs: three months of pretraining at an assumed $1 per GPU-hour would cost far less. Data and other expenses still matter.

That distinction supports Patel’s broader argument about AI businesses. He expects lower personnel-related research and development costs but higher service-operating costs than in conventional software subscriptions. Infrastructure efficiency directly affects service economics. He credits AWS’s custom infrastructure with cost advantages in storage, databases and CPU services, and sees Google’s investment in its TPU AI chips and optical switches as a corresponding advantage in AI.

Patel began semiconductor consulting in 2017, launched his blog in November 2021 and started hiring in 2022. SemiAnalysis connects software requirements to manufacturing: Google’s TPU expansion becomes a question about supply-chain orders, system builders such as Celestica, testing suppliers and chip-design intellectual property. Those orders offer a way to estimate capacity Google does not publicly disclose—and ask what it could do with that capacity.

512 H100s look different against an industry-wide expansion

Patel recalls a startup celebrating 512 incoming H100s and people comparing GPU holdings at parties. Against those allocations, he places his supply-chain estimates: roughly 400,000 H100s manufactured in the previous quarter and 530,000 being sold in the current quarter. A team’s hard-won allocation can still be small relative to the capacity coming online.

Speaking in November 2023, he forecasts more than a million H100 sales that year and well over three million total NVIDIA GPU sales in 2024. His question for developers is whether their work will retain value as that capacity arrives. He argues that larger models can perform better on a given dataset, at greater expense, and predicts accelerating progress from the major labs through 2025. These are contemporary forecasts, not shipment results or a measured comparison of progress.

That outlook drives two criticisms of engineering priorities: - Optimizing around scarce personal compute: Patel questions the value of fine-tuning on gaming GPUs and strongly criticizes batch-size-one inference on cloud H100s, while acknowledging that most people lack substantial resources. - Optimizing for leaderboard position: he argues that highly ranked models can disappoint in use and dismisses TruthfulQA as a poor benchmark, without offering a controlled model comparison.

His criticism accompanies explicit support for open-source work from Mistral, Meta and Mosaic. He wants those efforts to succeed and opposes labs seeking regulation that would force others to follow their preferred approach.

Software determines who can use the silicon

Available hardware does not automatically deliver accessible performance. Patel describes Google’s internal software stack as JAX and XLA, while judging external PyTorch/XLA support usable but imperfect. He distinguishes TPUv5e, primarily an inference chip in his account, from the TPUv5 discussed in his earlier compute analysis: v5e is about half the size. He reports very good Llama 70B inference through PyTorch/XLA and better performance through JAX/XLA, without specifying throughput, latency or cost.

The larger goal is specialization across software layers. Researchers using a high-level framework such as PyTorch should not routinely need to write custom CUDA kernels—the lower-level routines that make their work run efficiently. Framework, compiler and kernel specialists can optimize those layers, while occasional breakthroughs cross their boundaries. OpenAI Triton is presented as a gathering point for third-party hardware vendors, connecting this argument to SemiAnalysis’s analysis of CUDA, Triton and PyTorch 2.0.

Small-batch inference runs out of bandwidth before arithmetic

The central engineering problem is how much computation a model performs for each parameter it reads. The simplified account assigns roughly six floating-point operations per parameter to training and two to an inference forward pass. An INT8 or FP8 parameter occupies one byte; FP16 occupies two. At batch size one, each parameter read supports little computation, leaving much of the GPU’s arithmetic capacity unused.

Two utilization measures distinguish the bottlenecks: - Model FLOPS Utilization (MFU) compares useful model computation with quoted hardware throughput. Achieving 200 trillion floating-point operations per second on an A100 quoted at 312 trillion is offered as a very good training result. - Model Bandwidth Utilization (MBU) measures how much of the available memory bandwidth the model uses. It is the more informative target for the small-batch inference workload under discussion.

Increasing batch size lets more computation share parameter reads, moving the workload toward a compute bottleneck. Training can exploit large batches; interactive inference must also meet latency requirements. The discussion places an H100 crossover around batch size 256, without specifying enough of the workload to make that a general threshold. Choose batching around the user’s latency requirement, then optimize throughput within it.

Llama 70B makes the bandwidth requirement tangible. At INT8, its weights require approximately 70 GB of reads per forward pass in this simplified model, excluding attention-related work. A proposed chatbot target of around 30 milliseconds per token puts effective bandwidth requirements in the multiple-terabytes-per-second range. The quoted 2.1 TB/s is approximate: the calculation shifts between 30 tokens per second and 30 milliseconds per token. The latency target itself is a usability judgment about reading and skimming, not a universal requirement.

The recommendation is to investigate wasted bandwidth before assuming more GPUs are necessary. Some Hugging Face inference configurations, including Llama 70B on eight A100s, are criticized for roughly 15% MBU. The suggested target is above 60%, with reports of 70–80%. Those figures lack complete benchmark conditions, so they serve as optimization prompts rather than a general library ranking. Models spread across accelerators also face communication constraints.

The hardware forecast makes this mismatch more consequential. Memory is described as scaling more slowly than logic. H200 is expected to provide temporary relief through bandwidth improvements, while later generations are expected to increase arithmetic throughput faster than bandwidth. A newer accelerator can leave small-batch inference further from its advertised compute peak.

Fast matrix units need fast communication

The same utilization problem extends between chips. Google’s TPU partnership with Broadcom is described as combining Google’s substantial model-specific hardware and internal chip-design work with Broadcom’s networking and high-speed chip-to-chip connectivity expertise. Google contributes to networking too. Good matrix multiplication hardware is insufficient unless memory and communication keep it occupied—interfaces the speaker argues many chip startups have struggled to build well.

Network speed is described as improving more slowly than compute throughput and memory bandwidth, making communication a major constraint on scaling. Supplier relationships consequently run deep: the discussion reports that Google’s plan to move away from Broadcom by 2025 had failed, with 2027 becoming the target. A roughly four-year chip design cycle limits how quickly such a change can happen, and TPU networking differs materially from GPU networking for low-level programmers.

Limited compute changes which problems are worth solving

“GPU poor” is relative to ambition. Even the largest lab can want more compute, just as it can want better data. Mistral is offered as an example of producing a strong model with relatively few GPUs before scaling up, although no count or benchmark accompanies the example.

Smaller teams have opportunities shaped by constraints beyond frontier training scale: - On-device inference: phone and laptop memory capacity and bandwidth limit model size and speed, while security, openness and independence from cloud providers give local deployment value. - Architecture and training research: alternative models and asynchronous training may change how effectively limited resources can be used. - Distribution and underserved applications: reaching users, operating services and building useful fine-tunes can create value. Adult-content generation is the concrete example of a market major providers may avoid.

Medusa, presented as Together’s open-source work, and speculative decoding are proposed as useful directions for batch-size-one inference and local deployment. Their decoding mechanisms are not explained here. The speaker contrasts their usefulness with high-batch cloud serving, claiming they do not work at high batch sizes without providing a measured comparison. The estimate that Medusa could be developed on one eight-GPU server is explicitly speculative.

The endorsement of useful fine-tunes sits alongside a much stronger later objection to fine-tuning existing small models outside on-device use. That objection assumes a cloud business has enough traffic to batch requests efficiently: good bandwidth utilization and batching could make larger models cheaper to serve than an unoptimized comparison suggests. The evaluation objection is separate: comparing a fine-tuned seven-billion-parameter model with an untuned “3.5” model leaves out what task-specific fine-tuning could do for the stronger model.

That comparison leads to disagreement over 3.5’s size and architecture. One proposed estimation method combines latency with assumptions about hardware, how the model is divided across chips and effective memory bandwidth. It constrains parameters used per forward pass, not total parameters. The resulting guess of roughly 40–50 billion active parameters is explicitly described as potentially very wrong; neither the model’s architecture nor its total size is established.

The discussion predicts open-source models will match GPT-4 while vision, tool use and later systems keep moving the target. An estimate of about 7,000 H100s running for three months to surpass GPT-4 depends on an exceptional team, data quality and algorithmic improvements; it is not a demonstrated result against a defined benchmark. Tool use and agents are suggested as further opportunities, with the speaker disclaiming expertise in agents.

Chip designs bet on models that do not yet dominate

A nonpublic document attributed to Noam Shazeer introduces the importance of anticipating workloads. One speaker recounts its prediction that large language models would dominate deployed compute inside Google, describes initial skepticism and sees Gemini training and inference as vindication. Accelerator designers face a related problem: their architecture must fit workloads several years after the design decision.

Memory placement illustrates the bet: - Fast memory on the chip: the on-chip memory discussed as SRAM offers high speed but low density. Concentrating memory there can be superior for image networks and other models that fit entirely within it. - Larger memory outside the chip: the off-chip memory discussed as DRAM provides capacity as models grow, but the bandwidth connecting it to computation becomes critical. Capacity alone does not ensure useful performance.

The speaker criticizes Graphcore and SambaNova for combining substantial on-chip memory with comparatively slow external memory, while crediting NVIDIA with a better balance of external capacity and bandwidth. Cerebras illustrates how quickly the capacity bet can fail: even a wafer-sized chip with a stated 40 GB of on-chip memory can be outgrown by deployed models. MatX receives a more favorable assessment for informed architectural bets, though technical details are withheld and success remains conditional on future workloads.

The November 2023 prediction for AMD MI300 and Intel’s forthcoming accelerator is better hardware than H100 on paper, helped by arriving later and adopting a similar balance of compute, bandwidth and capacity. Programming difficulty is excluded from that comparison. Competitive design also requires interdependent decisions about model behavior, power delivery, cooling and physical layout.

Manufacturing capacity limits competition too. These architectures require high-bandwidth memory (HBM), large chips made on advanced manufacturing processes and the supporting supply chain. Lower supplier margins can let competitors offer more memory at a similar selling price despite higher manufacturing costs. Meanwhile, NVIDIA’s expected annual release cycle threatens to shorten their windows of advantage; the specific future performance and price figures offered in the conversation are hearsay.

A hardware discount must pay for the engineering

The practical platform recommendation separates three business cases: - TPUs for available capacity: Google’s increased external sales effort makes its infrastructure a serious option. - NVIDIA GPUs for optimized software: the extensive open-source ecosystem reduces the work needed to obtain useful performance. - Other accelerators for a compelling business case: strong low-level programming skills and a substantial supplier discount should justify the additional effort.

Large cloud operators have another incentive: credible alternatives can improve negotiating leverage with NVIDIA before becoming the best chips. For a frontier lab, however, supporting several accelerator platforms consumes scarce technical staff. Maintaining separate low-level codebases competes with improving utilization, designing the next supercomputer, developing multimodal models and making asynchronous training work.

Those costs complicate lab–cloud partnerships. The discussion predicts that OpenAI and Microsoft will become less close as their ambitions overlap, even if OpenAI continues using Microsoft’s GPUs. Microsoft’s then-upcoming chip is described as potentially cheaper for Microsoft internally despite lower performance than H100, but only for certain unspecified model sizes.

Apple presents a different tradeoff: polished releases versus learning from public deployment. The assessment is that Apple’s product culture conflicts with the feedback cycle through which model developers discover jailbreaks and improve behavior. Yet its exceptional distribution remains a major advantage.

Moving a fab does not move its supply chain

A question about Taiwan shifts the discussion from companies to national dependencies. Concern about conflict is described as mixed, with expectations of an invasion relayed as other people’s views. The argument about rebuilding semiconductor capacity in the United States is more concrete: domestic fabrication still depends on specialized suppliers abroad.

Two examples expose that dependence: - Small suppliers can be indispensable. Two unnamed Austrian tool companies, each described as earning less than $1 billion annually, are said to control technologies essential to advanced chip manufacturing. - A domestic fab can retain overseas dependencies. TSMC’s Arizona facility is described as requiring masks from Taiwan, Japanese chemicals and Austrian tools, with output still returning to Taiwan. The purpose of that return shipment is not specified.

Rebuilding the system requires accumulated expertise, research and capital across many specialized businesses. At the same time, international diffusion of technology helps the industry advance through contributions from the United States, Korea, Taiwan, Japan and the Netherlands. Geographic interdependence creates both capability and exposure.

Industry history explains why pivots matter

The closing reading recommendations span software, manufacturing and infrastructure: SemiAnalysis’s PyTorch 2.0 and Triton article, its Advanced Packaging series, Google Infrastructure Supremacy and the AMD MI300 analysis. The Google piece is singled out for connecting networking, chips and TPU history.

Book recommendations include a Chris Miller title transcribed as “Chip Wars” and Only the Paranoid Survive, whose authorship is unclear in the exchange. The historical interest follows a broad conviction that semiconductors underpin much of the Western world’s productivity growth since the 1970s, enabling improvements in cars, software and connectivity.

NVIDIA’s move toward AI is presented as a pivot cultivated through adoption. Anecdotes describe Jensen Huang giving GPUs to university researchers and conference attendees. A host draws a parallel with OpenAI’s movement from projects such as Dota toward GPT around 2019, explicitly offering a personal classification rather than an official corporate history.

Four hours of writing can rest on months of research

SemiAnalysis’s publishing process combines teammates’ specialized knowledge with topics prompted by excitement or a business event. Kokusai Electric’s planned listing prompted an explanation of manufacturing processes for flash storage, gate-all-around transistors and newer memory designs. Myron Shi is credited with most of the work, which reportedly took more than a month despite his existing expertise. These projects also support consulting and separately sold reports.

One AI article reportedly took four hours to draft, but drew on substantial prior work on Google’s TPU expansion, manufacturing capacity, memory supply and unit orders. The team describes attending more than 40 conferences annually, spanning chipmaking materials, manufacturing tools, networking and AI. The drafting time understates the accumulated technical work. Viral commentary and leaks attract disproportionate attention; the stated core business is understanding changing technologies, supply chains and where resources can create value.

Can training escape a single data center?

Asked which unknown could change his worldview, the closing speaker returns to networking. His position, based on what he had seen by November 2023, is that large-scale training requires one data center with very high-speed communication among chips. Every chip need not connect directly to every other chip, but fast communication throughout the system is essential.

The desired breakthrough is effective training across compute pools with much slower links between them. The concrete example combines 256 GPUs in a small data center with a larger cluster elsewhere. An algorithm that could use both effectively would relax the limit imposed by the power and chips that fit inside one facility.

Google, OpenAI and Anthropic are described as working on the problem, but the speaker does not know whether they have solved it. The anticipated orders-of-magnitude expansion is conditional on a solution. The final question is how to turn geographically scattered hardware into useful training capacity.