AI’s Bottlenecks Move Beyond the GPU — Part 3 of 3

Microsoft: supply competitors or defend the core?

Microsoft faces a concrete allocation problem: rent GPUs to companies that could undermine its software business, or direct capacity toward its own capabilities. This final part of the conversation opens with a critique of its retreat from a larger OpenAI infrastructure commitment. One participant interprets that decision as a conflict between shareholder expectations, return on invested capital and willingness to spend against a threat to the core business.

The criticism describes an innovator’s dilemma: preserve today’s cash flow or invest against the disruption of the business that produces it. The participant reads Microsoft’s strategy as treating AI as another productivity tool—an interpretation of management’s behavior, not a stated position from Satya Nadella. Claude for Excel and Claude for PowerPoint make the argument tangible: both participants see them as products Microsoft should have built using its distribution and OpenAI relationship.

The critic expects Microsoft to turn further inward through internal training, foundation models and model weights it can access, but doubts its execution. The other participant counters that Microsoft’s resources leave the outcome open. The disagreement is about direction as much as spending: defending Copilot while pulling back from Azure looks inconsistent to the critic, who explicitly remains willing to revise that judgment.

Oracle: debt markets constrain the pace of construction

Oracle presents the opposite problem. The same participant supports taking on more AI infrastructure business but criticizes the scale and sequencing of its commitments: aggressive announcements, large fundraising and then construction delays. The proposed alternative is phased deployment, with revenue from operating GPUs helping fund subsequent expansion. That would stagger financing needs while giving deployed capacity time to produce revenue.

The financing mechanism extends beyond Oracle’s balance sheet. A large debt offering must attract investors away from other securities. Better terms on the new offering can pressure the prices of existing debt. In this account, the market’s ability to absorb debt becomes an infrastructure bottleneck: a viable project can still demand more financing, more abruptly, than available buyers will comfortably supply.

The participant attributes much of the concern around Oracle’s credit default swaps to this supply-demand effect in a relatively illiquid market, while maintaining that Oracle can meet its commitments. Both participants believe Microsoft could have funded the infrastructure internally. One also claims Microsoft enjoys a borrowing-cost advantage over Oracle, though the discussion provides no dated, maturity-matched comparison or calculation establishing its size.

TPUs: convert a temporary advantage into repeat customers

Google’s move to supply current-generation TPU accelerators externally raises another allocation question. One participant interprets the change through the period before Gemini 3: keeping compute internal offered limited strategic value if consumer products were losing ground. Supplying Anthropic could instead turn Google’s hardware advantage into an external business.

The participant considers TPU v7 Ironwood to have its largest advantage over Nvidia on total cost of ownership, or TCO, at the time of the discussion. That creates two strategic choices: - Capture margins now: retain more of the economic benefit while the hardware is especially competitive. - Build an installed base: win deployments whose customers can later upgrade an existing TPU installation.

The second choice explains the Anthropic relationship in this account. An installed base creates future upgrade demand; a challenger must first win a new deployment before it gets that opportunity. The participant identifies this as a major obstacle for AMD and argues that TPUs now have the mature software, hardware and networking—and a capable external customer—to pursue broader adoption.

Favorable economics do not guarantee available supply. TSMC is tentatively identified as the largest constraint, with optical components also mentioned. Plans for a future TPU benchmark remain prospective; no measurements accompany the TCO claim. A separate question about JAX gaining ground on PyTorch receives no substantive answer.

The participant expects Google’s favorable window to last roughly one or two years, contrasting its stable, repeatable TPU pods with Nvidia’s pursuit of maximum system performance. They argue that GB200 would have decisively beaten v7 if it had arrived on time and stable. Looking ahead, they expect Rubin to benefit from memory improvements and Nvidia’s supplier relationships, including access to better memory, connectors and denser racks. These are forecasts, not measured comparisons with TPU v8; rising memory costs would affect Google too.

Memory demand displaces supply elsewhere

The memory squeeze spreads through both system design and manufacturing. Moving the model’s KV cache into memory outside HBM adds demand to those other tiers. Meanwhile, producing more HBM consumes capacity that could otherwise supply conventional DRAM. The participants describe a rough three-to-four-times production-capacity trade-off, partly associated with imperfect yields. The range is approximate: it is not a universal conversion factor, and no process-specific assumptions or consistent capacity denominator are given.

The fuel analogy is useful: producing a newly valuable, highly refined fuel consumes capacity previously used for ordinary fuel, leaving less of the latter available. HBM demand encounters an industry whose previous financial distress had discouraged investment in cleanrooms and equipment. One participant cites roughly two years to build cleanrooms and three-year lead times for some investments, predicting that supply will take about two years to catch up.

That participant forecasts another possible doubling of DRAM prices, but also describes demand destruction: prices rising enough that buyers purchase less. Power and construction delays can make postponement practical. In the example discussed, a data center expected in 12 months opens in 18 instead, giving its buyer room to defer expensive memory purchases and focus on energy infrastructure.

Ordering behavior can amplify the reversal. Buyers pull purchases forward and double- or triple-order to secure supply, inflating the demand producers see. Once power limits deployment, buyers slow orders and memory prices can fall. The participant still presents memory capacity—through SK Hynix, Samsung, Micron and semiconductor capital equipment—as a direct investment expression of the bottleneck thesis. The ordering cycle explains why that thesis does not imply uninterrupted price increases.

Scarcity also changes which engineering compromises become attractive. CXL enables CPU memory expansion and pooling; the proposed use is to collect older DDR4 and spare memory into racks and attach that capacity to newer systems. The participant sees a renewed opportunity for CXL because it can reuse available memory under shortage conditions, rather than because it offers leading memory performance.

Context capacity, usefulness and price are separate questions

Memory constraints bring the discussion back to software. One participant argues that mainstream context windows have stayed near a million tokens for roughly two years and predicts that physical constraints will prevent expansion to 100 million or a trillion tokens over the next five to ten years. This is a forecast, not a demonstrated ceiling. The pair imagine context rationing: a hypothetical thousand-token free window and a million-token tier priced a hundred times higher. Those figures illustrate a possible pricing structure, not an announced product.

A context-length headline can conceal several distinct questions: - Can the model process the stream? One participant notes that state-space models can process a hundred million tokens without applying full attention across them. - Can it use the information effectively? The other questions the benefit of an enormous window when performance deteriorates, drawing on personal experience and unspecified papers. - Can it reuse a smaller budget? Recursive language models are introduced as an approach that repeatedly reuses the same context window.

Algorithmic improvements could make context more useful even while physical capacity remains constrained. The expectation is better utilization over time, with continuing skepticism that models will effectively use everything their windows can hold.

Specialized chips must address the limiting resource

Fixing model weights in silicon attacks memory demand directly. The participants find custom chips with embedded weights appealing because they could avoid memory used to hold those weights and reduce data movement. Implementation remains unresolved, and one participant has not tried the referenced system. The claimed benefit concerns weight storage; how the design handles other inference state is left unspecified.

The commercial question is whether a heavily distilled production model delivers enough capability at an attractive efficiency trade-off to justify embedding it. One participant expects a more segmented accelerator market, with performance-focused inference serving using different hardware from training and other large workloads. Separating prefill and decode is offered as another example of dividing computation into specialized parts.

Asked about Etched, the participant distinguishes its large systolic-array approach from fixing weights in silicon. They interpret the array design as a bet that compute is the bottleneck, while the production problems they see concentrate in memory. Specialization pays when it addresses the resource that limits the workload. They acknowledge useful niches but remain skeptical of alternative accelerators’ broader prospects, treating Cerebras and Groq as exceptions to a history they view pessimistically. SambaNova receives a more favorable mention, qualified by limited recent familiarity.

Generated software still needs CPUs

The CPU discussion begins with speculation about perceived cloud instability. One participant suggests either poorly vetted AI-generated code reaching production or aging infrastructure stretched by budgets redirected toward AI. The hardware hypothesis places 2020–2021 purchases near a five-to-six-year refresh cycle. Neither explanation is causally connected to outages in the discussion.

The CPU-demand thesis combines several workloads: - Software created by coding agents: Claude Code and similar tools produce software that must execute somewhere, potentially raising CPU utilization. - Reinforcement-learning environments: simulating and running software during training consumes CPUs, even at a different scale from GPU demand. - Production agents: deployed agents add another source of compute demand.

Against an alleged two years of maintenance-focused spending, even modest growth could expose insufficient CPU capacity. The argument does not require CPU demand to grow as quickly as GPU demand. The technical discussion closes with a related allocation forecast: rising memory prices could make low-end phones or gaming GPUs unaffordable for some buyers. Scarcity could determine which uses get priced out, rather than simply raising every hardware market together.

Writing: prepare ideas, then compose

Asked how they write so much, one participant begins with reading. Rapid synthesis had been a personal advantage before large language models: reading friends’ PhD papers and pushing through semiconductor textbooks helped them learn unfamiliar subjects, even with incomplete comprehension. A sustained weekly publishing requirement developed the writing habit. Maintaining both SemiAnalysis and Fabricated Knowledge had since made that cadence harder to sustain.

Their preferred model workflow is use an outline to support personal composition. They supply disorganized thoughts, request an outline with bullet points, then read it while developing the argument and writing the prose themselves. They also use model-assisted editing guided by a style guide and lessons from On Writing Well, condensed into a reusable skill.

The participants compare two preparation habits: - Mental accumulation: let an idea develop until enough information has gathered for a rough outline, then draft in one concentrated effort. Fragments stored in Slack and elsewhere remain available for retrieval. - Mise en place writing: maintain a store of ideas, develop several in parallel and periodically compose from the prepared material. Separating passive collection from focused writing helps preserve useful details.

The first participant adds a deliberate pause: collect ideas and outlines, sleep, then open a fresh tab and write first thing in the morning. They estimate that this gets a piece roughly 60–75% complete, sometimes as an outline developed enough to make the remaining work clear. Prepare before sleep; compose with a fresh mind is their practical recommendation.

Six months away, and a clearer sense of personal limits

The closing exchange considers taking a break when absence feels costly. One participant fears falling behind; the other recalls feeling they missed a major year of Substack growth during a 2021 sabbatical. They nevertheless chose the Continental Divide Trail for their first thru-hike because it frightened them most and represented a long-desired accomplishment. They made the choice assuming another opportunity might never come.

The hike took six months and covered approximately 2,800–2,850 miles on their chosen route, mostly alone. Audiobooks became tiresome, then music did too, leaving substantial boredom. Hunger, fear and loneliness accompanied unusually intense emotional lows and highs. The participant still values the experience deeply.

The lasting benefit was a more precise understanding of personal reactions and limits: what feels too frightening, too difficult or beyond what they are willing to do. They recommend making room for a long hike while young and wish they had done it immediately after college. The conversation ends with self-knowledge and self-mastery as foundational tools worth developing.