The Four AI Wars: From Frontier Models to Deployable Systems — Part 1 of 3

Where should a country put its AI budget?

A country facing a shrinking workforce wants AI to improve productivity. Should it fund foundation models, data centers, GPUs or agents? Alessio opens with this allocation problem. In Singapore, senior financial executives had asked him about state-space models and alternatives to Transformers. He sees technical sophistication; Swyx worries that interest in possible replacements can bypass the immediate question: what will an organization actually put to work?

Swyx proposes AI engineering as a route for countries outside the US to participate without financing a $100 billion compute cluster. That distinction between building frontier capability and applying it runs through this Latent Space recap. The hosts revisit four categories from their 2023 year-end discussion: compute access, data quality, multimodality and ecosystem tooling, originally labeled GPU rich versus GPU poor, data quality wars, multimodality wars and RAG/Ops wars.

Claude changes the default—and the evaluation question

Claude 3.5 Sonnet represents a competitive shift for Swyx. At the time of the conversation, he describes it as leading some hard and coding benchmark categories on LMSys for more than a month. He also sees projects and tools being built specifically for Claude: developers are choosing Anthropic first, weakening the assumption that the ecosystem revolves around OpenAI.

The technical discussion turns to Scaling Monosemanticity and sparse autoencoders. In Swyx’s explanation, these tools identify internal model features whose activation can be increased to steer behavior without retraining the underlying model. His examples range from fixation on the Golden Gate Bridge to better coding behavior, though he gives no measured coding improvement. He sees applying interpretability to Claude 3 Sonnet as a significant step beyond toy models. The suggestion that this research substantially explains Claude 3.5 Sonnet’s performance remains a rumor; his informal contacts also pointed to other undisclosed techniques.

Alessio hopes a distributed open-model effort can combine interpretability with synthetic data from Llama 3.1, eventually bringing insights from a 405-billion-parameter model into models around seven billion parameters. That is a research ambition. Their immediate reasons for choosing Claude are practical: - Podcast work: Alessio generally prefers Claude when comparing it with OpenAI, explicitly describing this as subjective experience. - Summarization and instruction following: Swyx runs daily comparisons for AI News Summary and favors Claude on these tasks, while acknowledging that OpenAI sometimes wins.

These experiences motivate evaluation that distinguishes useful behavior. Swyx argues that the MMLU benchmark alone no longer captures the differences he cares about. The discussion also raises a question-quality limit: flawed questions may impose an error floor, so extremely high scores could reflect memorization. This is a criticism of benchmark coverage and interpretation, not evidence that a particular model memorized the test.

Llama’s value as a teacher

Before assessing Llama, the hosts pause on an editorial tension: prominent guests help grow the podcast, while researchers provide the technical depth its audience values. Their preference for that depth carries into the release discussion. Llama’s most useful contribution may be a large model that helps build specialized small models.

The mechanism is distillation through synthetic data: use a larger model’s outputs to train or fine-tune a smaller model for a specific task. One host connects this to a perceived shift in value toward applications serving particular industries or workflows. Improving a small specialist could matter more to a business than serving every request with a 405B model. Direct deployment raises separate, unresolved questions about self-hosting costs and whether providers such as Fireworks and Together can support demand from large products.

The useful question has moved from whether synthetic data works to which tasks it serves and how to use it. The Llama paper is praised for describing methods for code, mathematics, multilinguality, long context, tool use, speech recognition and voice generation. These methods concern post-training, broadening the earlier focus on synthetic data for pre-training. The hosts characterize the contribution as permission to distill combined with a published recipe, without examining specific license terms.

Competitive parity brings rapid depreciation

Anthropic and Meta releases now receive serious consideration as frontier contenders, the hosts argue. OpenAI’s newly announced SearchGPT is still behind a waitlist; expectations of a dedicated model or unseen innovations remain speculative. One proposed way to assess labs is the trajectory of successive releases. Perceived improvements from Phi-1 through Phi-3 and from Gemma 1 to Gemma 2 suggest momentum. Gemma 2’s favorable reception in an informal Local Llama poll is a community signal, not a controlled evaluation.

This pace creates an economic concern: competitive value can depreciate quickly after expensive model development. Mistral Large 2 becomes the example. The hosts welcome what they describe as GPT-4-class open weights under a research license, but find the release less distinctive than Llama’s. Better instruction following, structured output and multilinguality maintain competitiveness; one host argues that Mistral needs clear leadership in a dimension to stand out.

The commercial assessment is less certain. One host hears little about Mistral use at scale; the other points out that a US-centered network may miss adoption elsewhere. Mistral’s Mamba experiments and Mistral Nemo’s efficiency receive more enthusiastic attention, although no efficiency measurements accompany that assessment.

Inference costs shape hardware, training and architecture

Tri Dao’s FlashAttention 3 illustrates how a small group’s optimization work can affect an industry while reinforcing a hardware incumbent. The hosts describe the release as specific to Nvidia H100s. Its optimization, scaling and validation work would be expensive to reproduce elsewhere: software investment compounds Nvidia’s hardware advantage.

The counterargument is economic. If training runs grow from the cited $100–200 million range toward $500 million or $1 billion, custom chips could become justifiable. A single run could repay development costs if its savings were large enough. Halving run costs is the hosts’ hypothetical assumption, not a demonstrated result.

Character.AI illustrates optimization across the model and serving system. A post attributed to Noam Shazeer compares its inference traffic with 20% of Google search traffic. The hosts qualify that figure immediately: multi-turn conversations produce many messages, so comparable request counts do not imply comparable users or completed tasks.

Two techniques receive concrete attention: - Native INT8 training: a host corrects an earlier FP8 description. Training with reduced precision is intended to save memory while preparing for inference that will use reduced precision anyway. - Global-local hybrid attention: Character.AI reportedly uses a 1:5 ratio of global to local attention, described as effective for its long conversations. Both make eventual inference constraints part of training and architecture decisions.

One host predicts a broader division of labor: Transformer components provide global attention, while other components handle local processing, potentially including sliding-window Transformers. Hybrid architectures remain a research-stage direction in this discussion, with scaling still prospective. The relayed claim that Shazeer could make inference 30 times cheaper than Fireworks and Together likewise describes possible headroom, without matched workloads, hardware, quality targets or cost accounting.

Technical enthusiasm does not resolve the business question. Discussion of a possible Character.AI sale remains speculative, with little public financial information. One host’s suggestion that Shazeer alone should be worth $1 billion to Google or Meta is a valuation opinion, not a transaction report.

Local AI makes routing a platform decision

Llamafile is presented as a possible second act for Mozilla: open-source local AI emphasizing privacy and optimized CPU inference. The hosts distinguish two layers: - Execution infrastructure: Llamafile’s cross-architecture executable, alongside Mojo, llama.cpp and MLX. - Vendor integration: Gemini Nano inside Chrome and Apple Intelligence at the operating-system level, exposing model capabilities through existing products.

One host predicts that small models will converge enough that users stop caring which one runs. The other counters that Apple Intelligence itself is a router: it selects task-specific adapters—components loaded for a particular use case—or sends requests to OpenAI. Spellchecking makes the rationale concrete. Choosing an appropriately sized model could avoid unnecessary computation and support privacy, though neither benefit is measured here.

Control of routing creates bargaining power. A device platform could make external AI providers interchangeable. The hosts debate whether default AI placement might become more valuable than default search placement, while recognizing that the AI provider bears inference costs. A no-payment OpenAI arrangement and possible Google participation are discussed as rumors.

For web developers, browser access is a separate priority. One host considers Gemini Nano’s success particularly consequential for the open web, but has seen demos rather than systematic tests. The discussion then highlights DataComp LM, described as extending DataComp from vision into language with an open 7B reference model associated with Apple. Its claimed advantage over FineWeb comes without comparison conditions; openness in data, weights and model is the more concrete reason offered for expecting further optimization.

Shared weights and matrices are mentioned as ways to reduce small models’ file size and memory use. One host extends this into an economic thesis: intelligence too cheap to meter requires local execution, creating tension with metered API businesses. Speculation about an OpenAI phone follows, tempered by iOS lock-in and reluctance to buy an unproven device.

Data rights are also a competition problem

At the time of the recording, the New York Times lawsuit remains ongoing while licensing deals involving Stack Overflow and Shutterstock continue. One host supports the newspaper’s challenge ideologically but argues that other publishers’ willingness to license weakens its bargaining position. The account of negotiations nearing agreement before breaking down is OpenAI’s stated position, not an established history of the talks.

Reddit brings three competing interests into focus: - Access for new competitors: one host asks whether datasets gated behind contracts exceeding $100 million could prevent startups from competing. - Freedom to negotiate: the other considers state intervention in such contracts an overreach and defends voluntary dealmaking. - Payment for contributors: platforms can license user-generated content supplied by volunteers, complicating who should receive the proceeds. These are policy arguments, not an existing legal restriction on data contracts.

The hosts also discuss Scarlett Johansson’s criticism of OpenAI’s Sky voice and music-industry actions against Udio and Suno. One considers Johansson’s legal prospects stronger than broad training-data claims, while expecting costly, uncertain music litigation. Their larger forecast is that a ruling accepting AI training as transformative use could remove much of the litigation-driven incentive to license content. That is a legal position and economic prediction, not a reported ruling.

The creator-payment question briefly becomes personal: the hosts invite listeners to support Latent Space through an $8 monthly subscription to help cover microphones and travel. They describe producing the podcast as a labor of love whose financial return does not justify the time invested.

Capability needs a generalization step

Returning to reasoning, the hosts discuss AlphaProof through a reported result one point below the International Mathematical Olympiad gold-medal cutoff: full marks on four of six problems and failures on two. Their account does not specify time limits, problem preparation or individual components’ contributions. Related Hugging Face and Numina work concerns an easier AI mathematics competition and draws attention to search guided by verifiers—exploring candidate mathematical solutions with assistance from checks on the work.

Jagged intelligence describes the contrast between advanced success in one setting and elementary failures elsewhere. The hosts juxtapose Olympiad problem-solving with reports of language models mishandling the comparison between 9.9 and 9.11. These are not matched tests of the same system. The consequential distinction is that specialized mathematical success does not establish general intelligence.

The hosts differ on how that gap closes. One questions whether a single scaled model will do everything or whether a combination will be needed. The other expects future models to unite advanced reasoning with everyday capabilities, while conceding that using them for ordinary work could be uneconomical.

This motivates a revised development sequence: 1. Demonstrate capability in one domain. 2. Generalize the capability beyond that setting. 3. Make it efficient enough to deploy. Generalization is the added step in the earlier capability-then-efficiency view. The deployment recipe includes training a large base model, distilling it, and adding structured outputs, tool calling and long context. How many individual capabilities to improve before attempting generalization remains unresolved.

Voice deployment meets the acoustic environment

GPT-4o voice is still awaiting rollout in this conversation, with a beta expected the following week. Anticipated access has persuaded one participant to resubscribe to ChatGPT Plus, illustrating how an exclusive capability can restore subscription appeal. Yet its AI Engineer World’s Fair demonstration required work beyond the model itself.

Large conference speakers fed the system’s output back into its microphone. It had to distinguish human speech from its own voice and ignore the latter. OpenAI engineers tuned the setup for that stage. Roman, who reportedly helped deliver the demonstration after last-minute dropouts, was prepared to cancel if the audiovisual setup could not be made to work. Presentation quality depended on handling the actual acoustic environment.

Multimodality: adapters or joint training?

The hosts anticipate multimodal Llama, describing adapters that bring previously separate Meta research, including Voice Box and Seamless Communication, into the model family. One participant attributes earlier speech-system weaknesses relative to Whisper largely to training scale and hopes more investment will close the gap. That remains a forecast; the discussion does not demonstrate that additional audio alone would suffice.

Chameleon introduces the architectural choice: - Late fusion: freeze an existing language model and vision Transformer, then connect them with a thin adapter. The hosts describe multimodal Llama as following this approach. - Early fusion: train with modalities natively interleaved, as Chameleon is described doing for image generation, visual understanding and text understanding. One host expects demand for multimodal applications to justify greater investment in joint training and interprets GPT-4o as a sign of that shift. This is a forecast, not a comparative performance result.

Research capability can exceed what a release exposes. The hosts report that the open Chameleon release omitted image generation. Fine-tuning recipes then circulated to enable it, with project leads discouraging their use.

The discussion returns to business utility through a small PDF-processing system combining retrieval with a vision model to extract structured text. Its name is uncertain in the transcript, but the application is concrete: specialized multimodal models for document work. The relayed claim that it outperforms Amazon Textract is hearsay without evaluation conditions.

Developer access and the opening of “LLM OS”

The final practical concern is getting started. One host presents AI Studio as a more approachable alternative to Vertex, comparing the relationship to Netlify or Vercel versus AWS. The friction is concrete: layers of authentication, organization identifiers and policy permissions before an engineer can build. Reducing that setup burden is part of making model capability usable.

Part 1 ends with a proposal to rename “RAG/Ops wars” as “LLM OS.” The opening rationale is that the earlier tooling category reflected a chatbot-centered view of AI applications. The proposed name signals a broader platform discussion, but this part ends before defining its components or explaining how it would differ from existing tooling.