Latent Space’s 2024 Year in Review — Part 1 of 3: From Scaling Models to Shipping Systems

When every recommendation is the same

The lightning round had stopped producing surprises. Guests kept recommending Cursor, Perplexity and Midjourney, leaving the hosts asking what else people actually used. By Latent Space’s 100th episode, after almost two years and four studios, Alessio wanted more research-driven conversations like the interviews with Tri Dao and Jeremy Howard: a way to discover work beyond the familiar tools.

In this 2024 year-end conversation, Alessio treats the podcast’s growth as a rough proxy for the expanding AI engineering community. He recalls hundreds of people queuing for an applied AI talk at AWS re:Invent. swyx points to Gartner and GitHub adopting the term as signs that the category had arrived, while deliberately leaving its boundary with machine learning engineering open.

For swyx, AI engineering means translating research into production without reproducing every paper. Calling APIs may carry less prestige than developing models, but the tools and expertise required to build useful systems keep growing. That explains why an engineering podcast covers NeurIPS: research needs a route into deployed products. He argues that commercial outcomes offer a stronger measure of success than citations alone.

Field surveys lead to the scaling debate

Latent Space Live’s domain surveys addressed a related information gap. Research conferences organize attention around individual papers and recruiting researchers; visitors seeking an overview must supply much of the connecting context. swyx commissioned surveys of startups, vision, open models, post-transformer approaches, synthetic data, small models, agents and reasoning. Workshops and broader conference tracks help, he acknowledges, but speakers need an explicit assignment to synthesize a field.

The closing debate asked whether scaling had encountered a wall. swyx describes the difficulty of finding someone willing to challenge the prevailing pro-scaling position and says the opposing side won. A later clarification matters: the anti-scaling position was an assigned role. Its advocate still believed further scaling was possible, so the debate stance should not be read as a personal forecast.

The technical question becomes where additional computation should go. Inference-time compute is computation spent while using a model. The conversation favors that term over “test-time compute” because inference serves production work as well as evaluation. It also distinguishes Chinchilla-style optimization of pretraining compute from optimization that includes serving costs: the best training allocation can change once inference enters the budget.

Competition needs more than a leaderboard

The hosts describe late 2024’s leading frontier tier as a contest among Gemini, Anthropic and OpenAI. They see Claude 3 and Claude 3.5 Sonnet as catalysts for a shift toward Anthropic. They also argue that xAI’s slower API rollout and limited cooperation with benchmarkers made it harder to include in comparisons: access affects which competitors get measured.

Spending share and request share answer different questions. Two signals in the conversation need separate interpretations: - Ramp’s spending estimates: The cited dataset ended September 2, 2024. The hosts question how credit-card spending identifies individual API models, leaving the conversion from spending to production traffic unresolved. - OpenRouter’s request volume: A cited figure gives Gemini Flash half of requests on that platform. The speaker emphasizes small, cheap calls; this measures neither overall market share nor half of model spending.

Flash prompts another reconsideration. One host had expected the open-source long tail to produce the best small models, but now credits the large labs. Yet small no longer simply means cheap. The discussion distinguishes open models around one to three billion parameters from commercial offerings whose parameter counts may be undisclosed. A low API price does not establish a small architecture.

Scale’s SEAL leaderboard earns praise for private test sets intended to reduce benchmark contamination. But a host questions the production relevance of a large Llama model’s strong results: inference is described as slow and expensive, with customization constrained even when specialized serving is available. The host sees greater value in using it as a teacher for distillation—transferring capabilities into a smaller model.

There is also a measurement trap: a benchmark nearing its ceiling can make capability gaps look smaller mechanically. One host argues that o1 widened the meaningful gap between open and closed models, even while saturated charts suggested convergence. That assessment concerns the direction of the frontier, rather than a demonstrated ranking across every task.

Inference changes who pays

Inference-time compute changes the economics of openness. Funding a training run produces a reusable artifact; funding training does not pay for everyone’s future inference. If capability depends on substantial computation for each request, distributing weights leaves a recurring expense with whoever operates the model. The hosts see a possible advantage for large labs, while leaving the consequences for GPU clouds and the largest cloud providers open.

For a commercial provider, the attraction is tying compute expense to customer revenue. More usage incurs more inference expense and can generate more revenue through usage-based pricing. A host argues that adding a margin to attributable costs makes gross margins easier to control than a large upfront pretraining bet. The suggestion that financing pressure caused this shift remains conjecture. With cheap inference leaving little room for further price cuts, compute-intensive premium products offer another route to revenue.

One host plans to try Devin and ChatGPT Pro to discover what spending $1,000 a month on AI enables. The other supports substantial experimentation for people whose work requires staying current, with a practical qualification: subscriptions are easy to buy and harder to use. Google’s newly launched Deep Research offers another experiment, described at the time as inference-intensive and free during its preview.

Lindy provides a concrete workflow example: one host has built meeting preparation and a phone-call assistant with it. Its speech-to-text-based voice interaction is perceptibly less natural than advanced voice interaction, but still useful for tasks such as reservations.

Agents need to discover the instructions

Predictions of a coming “year of agents” repeat the previous year’s promise. A talk attributed to Graham offers a more concrete framing: unresolved frontier problems. One stands out for enterprise deployment: how does an agent learn an environment that was never public? Internal codebases and tools are only part of the answer.

Indexing and retrieval-augmented generation, or RAG, depend on having records to retrieve. Retrieval cannot recover a process nobody documented. Employees may know how work gets done through experience without that knowledge appearing in a searchable document. The hosts call the missing work business-process extraction: eliciting the practices needed to carry out a task. Their diagnosis is that current agents are better at following supplied instructions than drawing those instructions out of users.

The discussion then turns to systems that improve AI itself. The hosts characterize Jeff Dean’s talk as a feedback loop: use machine learning to improve systems, then use those systems to improve machine learning. They connect this to synthetic data across pretraining, post-training and evaluation, and to AlphaChip’s work on chip design.

The data constraint behind the pretraining argument

Returning to Ilya’s talk, the hosts recall emails released during OpenAI’s dispute with Elon. They interpret the 2016 correspondence as evidence that Ilya and Greg were already pushing GPU scaling during the Dota work. The constraint was concrete: enough compute for 1v1 training, but substantially more needed for 5v5.

A host narrows the earlier “wall” argument: data may not be growing as quickly as compute. On this reading, Ilya was questioning whether adding chips could reproduce earlier progress when the other ingredient failed to scale alongside them—not declaring pretraining finished.

The hosts contrast reinforcement learning’s strength at specific tasks with broadly trained language models that still need specialization. Reinforcement fine-tuning uses rewards to guide task-oriented improvement. Its appeal begins with a familiar barrier: telling users they can customize a model does not give them the dataset needed to do so. One host expects broader adoption if users can begin with a task and a way to judge successful behavior.

Hidden messages and overlooked research

Finding overlooked work is another reason to attend NeurIPS. One host values talking with authors away from the crowds around already-famous papers. Copies of Must-Read Papers of 2024, a document made by a friend identified as Yi, reportedly disappeared quickly from where they were laid out—a sign, to the host, of demand for guidance through the literature.

One discovery was Secret Collusion Among AI Agents: Multi-Agent Deception via Steganography, described as a DeepMind paper. Steganography hides a message inside visible content. A simple example is taking the first letter of each word to recover a second message. The hosts raise the hypothetical risk of agents coordinating covertly through ordinary-looking text; they do not report an observed case of autonomous malicious coordination.

The discussion separates two ways agents might coordinate: - Agreed encoding: Agents share a format for hiding and decoding messages. - Schelling-point coordination: Agents converge on an obvious convention with little explicit agreement. The analogy is choosing Grand Central Station when told only to meet someone in New York; common decoding methods might provide similar focal points.

The broader conference impression is a shift toward datasets and benchmarks as contributions accessible to researchers with limited compute. Larger developers can then adopt those resources. One host favors industrial use as a measure of research value and questions whether oral selections reflected it, citing DCLM and FineWeb. Schedule-free optimization, Shampoo and SOAP attracted attention too, although the host reports informal skepticism from people at large labs about their practical importance.

Synthetic data moves toward graded reasoning

The hosts revisit their Four Wars of AI framework, beginning with data: writers, artists, publishers and other intellectual-property owners on one side, synthetic-data proponents on the other. They distinguish a common use from a larger unresolved promise. Distilling a large model into a smaller one does not establish that synthetic data will produce the next frontier advance. One host leaves that question open, acknowledging limited visibility into frontier-lab practices.

Reasoning introduces another route. OpenAI’s announced reinforcement fine-tuning API is described as accepting user-submitted graders or preset graders. A host interprets this as a possible way to generate valid synthetic examples for improving reasoning paths. The next question is whether grader-guided reasoning generalizes beyond math and science.

Creative writing supplies an everyday example. One host reports using o1 for AI news and podcast intro-song lyrics, crediting stronger rhyme, counting and character-level instruction following. A recalled o1 Pro demonstration involved writing paragraphs without the letter `A`. These are personal impressions and a recalled demonstration, rather than controlled comparisons or measured success rates.

Protecting reasoning traces may not protect the lead

The data dispute turns inward. One host asks how OpenAI’s use of publishers’ material fits with withholding reasoning traces and restricting others from training on its outputs. The reply focuses on enforcement through API access. It also argues that hidden traces are difficult to extract and that an apparent leak need not be authentic. Those practical arguments leave the legal question unresolved.

A more consequential question is whether proprietary reasoning training creates a durable competitive advantage. A host offers several reasons for doubt: - Task-specific fine-tuning: Cosine’s work on GPT-4o is cited as outperforming o1, without a benchmark or evaluation setup. - Competing coding models: Claude Sonnet is described as outperforming at least o1-preview on coding tasks, again without a specified comparison. - Fast competitive responses: DeepSeek’s work on R1 is cited as evidence that competitors can move quickly. The reported two months of notice does not establish its actual development duration.

The host anticipates confusion between substantive inference-time methods and elaborate chain-of-thought prompting. Strong alternatives may also be undervalued against OpenAI’s reputation. The distinction is between a capability lead today and the durability of the methods behind it; earlier confidence in o1’s lead does not settle the latter.

The GPU smiling curve: machines and customers

The final stretch examines a polarized compute market. The hosts see less appetite for startup pitches that amount to raising $60 million and spending $50 million on GPUs, while the largest labs keep building. One host proposes a practical model-size wall shaped by training costs, data requirements and inference prices. It is an economic and engineering hypothesis, not an established parameter ceiling.

Research experiments and production growth both drive compute demand. New inference approaches require general-purpose capacity for experiments while existing models continue gaining deployments. Distribution matters too: the hosts see Amazon as an underappreciated competitor after re:Invent. They point to prepaid AWS commitments and Marketplace purchasing as reasons customers may favor products available through that channel.

For developers with limited hardware, the hosts offer different priorities: - Use hosted models to build products: One questions the effort spent squeezing models onto tiny clusters when cloud inference is cheap, while leaving room for home hardware as architectures settle. - Use small clusters to develop methods: The other sees experimentation as a stepping stone. Heterogeneous and distributed compute could make disparate resources useful to more people, without necessarily producing frontier models.

On-device products offer another form of access. Apple Intelligence notification summaries are an everyday example, with mixed results in the hosts’ experience. Gemini Nano in Chrome is described at the time as an alpha available behind feature flags. Both bring AI into familiar software without requiring users to assemble GPU infrastructure.

Businesses can also grow without owning large GPU fleets. The examples are Suno, using rented training infrastructure, and Bolt, building on cloud models. A host recalls Suno reaching roughly $20 million in annual recurring revenue during 2024 and Bolt announcing $20 million, up from the $8 million previously used in a podcast title. The figures have no common measurement date or interval that would support a growth-rate comparison.

This leads to the proposed GPU smiling curve: businesses can succeed by leading close to the machines or by leading close to customers, while companies between those positions have fared less well in the host’s assessment. Part 1 ends with that hypothesis and a recollection of an earlier suggestion that Google should buy Character for $1 billion—a proposed purchase price, not an actual transaction.