From Coding Agents to Inference Economics — Part 2 of 3

Putting agents to work exposes new failures

An AI agent can take over customer-support work and still make mistakes a human would never make. Early in this part of the recording, a participant cites Klarna’s reported replacement of roughly 700 support agents as a sign that specialized AI agents could spread across large companies. The forecast includes embarrassing and harmful failures: wider deployment requires careful rollout.

The opening predictions also include speculation about a Google leadership change, with Demis mentioned as a possible successor to Sundar. The participant considers three months too short a horizon and suggests six to nine months instead. The recording then moves to Thursday Nights in AI, the recurring San Francisco meetup run by Imbue and Outset Capital, for a conversation with swyx.

Learning in public becomes an editorial mission

Originally from Singapore, swyx came to the United States for college. At the time of the conversation, he had spent roughly 15 years there, divided between finance and technology. Tech suited his preference for learning in public: sharing mistakes at meetups, publishing code, and contributing pull requests to other projects. He contrasts that openness with finance’s trade-secret culture. His developer relations career took him through Netlify, AWS, Temporal, and Airbyte.

Stable Diffusion prompted Latent Space’s creation: running a capable large model on a desktop felt qualitatively different to swyx. About four months later, around the beginning of 2023, he and Alessio started the podcast. Interviews let them create the original material he was running short of as a writer. He credits their complementary strengths, guests’ willingness to participate, and Alessio’s cold outreach, including the early booking of George Hotz.

The editorial filter was what software engineers can use at work. Swyx acknowledges existential safety concerns but finds debates about general intelligence timelines and catastrophe probabilities less useful for that audience’s daily decisions. The same filter explains limited coverage of robotics and high-frequency trading, despite their substantial machine-learning activity. He sees the podcast as one way to deliver ideas, with the ideas determining the format.

Coding autonomy depends on human attention

Asked what an AI engineer is, swyx distinguishes three categories: - AI-enhanced software engineers use tools such as Copilot and Cursor in their own work. - AI product engineers expose existing model capabilities to users, often through APIs, without necessarily training the models. - Autonomous coding agents write code for people. He describes this non-human category as the hardest and furthest from realization.

For autonomous coding, he favors measuring how much human attention the system needs over predicting an arrival date. His analogy is self-driving: compelling demonstrations can precede dependable operation by years. Coding likewise spans constant supervision, accepting suggestions, and delegating work while away from the keyboard.

The desired workflow starts when a nontechnical colleague files an issue. An agent writes the code and passes the tests, then a human merges the result. Sweep and CodeGen are named as companies pursuing this direction, with Magic mentioned tentatively. The imagined tenfold or hundredfold expansion in effective developer capacity remains an aspiration; swyx explicitly says the workflow has not reached that goal.

A practical boundary separates two kinds of development work: - Inner loop: local, hands-on work between Git commits, where assistance must be fast and the developer remains present. - Outer loop: work after pushing to GitHub or GitLab, including issue-driven tasks performed while the developer is away. The outer loop requires substantially more autonomy. In this 2024 discussion, swyx questions whether the available models are capable enough to handle it reliably.

The weak link is turning intent into tests

An interviewer describes the awkward experience of receiving an automated pull request: someone now has to inspect 300 lines of generated code to find a bug. Asking the agent to repair it might produce 400 lines instead. The numbers are illustrative, but the failure mode matters: generating more code can increase the review burden.

A second agent could review the first agent’s work. Swyx worries, however, that agents could imitate the human habit of scrutinizing tiny patches while waving through large ones. He favors better planning at the outset over relying on a generator-and-reviewer contest, connecting that preference to Imbue’s thesis.

The alternative links the user story, the tests, and the implementation. The story describes intended behavior, generated tests check that behavior, and code implements it. Changes to any artifact should propagate to the others. Generating good tests from the story is the critical dependency: implementation can keep changing until it passes, but passing only helps if the tests capture what the user wanted.

Automatic test generation therefore offers a promising product entry point. It addresses work developers often avoid while improving the weakest part of the broader automation loop.

Those tests also need somewhere to run. Agents need sandboxes that return feedback: isolated environments where generated code or plans can execute, encounter failures, and be revised before returning a checked result. Fly.io and Cloudflare Workers are suggested as possible environments, depending on resource needs. Agent development brings familiar cloud-infrastructure problems back into focus, at a potentially much larger scale.

A profession built around practical fluency

AI engineering is presented as an emerging specialization, analogous to frontend, DevOps, and data engineering. The economic argument is that scarce research talent and GPUs concentrate model development in relatively few labs, while many more engineers build products using those models. A participant consequently predicts that AI engineers will eventually outnumber machine-learning engineers—a forecast about the profession, rather than an established workforce count.

The category is deliberately open enough for practitioners to shape it. It includes full-time specialists and engineers who spend only part of their time on AI. A shared title helps employers and practitioners find one another, without requiring a single prescribed career path.

For programmers overwhelmed by AI, the proposed starting point is a seven-day email course, with one hour and one project per day. Recalled topics include model API calls, image generation, code generation, speech recognition, open-source models and fine-tuning, and agents. The goal is moving from unknown unknowns to known unknowns: enough practical fluency to assess a request to build an AI feature. Seven projects provide a starting point, not mastery.

Conferences give the community a gathering place and speakers a deadline to prepare substantial technical material. The AI Engineer World’s Fair was planned to be four times larger than the previous Summit and to reflect AI engineering in 2024. Its organizing model combined NeurIPS’ celebration of researchers with KubeCon’s industry exhibition floor, putting engineers, their tools, and the difficulty of moving research into production at the center.

Long context simplifies prototypes; retrieval still has a role

Gemini 1.5’s million-token context window raises a practical question: if a model can accept an entire knowledge base, book, or film, does an application still need to retrieve selected material? Ten-million-token context was also described as under research. The immediate attraction is less work deciding what to include in a request.

The response preserves three reasons to use retrieval-augmented generation (RAG), where a retrieval stage selects material for the model: - Repeated input costs money. Sending a large body of material on every request incurs per-token charges. - Large requests take time. A larger input allowance does not eliminate processing latency. - Intermediate stages help locate errors. A retrieval pipeline offers places to investigate an incorrect answer; one large-context request leaves fewer visible boundaries for debugging. These tradeoffs depend on the workload; the discussion provides no measured price or latency comparison.

For a prototype making only a few calls, long context can postpone retrieval infrastructure. Avoiding context-limit errors and setup work may matter more than optimizing each request. Once the idea works, retrieval can be built or bought to address the workload’s other constraints. Context length expands the options available to an engineer.

Native video makes new applications worth attempting

Multimodality—the ability to work across inputs such as text, audio, and video—broadens those options further. The respondent acknowledges underestimating image generation’s business potential, citing Midjourney, then emphasizes native understanding of audio, video, and code. In this view, translating everything into English creates an information bottleneck: working across modalities can preserve information that a text-only interface loses.

Simon Willison’s reported bookshelf experiment supplies a concrete example. A video scan of the shelves went into Gemini 1.5; a JSON inventory of books, authors, and visible details came back. The output included hallucinations, so the example shows an application becoming practical to attempt, while leaving inventory accuracy unresolved.

Video adds time to image understanding, a theme the conversation connects to Sora. The respondent distinguishes the value of this understanding from the necessity of image generation for general intelligence, agreeing with a criticism attributed to Nat Friedman that image creation is not evidently on that critical path.

The podcast edit omits the meetup’s subsequent Q&A; its closing narration directs listeners to the full Thursday Nights in AI video for that material.

Fast inference for whom, at what utilization?

The next conversation turns to GroqChat’s inference speed and serving economics. How did it generate tokens so quickly, and could its advertised price of $0.27 per million tokens sustain the service? A participant describes generation as roughly five times faster than competing services at the time, without specifying a matched workload or benchmark.

A participant presenting a cost analysis estimates that Groq’s API was seven to ten times away from breaking even, though the underlying calculation is not included. The central engineering distinction is per-user latency versus aggregate throughput: - Latency-oriented serving prioritizes how quickly an individual user receives output. - Throughput-oriented serving prioritizes total work completed across concurrent users. The analysis describes competing GPU services as serving batches of 64 or 128 users, while Groq used 576 chips to serve far fewer users very quickly. Comparing response speed alone obscures the difference in utilization.

The speaker argues that GPUs could also be configured for low latency, but such services needed customers willing to pay a premium. Groq’s economic challenge, in this account, was increasing concurrency. Capacity for each user’s cached model state, referred to as the KV cache, limited how many users could be served together. Faster output for one user did not resolve the capacity constraint.

The speaker describes ten racks serving Groq’s API and reports an expectation that expanding to twenty racks could improve performance, while doubting its scalability for larger models. A comparison with a model called Mistral running on two GPUs lacks the exact model, precision, batch size, and performance target. Rack counts alone do not establish a matched comparison.

Compiler work determines usable hardware performance

Groq’s architecture is described as requiring substantial compiler work to support and optimize individual models. The speaker attributes part of that burden to the absence of buffers and other familiar hardware features, which makes coordination ahead of execution more demanding.

The claimed benefit is predictable instruction timing: compilation determines when instructions finish and how long execution takes. This is a timing property. Numerical reproducibility also comes up in the exchange, but the explanation does not establish a connection between the two.

Chip design consequently involves more than power, performance, and area: programming difficulty is another design tradeoff. Google’s compiler and software investment is offered as what makes its AI accelerators usable. For Groq, the speaker identifies two compiler responsibilities: - Extract performance: bring actual execution closer to the hardware’s theoretical capability. - Support new models: make each additional model straightforward to bring up and optimize.

An audience question introduces VLIW, or very long instruction word. The speaker presents this architecture as a route toward better power, performance, and area efficiency at the cost of harder programming, and describes Google’s matrix-multiply unit, MXU, as using it. The deeper question—Groq’s average VLIW bundle occupancy—goes unanswered: the respondent explicitly says they do not know.

Memory bets must survive changing models

The speaker characterizes an earlier generation of accelerator startups as betting on more on-chip memory than Nvidia, with different implementations: - Groq: take the approach to the extreme, using only on-chip memory. - Cerebras: pursue more on-chip capacity through a wafer-sized chip. - Graphcore and SambaNova: make related bets on greater on-chip memory.

The critique is that model growth outpaced feasible on-chip capacity, undermining that bet. The speaker proposes two questions for evaluating newer hardware companies: what do they do differently from Nvidia, and could Nvidia incorporate a version of that advantage into its next chip?

Buyers also face model-architecture risk over years of ownership. The speaker expects GPUs to retain some usefulness after four or five years but questions whether specialized accelerators will support future models. A hypothetical chip optimized for the model architecture called LSTM in 2016 illustrates the risk: BERT’s arrival represents the kind of workload change that could strand a specialized purchase. Graphcore’s strength in graph neural networks is offered as another example of technical capability whose commercial value the speaker doubts, without benchmark or demand figures.

The conversation briefly returns to long context. A questioner presents Gemini’s context length as a challenge to claims that alternative model architectures are necessary; the respondent declines to offer a coherent thesis. A professed dislike of RAG leads into jokes about a White House memory-safety announcement and a participant’s edited version telling people to stop comparing RAG with long context and fine-tuning. That comparison language is parody, not government guidance. The respondent explains the retweet as appreciation of the joke, rather than a technical position on retrieval.

The technical closing returns to memory movement. Keeping data in on-chip memory promises energy savings by avoiding transfers off-chip. But the savings depend on sufficient local capacity: if the working data does not fit, the system still has to move it repeatedly. The speaker argues that realizing the efficiency promised by an on-chip-memory-heavy design is therefore difficult.

The final question asks whether obtaining energy or obtaining hardware will become harder. The respondent expects power to be difficult to secure but leaves the comparison unresolved. Part 2 ends as an argument about historical growth in power production and consumption begins.