The Agent Lab Playbook: Build at the Frontier, Specialize Around the Work

An architecture stable enough to build on

An AI infrastructure company can build around one development pattern, then have to rebuild as the next arrives. Early in this Unsupervised Learning × Latent Space conversation, the interviewer raises that problem through his reading of Harrison Chase’s recent remarks: perhaps the infrastructure is finally stabilizing. The practical question is whether builders now have an agent architecture worth committing to.

Returning from AIE Europe in London, swyx identifies OpenClaw, harness engineering, and context engineering as leading priorities. His conference tracks reflect that judgment, alongside recurring work on evaluations, observability, compute infrastructure, multimodality, and generative media.

The emerging architecture is straightforward: a large language model uses tools in a loop, with a filesystem for retrieval and Skills—integrations packaged as a Markdown file with attached scripts. That minimal format gives swyx some confidence in convergence around the agent harness, the machinery supporting that loop. Real-time behavior, subagents, and memory may still require substantial adaptation.

His advice is to adopt what works, build a viable business around it, and change again when necessary. Predicting a permanent architecture matters less than turning the current one into something customers value.

From outsourced AI team to agent lab

That flexibility is easier to sustain when the customer relationship survives a rebuild. The interviewer argues that application customers give vendors room to adapt as models improve, while developers buying infrastructure are pickier. swyx adds that infrastructure churn can be comprehensive: even database workloads can migrate.

A domain-focused AI company such as Abridge can function as an outsourced AI team, applying current methods for customers who would otherwise need that expertise internally. swyx considers this responsibility resilient to changes in the underlying techniques. He also sees opportunities in services spanning many domains, particularly reinventions of cloud infrastructure. Sandboxes are his example: another form of compute, serving what he describes as enormous workloads.

The agent lab playbook extends that customer relationship into model development: 1. Start with frontier models and specialize the product for a domain. 2. Accumulate enough workload and high-quality user data to identify recurring needs. 3. Train specialized models to improve the economics of serving those needs. For high-volume workloads with well-defined tasks and little variation, the proposed next step is distillation: using larger models to help produce a smaller model the company can run itself. Specialization follows demonstrated demand.

The interviewer questions how much proprietary models deliver user value versus marketing value. swyx sees both. He points to users selecting Composer 2 among their preferred model-switcher options without subsidies, though he gives no usage counts or observation period. He makes a more targeted case for domain-specific search models, where he sees a clear purpose and increasingly accessible training infrastructure.

Cost, quality, and faster inference

The initial objection to in-house reinforcement learning concerns the durability of a quality advantage. General models might erase specialized gains with their next release, as earlier domain-specific pretraining efforts found themselves overtaken. Smaller models have a clearer cost advantage at scale. The response is that quality and cost belong in the same decision: maintain quality while reducing expense, or accept a modest quality loss for a much larger cost reduction.

Alternative inference hardware adds generation speed to that tradeoff. A participant cites Cognition and OpenAI as adopters of Cerebras, then contrasts thousands of tokens per second with fewer than a hundred. No model, workload, or latency definition accompanies those figures, so they cannot establish a comparable performance gain.

The product hypothesis is that a 2× speedup may have limited impact, while a 10× improvement can enable different usage patterns. The participant expects a multiyear investment cycle in alternative hardware but leaves open which applications much faster generation will make possible.

When the developer customer is an agent

Agents are not necessarily rational infrastructure buyers. A participant argues that they are vulnerable to prompt injection and tend to reinforce products already prominent in their training data. The same discussion recalls Malte Ubl reporting that bots accounted for roughly 60% of traffic to a Vercel configuration or administrative surface. The precise surface is unclear; the figure does not describe all Vercel-hosted traffic.

Serving these automated users largely extends familiar developer experience practices: - Expose functionality through APIs and command-line workflows. - Provide clear documentation and consistent, mostly stateless APIs. - Make capabilities discoverable through search and progressive disclosure. A polished graphical interface alone cannot serve agents working through programmatic interfaces.

Being usable does not guarantee being recommended. An unspecified survey reportedly found that Claude recommended Resend in roughly 70% of responses to a generic request for an email provider. Without a model version, sample size, or methodology, this remains an anecdote about a newer company entering default recommendations. The broader concern is concentrated distribution: agents may offer three choices rather than twenty, strengthening winner-take-most dynamics.

One suggested tactic is publishing integration articles showing how a product works with an established platform such as Vercel. The proposed mechanism is that these co-occurrences strengthen associations in material models encounter. Skepticism about gaming chatbot recommendations coexists with an acknowledgment that temporary advantages can compound.

Over a three-to-four-year horizon, one participant expects memory and personalization to influence selection more than raw mention frequency. Recommendations could increasingly reflect a particular user’s history, though the design of that system remains unresolved.

Coding’s expansion rewards exploration

Coding’s prominence invites a tempting startup thesis: pursue less developed categories because they have more room to catch up. A speaker challenges the assumption. Coding’s momentum could continue, making a bet on convergence costly.

The revenue figures offered convey the perceived scale of the opportunity, but carry different qualifications. One speaker puts Claude Code at roughly $2.5 billion in annual recurring revenue while questioning how that figure is recognized. Roughly $2 billion for OpenAI’s coding business is the speaker’s estimate; a similar figure for Cursor is described as a rumor.

The speakers characterize the period as capability exploration: organizations reward experimentation before establishing disciplined measures of efficiency or output quality. Token-consumption leaderboards capture the awkwardness. They show that employees use AI, but say little about the value produced. One participant considers this sloppy yet broadly useful because the tools remain underused.

A previous podcast guest reportedly consumes a billion tokens daily. The accompanying estimate of $10,000 per day uses hypothetical market API pricing, not a reported bill, and includes no model mix or price breakdown. The argument is that repeated experiments increase the chance of discovering capabilities others miss, even if much of the output is poor. Compute budgets can therefore constrain application discovery as well as frontier research.

At the time of the conversation, a speaker characterizes Anthropic as a premium provider with tighter limits and Codex and Gemini as heavily subsidized. The recommendation is to use those subsidies for exploration; a $200 monthly subscription is presented as enough to support useful experimentation.

The model frontier and the enterprise last mile

One baseline forecast is a coding market with two major players and a smaller long tail serving unmet needs. Changing that structure would require different economics, stronger brands, or new value propositions. Microsoft mounting a larger initiative through GitHub is offered as a hypothetical disruption. OpenAI’s super-app push raises another question: whether broader consumer ambitions or product consolidation will divert attention from coding.

Expansion into other markets could leave room for focused coding specialists such as Cursor and Cognition. Finance and healthcare are identified as likely next targets for model labs, with finance judged to have a clearer path to revenue and healthcare to be more difficult.

The distinction between model labs and agent labs becomes operational here. Agent labs promise the last mile of enterprise implementation; model labs are characterized as more minimalist, relying heavily on model capabilities. Neither approach wins every use case. Enterprises may want a dedicated implementation partner even while frontier capabilities strengthen their relationships with model providers.

The uncertainty reaches investing. One participant argues that even a large AI company’s prospects can change radically within a quarter, making late-stage investing less like pricing a bounded outcome. Anthropic’s coding success illustrates the upside: a timely market, suitable model, and strong execution are credited with transforming its prospects.

Product loyalty and the next agent experience

Product habits may counter some of that volatility. One speaker sees ChatGPT retaining a first-mover advantage despite finding no obvious superiority over Claude or Gemini; another argues that retention has not translated into continued growth. Neither offers usage measurements. The broader consumer challenge is framed as bringing in more users and increasing frequency beyond the knowledge workers who already find these tools valuable.

A similar disagreement surrounds Claude Code. One speaker is surprised by its apparent stickiness given reports that Codex is comparable or better. Another cautions that Claude Code is only about a year old and some users are already switching. Enterprise API buyers and enterprise product users may behave differently: habits around a product can persist after competing capabilities approach parity.

The product comparisons remain subjective. One participant still sees Codex as catching up while preferring Spark, its Skills integration, and its perceived speed. They describe these as minor impressions rather than objective assessments. The larger question is why Gemini and xAI have not established a stronger presence in the coding-agent discussion.

A compelling new experience could give another competitor an opening. A consumer agent combining computer use with other capabilities is floated as one possibility. OpenClaw serves as a preview of the broader thesis: after coding agents’ rise in 2025, 2026 could see them expand into other work. Software generation provides the route—agents that create software can acquire capabilities beyond editing code. Personal experiments are acknowledged to be easier than enterprise adoption.

Replacing SaaS means bringing the team along

The ecosystem has incentives to support several strong model providers, but demand for choice does not create successful labs automatically. The participants agree that new labs have struggled to break through over the preceding year. Their disagreement about NVIDIA’s support resolves into a distinction about control: helping establish another cloud provider is more directly tractable than creating a successful model lab.

For very small startups, one participant sees competent execution as career evidence that could lead to a lab hiring or acquiring the team even if the business fails. This is a founder-career argument, not an investor-return argument. Midsize companies look more exposed; Langfuse’s absorption into ClickHouse is cited as an example of infrastructure consolidation.

The debate over replacing software as a service becomes concrete with an event-and-sponsor-management system costing one participant $200,000 annually. They estimate that a custom version might cost $2,000 to build. The replacement remains prospective: those figures establish neither equivalent functionality nor ongoing operating costs. The stated obstacle is getting the team on board.

Both sides have a credible complaint. An enthusiastic builder may see colleagues resisting workflows that agents could simplify. Colleagues may see a weekend prototype covering roughly 80% of the need and leaving them to finish the job. Mature customer relationship management systems are easy to underestimate, especially when the builder is not the target user. Yet easier customization makes replacing a narrow slice of a large product more plausible. Who inherits the unfinished work becomes central to the decision.

That tension suggests an opening for an AI-native system of record optimized for rapid iteration before scaling into databases such as Postgres or MongoDB. Convex is floated as a possibility, though the participant remains unsure whether a convincing equivalent of Firebase for AI applications has emerged.

Restricted access and the next compute constraint

A dinner anecdote about differing awareness of biological misuse risks leads into Mythos and restricted releases. One participant considers Mythos a strong model but criticizes its marketing and questions whether access across dozens of large organizations meaningfully excludes malicious users. Another sees restricted distribution as a possible template for future releases and hopes it is safer than broad access.

The access policies receive competing explanations. One participant characterizes Anthropic as restricting access and potentially bundling products with models, while OpenAI favors broader availability. Another points to differences in available compute. Looking ahead, one speaker predicts that larger clusters arriving over three to five years will ease access constraints on models above 10 trillion parameters, with broad access in roughly two years. The reply is that rationing will move to the next generation.

An explicitly unconfirmed theory imagines Google using a larger internal Gemini Ultra to teach cheaper deployed models. The argument is economic: an expensive teacher can create value through distillation even when directly serving it is unattractive.

Parameter scaling remains an open question. One speaker has revised an earlier expectation that model size would cap around two trillion parameters and argues that much larger gains may require architectural improvements. They question the desirability of vast data-center expansion; another responds that sufficient returns from intelligence could justify it.

Context length and memory are proposed as persistent constraints. One speaker describes context growth from roughly 4,000 to one million tokens over about three years and expects memory to become a major bottleneck. This is a forecast about what limits future systems, not an established scaling ceiling.

Open models and software without human review

Asked what has changed their mind, one speaker points to improving prospects for open models. Public benchmarks are vulnerable to gaming, while OpenRouter usage reflects discounts as well as preference. Neither provides a simple answer about the capability gap or adoption. The speaker instead proposes separating leading agent labs from ordinary AI startups; the suggested top-20% versus bottom-80% split is an analytical lens, not a measured segmentation.

The argument for fine-tuning services follows workload maturity. Once a company knows a use case works, it can focus on delivering it more cheaply and quickly. The speaker sees leading agent labs moving toward open models and names Fireworks and Together as beneficiaries. This returns to the agent lab sequence: discover the workload, then improve its economics.

The next proposed transition is software dark factories, progressing from zero human-written code to zero human code review. A speaker recalls encountering the former at Cognition about five months earlier and says OpenAI is exploring the latter. They argue that removing human review is the scalable path, requiring substantial changes to the software development lifecycle and stronger testing and automated verification.

The expected payoff is cheap, disposable software that makes more experimentation affordable. Greater output could eventually improve quality by enabling more attempts. This remains a forecast, coupled with a challenge to engineers: shape generated software toward useful outcomes rather than dismissing the category as slop.

A temporary training advantage can still pay

Late in the conversation, the earlier skepticism about training for quality shifts. One participant had accepted cost and latency benefits but expected general models to erase capability gains within three to six months. Application companies, however, routinely build features they later replace. Post-training—further training to improve an existing model—can be evaluated on the same terms.

The decision rule is to compare post-training with the best alternative for improving customer outcomes during the change’s useful lifetime. More data through connectors or better backend engineering may deliver more value. If training is the strongest option for that interval, a general model catching up three months later does not make the investment pointless.

Discarding a result also differs from discarding the raw data. Keeping the data permits another run. The investment is compared with assigning four engineers for three months, without establishing an actual rerun cost.

Another participant points toward reinforcement learning over long trajectories, sequences spanning hundreds of turns. They argue that this permits more specific customization than traditional supervised fine-tuning or the shallower reinforcement learning used a year earlier, opening a path toward deeper domain specialization. Long trajectories and synthetic rubrics are suggested as research directions; the discussion does not explain the rubric method or provide a training recipe.

Beyond answering questions

The technical closing returns to memory, personalization, and world models: the broader effort, as framed here, to give AI an understanding of objects, matter, and physics. Walkable, static 3D demonstrations can leave a business-software engineer wondering what they contribute to an application. The speaker argues that these demonstrations do not define the full problem, which concerns improving intelligence beyond next-token prediction.

Fei-Fei Li’s essay on spatial intelligence is recommended for articulating that problem, without assuming she already has its solution. A Good Will Hunting analogy captures the distinction: someone can learn extensively from books without having lived the experiences they describe. Applied to language models, it expresses the proposed gap between verbal knowledge and experience of the world.

The speaker traces the analogy to one of Latent Space’s newly added daily write-ups. In the outro, Jacob Effron describes Unsupervised Learning as a nights-and-weekends project alongside his investing work at Redpoint at the time of recording. He closes by asking listeners to subscribe and share, crediting that support with helping the podcast attract guests.