Inside Poolside’s Model Factory

An early bet on neural networks for code

In 2015, the guest in this Latent Space conversation read Andrej Karpathy’s “The Unreasonable Effectiveness of Recurrent Neural Nets” and pivoted a startup overnight toward getting neural networks to write code. Over the next four or five years, the fully open-source company explored recurrent networks, convolutional networks over code structure, attention and eventually Transformers.

The company failed after roughly 40 people had spent years on the problem and used $12 million of investors’ money. The guest’s retrospective diagnosis is a promising direction without enough scaling, although scaling was far from an obvious prescription then. After about two years away from language models to focus on family, ChatGPT’s arrival felt like vindication of the original idea.

Poolside began with two convictions: model capabilities would keep compounding, and reinforcement learning would become their biggest driver of improvement. Reinforcement learning, or RL, features here as training that shifts a model toward desired reasoning and behavior. Openness was not initially central; the founders revisited it when they became concerned about a future in which only a handful of companies supplied advanced intelligence.

The guest would prefer 100 foundation-model companies to five, even if Poolside were among the five. Models can develop different behaviors and biases despite drawing from broadly shared data, making provider choice consequential. Poolside’s proposed contribution has two parts: - Open weights enable modification: others can use and change the model artifact. - Open research helps others reproduce development: weights alone do not explain how to recreate the underlying work. Sharing datasets and finding a sustainable business model remain unresolved.

Three weeks inside an optimizer bug

Poolside describes itself as an American company from inception, with a deliberately global recruiting strategy. It initially avoided hiring Bay Area researchers, anticipating competition for that talent. The remote organization opened offices including Paris and London and later added people in Silicon Valley. Starting outside an established lab also meant learning without much of the informal training knowledge circulating inside one.

The cost was tangible. With about five people in the company, the team wrote its first training codebase from scratch, then spent three weeks debugging an optimizer problem that made training unstable. The guest recalls an issue involving Adam’s denominator epsilon: after fixing it, the team no longer needed the unusually high values it had adopted from published recipes. The exact bug and setting were not recalled; the lasting lesson was to investigate assumptions embedded in a training recipe.

That experience helped establish a culture of learning through experiments. Model building repeatedly defeats apparently sound intuition, sometimes without a clear explanation. The guest characterizes it as “90% engineering” because so much daily work consists of inspecting data and writing code, rather than as a measured division of labor.

Optimize the path from idea to trusted result

The model factory is Poolside’s system for making model development repeatable. It spans web ingestion, data filtering and transformation, analysis, distributed training, and the subsequent training that refines a model’s behavior. Distributed systems engineers participated in research from the beginning. Reliability was integral, with each hardware generation introducing new operational problems.

Its target metric is time from research idea to trusted result to the next training run. The guest estimates that fewer than 70 researchers, supported by another 35 engineers, were running roughly 10,000–20,000 experiments per month, while acknowledging that the latest count had not been checked. At that volume, infrastructure must make results trustworthy as well as make jobs executable.

The reported release intervals show what the factory is intended to buy: five weeks from the start of pre-training to launch for the earlier XS release and eight weeks for the model under discussion. Those clocks exclude the preceding research and infrastructure development. Reaching this point took three years, the guest says; intervals between successive models were now measured in weeks.

Good interfaces also made the factory accessible to agents. Researchers reportedly used multiple agents to write code, launch jobs, inspect results and make changes, especially in pre-training, post-training and synthetic-data pipelines. Humans still supplied ideas and helped debug. The guest sees this as an early form of recursive self-improvement—models helping develop subsequent models—while the workflow remains agent-assisted research under human direction.

Experimental results from the previous day can enter a new run without a lengthy configuration freeze. Laguna S reportedly had no on-call events, but startup still needed staffed intervention: configuration mistakes commonly surfaced during the first roughly six hours of a run. Reliability allowed rapid incorporation of changes without eliminating startup debugging.

Stream the data; preserve its lineage

A bottleneck appeared before training even began. Preparing an entire dataset meant tokenizing, packing, transferring and distributing it across the cluster. Every correction or mixture change could repeat that work. Larger clusters could require torrent-like distribution algorithms. Poolside’s alternative was to feed training at its consumption rate, allowing it to start before the complete dataset was ready.

The design starts with tokens consumed per second. The guest argues that this is a manageable rate from a data-transfer perspective, so separate Spark-based processing infrastructure can supply the training cluster just in time. Poolside’s infrastructure software can run in an AWS account or on other hardware.

Blender makes data mixtures configurable. A run specifies source proportions, how often data repeats and how it is shuffled. Training begins while the remaining data is still materializing, reducing the operational work required to try another mixture.

Streaming is paired with immutable data and versioned experiment code. Poolside says it can trace an individual token to its input cursor position and the code version used in a run. The guest reports being able to reproduce two-year-old experiments, though the claim does not specify numerical equivalence. This lineage gives researchers a known data and code history against which to investigate changed results.

The machinery initially supported relatively ad hoc experiments. About a year to a year and a half into the company, its purpose became more explicitly scientific: each experiment should isolate a change and help explain its effect. Publishing those engineering lessons is part of the openness strategy. The guest credits other organizations’ research, including Chinese labs, with accelerating Poolside and sees a corresponding obligation to contribute back.

Persistence expands a smaller model’s useful capability

The factory also lowers the barrier to becoming a researcher. In the guest’s account, an engineer who initially built agents became a productive reinforcement-learning researcher in about six months. The work remained concrete: inspect data, write code, run experiments, read plots and revise an intuition.

Laguna S is the first release described as meaningfully contributing to Poolside’s own work, while still falling short of the overall frontier. In a message relayed by the guest, Pang Ming attributed much of its improvement to learned behavior: - Verify more: check the work instead of taking assumptions for granted. - Avoid premature success declarations: continue until the task is actually resolved. - Persist through difficulty: keep working when an initial approach fails. The guest attributes these gains to post-training.

One request was for a Mac Wi-Fi scanner without external libraries. Without internet access, the model reportedly kept working through the Core WLAN API to understand how to proceed; the account does not establish a finished, verified scanner. Many hours spent reading model trajectories reinforced the guest’s optimism about backtracking and effective interaction with an environment.

Persistence consumes reasoning time. Laguna S reasons extensively, and efficiency and support for different reasoning modes still need work. For everyday programming, the guest argues, debugging, consulting documentation and trying another approach can matter more than connecting ideas across distant scientific fields.

That raises a task-dependent economic question: when does a larger model’s additional capability stop justifying its cost? The guest already chooses smaller models for some coding tasks, but offers no established optimum. The suggestion that ordinary knowledge work might reach that point around one to ten trillion parameters is explicitly hopeful and speculative. Poolside’s strategy still calls for scaling to compete with the largest models.

Teach reasoning earlier through a continuous curriculum

Gains from post-training leave substantial questions about the earlier training process. Poolside has spent years investigating how web data could teach reasoning beyond next-token prediction. The motivation is the web’s enormous variation in quality, from poor material to scientific papers, and the suspicion that current training extracts too little from its best content.

The guest considers distillation and additional training environments effective, but questions whether continually adding environments is sufficient for general intelligence. A host supplies the furniture example: understanding instructions for assembling a table should not necessarily require a dedicated furniture-building training environment. The question is how much more models can generalize from existing information.

The discussion contrasts two ways to organize learning: - Mid-training as a coarse curriculum: continue pre-training on better or domain-specific data, then apply reinforcement learning. - A continuous curriculum: optimize the ordering of data and learning objectives throughout the run. The guest argues that organizational boundaries help preserve discrete stages: a dedicated mid-training team creates a distinct mid-training effort. Poolside wants its factory to support broader experimentation, but its organization and compute setup do not yet support the proposed continuous curriculum.

Poolside has not disclosed its alternative pre-training objectives or results. Pursuing them requires balancing established recipes with speculative research, adjusting the allocation as the company’s competitive position changes.

Laguna S: architecture, precision and measured scope

The architecture discussion returns to compute efficiency. Lower-precision training opens a tradeoff between larger models using less precise numerical representations and smaller models using more precise ones. Where that tradeoff is beneficial remains an empirical question.

Laguna S has 118 billion total parameters and 8 billion active parameters. It scales the XS architecture and uses a three-to-one ratio of sliding-window attention to global attention. Training reportedly used the FP8 precision format except for all-to-all communication between devices. The subsequent run extended FP8 to that communication too; the earlier exclusion reflected readiness at the cutoff date. Poolside reported operating a cluster of 10,000 H200 GPUs.

The size was chosen to deliver progress quickly and economically. The guest also reports running it on a DGX Spark at roughly 30–40 tokens per second, without specifying precision, context length or other deployment conditions. That throughput is a reported observation, not a reproducible benchmark configuration.

A host suggests overall state-of-the-art performance on a Tau benchmark. The response narrows the claim to probable leadership within its size class and some larger classes, without confirming the Tau-3 banking leaderboard position. No scores or evaluation configurations accompany the exchange.

Internal use complements formal evaluation: an early post-trained checkpoint can quickly give its developers an intuitive sense of quality. But attachment to one’s own model can hide rough edges. Open-weight releases bring external feedback about both strengths and failures while the team continues work on the next model.

Control the training harness, then polish compatibility

An agent harness supplies the instructions, tools and data access through which a model works. Poolside separates two responsibilities: - Capability development: concentrate reinforcement learning and synthetic data in the harness the team understands best. - Compatibility polishing: adapt reasoning behavior and tool-call formats so capabilities transfer to other harnesses. For smaller models, the guest says a modest amount of training across harnesses helps, though no training volume or measured transfer gain is given.

The rationale is experimental control. Adding harnesses introduces different prompts, sub-agents and tool behavior, making it harder to isolate what improved the model. Open-source harnesses can be inspected, as a host points out, but understanding them still takes time. Poolside’s own setup has roughly half a dozen tools covering shell execution and control, writing and web fetching.

Application developers can go further. The guest expects a specialized harness to outperform Poolside’s lightweight setup through better instructions, data sources and tools. Equal ability to finish a task does not imply equal efficiency: a tailored workflow can avoid rediscovering how to do the work. Poolside reports successful use in OpenCode and KiloCode, without quantitative comparisons.

Poolside’s goal extends beyond coding. Long software tasks require sustained reasoning, external tools and data, making them a proposed path toward broader intelligence. The guest describes effective personal use of the model for deep research inside a coding harness. Promising initial LegalBench runs were withheld from publication because the team had not validated them sufficiently.

A useful evaluation and a trusted evaluator solve different problems. The latter must execute consistently across competitors and check against cheating. Poolside supports third-party evaluation while noting that infrastructure failures and tool configuration still require coordination. A host also presses the product implication: a model company should invest in excellence within external harnesses. The guest agrees, with general capability across harnesses as the eventual goal.

Generated code as the interface

The guest favors generated code over a large catalog of predefined tool calls. Give a model a virtual machine or container with installed binaries, a working codebase, documentation, data access and a writable memory directory. It can express loops, conditionals and complex operations in scripts instead of chaining narrowly specified calls.

Poolside nevertheless supports tool calls, and the discussed model adds parallel tool calling. The guest predicts large tool catalogs will disappear from system prompts; a host expects compatibility to persist because users already depend on it. They agree that writing code is a general way to accomplish tasks, without offering comparative efficiency measurements.

The modality priorities are explicit: - Vision comes next: the models discussed do not yet have visual understanding, but work has begun. The guest sees it as supporting stronger reasoning and unlocking useful capabilities today. - Audio is deferred: researchers and compute remain focused on reasoning and longer tasks. Audio is considered eventually necessary, but less useful to that immediate direction.

The language-first argument concerns information density. A short physics paper can compress extensive intellectual work; processing a 50-minute lecture as video may spend more computation on signals unrelated to its reasoning content. This is a qualitative resource-allocation argument, not a measured modality comparison. The team continues to debate whether reasoning should occur in language tokens or internal latent representations.

The final run is only part of development cost

Laguna M had begun training the day before the conversation, with a planned 39-day pre-training run. It was described as substantially larger than the previous medium model; L and XL were future plans. These size labels are relative across generations rather than fixed parameter specifications.

The launch itself reportedly amounted to a Slack message with links for following evaluations. That anticlimax captures the factory’s purpose. A final-run compute bill excludes the work of discovering how to train the model: research, infrastructure and data preparation come first. The guest uses discussion of DeepSeek’s training cost to challenge treating a final-run figure as total development cost.

The company’s name carries a related founding story. Snowball Labs was dropped after the founders encountered an Amazon trademark; the next suggestion was Bedrock. Before incorporation, they were discussing a compute partnership with an unnamed large technology company. Moving that conversation to a poolside restaurant supplied the eventual name. The deal never happened, and the guest describes Poolside as a reminder to preserve the company’s ambitions while pursuing independent funding.

During the $500 million fundraising round, roughly a year and a half before the conversation, the guest says many investor discussions still centered on whether model capabilities would keep improving. Poolside sought investors who believed AI would underpin economically valuable and scientifically interesting work. The thesis was becoming a supplier of that intelligence, a task increasingly requiring infrastructure as well as models.

The guest expects demand to grow while prices and margins decline, with customers continuing to want multiple providers. A sustainable business around openness remains an open question for Poolside.

Openness remains conditional on risk

The guest’s support for open models is conditional on emerging evidence of misuse risks. Future real-world risks could justify changing how a model is trained, releasing different versions or imposing restrictions. The guest rejects both a permanent commitment to unrestricted release and immediate blanket closure, arguing that release decisions must be reevaluated as capabilities and evidence change.

Another participant stresses the irreversibility of release: initial restrictions can be relaxed later, while releasing a model can be a one-way decision. The guest agrees. Other participants then raise the competition problem: restraint by one company or country does not stop development or release elsewhere. The guest favors international agreement on specific capabilities to limit or withhold, rather than blanket decisions about whether models may be available at all.

RL’s bottleneck is calendar time

The late technical discussion turns to co-designing models and hardware. Poolside identifies reinforcement learning as its largest wall-clock bottleneck. In the guest’s account, a finite pool of training tasks constrains useful batch growth, so adding GPUs eventually stops delivering useful RL scaling. Pre-training over web-scale data offers more flexibility to increase batches.

The proposed direction is heterogeneous hardware: assign the inference phases called prefill and decode to different kinds of chips, borrowing from inference-system designs. The intended payoff is shorter elapsed training time. The phase mechanics and specific hardware assignments remain unspecified, so this is a research direction rather than an actionable design or a deployed Poolside result. The guest also qualifies the suggestion that nobody is yet doing it for RL.

Lower precision offers another possible route, provided training remains stable. Poolside reports using FP8 for RL and expresses interest in going lower, without reporting a speedup or stability result. The criterion is consistent with the rest of the factory: competitive progress depends on calendar time.

The guest reports an asymmetric transfer pattern between model sizes: - Small-to-large recipes need substantial rethinking: post-training that works on a smaller model often does not transfer directly upward. - Large-to-small recipes often provide a strong baseline: improvements developed for larger models can quickly improve smaller ones, with further tuning still useful. Scaling therefore offers another potential payoff: lessons from larger models can improve smaller models already useful to developers.

Poolside was not using a larger teacher model to distill its released smaller models, although it had run distillation experiments. These releases serve two purposes: public products and tests of whether the factory improved. Smaller models permit faster research iteration; larger runs expose new stability problems.

Frequent end-to-end training makes improvements easier to attribute. Earlier six-month gaps between major runs accumulated so many changes that the team struggled to identify which ones mattered. Shorter cadences help researchers connect interventions to outcomes in both pre-training and post-training.

Measure delivery, and give autonomy a boundary

The closing discussion applies the same logic to software teams. The guest proposes two measures: - Lead time from user need to shipment: measure how long it takes to deliver something valuable. Code volume and pull-request activity are indicators along that path. - People required to deliver: measure how many people participate between identifying the need and shipping the result, with individual builders closer to customers and able to deliver more themselves.

Fixing a bug in two hours instead of leaving it in a backlog for three weeks illustrates the intended benefit; it is not a reported Poolside measurement. For hiring, the guest looks for agency demonstrated through action in professional work and personal projects.

A host raises the coordination problem: people with high agency may pursue different directions. The response is a shared outcome with explicit boundaries. Researchers should know the company’s priorities before joining and have substantial freedom within them. Poolside’s focus on reinforcement learning and limits on modality work illustrate those boundaries. Comparatively little purchased external data was another constraint the guest believes pushed the team to improve elsewhere.

At the time of the conversation, Poolside was recruiting across applied research and engineering, including pre-training, evaluations, post-training and architecture. The closing pitch emphasized individual impact and mission alignment: fewer than 70 people reportedly built the model, with fewer than 115 involved in the broader research and engineering effort. The opportunity was to help a small team turn more ideas into trustworthy experiments and useful releases.