From scaling models to building reliable agents

Organizing a lab around outcomes

On David Luan’s second or third day at OpenAI, Greg and Ilya asked him to take over their direct reports so they could concentrate on individual-contributor work. He recalls joining around employee 30–35 and spending roughly two and a half to three years leading engineering. His immediate problem was to bring existing efforts, including a substantial Dota project, into teams with a clearer research direction.

In this conversation published in March 2024, Luan describes a shift in what made an AI lab effective. In his telling, a few researchers could write a field-changing paper during 2012–2017; around 2017, scale became decisive. His response was to organize OpenAI around a few ambitious scientific outcomes, retaining room for bottom-up research while concentrating resources on the largest bets. Novelty mattered less than achieving the outcome.

Luan’s next proposed transition—and a reason he gives for founding Adept—was the joint development of models, products and user feedback. Adept’s goal was an agent that translated natural-language requests into actions across existing software. A useful product could therefore do two jobs: help people complete work and generate feedback that improved the underlying AI.

Learn from humans, then pursue goals

Luan favors a practical definition of artificial general intelligence (AGI): a model that can do anything a human can do on a computer. Definitions centered on replacing people or outperforming them at economically valuable tasks, he argues, already assume a particular future role for humans. His definition instead points toward agents that act to accomplish goals.

Reinforcement learning (RL) supplies a formulation for goal-directed behavior: act to maximize reward. Standard language-model training does not provide that objective out of the box. But Luan considers learning everything from scratch through RL wasteful. Recalling a disagreement with an unnamed Berkeley professor, he rejects the idea that AGI must reproduce all the computation expended by evolution. Human knowledge offers a shortcut.

He uses behavioral cloning broadly to mean learning from human behavior and its recorded outputs. Language models learn from written knowledge; multimodal models extend that process into visual information. His envisioned universal byte model would learn mappings across data types, such as text to voice or images to video. The proposed next step is to combine those learned patterns with RL’s methods for pursuing goals.

Compute needs organizational commitment

Luan later led Google’s language-model efforts and served as a Google Brain lead. He describes research leadership as portfolio allocation: gather critical mass around promising ideas, move resources toward those that work, and wind down those that do not. In his assessment, Google lacked that organizational capability during his time there.

While OpenAI trained GPT-3, Luan worried that Google would finish a competing model first. It had abundant compute, researchers behind the underlying technology, and Noam Shazeer’s ambition to build a trillion-parameter model. Luan says he later learned that the effort had struggled to assemble critical mass. His suggestion that assigning Shazeer half of Google’s TPUs could have let Google beat OpenAI remains a counterfactual judgment.

The Brain Credit Marketplace illustrates the coordination problem. Researchers received credits whose purchasing power in chips depended on supply and demand. As described in the conversation, assembling a large training job could require persuading roughly nineteen or twenty colleagues to forgo their own work. An organization could possess substantial compute while making it difficult for a single project to use enough of it.

Hardware access also had to translate into useful computation. OpenAI’s collaboration with NVIDIA brought together supercomputing expertise and efforts to push chip utilization. Advance knowledge of A100 sparsity capabilities prompted consideration of whether they could help training, though adoption is not established. The engineering objective was to map useful model computation efficiently onto the hardware.

GPT-2 captures the tension between this work and conventional research expectations. One recollection describes a modeling section roughly a paragraph long, presenting a largely standard decoder-only transformer. The authors worried that the field would dismiss it for lacking novelty. Yet scaling programs depended on hard-won knowledge that was often unpublished or difficult to express in a conventional paper.

A second GPT-2 story concerns the final Microsoft pitch before its investment in OpenAI. The technical presenter had to demonstrate a slightly larger model version prepared only a day or two before the Seattle trip. Canned prompts offered reassurance; letting executives type arbitrary input did not. The anxiety illustrates checkpoint-dependent behavior: familiarity with one saved version of a model does not guarantee predictable behavior from the next.

When an agent dispatches a truck

At the time of the conversation, Adept positioned itself as an enterprise company automating complex employee workflows for large startups and Fortune 500 companies. Its business was ready-to-use automation rather than API sales, open-source model sales or a self-service browser extension.

The early ACT-1 demonstration showed a house-related task on Redfin and interactions with Google Sheets. Dependable business use posed a harder problem. Agent demos were characterized as working about 60% of the time, against enterprise expectations of reliability “in the nines.” These were descriptions of the perceived gap, not benchmark results or a measured Adept success rate.

One reported workflow ended with a physical truck being dispatched. An incorrect destination makes an agent error a direct operating expense, explaining Adept’s emphasis on reliability. The truck-specific monitoring and approval process was not established in the discussion.

An interviewer describes agents as “services as software”: software performs work otherwise delegated to a person. Adept’s account emphasizes a shift toward supervision. Workers could run tasks in parallel, open individual execution trajectories—the sequences of actions taken—and troubleshoot harder cases. The reported enthusiasm came from removing rote work from their jobs.

Many failures were described as visibly broken trajectories that the agent recognized, allowing a person to intervene. That does not imply every error is detectable. Adept’s augmentation strategy deliberately targets tasks just beyond the model’s current ability: human oversight, clarification and expert intervention provide training data for capabilities the model has yet to acquire.

Adept reported being fully booked for Q1 customer onboarding, with onboarding bandwidth as the constraint. Its research ambition was to support increasingly abstract delegated tasks while maintaining reliability, using feedback at each level to help reach the next.

Multimodal models for knowledge work

The proposed standard for an agent is completing a goal-directed workflow correctly in as few steps as possible. Chatting or making a function call is insufficient by itself. ACT-1 addressed the initial challenge of teaching a model to operate a computer across different instructions. Production use added company-specific requirements: similar tasks could follow different procedures, so specifications and guardrails had to express the intended behavior.

The Fuyu architecture grew from Adept’s need for fast multimodal models that understood screens and knowledge work. The proposed training corpus followed that purpose: - Charts, graphs and tables. - Invoices, PDFs and receipts. - Unstructured business data and user interfaces. This emphasis differs from a corpus centered on natural photographs. Low latency and dense optical character recognition (OCR)—reading the many small pieces of text on a screen—were treated as essential agent capabilities.

Adept said its public Fuyu releases, including Fuyu-8B, used other training data rather than its actual knowledge-work corpus. Public model behavior therefore did not directly demonstrate what that internal corpus produced.

Product design is part of the learning system because the interface shapes the feedback users provide. Adept described integrating models, agent execution and interface design around that relationship. Adept Experiments tested design ideas with users; its codebase also underpinned the actual product and provided a framework for deploying different cards.

Operate the software people already use

Adept combines human-like computer operation with API calls. The analogy is humanoid robotics: operating in an environment built for people allows a machine to use what already exists. In software, control of the user interface fills gaps where a workflow lacks an API for every step. It also makes human computer usage available as training demonstrations without translating every action into a custom API operation.

The longer-term prediction is an agent interaction layer above existing applications. Initially, agents would operate graphical user interfaces and APIs. With greater capability and trust, they could break down goals, execute workflows and present information tailored to the task. Underlying applications would increasingly store authoritative records or run specialized workflows, while direct application use remained useful for power users.

Rabbit comes up as a possible example, but the endorsement concerns this broader interaction model. The response explicitly declines to assess Rabbit’s real-world experience without knowing more about it.

Debug the controls before blaming the model

Self-driving supplies a caution about easy demonstrations. A car traveling down Highway 101 on a sunny day says little about broader reliability. The discussion acknowledges Waymo’s accomplishments while emphasizing the long interval between early demonstrations and mature deployment. Adept’s ambition is to collect useful feedback immediately and improve continuously, avoiding a development path dominated by distant autonomy milestones.

The concrete debugging lesson concerns sensors and actuators: the interfaces through which an agent observes and acts. Adept reported apparent model failures caused by unreliable action interfaces. Even a human using the same controls could not complete the task reliably. That suggests a useful diagnostic: try the failing action through the agent’s controls yourself. A model improvement cannot, by itself, repair a control that cannot execute the action.

Reliability, speed and cost can conflict with generality at the Pareto frontier, where improving one objective requires sacrificing another. The argument here is that many systems have not reached that frontier. Adept proposed treating each new use case as additional training data, allowing capabilities to expand without prescribing a narrow set of permitted steps. The acknowledged risk is overfitting to those end uses.

The customer-customization architecture remained undisclosed. An interviewer’s proposal of a shared base model with customer-specific fine-tuning was not confirmed.

Combine demonstrations with generated experience

Two complementary routes to greater capability bring the discussion back to learning: - Turn compute into data through search, reinforcement learning and self-play. - Collect expert human demonstrations of difficult work. The proposed relationship is that the first route improves sample efficiency for the second: the system learns more from each valuable human example. Rapid progress, in this view, requires both.

Computer use offers an environment in which agents can act, making it an attractive setting for exploring synthetic data. No generation procedure or measured result is given here; the stronger suggestion that everything can be simulated comes from the interviewer.

Adept’s competitive argument rests on a vertically integrated agent stack: developing the model, execution system and product together. That control lets the company prioritize model speed and agent capabilities. Internally, foundation-model research is judged by whether it produces a better agent.

The accompanying business prediction is that improving open models and large companies’ compute investments will commoditize general language models, followed by general multimodal models. It assumes predictable gains from reasonably executed training on public web data. Adept presents specialization in useful agents as protection against competition driven by ever-larger training budgets.

Customer failures become research targets

A late turn to robotics returns to behavioral cloning: learn from robot behavior, then teach the model to solve new problems. This is expressed as a confident prediction, not an experimental result. Figure’s strategy is left unassessed; the question worth asking is how it plans to reach reliable deployment. Enthusiasm for robotics models is tempered by less enthusiasm for a business framed around replacing workers.

Asked why someone should join Adept rather than Imbue, the response rejects a zero-sum competition: the opportunity is large enough for companies to succeed in different areas. Adept’s recruiting premise is useful agents, with research serving that goal and customer feedback helping accelerate progress.

The concrete mechanism is turning unmet customer requirements into evaluations. Tasks an agent cannot perform become explicit research targets. Reasoning, planning and generalization then have a practical test: can the system now do work a customer needs? Familiar benchmarks may be saturated, while an unsaturated benchmark is not necessarily useful. Customer requirements give the next capability a purpose.

The closing argument places this work within the industrialization of AI. Compute, data and methods for exchanging one for the other increasingly determine progress. Research advances still matter, but the field must also embrace the industrial work of building and improving systems at scale.