AI Engineering Starts at the Model Boundary

A few API calls get you started. Production needs a specialty.

A developer can get an AI product running with a prompt and a few API calls. Then come the harder decisions: how to evaluate its behavior, operate it reliably, and improve it. Shawn Wang’s argument is that productizing model capabilities is a full-time engineering specialty, even when the team does not develop the underlying model.

In this High Agency conversation, Raza Habib interviews Wang, also known as swyx, roughly a year after “Rise of the AI Engineer”. The interview looks ahead to the AI Engineer World’s Fair; the accompanying June 25, 2024 announcement reports that it has sold out.

Habib opens with Wang’s essay, whose comparison of years of research with an afternoon using API documentation illustrates a shift in accessibility, rather than a measured speedup. APIs and open-source foundation models put previously research-intensive capabilities within reach of software developers. Evaluation, application, and delivery still require substantial work.

Wang calls the dividing point the API line: the boundary between developing a model and building a product with it. He argues that rising model-development costs and increasingly closed labs push that boundary between companies. A product business can outsource model development while employing specialists who understand the deepening application stack. The model becomes an input; making it useful remains their responsibility.

Choose skills according to the model work you own

Wang deliberately leaves the role’s specification open. Machine learning (ML) engineering skills can make an AI engineer more capable, but they are not an entrance requirement. The practical distinction is how much model work the product takes on: - Use a hosted model. A product built around GPT-4 APIs can be commercially viable without extensive in-house ML engineering. - Take greater control. Investment in fine-tuning, inference, operations, and evaluation deepens the technical stack. Taking on more model work or control over open-source models increases the need for ML expertise.

He also distinguishes initial product creation from optimization of an established problem. His hypothetical ML engineering example is reducing fraud from 8% to 5% when a company already has data, a baseline, and enough scale for the improvement to matter economically. Cursor, Copilot, and Photo AI illustrate the other question: what useful product can someone build around a model’s capabilities?

These are tendencies in skills and working style, and individuals can succeed across both. Wang argues that agent development particularly rewards different assumptions and a dedicated focus: traditional ML experience contributes, but does not confer a unique advantage.

Product judgment determines what the team should build

The team-composition discussion proposes roughly four AI engineers per ML engineer for a mature engineering team. Product engineers would handle product work and long-tail gaps, giving ML engineers room to develop model-based advantages. Four to one is a staffing hypothesis, not a measured optimum; how teams should evolve beyond their first viable product remains unresolved.

The responsibilities extend beyond engineering: - Product managers and domain experts supply direction. Customer understanding matters because better implementation cannot repair a bad product decision. - AI engineers translate requirements into available capabilities. On a small team, one person may combine product judgment, some ML knowledge, and enough engineering skill to build. - Domain experts can contribute directly to implementation. The HumanLoop product thesis is that subject-matter experts can write prompts and create artifacts that become part of the product, moving beyond specifications.

That direct collaboration motivates a business bet on tools connecting experts and engineers. Several teams can pursue the same insight; execution determines the outcome.

Does this work require a separate title? One objection is that every software engineer will eventually become an AI engineer. The response distinguishes general AI literacy from sustained specialization: developers can learn AI while continuing to specialize in databases, frontend work, or distributed systems. The title’s immediate purpose is to help employers and practitioners find each other. A settled skill ladder can follow.

Employers already interpreted the title differently. An OpenAI posting discussed in the interview reportedly required five to seven years of ML engineering experience for fine-tuning work on its custom models team. The respondent questions that prerequisite for work without pretraining, while accepting that the posting occupies the ML-heavy end of the spectrum.

Early learning compounds; early attention can disappear

Entering the field early can reduce the catch-up burden as papers, prompting techniques, APIs, and historical context become assumed knowledge. But being early creates a learning advantage, not a right to win. AutoGPT, a sponsor of the previous conference, supplies the cautionary example: the respondent’s assessment at the time was that extraordinary early attention had failed to become durable success. Users attracted by AI’s promise can leave quickly when a product fails to deliver.

The AI Engineer World’s Fair gave the specialty a place to develop. Its expansion from one track to nine reflected distinct areas of practice: retrieval-augmented generation (RAG) and code generation remained core topics, while multimodality, evaluation and operations, and agents received more dedicated coverage.

Two additions broadened the program beyond technical implementation: - AI in the Fortune 500 sought larger deployment stories beyond the startup circuit. Coinbase and Salesforce were cited as participants, with the qualification that not every participant was a Fortune 500 company. - VPs of AI addressed strategy, team growth, organization, and operating principles.

The organizer wanted peer conversations valuable enough to justify attending on their own, alongside practical benefits in learning, hiring, job searches, and launches. At recording time, about 1,500 people were expected against a target of 2,000, and three unnamed foundation-model launches were tentatively anticipated. These were preview expectations.

Ship to start the feedback loop

The advice for builders begins with a concrete challenge: if the plan takes three months, what would make a useful first version possible in three weeks? A rough prompt-based implementation may test the idea before a more elaborate system is justified. The proposed optimization loop runs through market feedback: improve the product using what people do with it, without assuming the first task must be collecting data and optimizing model weights.

The interviewer adds the mechanism. A good-enough first version produces real usage, evaluation data, and feedback. Those observations help improve the next version, creating further opportunities to learn. Deployment supplies feedback that an unshipped product cannot generate.

Asked about hallucination risk, the respondent argues for greater tolerance of experimentation at many companies, with expectations set around a beta generative product. He acknowledges the concern and distinguishes those businesses from Google’s greater responsibility.

Vertical products compete through domain knowledge

The stronger business preference is for vertical AI: products focused on a particular customer domain or workflow. The proposed advantage combines proprietary data, customer-specific knowledge, and urgent problems for buyers who may be less price-sensitive than developers purchasing infrastructure. An unnamed construction startup illustrates the opportunity: difficult real-world workflows can yield knowledge that developers focused on APIs overlook.

The examples show several kinds of specialization: - Midjourney serves creative users. A vertical can be defined by its audience and work. - Brightwave assists hedge-fund research. The argument is that language models can accelerate research analysis, while still making errors. - Virtual staging helps sell houses. Real-estate agents need images of staged properties; the product name is tentatively recalled as Interior AI. - News summarization can be a standalone product. The respondent describes pursuing it as a focused business where others treat it as a feature.

The proposed sign of strong product fit is that customers consider the tool necessary to remain competitive. Harvey, Cursor, and Copilot illustrate the sentiment the respondent expects, rather than a customer-research finding. His competitive bet is that domain knowledge and focused workflows make businesses less vulnerable to a foundation-model provider’s next launch, while horizontal vendors may face a shakeout.

Buy shared infrastructure; learn where you differ

For teams buying AI infrastructure, the recommendation is buy first, then selectively build once requirements are clear. Existing products incorporate problems other customers have encountered. Using them lets a team discover its own needs before paying for a custom implementation, while keeping lock-in in view.

Suggested early purchases cover recurring needs: - Evaluation platforms support assessing model behavior. - API observability and monitoring make interactions inspectable, including inputs and outputs that can support evaluation or fine-tuning. - Key management handles time-limited API keys and key rotation. A custom replacement becomes reasonable when the team understands the problem and finds available tools unsuitable or substantially overpriced. Buying first shares development costs with customers facing similar requirements.

Customer-product infrastructure is distinct from internal productivity tools such as coding assistants and meeting summarizers. Beyond those lies the more speculative idea of AI employees handling bounded portions of a job, with responsibilities expanding over time. The anticipated benefits include greater capital efficiency and work continuing overnight. The respondent says this vision has not yet been broadly realized; Lindy AI is the closest example he has seen, without massive adoption.

The employee analogy has limits. Humans and AI may learn differently and excel at different tasks, so useful AI workflows need not reproduce human jobs. The interviewer similarly challenges a single “human-level” milestone: a system can exceed human abilities in some dimensions while remaining weaker in others. Design around the capability profile, rather than expecting a uniform substitute for a person.

Separate competitive bets from research priorities

The closing discussion offers two filters for AI news: - Commercial battlegrounds: data, graphics processing units (GPUs), general-purpose versus domain-specific models, and RAG and operations. The criterion is a conflict in which participants cannot all win. - Research priorities: long inference first, synthetic data second, alternative architectures third, mixture of experts and model merging fourth, and online large language models (LLMs) fifth. These are proposed priorities; the conversation names the research directions without explaining their mechanisms.

The distinction matters because technical importance does not automatically make an area a competitive battleground. Open source is characterized as politically contested but broadly supported within technology. Code generation remains important and exploratory, with no settled successor to Copilot; Devin is offered only as a possibility.

Falling costs and faster inference change product choices

The economic claim concerns falling cost at a fixed capability level. The respondent proposes a rough annual five- to tenfold decline, illustrated by the cost of reaching a score of 70 on MMLU, a benchmark described as a collection of professional-exam-style tasks. Without a defined model set, pricing method, or benchmark configuration, that rate remains a planning hypothesis. The discussion also cautions against fixating on a benchmark that may be replaced as it saturates.

Cheaper capability and higher expectations can coexist. A capability tier becomes less expensive while a stronger model raises expectations and starts a new, more expensive tier. A then-future GPT-5 serves as the hypothetical example. This motivates considering products that lose money initially, but their business case depends on costs continuing to fall and on the capability customers require.

Other changing constraints can alter the product itself: - Inference speed changes interaction design. At a hypothetical 5,000 tokens per second, the respondent argues that incremental token streaming would no longer be necessary for the experience he imagines. The Groq figure mentioned is an unnamed-source target, not an achieved result under specified workload conditions. - Longer context expands possible inputs. The discussion contrasts roughly 4,000-token contexts with million-token contexts while leaving their best uses open. - Multimodal input and output expand the interface. The proposed direction is toward accepting and producing multiple kinds of media.

Generate possibilities, then test them

The final technical turn concerns variance. “Temperature zero” and “temperature two” name contrasting application objectives here, rather than universal settings: - Constrained work values adherence. RAG supplies the example of asking a model to stay within a bounded workplace task. - Exploratory work values unexpected possibilities. An invented idea can be useful because the team had not considered it, making additional creativity worth paying for. The distinction is whether invention violates the task or contributes to it.

The interviewer extends this into a proposed mechanism for knowledge creation: use the model to generate conjectures, and a separate process to test and measure them. Varied generation supplies candidate explanations; testing determines which deserve confidence. The proposal requires more than an LLM alone and gives creativity a place without treating an interesting output as established knowledge.

At the close, Shawn directs listeners to AI Engineer for the active conference and offers highagency for last-minute tickets at the time of recording. He points to @swyx and Latent Space for more conversations, adding that a listener donated the podcast’s domain. The host closes with an invitation to explore more High Agency episodes and send feedback.