The AI Engineer’s Moving Boundary: From Model APIs to Personal Memory

A model is available. Who turns it into a product?

An ordinary startup wants an AI feature but cannot assemble a foundation-model research team. It can, however, access a model through an API or use an open-source model. Swyx locates the AI engineer in that gap: a software engineer specializing in AI and its application stack, primarily working downstream of an available production model. The work can include fine-tuning—additional training on an already pretrained model.

In this Cognitive Revolution conversation, published ahead of the first AI Engineer Summit in October 2023, Swyx rejects the suggestion that he coined the title. He sees himself as identifying an emerging specialization and betting on its name. His distinction follows the work’s constraints: research scientists focus on research problems; machine learning engineers and data scientists bring models toward production; AI engineers build applications around them; full-stack engineers deliver the user interface and experience.

He expects most entrants to be software engineers moving toward AI specialization, though researchers can move toward applications too. He cites Raza Habib’s work on Humanloop, which builds tooling for operating large language models (LLMs). The interviewer supplies an application example: including GPT-4 reasoning in a GPT-3.5 fine-tuning dataset reportedly improved results, with Humanloop simplifying the workflow. The improvement is a personal report without a specified task, evaluation method or quantified gain.

Learn enough to judge feasibility

Swyx’s proposed curriculum starts with a concrete action: call the GPT API directly. ChatGPT and Copilot hide some controls and context limits that application developers need to understand. His free introductory email course, Latent Space University, proceeds through API basics; prompts, tooling and memory; code generation; image generation; speech-to-text; fine-tuning and running open-source models; and building an agent.

The goal is to answer a product manager’s feasibility question: work out how to build the proposed product, or explain the concrete obstacles. The interviewer worries that developers give up because they use ineffective prompts or the wrong model. But the course’s promise is deliberately limited: roughly an hour a day covers fundamentals, with links and side projects for deeper exploration. Building commercially useful applications and staying at the frontier require different commitments. Following niche technical discussions can itself become a full-time job.

Keeping up becomes an engineering problem of its own. Resources include curated communities and the podcast and newsletter list in swyx’s AI Notes. One participant describes a personal project that scans Discord discussions daily and emails summaries. The other, overwhelmed by roughly 50 communities, proposes using notifications to select material instead of summarizing every message. The problem is selection before summarization: unrelated discussions make a comprehensive digest noisy, while useful conversations may sit in private channels. The notification filter remains a proposal.

Buy first, then learn what to replace

A participant speaking for Waymark, a video-making business, describes the timing problem in build-versus-buy decisions: shipping early can require custom development, while waiting may let vendors supply the missing components. The recommendation is to buy first and become capable of replacing what you buy. Existing tools offer established practices and communities that have already encountered similar problems. Once the domain is understood, custom implementation becomes a better-informed choice.

LangChain illustrates the tradeoff. Developers begin with the framework, discover a mismatch, remove it and dismiss it. The response argues that this overlooks the project’s rapid evolution and the foreseeable cost of replacing dependencies in an immature field. Learning has a complementary rule: Swyx recommends implementing some functionality yourself before adopting packaged abstractions, so you understand what they do.

The tools occupy overlapping parts of the application stack: - Humanloop manages the prompt lifecycle. Develop a prompt in a playground, save it behind an API accepting configured variables, and update it independently of application code. Logs support evaluation, export and potentially the selection of fine-tuning examples. - Guardrails emphasizes output validation. Checks on generated SQL and JSON illustrate its focus on safety and production readiness. - LangChain emphasizes orchestration, coordinating application operations. Its framework also overlaps with validation tooling.

Operations vendors compete to own the application’s LLM interface, converging on prompt versioning, evaluations, cost tracking and latency monitoring. Model providers can absorb some of that surrounding work. One participant argues that OpenAI’s JSON-related capabilities displaced framework validation, but identifies neither a specific feature nor conditions under which validation becomes unnecessary. The boundary between model capability and application tooling keeps moving; that does not establish that output checks can be discarded.

Hire for initiative; gather around production work

Hiring extends beyond familiarity with tools. The proposed baseline combines technical competence with observable initiative under ambiguity: proposing ideas, shipping independent projects and putting new techniques into practice. Suggested recruiting venues include people publicly shipping projects, Discord hiring channels, Hacker News hiring posts and conferences. Adopting a new job title alone does not demonstrate those behaviors.

Asked why more developers have not entered the field, the response identifies skepticism after crypto, concern that AI is another fad, and doubts about qualifications. The argument for AI engineering is that a PhD is unnecessary to contribute through applications. Coding assistance offers a spectrum from autocomplete to increasingly autonomous generation: choose the level of autonomy that is useful now, while expecting that boundary to move.

The AI Engineer Summit aimed to give this professional community a recurring gathering, combining new product demonstrations with production experience. The planned program included established technology companies and accounts from Hex and Prefect of adding AI to existing businesses. Its organizer envisioned an engineering counterpart to the International Conference on Machine Learning (ICML), built over roughly a decade. The initial two-day, single-track program left acknowledged gaps in image generation and infrastructure.

Autonomy means completing the delegated task

Agents—systems that act toward a delegated task—bring the central disagreement into focus. One participant distinguishes open-source projects pursuing impressive demonstrations from commercial products pursuing mundane, recurring work worth a subscription. Both can contribute, but an impressive run and a dependable daily service are different targets. The described AutoGPT workflow asks for confirmation before each action; the speaker wants to delegate a task and stop supervising it.

The simplest example contains no LLM. Send someone a SavvyCal or Calendly link, and they can book, reschedule or cancel without further coordination from the calendar owner. Unattended completion within a bounded task is the useful property. An agent could deliver that value without achieving human-like insight or artificial general intelligence (AGI).

Code Interpreter, also called advanced data analysis in the discussion, provides a concrete feedback loop: 1. Generate code for the task. 2. Execute it. 3. Inspect the output. 4. Decide whether to repair the code or stop. The system uses the result of an action to choose its next action. Broader autonomy would require planning, prioritization, memory, more tools and ways to act on the outside world, including manipulating interfaces.

The forecasts differ sharply: one participant entertains five to ten years for broad autonomy, while the other expects useful agents to become reasonably commonplace within about a year. The disagreement narrows around Codium AI’s test-generation workflow, in which a user accepts or rejects generated tests. The cautious participant accepts the near-term outlook for that bounded task, while expecting longer waits for systems with more autonomy and more ways to fail. These are forecasts, not measured reliability results.

Planning, permissions and the specification problem

Lindy’s text-to-automation demonstrations make planning concrete. A short instruction specifies an email-domain trigger, a lookup, a drafted response and calendar actions; the platform appears to assemble a workflow resembling a Zapier automation. The description comes from posted screenshots, so it establishes the proposed interaction rather than reliable end-to-end execution.

Two possible routes to more reliable execution emerge: - Train on task decomposition and planning. One participant proposes fine-tuning GPT-4 on documented reasoning and the steps needed to carry out moderately complex tasks. - Give the model more inference time. The response treats longer computation when producing an answer as a possible frontier, while pointing to a lack of planning and prioritization benchmarks. Neither proposal is presented as a completed experiment.

Reliability is only one constraint on an always-on assistant; economics and security also matter. One participant advocates composing smaller, domain-specific models into software that prioritizes people, placing substantial responsibility on engineers. Their stronger assertion—that this approach has no safety concerns except unrestricted internet access without permissions—is a stated position, not a safety guarantee.

The other participant favors accelerating adoption while pausing extreme scaling: obtain benefits from current or modestly improved systems while remaining cautious about much larger models whose behavior is hard to anticipate. They connect this to Eric Drexler’s comprehensive AI services vision—specialized systems, potentially superhuman within their domains, performing many jobs and providing a buffer against more general systems.

The proposed test of progress then becomes issue-to-pull-request automation. The skeptic argues that simple copy-edit demonstrations from systems such as Sweep.dev do not establish competence at architecture or translating requirements into complex changes. MetaGPT, which assigns agents different software-development roles, raises a related concern: reported evaluation gains may omit much of what real software must optimize. The other participant expects fairly complex prompt-to-PR work within two to three years, but a wager remains undefined without criteria for codebase size, change size and required context.

The deepest disagreement concerns who supplies the specification. If a technical user must write detailed English instructions or specialized YAML, the interface may function as a higher-level programming language. The skeptic sets a stronger bar: a nontechnical person should be able to manage a software project by filing issues. Code generation from a carefully engineered specification can be useful while still leaving that bar unmet.

Vision changes feedback; voice changes availability

An upcoming Summit talk on DALL-E 3 and GPT-4 Vision turns the discussion toward multimodality: working with images and speech as well as text. A participant describes building a React module despite having no previous React programming experience. They asked the assistant to explain the application’s structure and unfamiliar file types, then obtained a command to print the repository tree and supplied the result as context. Screenshots added another feedback channel, reportedly helping the assistant recognize visual defects and modify code.

The participant acknowledges that technical involvement remained essential. The promise is more specific: showing current behavior may reduce the burden of specifying a fix, if the model can infer the needed code changes from the visible problem and the desired behavior.

Voice prompts a different reassessment. Reading may be faster than listening, but hands-free availability changes when an assistant can be used. One participant endorses a view attributed to Ben Thompson: vision may produce more striking individual demonstrations, while voice becomes a daily interface. A Russian-language tutoring demonstration inspires enthusiasm for patient, personalized teaching, though the prediction that it could outperform language apps and teachers comes without comparative learning results.

The integrated API and the platform dilemma

New capabilities also expose platform risk. One participant asks whether OpenAI will absorb adjacent products as it responds to customer needs, citing overlap between its fine-tuning jobs interface and Humanloop. Voice products offer a possible distinction: after cloning their own voice with ElevenLabs and Play.ht, the participant favors ElevenLabs for ease of use and Play.ht for creative direction and emotional expression. The suggestion that this could affect their exposure to OpenAI is a positioning hypothesis.

The appeal of consolidation comes from two integration burdens: - Voice requires a chain of services. Transcribe audio, send the text to a language model, receive text, then synthesize speech. The desired interface accepts audio and streams audio back. - Retrieval requires decisions about finding relevant documents. Choose a vector database for similarity search, divide documents into chunks, ingest them and construct queries. A user’s question may not resemble the passage containing the answer, making query construction a substantive problem.

The imagined service accepts documents plus image or audio questions and returns streaming audio, handling the intermediate work. Integration saves application work while threatening component vendors. The response compares this with Apple’s incorporation of community-app functionality, often called Sherlocking. A proposed startup strategy is to demonstrate strong engineering, choose interesting problems and ship quickly enough to become an acquisition target, though some companies would still be displaced.

Integrated multimodal APIs by the end of 2023 remain a forecast here. The claim that ChatGPT Enterprise already has a native vector database is also unsettled: when asked for details, the participant describes an uncertain understanding of forthcoming connections to company data, without knowing whether they have launched.

One participant nevertheless expects text to remain a central intermediate interface for speech and image workflows. They treat this as a temporary advantage, supported by a better understanding of high-quality text data, rather than an inherent privilege of text over other modalities.

Who pays for inference—and who remembers the user?

The proposed login with OpenAI addresses inference costs as well as authentication. Waymark’s representative estimates about $0.15 in direct cost for each user trying its free product, including language models, image understanding and some Amazon-hosted infrastructure. A shared login could let users bear inference costs across third-party products, reducing the developer’s expense of offering demonstrations. This is a proposed billing arrangement.

The response questions whether operating identity infrastructure fits a research organization’s mission, especially as user-data management grows more complex. Universal login could signal a choice to become an AI cloud. A related ambition is personal context that follows the user. One participant sees Character AI’s chatbot-product experience as relevant, distinguishing that product expertise from enthusiasm for OpenAI’s model progress.

The comparison of model providers keeps release activity, practical usefulness and future expectations distinct. OpenAI dominates one participant’s monthly news recap; the other praises Claude 2 for long-context work. Discussion of Google’s ability to ship products and its confidence in Gemini remains speculative, and one participant declines to judge PaLM 2 definitively without more experience. These are contemporaneous assessments rather than controlled comparisons.

Persistent context leads to the episode’s AI Horcrux: an always-carried device that logs conversations and makes those memories conversationally accessible. The desired digital twin could begin as a private note-taking tool and potentially become something other people interact with. Pendants, watches and clips are proposed as consumer form factors beyond the phone, with AI engineers building the experiences around them.

The app bundle: occasional use without another subscription

The closing discussion returns to application economics. The bundle proponent’s AI subscriptions are approaching or exceeding a former cable bill, yet many useful tools remain outside it. Waymark faces the other side: users need to try its AI video creator, but trials cost money. At the reported $0.15 per trial and $30 monthly plan, one paid month covers 200 trials’ stated costs, before other expenses or profit. Immediate cancellation can reflect an occasional need for a video rather than dissatisfaction.

The proposed AI app bundle charges an illustrative $100 a month for 1,000 applications, mixing recognizable flagship products with occasional-use tools. Equal allocation gives each app $0.10 per subscriber per month, with opportunities to upsell power users. At one million subscribers, those inputs imply $1.2 million annually per app; the speaker rounds this to roughly $1 million. All of it is hypothetical. The respondent cites Setapp as an existing analogue and asks the decisive question: are apps paid even when subscribers do not use them?

The proposal has several competing incentives: - Consumers gain discovery and occasional access. A single payment could make tools such as Gamma attractive even to someone who rarely creates slides. - Suppliers gain a revenue baseline. The proponent favors payments to every participating app, with shares changing as usage patterns evolve. - Vendors give up some control. Owning customer relationships and changing prices can conflict with a bundle operator’s rules. - Consumer apps fit better than enterprise infrastructure. Vendors pursuing large annual contracts have less reason to join a consumer subscription bundle.

The final practical advice is to take the smallest step that produces useful information—perhaps assemble five products and test demand. The difficulty is that a bundle’s appeal may depend on scale and recognizable anchor products. Hundreds of unfamiliar apps can be less compelling than a few services a buyer already wants, accompanied by tools to discover later.

At the recording’s close, physical Summit tickets were sold out, with online participation available through a livestream and Slack community. The closing directs listeners to ai.engineer for the Summit, newsletter and job board, and to latent.space for the podcast.