Software 3.0 and the Rise of the AI Engineer

Start with a model, then collect application data

Computer vision and voice recognition expose a limit of hand-written software: specifying behavior through conditions and loops becomes impractical. Machine learning changes the task. Developers specify a model’s architecture, train it on data and obtain weights—the learned parameters that encode what training has captured. Foundation models change the starting point again: an application developer can begin with an already-trained model.

The Software 3.0 framing extends Andrej Karpathy’s Software 2.0 idea, with no conceptual lineage from Web3. The versions distinguish where application behavior comes from: - Software 1.0: write the instructions. Engineers express behavior directly in code. - Software 2.0: learn from collected data. Engineers define a model’s structure and train its weights. - Software 3.0: use a foundation model. Engineers build on an existing model, then collect application data for optional further training, or fine-tuning.

Conventional product development already learns from data: engineers ship, read analytics and change the code. Software 2.0 moves part of that learning into training. Foundation models move the initial training outside the application team. Model access can precede application-specific data collection, allowing a team to try an AI product before undertaking its own training process. The claim that this shortens development by orders of magnitude is an estimate, not a measured comparison.

Code, weights and training data are different assets

A model architecture specifies a model’s structure, including its layers. In the simplified development workflow, an engineer defines that structure in a framework such as PyTorch, trains it and deploys the result. AI engineers need to understand how model choices affect their products; they need not acquire all the expertise required to create those models.

Llama 2 illustrates the shift in value toward training data and learned weights. Its reference implementation is described as only a few hundred lines, but Meta did not release the training dataset. The weights were available for many commercial uses, though the guest qualifies their status as not fully open source. Access to implementation code, weights and training data are separate questions; commercial usability does not settle all three.

Foundation models supply capabilities that transfer to application tasks without the developer first training a model. Examples include GPT-3, GPT-4, Claude, Whisper, Segment Anything and Stable Diffusion. A trained model’s binary weights are loaded for inference—the use of a trained model to produce outputs. In a product, those outputs become capabilities such as generating text or images. The engineering question is what those capabilities let a customer do.

Production ranges from API integration to GPU infrastructure

Putting a model into production can mean three different operating situations: - Hosted API: call a provider’s model much as another application API, while managing context limits, sensitive information, rate limits and token costs. - Local personal use: run tools such as Llama.cpp or Whisper.cpp on your own machine, managing the local runtime without serving an outside audience. - Serving external users: operate custom models for many users, taking on the responsibilities of a model infrastructure provider.

An exposed model endpoint can become an exposed bill. Bots can discover exposed endpoints or leaked credentials and consume expensive model calls. Using a hosted model leaves application developers responsible for access, user information and spending.

Operating the serving infrastructure adds questions of component bandwidth, saturation and GPU utilization. A FLOP is a floating-point operation; repeated matrix multiplication is central to the computation discussed here. A GPU’s theoretical capacity does not establish the throughput an application will achieve. High utilization requires specialist infrastructure work, while batching work across tenants is a key mechanism for reducing serving cost per tenant.

The API separates model development from product development

Foundation models make it possible to ship an AI product without an in-house machine learning (ML) team. The proposed division of work follows a spectrum: - Research scientists develop models: they work on algorithms and architectures. - ML engineers operate models: they handle infrastructure, serving and data pipelines. - AI engineers build with models: they incorporate model capabilities into applications and add the surrounding product logic. The API forms a permeable boundary: it can separate teams within one company or connect an application team to an external model provider.

An AI engineer is thus a software engineer who uses foundation-model APIs as application building blocks. The specialization includes following new techniques, putting them into production and judging when an AI application is not ready to pursue. Existing software engineering seniority remains relevant. The guest’s estimate that the work is roughly 90% software engineering expresses that continuity; it is not a measured task breakdown.

The economic thesis is that demand for AI products exceeds the supply of ML specialists. Because software engineers are more numerous, the expectation is that most AI engineers will come from that population learning to work with models. A shared title helps employers and practitioners find one another and organize around tools, papers and skills. In this 2023 account, the category remains small and poorly defined; its scale, curriculum and career ladder are still predictions.

Lower barriers do not remove the need to understand customers. One proposed opportunity is a cheaper, lower-quality substitute for a professional service. The copywriting example contrasts a $20 monthly ChatGPT subscription doing an “80%” job with a specialist charging $3,000 an hour—illustrative figures, not measured quality or market pricing. The business question is whether the cheaper result meets the customer’s needs. The further expectation that displaced specialists will move into higher-value work is a forecast, not an assured transition.

Retrieval supplies the information a model lacks

The Software 3.0 stack is a map of tools practitioners might need, organized around three responsibilities: - Foundation-model access: obtain the capability used by the application. - Retrieval-augmented generation: connect that capability to relevant data. - AI user experience: design how people interact with it. The model layer is termed a system of reasoning. It includes closed models accessed through OpenAI or Anthropic, alongside open-model options involving services such as Hugging Face, Replicate, Modal and Lambda Labs, where hosting requires more attention.

A broadly trained model may lack the information needed to answer a question about your company, product or personal records. Retrieval-augmented generation (RAG) retrieves relevant information and places it in the model’s context, the material available to it while generating an answer. The application supplies information at answer time instead of assuming the model’s training contained it.

The described workflow connects data storage to answer generation: 1. Connect the sources. Orchestration frameworks such as LangChain and LlamaIndex connect a large language model (LLM) to material in Notion, Slack, Gmail or Google Drive. 2. Prepare material for retrieval. The source material is embedded and stored in a vector database, with Pinecone, Chroma, Weaviate and Milvus among the examples. 3. Insert retrieved information into context. When an answer is needed, the application retrieves material and supplies it to the model during generation. This is the role of the retrieval components in the stack; the mechanics of embeddings and vector search are a separate prerequisite for implementing them.

Assembling these tools was characterized as cumbersome but the strongest available approach at the time. Contextual AI pursued retrieval integrated into the model, offering an alternative to stitching together separate components. The guest also treated fine-tuning new knowledge into existing models as unresolved. That broad assessment was offered without experiments and should not be read as a demonstrated limit on fine-tuning.

Interfaces make model capability usable

AI user experience (AI UX) concerns how people interact with models. It is presented as the least established part of the stack and an opportunity for frontend engineers. ChatGPT’s back-and-forth conversation unlocked value beyond the earlier playground interface. But calling its launch purely a UX innovation overlooks the accompanying model change: the guest describes it as a slightly better GPT-3.5 model paired with a new delivery format. The product principle is to bundle a new model with a new mode of interaction.

Three examples place model output directly within a task: - GitHub Copilot puts suggestions inside the editor. It observes typing and offers inline completions, letting developers assess suggestions alongside their code without repeatedly switching to chat. The guest describes more than six months of deliberate engineering behind that experience. - Vercel’s v0 turns a UI request into editable code. The then-recently announced product generated React and Tailwind code that users could copy into their projects. - ChatGPT voice removes manual submission between turns. The interviewer highlights automatic submission when a person finishes speaking as a detail that supports a continuing spoken exchange.

The then-new vision and conversational speech features in ChatGPT suggested a shift toward OpenAI becoming more of a product company. At that point, those capabilities were being delivered inside ChatGPT without corresponding APIs, code releases or papers.

Voice interaction already existed in Google Assistant, Alexa and Siri, so novelty alone does not explain the opportunity. A trusted product can become the user’s default without a fresh comparison of every alternative. The Apple analogy captures the distinction: customers may accept a later feature because they expect the finished experience to be good. Matching a feature and earning that expectation are different accomplishments.

Separate AI blame from skepticism about capability

Swyx distinguishes two arguments that can sound alike: - AI blame credits AI with disruption. He uses the term for blaming AI for problems already underway. His example is Chegg: he argues that its decline preceded ChatGPT even though the company blamed poor performance on it. - Capability skepticism questions what models can do. Claiming that AI is powerful enough to damage a business implicitly credits its capabilities; questioning whether it possesses intelligence is a different argument.

Swyx characterizes the Stochastic Parrots critique as questioning whether plausible language amounts to thinking or knowledge grounded in experience. He partly accepts that concern, then asks whether a sufficiently good representation can reflect the world. His position is that demonstrated performance should count toward judgments of intelligence, even when a model’s process differs from human thinking.

He invokes academic and professional exams to support that position. His sweeping claims of superhuman performance and a conclusively passed Turing test, however, lack model versions, scores, comparison groups and test protocols. They express his argument about recognizing capability, rather than establishing a reproducible evaluation result.

Learn by building something useful

Swyx also considers skepticism about overheated expectations warranted, attributing some exaggeration to media and creator incentives. His practical response is to build around a customer need or a task that improves your own work. One personal project mostly generates Chrome extensions from prompts; he finds it useful without intending to sell it. Personal utility is enough to make a project worthwhile.

That approach carries into his educational work: a free seven-day email course, developed with his friend Noah, offers guided projects across major AI modalities. Wes Bos’s JavaScript 30 supplied the inspiration—learn through repeated, concrete projects. Swyx also acknowledges the newsletter format’s benefit for collecting subscribers. The companion article separately previews the Latent Space University course, with its first live testing planned for the Summit workshops.

The closing invitation is to the AI Engineer Summit, bringing together participants from organizations and projects including OpenAI, Amazon, Microsoft, Vercel, Notion, LlamaIndex, AutoGPT, Supabase and Guardrails. In-person attendance was full, with livestreaming planned. The companion article announced October 8–10 with free online attendance: an occasion for the emerging field to gather around its tools and practices.