Sam Schillace: Think with the Model, Plan with Code

An engineering community needs somewhere to meet

More than 5,000 people applied to attend the first AI Engineer Summit in person; the organizers accepted about 400. The AI Engineer World’s Fair would give that emerging community a larger gathering: an expo where builders could exchange ideas and discuss what they were learning.

Planning began after the organizers met at a hackathon in February 2023. They announced the Summit that June alongside Swyx’s The Rise of the AI Engineer. By this March 2024 conversation, they were seeing more attendees use AI engineer as their job title. Swyx wanted a gathering where engineers moving AI into production could take center stage.

Vendor neutrality meant putting competitors together. Swyx wanted Microsoft beside smaller AI startups so engineers could compare where companies were innovating. The organizers argued that an independent conference could prioritize technical relevance across vendors. The World’s Fair name invoked grand expos showcasing achievements across industry and culture, while the gathering itself offered time for conversations beyond the Twitter news cycle.

The announced program had more than 100 talks and workshops across nine tracks: - Models and infrastructure: Multimodality, Open Source Models, and GPUs and Inference. - Application engineering: Retrieval-Augmented Generation, Evals and LLM Ops, Code Gen and Dev Tools, and AI Agent Applications. - Organizations: AI in the Fortune 500 and AI Leadership, with a separate VP pass required for leadership access. Microsoft was the presenting sponsor, with keynotes also confirmed from OpenAI, AWS, and Google. The conversation then welcomed Sam Schillace, introduced at the time as Microsoft’s CVP and Deputy CTO.

From collaborative documents to programming with meaning

Schillace still had the first Google Doc in his personal account, containing collaboration tests with Steve Newman. Its continuity was conceptual: to his knowledge, both frontend and backend had each been rewritten at least twice.

His early experiments with GPT-4, which he recalled accessing around September 2022, explored another kind of expression. Using ABC notation, a plain-text representation of music, he asked the model to compose a fiddle tune. He reported that it did, then changed the tune’s key on request.

That experience suggested that models could perform some of the translation from intention to computation. Programmers traditionally mediated between the world and the machine through syntax, schemas, and programming languages. Models could now accept semantics—meaning, intention, nuance, and ambiguity—and perform some of that mediation themselves. Schillace saw the beginning of a second era of computing, recalling the possibilities that cloud software had opened for him.

He also recounted, secondhand, an AP Biology demonstration that changed Bill Gates’s assessment of AI after Gates challenged Kevin Scott to demonstrate that capability. Schillace’s own response was to start building developer tools. He and a small team created Semantic Kernel, an early orchestrator that combined conventional code with model inference. Its Unix-like pipelines passed one module’s output into the next module’s input.

A correct answer can still violate the interface

The [Schillace Laws](https://devblogs.microsoft.com/semantic-kernel/early-lessons-from-gpt-4-the-schillace-laws/) grew out of a fall and winter spent building applications with models. John Maeda attached Schillace’s name to the principles and publicized them. Schillace approached the work as an application builder consuming models. One early lesson was to break difficult tasks into pieces, even when inference costs and scarce GPUs encouraged developers to squeeze everything into one prompt.

A small formatting failure made the lesson concrete. A question required reasoning through a book, its author, and when that author had lived in a particular country. The expected answer was 2000. GPT-4 could produce a paragraph reaching that answer, but Schillace could not reliably get it to return only the year. The paragraph was correct but unusable as the next pipeline stage’s input.

The repair was to separate reasoning from answer extraction: 1. Let the first model call produce a paragraph reaching the answer. 2. Pass that paragraph to a second prompt that extracts and returns only the answer. The second call had a simpler task: identify an answer already present in its input. Schillace reported that extraction always worked in this experiment, without giving a trial count or evaluation protocol. It was a useful decomposition pattern, not a general guarantee of perfect accuracy.

His intuition was that generating intermediate reasoning helped the model solve the first task. Considering how a person would divide the work could therefore help design the program without treating the model as human. He also described prompting as directing a model toward relevant parts of a large representational space: simplistic questions tend to elicit simplistic answers, while thoughtful questions can elicit more sophisticated responses. This was an intuition for designing prompts, rather than a measured account of model internals.

Long-running agents make memory a planning problem

For larger processes, Schillace’s prescription was think with the model, plan with code: represent the overall workflow in software and let the model supply reasoning or content within its steps. The lesson came partly from Infinite Chatbot, an internal successor to Semantic Kernel work that explored retrieval-augmented bots and then multi-agent autonomy.

He compared conversational continuity to stop-motion animation. Separate interactions appear continuous, but a memory system must supply the information that sustains that impression. Reconstructing context can disturb the plan itself when a long-running agent depends on that reconstruction to decide what to do next.

His experiments exposed several failure modes: - Missing context: models could invent details when relevant information was absent. - Accumulated near-duplicates: similar memories could clog a naive retrieval system. - Self-reinforcing interactions: agents sometimes became stuck complimenting one another, or adopted a frustrated stance and refused to proceed. These were anecdotal failures, with no measured frequency. They suggested that higher-level planning was vulnerable to stochastic noise—the variability in model responses—as well as errors in individual answers.

Encoding the larger process in software reduces how much the system must reconstruct through conversation. The model still does useful work within each stage; the program preserves the intended workflow. That division of responsibility was Schillace’s practical response to drift in the systems he was building.

Textbook Factory: a thousand calls, one explicit workflow

Textbook Factory made the division concrete. Schillace described a Jupyter notebook with roughly 200 lines of Python and six short prompts. It extracted a subject and educational level from a request such as ninth-grade US history, then made approximately 1,000 API calls to generate a full-year textbook.

The notebook encoded the high-level procedure: 1. Determine the curriculum for the requested subject and level. 2. Divide it into chapters. 3. Generate lessons and lesson plans. 4. Produce a teacher’s guide with answers and a table of contents. The model supplied the content within this structure.

Reliable output type did not establish content quality. Schillace said the program reliably produced textbooks, but could not produce cookbooks or novels. Its narrow structure was both useful and brittle. The reliability claim concerned staying within the intended output domain; it did not establish factual accuracy or educational effectiveness.

The broader aim was more output per unit of human effort. Many AI products still required people to guide the model through each step. Where developers could identify a sufficiently clear problem pattern, encoding it could reduce that supervision. Schillace cited sales support and codebase tuning as other promising domains, without reporting measured outcomes.

His proposed next step was to give models reliable building blocks through domain-specific languages and richer planning operations. He described discussions about wrapping Power Automate and other Microsoft functionality into components models could invoke.

In his illustrative architecture, a model generates a 10–20-line plan that invokes existing implementations. An executor runs the plan, calling APIs that might encapsulate 100,000 lines of code. Those numbers illustrated an abstraction boundary: reuse substantial functionality and concentrate generation on selecting and composing it. Better future models could produce better plans without regenerating the underlying components.

Build while the implementation is still painful

Asked how senior technical leaders should remain hands-on, Schillace offered personal motivation rather than a time-allocation formula: he enjoys making things. He called programming with semantics and natural language the most interesting development in his 35-year career. He nevertheless avoided directly taking on production code because it would absorb his attention, working with engineers and designers on broader implications, patterns, and experiments.

His experience building Writely explained why rough tooling did not discourage him. Early JavaScript performance constraints made character-by-character string comparison impractical. To locate edits, he recalled using binary searches at both ends of a string, while inconsistent browser implementations added difficulty.

His opportunity test was visible potential, continuing controversy, and painful implementation. When engineers can imagine what a technology enables but still feel uneasy about how hard it will be to build, there may be room for consequential work. Once implementation becomes easy, he argued, the field is likely to be crowded.

The same stance limited the authority of his laws: they were provisional guesses distilled from building. He urged people to attempt real tasks and encounter failures before forming strong opinions about AI. Another participant agreed that the podcast could help listeners discover what to investigate, but listeners and hosts still had to do the practical work themselves.

Multimodality challenges text as the common interface

A participant challenged one principle directly: could text remain the universal wire protocol—the common format passed between components—as models handled images and other media? Schillace accepted that the principle might not hold. But text need not mean natural language. He offered TypeChat as an example of eliciting more precise structured responses from large language models, describing it as producing syntactically correct JavaScript.

The practical obstacle was programmatic control over multimodal operations. His desired interaction was to locate a dog in a picture, mask it, and enlarge it through textual instructions. Some of those capabilities were emerging, he said. The useful interface would let software direct specific operations on an image.

He also revisited a failed prediction. When DALL-E appeared, he expected generating images from descriptions to threaten Photoshop. By this conversation, Photoshop was incorporating more tools instead. His revised architectural guess returned to Unix: pipelines carrying both text and binary data, preserving compatibility with existing systems. Text could remain useful without being universal.

Research, products, and principles still open to revision

Asked how an outside AI engineer should understand Microsoft’s portfolio, Schillace described a spectrum: - Research: projects such as AutoGen and Phi, with Phi exploring small language model efficiency. - Research-to-product work: groups such as his own investigating patterns and carrying ideas into product development. - Developer and end-user products: Azure OpenAI alongside AI features in Office and Windows. He pointed to Azure AI Studio as a way to build and deploy long-running chatbots using an organization’s data through retrieval-augmented generation, or RAG. He did not present an implementation or evaluation.

He mentioned Azure credits through Microsoft for Startups Founders Hub, tentatively recalling roughly $2,500 initially and up to $150,000 over four years. These were qualified recollections of the offer at the time. He also described work spanning everyday Copilot use, silicon, models, and AI for science, expressing enthusiasm for forthcoming science work without identifying projects or results.

On leadership, Schillace praised Satya’s engagement with scientific details alongside product and business questions. For his own conference participation, he preferred Q&As and direct conversations with engineers to keynotes. He said he would probably attend the World’s Fair to hear what others were building, without confirming a stage appearance. The organizers described the leadership track as a closed-door unconference where participants could discuss their issues.

His closing qualification governed the technical advice: the laws were his best guesses. Some ideas would prove useful and others wrong. Working out which was which would require messy experiments, false starts, and conversations with other builders.

The episode closed by announcing the AI Engineer World’s Fair for June 25–27, 2024, in San Francisco, with an April 8 deadline for proposals for talks, workshops, and discussion groups.