Instructor: Typed Outputs, Explicit Workflows, and the Direction of Agency

Start with the maintenance problem

At Stitch Fix, one recommendation team served small, real-time workloads with FastAPI and pandas while another processed larger workloads in Spark batch jobs. Jason Liu kept refactoring their bespoke systems separately. Flight, the recommendation framework he built, centralized that maintenance: shared pipelines let one implementation support improvements across teams.

Standardization also made individual pipeline operations easier to inspect. Liu could identify slow Python code, missing caches, and duplicated computation, then work with the responsible teams to fix them. That connection between explicit structure and practical improvement runs through the Latent Space conversation.

Liu initially doubted that language models justified their infrastructure costs. Stitch Fix used transformers for recommendations before GPT-3, but he preferred classical models and questioned sending an API request for every customer comment. ChatGPT changed his assessment: he describes apologizing to colleagues and shifting from computer vision and recommendations to large language models. Their growing use for retrieval felt like a return to recommendation-system problems; swyx argues that developers should study that field before reinventing its concepts.

His earlier work used ResNets—image models fine-tuned here to predict merchandising attributes and text representations of clothing comments. The resulting image vectors augmented recommendations for similar pieces, complementary items, and outfits. These were custom embeddings, rather than off-the-shelf GPT-3 embeddings. With roughly three million inventory pieces and more than a billion user–clothing interactions, Liu expected fine-tuning to outperform a generic model, though he supplies no comparative benchmark.

Liu says displaying another 15–20 products on a product page increased revenue, without quantifying the lift. Yet recommendations still struggled with personal narrative: his enthusiasm for finding an unusual denim shop in Japan includes the journey and the story. His contrast between selling a $20 shirt and a $500 shirt asks how much of that motivation an item-matching system can capture.

From clothing requests to typed responses

A hand injury around 2020 interrupted Liu’s coding for roughly one to two years, split between medical leave and technical leadership. After winding down technical work in New York, he spent about five months experimenting with prompts. Startup friends seeking an AI strategy gradually turned those experiments into full-time work. [Instructor](https://github.com/jxnl/instructor) began as a small wrapper around OpenAI’s software development kit, written on a bullet train in Japan.

A clothing-search workflow captures the problem it addressed. A request for jeans of a particular fit and size needed to become JSON for an internal search API. Application code still had to parse the response and check field types. Function calling let developers specify the output schema—the expected fields and structure—separately from instructions and input data. Pydantic represented those schemas in Python, with methods and validators for working with and checking the results.

Instructor’s boundary was deliberately narrow: handle response models and routine parsing while preserving access to chat messages. Libraries that controlled string formatting could become awkward around chat APIs with explicit message structures. The design ideal was Python’s Requests: an obvious dependency that does not define the application’s identity. Instructor could sit inside another framework just as Requests could sit inside Django; success would mean developers expected its behavior directly from model-provider SDKs.

Typed responses are the payoff: objects whose fields support autocomplete and editor feedback when used incorrectly. Function calling and JSON mode are ways to get there. In the APIs discussed, JSON mode could produce valid JSON without specifying the required structure. A list of people with names and positive ages demands more than successful parsing. Function calling was therefore preferred when schema mattered, while JSON mode could save the prompt tokens consumed by function definitions.

Separate extraction, selection, and execution order

For extraction, one schema describing the whole result can express relationships among entities and return a directly usable object. Choosing separate actions, such as checking weather or controlling lights, creates a different case for multiple function calls. Instructor’s agent support was limited, and the practical experience offered with large tool sets was modest: one participant had rarely needed more than six or seven definitions.

For larger tool sets, one proposal is to retrieve candidates first: store tool descriptions in a vector database, rank them against the request, and expose only the top k candidates to the model. But similar descriptions can frustrate similarity search. In a reported integration project with approximately 780 endpoints, many specifications used nearly identical wording; Opus reportedly selected relevant endpoints well where vector similarity struggled. No quantitative comparison accompanies that experience.

A counterproposal uses a trained XGBoost ranker, taking query and tool-description embeddings as inputs. Its objective would be to keep needed tools near the top of the candidate list. This remains a proposed approach. Selection also leaves execution order unresolved: when actions depend on one another, the suggestion is to return the entire dependency graph as one JSON object. How reliably models could generate the required ordering remained uncertain.

Production needs usable data, not just valid JSON

Production schemas exposed failures that tests had missed. Workflows building and updating lists received numbers as strings, or lists whose elements were JSON-encoded strings rather than objects. Those elements required another parsing step. One participant reports problems in roughly 0.5% of production traffic, including conversational answers to JSON requests and function-call parse errors. That figure comes from private client workloads, with no workload size or measurement period specified; public benchmarks had not been reproduced.

Those failures coexisted with enthusiasm about Sonnet’s cost. A roughly 70% reduction was discussed as potentially making consumer applications viable or enabling better pricing, although the baseline model and calculation were unclear. The practical distinction was between low-priced consumer products, where per-request savings could change the business, and expensive professional services, where the same savings mattered less.

Instructor’s broader usefulness follows from giving ordinary algorithms data structures they can manipulate. Three applications illustrate the range: - Structured extraction: convert a receipt image into checkout items with prices, fees, and coupon codes. - Graph extraction: produce nodes and edges for task dependencies or family relationships in a story. - Query understanding: translate a natural-language request into an existing API’s schema.

Query understanding shows why structure matters. A request about the latest events this week is interpreted as a date filter covering today and the preceding seven days. Comparing writing this month with last month calls for grouping by month and summarizing the groups together. Similarity alone does not specify those operations. A model can produce the structure that tells application code what to filter, group, and summarize.

Make plans inspectable and repairable

The workflow proposal extends this principle to execution. A workflow here is a directed acyclic graph, or DAG, whose nodes represent work and whose edges describe dependencies, in the style of Prefect or Zapier. The advocated approach generates a complete workflow for a request. Request–workflow pairs then offer a concrete fine-tuning target, whereas the speaker finds the training objective for an open-ended reasoning loop less clear.

An explicit plan creates a smaller unit of recovery. If the connection between nodes x and y fails, repair that component and continue. With enough examples, retrieval could supply relevant demonstrations to guide the repair. This suggests a progression from modular workflows to better prompts, examples, and eventually fine-tuning for individual components.

Agent experience complicates the picture. One participant reports Devin succeeding on simple tasks but churning on work described as taking ten or twenty hours; another mostly runs sub-minute executions. An AutoGPT anecdote makes the concern concrete: an overnight attempt to buy a bicycle spent $7 in credits without finding one. Runtime alone is a poor measure of progress. More promising was Devin’s reported ability to revise a plan in response to its performance, rather than persist indefinitely with one approach.

A changing plan also poses an interface problem: how should a graph display both revisions over time and uncertainty about what comes next? The discussion leaves the eventual interface open, while favoring modular components that can be improved individually.

Observability must cover the surrounding system as well as the model. One participant uses Datadog or Sentry for system monitoring and stores model inputs, outputs, latency, and token costs in Postgres. That simple instrumentation met their needs for mostly extraction and workflow calls, drawing on prior recommendation-system experience. The preference was specific to that workload, which did not involve complex agent loops.

Let a focused library stay small

Instructor’s narrow scope also informs the case against venture funding. A utility modeled on Requests need not acquire a vast integration catalog. The stated strategy was to prioritize OpenAI and accept contributed platform support. Consulting offered access to interesting data and applied problems: construction and insurance automation, memory systems, and reports assembled from interviews.

The alternative business goal was explicitly aspirational: a consulting practice earning a million dollars with roughly three staff would be satisfying. A billion-dollar company was not the necessary destination for a useful library. Another participant challenges the assumption that technical insight and open-source success should automatically lead to fundraising.

Fundraising and searching for a required cofounder can also displace building. The proposed threshold for raising money is conviction about a particular problem, the ability to proceed independently, and a concrete plan for spending the capital. Keeping the project small is part of the technical thesis: a focused tool can be enough to do useful work.

Agency needs direction

The career discussion turns to feeling too late to enter AI. The response draws on jujitsu: waiting until one is fitter or feels ready postpones practice while time passes anyway. Rock climbing offers a similar contrast. Renting shoes and falling repeatedly creates experience; researching ideal equipment can become a substitute for starting.

Agency has direction as well as magnitude. Acting readily matters, but effort alone does not select worthwhile work. Pottery supplies a process metric: track pounds of clay used rather than finished cups. Software analogues include consistent commits, open-source contributions, and documentation. When a questioner asks whether this justifies counting lines of code, the response rejects the metric: expanding a loop could inflate it without improving anything.

The more precise proposal comes from a science perspective: value experiments that reduce uncertainty. Three or four unsuccessful experiments may eliminate unpromising avenues without immediately increasing revenue. Agency increases the volume of attempts; experience helps choose their direction. The point is to keep learning through attempts, rather than expect a guaranteed success rate.

Failed experiments need a record because tools and constraints change. An engineer can remain discouraged by an old result long after its cause disappears. Document the conditions of failure: - The conditions under which the experiment ran. - The inputs tested and outputs observed. - The changes that would justify trying again. The last item gives the record a future use. An earlier rap-lyrics experiment, for example, was attributed to poor training data; an approach no longer requiring task-specific training data could make it worth revisiting.

Discover the work before automating it

Personal assistance exposes a problem that more connectors cannot solve. After finding AI writing and assistance inadequate, one participant hired a writer and a personal assistant to discover reproducible steps that could eventually be automated. Connecting Gmail, Calendar, and Notion would provide access; it would not define what useful assistance meant.

Working with people revealed concrete requirements: turn meetings into useful weekly client updates, notice an overloaded Monday, coordinate rescheduling, and protect preparation breaks and lunch. These were needs the speaker previously would not have known to include in a prompt. Understanding them was progress, though reliable automated execution remained unresolved.

That discovery process explains the preference for keeping prompts close to developers. Instructor separates instructions, data, and output types: define the output model in code, send its definition to the language model, and receive a data structure. As a product’s objectives become clearer—selling supplements, booking fitness appointments, or supporting upselling—developers may need substantial prompt changes. Prompts embedded deeply in a framework can obstruct those adaptations.

There is still a substantial audience for managed products. Instructor serves developers building their own tooling; many downstream buyers want the work handled for them. A hypothetical organization might decline a 10% improvement if it required hiring another 20-person team. The preference for direct control is acknowledged as personal and specific to a small niche; many companies are happy to select vendors that help them move quickly.

Optimize against the feedback you can obtain

A questioner introduces DSPy, a contrasting approach to prompt optimization, and argues that AI can be a better prompt engineer than a person. The response is conditional: optimization can work when success has a short feedback loop, such as detecting personally identifiable information. Reducing customer departures and improving retention produce slower signals. Even judging whether a summary is useful can take longer than an automated optimization loop assumes.

Model changes add another constraint. The respondent reports difficulty preserving behavior across models and substantial infrastructure work when migrating between Anthropic and OpenAI. Prompt optimization addresses only part of that maintenance burden.

Hire for the application and its outcome

The closing hiring discussion asks what skills these applications need. One participant argues that startups without data, tests, or clear business outcomes can frustrate machine learning engineers by assigning them ordinary application repairs. Meanwhile, motivated internal engineers who built promising AI demos may be pulled back to calendar integrations and interface work. The recommendation is to examine that internal talent and ask whether the role actually requires writing PyTorch code.

AI engineering is presented as a legitimate software engineering subspecialty, with product engineering and data science increasingly important for defining and measuring success. Reduced churn is a business outcome; building an AI application is a means to pursue it. Existing data engineering and business intelligence tools remain relevant to that measurement.

The conversation closes with the proposed World’s Fair talk title “Pydantic Is Still All You Need” and the observation that Pydantic is someone else’s project. The final recommendations point listeners toward Instructor and InstructorJS—a fitting close to the case for useful components that developers can make their own.