Building LLM-native products before the ChatGPT API

The infrastructure behind developer success

Logan Kilpatrick recalled trying to create a Wikipedia page for NumFOCUS and having it rejected for insufficient notability. The nonprofit provides legal and fiscal support for scientific software projects including Julia, Jupyter, pandas, and NumPy. He pointed to tools used in the first black-hole image and Mars-rover work as examples of a peculiar gap: widely used projects, but little recognition for the organization supporting them.

Community support connected Kilpatrick’s engineering and advocacy work. After roughly two and a half years teaching Django through Harvard’s Division of Continuing Education, he joined the Django Events Foundation North America board and chaired DjangoCon US. Path AI’s use of Django connected those contributions to his job. Julia community work similarly gave him an outlet for advocacy while his Apple roles involved software engineering and training machine-learning models.

Moving into full-time developer relations at OpenAI made that volunteer work harder to sustain. It also changed the immediate problem: OpenAI already had abundant attention. Kilpatrick’s priority was helping developers build successfully, chiefly through better documentation. Colleagues had accumulated useful knowledge alongside other responsibilities; his job was to make that knowledge accessible to people building products on the API.

What conversation adds to a model

In this conversation published in February 2023, the ChatGPT API was still forthcoming. Host Alessio cited rapid user growth and unofficial API wrappers as signs of demand. Kilpatrick was especially interested in the chat-first experiences developers might build, while remaining uncertain whether conversation would endure as the preferred interface. Earlier chatbot enthusiasm had faded; a natural interaction style did not guarantee lasting appeal.

Kilpatrick distinguished ChatGPT, then a research preview, from its underlying GPT-3.5 model, which he said was not a research preview. He believed hallucinations—plausible but untrue responses—could be reduced. Developer experimentation was offered as a promising route, though no concrete mitigation or measured improvement accompanied that expectation.

The forthcoming API was described as conversational first, with tuning and an interface intended for conversation context. By contrast, text-davinci-003 accepted a prompt and returned a response. The respondent was unsure which differences had been publicly announced, so this was a provisional description. Developers could already construct a conversational layer around 003 or 002 through response instructions and data formatting.

That implementation claim was narrower than saying developers could have built ChatGPT six to nine months earlier. When a participant recalled that assertion, another challenged the timeline because davinci-003 had not been available that far back. The respondent conceded the stretch. Prompting could support conversational applications; that did not establish equivalence between earlier offerings and ChatGPT.

Examples teach users what to ask

A blank prompt box creates a discovery problem: users may not know which tasks a model can perform. One participant needed to see a task demonstrated before thinking to request it. The OpenAI Cookbook collected those demonstrated uses, while research citations connected practical guidance to the underlying literature. The respondent credited Ted and Boris with much of its tutorial work.

Examples also made operational advice usable. Rate-limit documentation paired retry and exponential-backoff guidance with code using three libraries. The respondent declined authorship credit: the material already existed in the help center. Their contribution was supporting a documentation-first culture and bringing existing knowledge into developer-focused documentation.

Prompt engineering—shaping instructions and input formatting to elicit useful responses—was described as more art than science at the time. Systematic approaches existed, but understanding capabilities and reliably invoking them still required experimentation. One speaker expected reusable prompt collections to make specialized prompting expertise less scarce. Their experience with DALL-E illustrated the mechanism: suggestions displayed during image generation introduced pixel art and cyberpunk styles that they then began requesting regularly.

Conversation reduces the burden on the first prompt. Users can refine requirements after seeing an answer. A participant argued that a suitable retry interface could have supported similar interactions with 003. The accompanying reliability argument was conditional: imperfect generations can remain useful when retries are cheap and mistakes have no critical safety consequences. The hypothetical example of a model being wrong half the time illustrated those conditions; it was not a measured error rate.

Access creates opportunities; shared models demand differentiation

Personalized tutoring offered a more ambitious application. A speaker predicted that within three years, technology could support a chat-based education experience for every student, adapting curriculum to questions and engagement. The envisioned system would supplement teachers and potentially handle grading. Similar optimism extended to low-cost mental-health services. These were forecasts about access and future products, without demonstrations of educational or clinical effectiveness.

Legal assistance brought institutional constraints into view. Participants recalled a DoNotPay proposal to pay someone to wear an AirPod and repeat model-generated guidance in court. The account came from a social-media thread, and the speaker was uncertain which legal body had allegedly threatened consequences. Beyond that episode, the opportunity they saw was broader: helping people who could not readily obtain legal representation or answers about what to do next.

LLM-native products are products designed around large language models from the outset. Their proposed short-term advantage was a coherent experience: a new product could organize its workflow around generation, whereas simply adding a prompt beneath an established search bar could feel awkward. For longer-term differentiation, the respondent emphasized unique domain data that could support fine-tuning—adapting an existing model—or specialized models, using legal and educational data as examples.

A question about whether Perplexity would defeat Google received a restrained answer. The respondent had not used Perplexity; novelty might attract trials, but lasting success required continued differentiation. Their demanding founder heuristic was to identify ten reasons the business was different. The underlying concern was straightforward: competitors could access the same API.

Jasper made interface value concrete. A participant who had been a customer before joining OpenAI valued its positioning, simple interface, and prepared prompts for tasks such as product descriptions. Customers could specify what they wanted without discovering the prompting method themselves. That explained its immediate usefulness; its durable competitive advantage remained an open business question.

Embeddings have an ongoing query cost

For builders exploring the available tools, the respondent recommended focused playground examples, such as explaining a topic to a second grader, and experimenting with embeddings. Their mechanics were assumed rather than taught in the conversation. Readers new to them can start with the companion’s linked introduction to word and sentence embeddings, a distinction participants highlighted as useful background.

The enthusiasm included a relayed $50 million estimate for embedding the whole internet. With no defined corpus, token count, or pricing assumptions, it was a speculative scale illustration, not a usable budget. Claims that OpenAI embeddings were cheaper and better on academic benchmarks likewise lacked comparative prices and benchmark scores. The respondent recommended OpenAI’s newer embedding-model announcement for changes and metrics.

The practical cost distinction was between processing source data and serving queries. In the workflow discussed, incoming queries also required embeddings, adding recurring expense and another API call’s latency. The reply emphasized a small incremental monetary cost, but the questioner stressed that an inexpensive request could still add time to every interaction.

Evaluate model upgrades against real product usage

The GPT-4 discussion rejected circulating graphics and exaggerated expectations without offering confirmed specifications. The more useful question was how to adopt a replacement model. Developers were advised to expect trade-offs and breaking changes, including changed responses to existing prompts. The embedding-model transition illustrated why a replacement needed testing in the application’s own environment.

The proposed evaluation process started with the product: 1. Identify how people actually use it. 2. Turn those interactions into automated evaluations. 3. Compare candidate models, including other providers’ models, on those use cases. The respondent described comparison tooling as missing at the time and predicted that academic benchmarks would saturate. Application-specific evaluation would answer the consequential question: which model works better for this use case?

LangChain and DUST were discussed as frameworks supplying software-engineering structure around volatile model behavior. Repeatability presented a related challenge: participants reported that identical API calls could return different outputs even at temperature = 0. One reported a documentation update making this explicit. Their suggested explanation involving floating-point GPU behavior was tentative and secondhand; the observed behavior was the firmer basis for engineering decisions.

Working code can still answer the wrong question

A daily Codex user described asking for a function to retrieve a Twitter bio. The generated code looked plausible, but called an endpoint the developer could not find in Twitter’s documentation. Coding assistants consequently invited two interpretations: a junior engineer whose output needs correction, or a more experienced helper enabling a junior developer to make progress.

Conversation inside VS Code offered a way to refine suggestions beyond autocomplete. A participant also appreciated Copilot presenting multiple candidate completions. They expected larger models and longer context to improve coding assistance rapidly, potentially reaching principal-level help, but this was an informal forecast rather than a benchmark result.

A React-hook example exposed a deeper limitation. ChatGPT helped one developer get the code working, but reviewers in Discord questioned its quality and approach. Another participant with little React experience reported successfully using ChatGPT to explain and modify existing code. These experiences showed different aspects of usefulness: producing expected behavior does not establish sound design.

The hook’s reviewer called the problem sycophancy: the model accepted the premise of a narrow repair request, while a human engineer questioned the approach itself. A proposed improvement was control over the model’s interaction style, allowing users to request challenges to their assumptions and clarifying questions. This was a desired capability, not an available feature in the form the participant wanted.

Personalization needs an understanding of the learner

Returning to education complicated the earlier optimism. Saving conversation history is different from understanding a learner: what they know, where their conceptual gaps lie, and when to challenge them. One participant saw this as a major barrier to AI tutoring, since learners cannot reasonably enumerate everything they do not understand. The related essay “What is AGI-hard?” was raised in connection with that problem.

A proposed tutoring workflow made the ambition concrete: 1. Supply previous work, such as calculus notebooks and assignments. 2. Infer possible gaps and generate diagnostic questions. 3. Add concepts the learner cannot explain to a personalized curriculum. Khan Academy provided an existing comparison through assessments that directed learners toward relevant videos and assignments. The conversational language-model layer remained the proposed addition.

Creation, adaptation, and everyday use

The closing product discussion focused on reducing the work of making educational video. A proposed workflow combined a ChatGPT-written script, an avatar presenter, and automatically generated slides; ChatBCG was named as a slide-generation demo. Another ambition was to turn an existing blog post into TikTok and YouTube pieces presented by a personal avatar. These were envisioned workflows, not demonstrated end-to-end results.

An apparently human presentation could leave viewers feeling deceived when they discovered it was generated. The suggested response was disclosure at the beginning, alongside a prediction that some brands would make AI-generated content an explicit part of their identity.

The infrastructure theme returned with recognition for the machine-learning operations community, or MLOps, which participants felt generative AI had overshadowed despite its continuing importance. Asked what would surprise people a year later, a respondent predicted rapidly increasing personalization and many more AI interfaces, including experiences from Facebook and Google.

The tool they would pay for was easier transfer learning and fine-tuning: adapting existing models with users’ own material instead of starting from scratch. A participant said their own machine-learning work at Apple had used transfer learning. They wanted accessible products that would let more people apply that approach to personalized AI.

A questioner offered Stable Diffusion customization as an analogy: sharing adapted model variants and training concepts through techniques called textual inversion and DreamBooth. The respondent acknowledged limited familiarity with the image-generation community, while anticipating marketplaces for reusable, specialized model adaptations. A free sharing service was mentioned with the name transcribed as “Civit.ai”; the companion links to Civitai. PromptBase raised a related disagreement: whether prompts should be freely available text or could be sold as knowledge.

The closing advice was to incorporate AI tools into everyday workflows. The speaker expected change to accelerate and hoped wider access would spread the benefits while reducing negative consequences. A final exchange about Twitter tempered the pace of that outlook: expectations of a broad departure had faded, and network effects remained hard to break.