From impressive outputs to practical AI

An impressive answer still needs a reliable workflow

An enterprise tries ChatGPT, gets an impressive answer, and encounters two harder questions: can it use the model under its privacy requirements, and can its software reliably consume the output? Free-form text may delight a person while frustrating an automated extraction pipeline. Dan Whitenack identifies inconsistent output as an integration problem that remains after a model demonstrates its capabilities.

Whitenack brings another constraint to the Latent Space–Practical AI crossover. He spent the preceding five years building a team at SIL International, an international organization doing language-related work, to develop translation and speech recognition for unsupported languages. For those users, the first question is whether the technology serves their language at all.

Practical AI grew out of his connections in the Go community: a conversation with Changelog’s Adam Stacoviak, followed later by Chris Benson’s proposal to co-host a show. Whitenack wanted listeners to learn something useful in daily work; Alessio identifies the same priority for Latent Space. The crossover examines that promise through audience favorites and personal episode picks.

Whitenack’s [Prediction Guard](https://www.predictionguard.com/) began with large language model (LLM) evaluation and selection, then shifted toward enterprise adoption. He describes two responsibilities: - Operate within enterprise requirements: address compliance concerns in how models run. - Structure and validate output: combine approaches associated with Guardrails and Guidance with self-consistency and factuality checks. He does not specify compliance standards or checking algorithms, or quantify their effectiveness.

Record expectations before outcomes are clear

Swyx values podcast archives as an oral history of AI: they preserve what practitioners thought before outcomes were clear. Practical AI’s guest-free “fully connected” episodes build practical mental models of training, behavior and outputs through subjects such as ChatGPT, Stable Diffusion and AlphaFold. Its AI for Africa series highlights communities building translation, speech recognition and agricultural applications for their own circumstances.

The favorites capture excitement and its limits. One participant recalls Mike Conover’s enthusiasm for the Red Pajama dataset, which reportedly appeared the morning of their interview. A Twitter Spaces discussion of ChatGPT plugins drew roughly 4,000 listeners after about an hour of preparation. By the crossover, plugins had disappointed one participant relative to their initial expectations—but the recording preserved those expectations for comparison.

Excitement does not predict lasting use. One participant had considered leaving natural language processing—the field concerned with language tasks—during the Stable Diffusion wave, then found themselves working entirely on language models. Another built a Stable Diffusion web interface but subsequently went six months without opening the model. This distinction between attention and working habits returns in the discussion of enterprise adoption.

LLMOps changes the work around the model

Metaflow, a Python package developed at Netflix, addresses versioning, orchestration and deployment. The notebooks-to-Netflix episode reportedly received roughly 30% more listens than any other Practical AI episode. Its appeal was hypothesized to come from the problem of getting notebooks into production.

Successful inference is only one part of a model’s lifecycle. Producing an answer leaves teams responsible for tracking versions, coordinating workloads on their infrastructure, incorporating their datasets and fitting data science tooling into an organization that also runs conventional software.

LLMOps—operations around large language models—changes the emphasis of machine learning operations, or MLOps: - Task-specific modeling: gather data, write training code, serialize the resulting model and deploy it. - Foundation-model applications: select a released pretrained model, design prompts and chains of processing, and consider fine-tuning to adapt it with additional data. Operational work persists, but more of it happens around an existing model. The participants still expect task-specific models, including scikit-learn models, to retain a role where explainability matters, with finance offered as an example.

Evaluate the behavior the application needs

Replit’s code-model episode leads to a consequential mismatch. Recalling an earlier interview, a participant attributes to MosaicML’s Jonathan Frankel the observation that many benchmarks ask multiple-choice questions while production applications often require open-ended generation. Selecting an answer and generating useful text require different evaluations. A strong score on the former does not settle the latter.

Model-based evaluation gives an evaluator model a context and a response, then asks it to assess the response. The discussion connects this approach to COMET, Unbabel’s learned evaluator for machine translation. It also revisits benchmark-driven development: as models catch up with tests, designers must find more demanding evaluations. Strong GPT-4 benchmark performance is not treated as proof of general intelligence; model-generated evaluations remain unsettled.

Models also participate upstream by helping create training data. A participant uses [Segment Anything](https://www.latent.space/p/segment-anything-roboflow#details) to illustrate extensive AI assistance in data creation. This creates a loop: models help generate data, models train on that data, and models may then evaluate the outputs. Synthetic data—data generated or simulated rather than collected directly—is longstanding; the notable change is the scale and flexibility of combining models during training and use.

Synthetic data cannot supply every missing language

That loop raises a concern one participant calls mode collapse: repeatedly building on model outputs may favor common use cases at the expense of less represented ones. The proposed response combines broader linguistic coverage in foundation-model training with community-created datasets for adaptation.

Arabic vernacular adaptation makes the mechanism concrete. A participant explains that standard Arabic in pretraining makes subsequent fine-tuning for a vernacular easier. But generating more model output will not necessarily supply Central Siberian Yupik if the model rarely produces it. Broader pretraining and local data collection address different parts of the gap. A community effort from Cohere for AI to improve multilingual representation is offered as a promising direction.

Masakhane, a grassroots organization of African natural language processing researchers creating technology for Africa, also challenges assumptions about what to build. Translating Wikipedia into every language may sound universally useful to a text-oriented developer. Some communities instead prioritize speech because their language use is primarily oral or literacy is limited. Local knowledge determines the useful form of communication, as well as the language.

Agriculture illustrates the same need to choose the application locally: - Precision spraying: a participant describes some John Deere tractors running models on an onboard Kubernetes cluster, with cameras identifying individual weeds for targeted spraying. - Other local priorities: disease detection, drought identification and disaster relief may better serve particular African settings. The argument is for technology developed by people who know the domain. The discussion also encourages contributors outside those regions to help grassroots efforts.

Choose models by constraints; keep operations proportional

The Capabilities of LLMs episode supplies independent questions for navigating the growing model landscape: - Downloadability: can you obtain the model weights? - Commercial permission: does the license allow the intended use? Downloadability alone does not answer this. - Language coverage: does the model support the required languages? - Task specialization: is it suited to code generation, general text generation or another task? These distinctions help frame whether to run a downloadable model or use a hosted provider such as OpenAI, Cohere or Anthropic.

Hugging Face download statistics can suggest models worth trying, including forks and fine-tunes that receive little attention elsewhere. A participant uses repeated downloads as a discovery signal, not a quality measurement; licensing still needs investigation.

Sharing datasets can also extend local work into larger releases. A participant reports that SIL, Masakhane and a speech technology company created datasets and models for roughly six African languages. Meta later found the data on Hugging Face and incorporated it into newer models. The account leaves the specific Meta release unnamed.

The earlier MosaicML discussion introducing MPT-7B describes another route to participation: making it easier for people to train their own models. Small teams still need proportional infrastructure. Recalling Kirsten Lum’s advice, a participant recommends treating curation, training, fine-tuning and deployment as time-boxed tasks within a broader role. If version tracking consumes disproportionate effort, recording a number in a Google Sheet may sometimes suffice.

Build beyond the first prompt

At the time of the conversation, a workshop practitioner sees widespread ChatGPT use but much less integration across enterprise technology stacks. Workshop participants recognize potential value yet struggle to create problem-solving workflows. Their familiar sequence—collect features, train a scikit-learn model, deploy with Flask—no longer supplies the whole recipe.

The suggested progression, recalling a hierarchy discussed by Travis Fisher, explores increasingly involved forms of adaptation: 1. Improve the prompt, including examples or demonstrations of the desired behavior. 2. Augment with data, bringing relevant information into the workflow. 3. Fine-tune a model with company-specific data when appropriate. 4. Train a generative model if the application warrants it. Output controls and chains of processing belong in this exploration too. The practitioner mentions regular expressions, guardrails, Prediction Guard, LangChain and LlamaIndex as available tools. A disappointing first prompt should not automatically disqualify a use case.

This gives substance to the debate over “prompt engineering.” One participant considers the term overhyped when it means selling an individual prompt, while seeing real engineering in combining prompts, data and processing steps into useful workflows. Another proposes AI engineering: an emerging software specialty built around working effectively with AI APIs, research and tooling.

The transition creates different challenges: - Software engineers face uncertain behavior and limited control. Hosted models produce nondeterministic outputs, and developers do not control the underlying training or changes. Possible GPT-4 quality regression comes up as an unresolved question. - Data scientists must reconsider when to train. They know uncertain outputs, but often from systems whose datasets and training they controlled. Prompting, chaining and data augmentation may deliver value before further training becomes necessary.

The interface makes capability usable

AI-UX—user experience for AI—makes model capabilities accessible and usable. A participant tentatively suggests that this work may be as valuable as training itself. ChatGPT illustrates the relationship: its conversational interface matters, and that experience depends on underlying model and dataset work.

Two examples show AI integrated into an activity already underway: - Gmail autocomplete: gray suggestions appear while a person writes an email. A participant recalls immediate value without first encountering a separate AI opt-in pitch. - GitHub Copilot: GitHub reportedly spent six months refining the interface around the initial Codex model, arriving at suggestions embedded in code as the user types. This is recognizable software and interaction design. The conversation identifies it as a potential gap for data scientists whose interface experience has centered on visualization.

The same principle applies to model-building tools. Label Studio, an open-source data-labeling framework, is discussed for its then-recent generative-model customization tools and its combination of humans and LLMs in the labeling loop. Prodigy is cited as an earlier example of related human-in-the-loop tooling.

One client already knew how to train and fine-tune models but struggled to understand the workflow for creating an instruction-tuned model. The practitioner explained familiar ingredients—dataset curation, training and reinforcement learning—arranged around pretraining, human feedback and subsequent reinforcement learning. The tooling opportunity is to make those stages approachable without requiring users to master the underlying algorithms first.

Data mixtures make model selection application-specific

One participant proposes a distinction in how enterprises approach customization: - Bring data to an existing model: adapt a capable model through augmentation, fine-tuning or training. - Buy a platform for custom model development: start with infrastructure for building models around company data. Their view is that enterprises often start with the first approach, making an API that accepts training examples appealing. Availability was model-specific: GPT-3.5 and GPT-4 did not offer fine-tuning at the time of the discussion.

For teams training models, opaque data mixtures leave a consequential variable unresolved. When reproducing a leading model falls short, its undisclosed training mix becomes one candidate explanation. The discussion separates two strategies: - Filter and mix existing datasets: address quality problems, biases and unwanted content through selection. - Add synthetic data: supplement those datasets with outputs from another model. The best combination remains unresolved. A participant’s recollection of a Common Crawl study illustrates the concern: data labeled as Arabic written in Latin characters reportedly was not Arabic. The study and its sampling scope are unspecified, so the anecdote does not establish an error rate for Common Crawl overall.

Different training mixtures can produce different task strengths, so test several models on the actual application, including models below the leaderboard leaders. A participant describes a smaller Writer model that produced useful marketing copy with workflow support and output structuring, was small enough for them to host and run, and was unsuitable for other tasks. This was a qualitative experience; the proposed connection to Writer’s training focus was a guess.

Explore generality without losing sight of coverage

The closing lightning round asks what arrived sooner than expected. One participant points to LLM usefulness beyond familiar language tasks. At a fraud-detection workshop, they chained prompts and supplied insurance-transaction examples to try to identify a fraudulent client, reporting substantial progress. The experiment suggests a use case to investigate, without an accuracy measurement or comparison against the workshop’s statistical models.

The unsolved question returns to language coverage. A participant describes a sharp performance drop beyond well-supported languages, including weak results for some languages spoken by millions. This is a broad observation rather than a model-specific comparison. The problem also extends beyond text: written language represents only part of human communication. Sign language and other forms of communication, combined with broader multilingual coverage, offer opportunities to understand both communication and current models’ limits.

The final recommendation is to build intuition through hands-on work: move beyond a simple prompt, inject your own data, build a query index, connect processing steps and try agents. Exploring those tools is how developers learn what models can do and how to assemble them into useful workflows.