Humanloop and the engineering of a feedback loop

Choosing the next experiment

A car in a wind tunnel offers many possible experiments. Adjust the ride height at each corner and test every combination, and much of the information will overlap. While helping build Monolith AI, Raza Habib worked with McLaren on a more selective approach: use the latest results to recommend the experiments likely to teach the team the most. This is active learning—choosing what to measure next so a model learns more efficiently.

Habib’s UCL PhD, spanning 2017–2022, combined Bayesian and graphical-model approaches with deep learning. Alongside research, he worked on Google AI’s speech-synthesis teams and helped Monolith apply machine learning to physical engineering. At McLaren, that included surrogate modeling: investigating whether neural networks could replace expensive simulation solvers. He credits company work with grounding his research in practical problems, while cautioning against copying his overlapping commitments.

Early Humanloop used similar techniques, but Habib identifies two obstacles to selling active learning: - Integration depended on the model. Supporting arbitrary customer models was difficult; owning the model made integration easier. - Avoided labeling costs were hard to prove. Customers rarely paid for random labeling as a control, leaving the savings difficult to measure.

Language models then reduced the need for annotated data. Humanloop’s eventual focus, explored in this September 2023 Latent Space conversation, was helping teams measure and improve large language model (LLM) applications through prompt management, evaluation and collaboration.

Better models move the bottleneck

Humanloop emerged from side projects involving Peter, Jordan and the responding cofounder. Acceptance into Y Combinator turned exploration into a company before the cofounder’s PhD was finished. Completing it later honored a grandfather’s wish and preserved a possible return to university teaching.

The original thesis preceded GPT-3. BERT and ULMfit suggested that pretraining on unlabeled text, then adapting the model, could enable many more enterprise natural-language processing (NLP) applications. Humanloop initially targeted two barriers: scarce machine-learning expertise and insufficient annotated data.

GPT-3’s ability to learn from context weakened the data-labeling thesis. InstructGPT made the new direction feel practical, according to the guest, prompting a pivot despite paying customers, reasonable growth and funding. The team reused the YC application as a compact business plan: what to build, whom to serve, how to charge and how large the opportunity might become.

The estimate was only 300–400 potential customer companies worldwide, mostly startups. The guest defends that estimate as roughly right at the time; what surprised the team was how quickly the market expanded. Its conviction drew on the “bitter lesson”: general learning methods with substantial compute and data often outperform specialist systems. When the interviewer questions whether confidence in continued scaling is understanding or belief, the reply acknowledges both.

The pivot changed the customer from ML specialists to product-focused software engineers, whom Humanloop found more willing to buy tools to reach useful outcomes quickly. Before committing, the team set a target of ten paying design partners in two weeks and reported reaching ten on day two. They were selling participation in developing a proposed prompt-engineering and evaluation tool, not a finished platform.

Customers’ workarounds explained the demand. They stitched together Excel, Grafana, Mixpanel and the OpenAI playground to version prompts, compare outputs and collaborate. Prompts substantially determine application behavior, but outputs are stochastic—they can vary between runs—and quality is often subjective. Conventional unit tests alone could not establish whether a change improved the product.

Three evaluation jobs, different evidence

Humanloop separates three evaluation jobs: - Development comparisons: determine whether changing a prompt, context or model improves the application before release. - Production monitoring: check whether behavior holds up across real customers’ varied inputs. - Regression checks: detect whether a change damages behavior that previously worked.

At the time of the conversation, production monitoring and end-user feedback were established parts of Humanloop. Offline development evaluations and regression testing were newer capabilities in private beta, with a public launch planned.

These jobs cannot always use the same signals. For a subjective product experience, the guest treats customer acceptance as the ultimate test—for example, whether a Copilot user accepts a code suggestion. Development evaluations use proxies for future user behavior. They can approximate that response, but cannot directly observe it before release.

Evaluation also crosses professional boundaries. Engineers, product managers and domain experts may all contribute to prompt design and judgments of quality, even when only some write application code. Humanloop emphasized larger organizations because they needed both shared tools and greater confidence in behavior before launch.

Capture feedback, then replay failures

Humanloop’s feedback features grew from systems early customers had already improvised. The product distinguished three signals: - Votes: explicit ratings, such as thumbs up or down—useful, but sparse. - Actions: implicit feedback from what users do with an output. - Corrections: edits that show how users changed generated material before using it.

Sudowrite’s fiction-writing editor makes behavioral feedback tangible. It tracked whether writers accepted suggestions, regenerated repeatedly, edited text before incorporating it or shared the result. The guest reports that these signals correlated with prompt and model quality, and that repeated experiments accompanied better traction and user enthusiasm—a qualitative account, without a measured effect size.

Capture useful edits before you need a training dataset. In the sales-email example, a model generates a draft, a person edits it, and the application records the edited version before sending. Those corrections can later support fine-tuning—further training on selected examples—as well as validation of prompt changes.

Humanloop connected production records to an interactive playground, where developers could reopen failures and run counterfactual debugging experiments: change a suspected cause and observe the result. - Test the retrieved context: supply the intended context and see whether the model succeeds. - Test the model choice: rerun a GPT-3.5 failure with GPT-4 and compare behavior. Either experiment could help explain a failure; neither substitution was guaranteed to fix it. Captured data could also be formatted and exported for fine-tuning.

Keep orchestration in code, make iteration shared

Humanloop positioned itself between base-model providers and application code or orchestration frameworks. Its philosophy favored keeping most orchestration in code, while providing dedicated tools for prompts, evaluation and data exploration. It described itself as model- and platform-agnostic, supporting developers using OpenAI, Cohere, Anthropic or open-source models.

LangChain’s emerging prompt-management and observation tools created overlap. The guest saw tight framework integration as an advantage for LangChain users but potential friction for teams using something else in production. Humanloop’s intended distinction was collaboration inside larger companies. Framework neutrality still allowed opinionated defaults, such as feedback categories drawn from customers’ production experience.

Planning for that product meant anticipating better models. The guest argued that GPT-4 with vision already left many useful applications to build, while expecting substantial further gains within a year or two. This was explicitly extrapolation from past progress, not a theory-based prediction. The strategic question was what developers would need if capabilities kept improving.

Cheaper inference—the cost of running models—was another expectation, with model quantization and more efficient hardware offered as routes. A hypothetical drop in frontier-model cost from $500 million to $1–2 million illustrated how access might broaden to startups. Those figures were a planning scenario; the participants did not establish an actual cost baseline or measured savings.

Enterprise demand changes the work

Move upmarket when demand justifies the overhead, the guest advises. Humanloop’s enterprise shift brought recurring security reviews and customer-specific questionnaires. It reported regular penetration tests and an ongoing SOC 2 audit process. Asked whether a Vanta trust report could replace that process, the response was uncertain: customers still expected SOC 2, even if Vanta made the work easier.

Build versus buy can mean doing both. Enterprises can purchase packaged tools while developing AI features for their own products. Gusto’s job-ad drafting feature illustrates the latter: a user provides a short description, and GPT-4 receives relevant state-specific information requirements alongside the drafting task. The guest recalls it reaching production within weeks, without claiming an exact timeline.

Humanloop offered virtual private cloud (VPC) deployments where needed, but not full on-premises deployment. Its first question was which concern the requested deployment must address. Contractual commitments or controls in Humanloop’s own cloud sometimes sufficed; financially regulated customers brought more sensitive information and additional auditing demands.

Applications were also becoming harder to classify by individual NLP tasks. The guest favored end uses, such as education, because one system could blend extraction, summarization and classification. The interviewer countered that simple applications might still dominate usage, but offered no measured distribution.

Humanloop’s observed progression began with writing assistants, followed by document chat using retrieval-augmented generation—supplying retrieved information as context for generation—and then broader applications. Duolingo Max illustrated expansion within one company: a conversational experience, language models used to evaluate it, and model-assisted content creation.

Logs become valuable through intervention

Humanloop planned a free tier to lower the entry cost for nonprofits and small companies. It would include almost the whole product, with charges growing alongside team participation and logged data. That raised a business question: would customers eventually outgrow the platform as logging became a commodity?

The guest conceded that each additional feedback point becomes less valuable at high volume, but argued that the workflow around the data becomes more valuable. Product managers, linguists and engineers could reopen cases, rerun them and add evaluations. Customers who proposed storing logs in S3 reportedly returned after struggling to connect an output to the model that produced it and quickly test an alternative.

Retrieval exposes the gap between storage and debugging. A log may contain an answer without recording which document supplied its context. When the answer fails, the team lacks the provenance needed to investigate. Humanloop’s product argument was to couple observation to intervention: inspect a failure, change a prompt or retrieval system, and run an experiment in the same environment.

The company-building advice emphasized focus before product-market fit: finding customers whose pressing needs the product meets. Humanloop reportedly stayed at four people for almost two years, hiring when demand overwhelmed its capacity. The guest acknowledged painful understaffing while defending deliberate hiring and a high bar. Following every customer request carried another risk: incremental improvements could distract from a stronger view of what future developers would need.

The planned free tier also made onboarding, documentation, demos and community support more urgent. At the time, Humanloop was hiring in London and San Francisco, with a developer relations engineer its most pressing opening. The guest considered helping developers succeed a dedicated responsibility, rather than someone’s part-time task.

Humanloop described product-market fit within specific niches, especially prompt management for teams in larger companies. Evaluations were the hoped-for next stage, with about ten companies in closed beta. Evaluators could be written in Python and call language models. Hosted execution used what the guest described as a protected environment; customer-infrastructure execution with logs sent back was planned. No isolation mechanism was detailed.

Foundation Model Ops and the prompt artifact

The guest preferred Foundation Model Ops to LLMOps, or large language model operations: the meaning of “large” would keep moving, and image capabilities would make a language-only name too narrow. The proposed category covered the tooling needed to build and evaluate applications around these general models.

The argument for separating it from machine-learning operations (MLOps) rested on two changes: - Different users: generalist software engineers bring different knowledge and goals from ML specialists. - Different evaluation problems: operating a small recommender at enormous scale differs from judging a subjective generative application. The guest expected incumbents to struggle with serving both customer profiles and adapting infrastructure organized around numerical metrics such as accuracy, while considering the transition possible.

Multimodality already implied concrete interface work. Humanloop had begun working on vision support, including accepting and rendering images in its interactive playground, with timing still unsettled.

The question of whether prompt engineering was dead required separating a job title from an artifact. The guest argued that effective prompting could be learned quickly, while insisting that natural-language instructions deserve the seriousness given to source code because they change product behavior. AI engineering was the broader skill set: building products, eliciting capabilities, evaluating outputs and understanding stochastic behavior. Code remained necessary to manage prompts and evaluate and run generated code.

Language Modeling Cascades offered a possible foundation for composing those systems. The guest described the Google paper as treating LLMs as inference engines inside probabilistic programs, where probability and random variables are first-class constructs, with operations for conditioning, prediction and inference. Its appeal was a principled set of building blocks for complex LLM programs. The guest favored a code framework and welcomed discussion with people pursuing one.

Security, fine-tuning and a changing base model

Few Humanloop customers were asking about prompt injection at the time. The guest connected that to deployment maturity: most were still tackling hallucinations and behavioral guardrails, and few had models taking actions. The guest regarded injection as less consequential for read-only applications and expected it to become much more serious with read/write access to data. That assessment was tied to the deployments being observed, not a guarantee that read-only systems were secure.

Fine-tuning had followed a different cycle. Humanloop reported substantial GPT-3-era use, then a sharp decline after GPT-3.5 and GPT-4 arrived. The guest attributed it partly to better base models and partly to fine-tuning being unavailable for the newer ones. After a correction during the exchange, the discussion proceeded on GPT-3.5 fine-tuning being available, with GPT-4 fine-tuning still unavailable.

A workflow already observed among users was to generate outputs with GPT-4, filter them using evaluation or feedback criteria, and fine-tune smaller, faster models on the retained examples. Filtering selected the generated material worth training on. Humanloop described support for filtering, fine-tuning, evaluation and deployment decisions within one product, without reporting measured cost, latency or quality gains.

For portability, the guest expected open standards and commercial platforms eventually to coexist, but considered the field too unsettled for agreement yet. Datadog and OpenTelemetry served as the analogy: common infrastructure could coexist with demand for a company’s support and expertise.

Asked whether GPT-4 had become less capable, the guest accepted that its behavior had changed without accepting a general decline in intelligence. The practical requirement was regression testing that helps distinguish causes: - Prompt changes: did the application’s instructions deteriorate? - Retrieval changes: did the system supply worse context? - Input changes: are users asking different kinds of questions? - Model changes: did the provider’s model behavior shift? Version pinning could not defer change indefinitely. Dependence on evolving third-party APIs also motivated considering where open-source alternatives might fit.

Research strength and builder density

The geographic discussion distinguished research talent from product-building density. The guest described London and Paris as strong AI research hubs, while finding a richer concentration of people experimenting with LLM products in San Francisco. Humanloop’s stated move toward San Francisco followed repeated useful encounters with customers, investors and other builders.

Humanloop had worked to build a London community by sharing an office with Bloop, which worked on LLM-based code search, bringing companies together and organizing an AI Tinkerers event. But the guest saw startup geography as self-reinforcing: people move where others are moving, making established talent density hard to match. The interviewer’s recollection of ten AI meetups on one Thursday supplied the counterweight—builders still needed time to build.

Learning from a history of learning

In the closing research questions, the guest recalled repeatedly underestimating progress, including GPT-4, ChatGPT and an early encounter with Claude. The unresolved problem that stood out was continual learning: enabling systems to keep learning after initial training. Retrieval could supply information, but the guest objected to treating that as a complete answer to models that otherwise stopped learning.

The guest also expected richer shared representations across vision, audio and speech. That led to two contrasting ways of training a transformer on reinforcement-learning experience: - Learn accomplished behavior. The guest described DeepMind’s Gato as learning from observation, action and reward sequences produced by trained agents, allowing one model to perform their constituent tasks. - Learn the history of improvement. A second DeepMind approach recorded an agent’s full learning trajectory, from poor initial performance to improved behavior, and trained a transformer to predict that sequence.

The second approach suggested a more provocative mechanism: predicting improvement might require simulating a learning algorithm. The guest reported adaptation to an unseen task through reward signals supplied in context, interpreting this as optimization occurring within the transformer’s context. The paper’s exact title was not recalled, and no quantitative results or task details were given. Making the approach practical through longer context windows remained a speculation.

Why not pursue that research directly? The answer returned to choosing the company’s purpose. Fundamental model research remained compelling, but the immediate motivation was helping customers deliver useful products, including tutoring and language-learning applications. A conceptual understanding of machine learning could guide tooling decisions and anticipation of future capabilities without becoming the company’s research agenda.

Humanloop’s closing invitation was specific: teams building serious LLM applications could use it to select and optimize prompts, evaluate models and collaborate with nontechnical colleagues. Those shared decisions were the product’s intended contribution to getting useful applications into people’s hands.