Humanloop

Humanloop built a collaborative development platform for teams creating applications with large language models. It combined prompt management, evaluation and production observability, allowing engineers and domain experts to test changes and investigate model behavior. In 2025, Humanloop announced that its team was joining Anthropic and scheduled the standalone platform's sunset for September 8.[6][3][4]

The transition announcement, signed by Raza, Jordan, Peter and the Humanloop team, paired the move to Anthropic with plans to sunset Humanloop's platform. An August 13, 2025 notice said billing had stopped on July 30 and instructed customers to export their data before September 8. It warned that the platform and all data would become permanently inaccessible after that date and offered prorated refunds for annual subscriptions extending beyond the scheduled sunset.[3][4]

The platform scheduled for closure had begun with a different task: helping people annotate training data for language models. Humanloop was founded in 2020 and joined Y Combinator's Summer 2020 batch.[8][2][4]

Raza Habib, Jordan Burgess and Peter Hayes formed Humanloop's operating founding team. Habib recalled that a side project they worked on together became the company, with YC admission prompting them to pursue it. Burgess took the lead on product and developer experience, Hayes on engineering, and Habib on customer-facing work. Index Ventures' 2022 account also credited Habib's academic supervisors as co-founders and identified Professor David Barber, head of UCL's AI centre, as Humanloop's chief research officer at that time.[1][5][6]

The early product addressed the expertise and annotated-data requirements of adapting NLP models. Programmatic, described by Index in July 2022, let domain experts express knowledge as simple rules that the software applied across a dataset. Index also described Humanloop selecting high-value examples and asking humans to resolve edge cases, rather than treating every example as equally useful. In the same announcement, Index reported that Humanloop had raised $2.6 million in seed funding, led by Index Ventures.[1][5]

Habib associated the pivot toward LLM applications with InstructGPT making the new approach practical. In his September 2023 interview, he recalled abandoning a business with paying customers and growing usage a little over a year earlier. The shift also changed Humanloop's audience: from machine-learning specialists toward engineers focused on building useful product experiences.[1]

Before committing, the team tested mockups with companies already building on language models. Habib said they set a target of ten paying development partners within two weeks and reached it on the second day. Those prospects were stitching together spreadsheets, Grafana, Mixpanel and the OpenAI Playground to iterate on prompts, preserve versions and measure subjective outputs. The proposed product brought those activities into a shared workflow.[1][8]

By September 2023, Humanloop managed and versioned prompts and collected production feedback. Habib distinguished three evaluation needs: checking improvements during design, monitoring performance on real customer inputs, and testing for regressions after changes. Early Humanloop evaluation depended on end-user feedback; offline development and regression evaluation was still forthcoming, with about ten companies in closed beta, according to Habib.[1]

Humanloop supported votes, actions and corrections. Votes captured explicit ratings; actions captured signals such as accepting or regenerating an output; corrections preserved edits that could later support evaluation or fine-tuning. Habib used James Yu's fiction-writing product Sudowrite to illustrate implicit feedback: whether writers accepted suggestions, regenerated them or edited them before inclusion. He also described customers retaining edited sales-email drafts.[1]

Production logs could be reopened in Humanloop's interactive playground and rerun with different models or context. This let a team investigate a failed response by testing whether corrected retrieval context or a different model changed the result. Humanloop also supported filtering generated outputs using evaluation or feedback criteria, fine-tuning a model, evaluating it and deciding whether to deploy it.[1]

Humanloop announced enterprise general availability on November 21, 2024. Its evaluation framework supported automated code checks, LLM judges and human review against versioned datasets, with reports, CI/CD integration and evaluation results attached to production logs. The platform organized applications around four versioned, callable file types—prompts, tools, evaluators and flows—managed through code or a UI and executed on Humanloop's runtime or the customer's own.[6]

Collaboration remained central to the product. Habib described Duolingo linguists refining prompts, Gusto's customer-support team contributing evaluations, and Filevine legal professionals developing industry-specific AI features. The shared editor and prompt repository brought production interactions into development while tracking changes and allowing prompts or tools to be deployed as callable functions.[6]

In his December 2024 review, CTO Peter Hayes described git-like prompt commits, diff views, attribution and .prompt files for source-control integration. A model proxy unified access across eight providers. Flows recorded multi-step application traces, with OpenTelemetry instrumentation and both component-level and end-to-end evaluation. Teams could curate evaluation datasets from production logs and replay the underlying model interactions.[7]

At general availability, Habib reported $8 million in total funding from YC Continuity, Index Ventures, Localglobe and angels including founders of Datadog and Segment. On December 10, Hayes reported millions of LLM logs processed daily across thousands of AI products in production. He also reported over 300 Humanloop production deployments culminating in 50 product releases during the review period; these described the company's release activity.[6][7]