Hamel Husain

Hamel Husain is a machine-learning engineer, independent AI consultant, and educator whose work focuses on product-specific AI evaluations: measuring whether a system succeeds at its users’ tasks. He helps teams debug, analyze, and measure LLM applications and co-teaches AI Evals for Engineers and PMs with Shreya Shankar. His technical writing connects evaluation to practical decisions about prompts, fine-tuning, data curation, and software development.[2][3][4]

In Your AI Product Needs Evals, published in March 2024, Husain described working on Lucy, Rechat’s conversational assistant for real-estate professionals. Early prompt engineering produced rapid progress, but expanding the assistant’s responsibilities exposed a plateau: fixing one failure introduced others, prompts accumulated edge cases, and the team lacked visibility into performance across tasks. Husain and the team responded with a systematic evaluation process.[4]

Husain organizes this process into three levels: fast unit tests, human and model evaluation, and A/B testing. Scoped assertions check individual behaviors—for example, whether a listing search returns the expected number of results or whether a response exposes an internal UUID. Synthetic user inputs exercise those scenarios before sufficient production data exists. He recommends updating tests as new failures emerge, while treating the acceptable pass rate as a product decision rather than automatically requiring perfection.[4]

The Lucy team also built a trace-review interface that brought together the conversation, the feature and scenario under evaluation, and links to relevant CRM records. They made final outputs editable so reviewers could correct small mistakes and curate examples for fine-tuning. This illustrates Husain’s argument that evaluation, debugging, and training-data preparation share much of the same infrastructure: searchable traces, failure checks, human review, and a way to test whether a change works.[4]

Husain calls his approach to building an LLM evaluator “Critique Shadowing.” In his LLM-as-a-judge guide, first published in October 2024 and updated in September 2026, a domain expert makes pass/fail decisions and explains the reasoning behind them. Those critiques help clarify expectations and supply examples for the judge prompt. Husain recommends fixing obvious product errors uncovered during review before proceeding to automate evaluation, then using error analysis to decide where specialized judges or code assertions are needed.[5]

While helping Honeycomb build its Query Assistant, Husain iterated an evaluator prompt using domain expert Phillip Carter’s judgments and written critiques. They exchanged spreadsheets containing natural-language requests, generated queries, critiques, and pass/fail outcomes. Husain reports reaching greater than 90% agreement between the model and Carter after three iterations on a roughly balanced dataset. That result describes their calibration exercise; his guide warns that raw agreement can conceal poor error detection when failures are rare.[5]

The updated guide treats the judge as a binary classifier: prompt examples belong in a training set, prompt revisions use a separate development set, and a hidden test set provides the final estimate after the prompt is frozen. It recommends reporting true positive and true negative rates separately. Husain’s emphasis remains on the analysis itself: “The real value of this process is looking at your data and doing careful analysis.” He presents the method as adaptable, including cases where manual error analysis is sufficient without an automated judge.[5]

Husain and Shankar extended their evaluation teaching into Evals Skills for Coding Agents, jointly announced in March 2026. The updated announcement describes an entry point that routes agents toward auditing an existing evaluation pipeline or discovering errors in traces. Other skills support custom review interfaces, diverse synthetic test inputs, judge prompts, and evaluator calibration against human labels. The authors distinguish these product evaluations from general model benchmarks and stress that the skills need adaptation to each team’s data and domain.[8]

Husain and Shankar also appeared together on Latent Space’s “Evals: How to Improve AI Consistently” lightning podcast.[1]

Before this evaluation-focused consulting and teaching, Husain worked on semantic code search and notebook-based developer tools. Those earlier projects supply context for his experience building evaluation datasets and reconsidering development workflows.[9][6][7]

Husain coauthored CodeSearchNet Challenge with Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. First submitted in September 2019, the work addressed semantic code search: retrieving relevant code from a natural-language query. The challenge paired 99 queries with about 4,000 expert relevance annotations, alongside a corpus of roughly six million functions across Go, Java, JavaScript, PHP, Python, and Ruby. Its evaluation task required bridging the vocabulary of human requests and the abbreviated, technical language of code.[9]

Husain joined nbdev in 2020 while at GitHub and helped lead its complete rewrite in 2022. He worked on the project through 2023 with Jeremy Howard and, later, Wasim Lorgat. nbdev uses Jupyter notebooks as the source for code, documentation, and tests, then generates a Python library and documentation website. Husain also built fastpages in 2020, a notebook blogging system that informed nbdev’s documentation approach, and contributed to ghapi and fastcore.[7][6]

Husain left a full-time job to work on nbdev and explored commercialization with Howard and Lorgat. Their proposed “WordPress for developers” would let people create and share projects without installing the tools. In his May 2023 retrospective, he described several reasons for stepping away: onboarding and notebook-review friction, competing priorities across developer tools and generative AI, and his lack of conviction in the business model. He concentrated on consulting; following ChatGPT’s arrival, he recalled that client demand led him to work exclusively on operationalizing language models.[6]

A separate decision followed in January 2026, when Husain announced that he had stopped personally using nbdev. In his experience, AI coding tools struggled to distinguish notebook source from generated code, making the workflow harder to use with an agent. He favored conventional development environments and reported using Amp, Cursor, and Claude Code. His change was specific to software development: he continued to value notebooks for data analysis, machine learning, and exploratory work. He also argued that careful thinking and good documentation depend on the developer’s effort rather than being guaranteed by a tool.[7]