Eugene Yan

Eugene Yan builds applied AI systems and writes about the engineering needed to make them reliable. His work spans retrieval, recommendation systems, and LLM evaluation, including the AlignEval application and experiments combining language models with catalog recommendations. He joined Anthropic as a Member of Technical Staff in 2026, following work at Amazon from 2020 to 2025.[3][5][7]

At Amazon, Yan built real-time retrieval, bandit rankers, recommendations in search, and AI systems for summarization, translation, and question answering. In 2021, he was part of the team developing personalized book-recommendation widgets, using browsing signals such as a reader’s interest in a particular book, genre, or topic. His later role was Principal Applied Scientist.[3][4]

Yan introduced AlignEval in October 2024 to help people build task-specific LLM evaluators. Its workflow begins with uploading input/output examples and labeling them pass or fail. Users then define criteria, compare an automated evaluator’s predictions with their labels, and optimize it using development and test splits. The order reflects his view that people should inspect actual outputs before deciding which defects to measure.[5]

AlignEval measures evaluators with precision, recall, F1, Cohen’s κ, and counts of correct and incorrect classifications. Optimization runs trials on development data before testing the revised evaluator on unseen examples. Yan cautioned that small samples produce unstable metrics and can let an evaluator overfit an unrepresentative development split; he recommended labeling more diverse examples that resemble real use.[5]

Building AlignEval was Yan’s first TypeScript project. He tried several frontend approaches before choosing Next.js, paired with Python and FastAPI because Python’s analysis and machine-learning ecosystem made optimization experiments easier. He selected small models for cost and speed, and chose Railway hosting over configuring a VPS so he could concentrate on the interface, features, and optimization. He credited Shreya Shankar, Hamel Husain, Kyle Corbitt, David Corbitt, Saumya Gandhi, Swyx, Eugene Cheah, and Dennis Taylor for discussions and early-demo feedback; Husain also read drafts of his write-up.[5]

In his April 2025 essay, An LLM-as-Judge Won’t Save The Product—Fixing Your Process Will, Yan argued that evaluation is an ongoing practice of observation, annotation, hypotheses, experiments, and error analysis. He advocated evaluating a baseline and measuring each subsequent change, whether to prompts, retrieval, or models. Automation still requires people to sample outputs, annotate failures, and examine customer feedback: “Automated evaluators amplify our existing annotation and feedback processes.”[6]

Yan also coauthored the June 8, 2024 guide What We’ve Learned From A Year of Building with LLMs with Bryan Bischof, Charles Frye, Hamel Husain, Jason Liu, and Shreya Shankar. He led much of the integration and overall structure, contributed a substantial portion of the lessons, and took primary editing and document-direction responsibilities. The authors jointly advocated task-specific evaluations, data collection, and human-centered workflows, arguing that lasting product value comes from the system surrounding a model—its guardrails, caching, evaluations, and feedback loops.[8]

In September 2025, Yan described an LLM–recommender hybrid built with approximately 66,000 video-game products and 78,600 interaction sequences from Amazon Reviews 2023. Applying existing semantic-ID and residual-quantization methods, he converted product-metadata embeddings into hierarchical token sequences, where similar products share prefixes. He appended a fourth token to distinguish products whose first three tokens collided, then extended Qwen3-8B’s vocabulary and fine-tuned it to understand product descriptions and generate recommendations from interaction histories.[7]

Yan’s demonstrations showed one model handling both item IDs and ordinary language: recommending from a purchase history, responding to platform or genre constraints, explaining suggestions, and naming product bundles. These were prototype demonstrations with limited fine-tuning and substantial prompting sensitivity. In a separate validation comparison using 500 negative candidates per positive next item, his regular-ID SASRec baseline outperformed the semantic-ID variant. That comparison tested the recommendation representations and architectures; it did not establish the final Qwen hybrid’s comparative accuracy.[7]

For the experiment, Yan credited Devansh and Shrikar for discussions, Luke and Runpod for compute, and Vibhu, Shawn, and Cheng for further conversations. He also hosted the RecSys track at AI Engineer World’s Fair 2025 and appeared in Latent Space’s Paper Club discussion of Who Validates the Validators?.[7][1][2]

Before his Amazon and LLM work, Yan moved from policy analysis into data science. He studied psychology at Singapore Management University and worked for the Singapore government examining legal cases and trade agreements. Recalling the transition in 2021, he said he missed working with numbers and data about human behavior. He joined IBM in 2013, where his work included job forecasts and recommendations.[4][3]

Yan worked as a data scientist at Alibaba/Lazada from 2015 to 2017 and as its VP of Machine Learning from 2017 to 2018, building systems for product ranking, push notifications, and product and review classification. He then led machine learning at the healthcare startup UCARE.AI; his personal timeline places that healthtech role in 2018–2019. While at Lazada, he pursued a computer-science master’s at Georgia Institute of Technology to strengthen his fundamentals, graduating in 2019. He relocated to Seattle to join Amazon in 2020.[3][4]

His public writing grew from a deliberate effort to become a better communicator after mentors repeatedly recommended that skill. He published on WordPress before launching eugeneyan.com in 2020. After briefly writing for mass engagement, he chose more technical subjects he wanted to learn and readers who might become friends or colleagues. As he put it in Amazon’s 2021 profile, “Each comment and real relationship are worth more than 10,000 likes”. His technical articles’ citation blocks also use the publication name Ziyou Yan.[4][3][5][6][7]