Chai’s feedback loop: crowdsourced models, human preferences, and inference economics

Small teams need a place to compete

A trading opportunity earning $100,000 a year yields 1% on a $10 million fund but 100% on $100,000. William Beauchamp uses that illustrative calculation to explain his start: after graduating from Cambridge in economics in 2012, he traded capital accumulated playing poker. He taught himself Python and spent roughly three years building neural networks and random forests to predict asset prices.

The operation grew into a company making approximately £5 million annually with about 15 mathematicians and physicists, Beauchamp recalls. Trading their own capital gave them freedom, but around his 30th birthday he began wanting a broader purpose for the team. Experiments with language models and published research from Google and OpenAI persuaded him to pursue conversational AI.

[Chai Research](https://www.chai-research.com/) began as a platform for specialized language models: small teams could build AI for particular purposes and reach users through a shared marketplace. Quantitative finance supplied the specialization analogy; YouTube and Wikipedia showed how outside contributors could expand a platform’s offerings. This vision also rested on a belief that model progress would follow an S curve—improving rapidly before leveling off—rather than increasing indefinitely with concentrated data, compute, and researchers. That was a motivating hypothesis, not an established limit on machine learning.

The bot people actually wanted

Chai’s first developer interface hosted and executed submitted Python scripts under a text-in, text-out contract. An early bot fetched popular Reddit news and called an external model API so users could discuss the stories. It attracted little demand. Recipe advice, dad jokes, pub quizzes, and fun facts also failed to catch on, while the developers approached were largely uninterested in building chatbots.

The first encouraging signal came from a therapist bot that the builder’s sister created while learning Python. The next day, the app reportedly had 20 active users averaging roughly 20 minutes, with that engagement attributed to her bot. The small discovery changed the product hypothesis: immediate, judgment-free conversation had a distinctive appeal. At 3 a.m., someone could speak freely and receive a reply without waiting for a friend.

The reference point shifted from search toward Instagram and social interaction. The proposed explanation was that reciprocal storytelling lets people participate rather than only consume. Chai also received accounts from users who felt the AI helped them through depression or eating disorders. These were user anecdotes, not clinical outcomes; they supported the team’s interest in conversation without establishing a health benefit.

Team-created celebrity and influencer personas still drew little interest. The next breakthrough was to give consumers the creation tools: a prompt, image, and name on top of an API-backed model, tentatively recalled as GPT-J, a six-billion-parameter transformer. Users supplied antagonistic characters and romantic archetypes the team had never imagined. That variety clarified Chai’s social-AI proposition: users create conversational experiences, share them, discover others, and interact.

Long conversations make performance per dollar decisive

Chai reached roughly 100,000 daily active users with approximately £2 million from its sole investor, according to the account. Competition then exposed a capability gap. After initially dismissing Character AI’s product, Chai began receiving complaints about its own six-billion-parameter model. The guest connects that pressure to the rival’s ability to fund more inference compute—the computation used to generate replies—and describes Chai’s subsequent move to Silicon Valley and fundraising.

The guest illustrates Chai’s workload with a typical session of approximately 90 minutes and 150 messages, far more requests than a question with a few follow-ups. An accompanying cost example uses one million users and $1 million of AI spending over a year: an aggregate dollar per user for that period, not a per-message price. Long conversations make small differences in serving cost consequential.

Additional compute can improve the experience in several ways: - Larger models can improve capability. - Longer context gives the model more conversation to use as memory. - Rejection sampling generates several candidate replies, then uses a reward model—a model that scores responses—or another scoring system to select one. Chai’s proposed target is therefore experience quality per dollar. A benchmark score alone does not establish how much good conversation the product can afford to serve.

This explains the guest’s interest in DeepSeek’s inference efficiency and whether the then-future Llama 4 could match it. No measured savings are provided. By the January 2025 episode, Chai reported roughly 1.4 million daily active users; the introduction also reported more than $22 million in revenue, without specifying its reporting period or accounting basis.

Repair reliability, then shorten the learning loop

Chai’s 2024 growth story starts with backend trouble. Firebase let the team concentrate on AI through roughly 500,000 daily active users, but Chai’s particular usage stopped scaling beyond that point. About three months went into deciding which requests to retain on Firebase and which to move elsewhere. The resulting backend reportedly supported approximately 1.5 million daily active users without further issues related to Google Cloud Platform. The threshold describes Chai’s workload, not a general Firebase limit.

Outages can damage acquisition after service recovers. The guest describes new users encountering a broken app and leaving, weakening retention, spending, and ratings. Poorer app-store placement then slows organic acquisition, so commercial recovery can lag the repair. A separate suggestion that Character AI reduced acquisition spending and gave competitors room to grow remains explicitly speculative.

The next bottleneck was model evaluation. Chai had waited for cohort tests to reveal how many users returned after 30 days. It moved model iteration to user ratings available in roughly three hours, reporting 20–50 models evaluated daily and at least 100 iterated through weekly with about five AI researchers. The guest credits this process, mature around the end of Q2 2024, with accelerating fine-tuning, prompt changes, blending, rejection sampling, and reward-model development through Q3 and Q4.

Three-hour preference feedback is distinct from day-30 retention. The faster signal ranks engagement and entertainment; its accuracy in predicting long-term retention is not established in the conversation. The operational gain was the ability to compare and revise models much more quickly.

Paid acquisition followed. Chai reportedly reached one million daily active users without advertising, then hired a growth specialist and increased acquisition spending from about $10,000 to $20,000 and eventually $40,000 daily. The guest estimates that this moved its annual growth pace from roughly 2× to 3×, not a completed annual result. With low switching barriers between chat apps, the argument remained that a better experience was essential to retaining those users.

Voice changed too little of the experience

Chai spent roughly three months building voice, addressing latency, cost, quality, and activation. The interface had to avoid unsolicited speech without requiring a button press for every response. When the A/B test showed no change in the tracked numbers, the team suspected a bug and spent another week checking the implementation.

The explanation was adoption multiplied by usage. Only an estimated 10–15% of users activated audio, and they used it for roughly 10–15% of their time. In the guest’s simplified arithmetic, one-seventh of users using a feature for one-seventh of their time changes about 2% of the experience. Even a large improvement within that slice had little aggregate effect on retention, engagement, or monetization. These approximate results concern Chai’s implementation; test size, duration, and statistical uncertainty are unspecified.

An interviewer suggests inadequate models might explain the result. The guest rejects that diagnosis: voice quality was good, but the feature felt like a convenience. Chai removed it. The resulting decision rule was to seek a substantial improvement for most users across most of their experience. Voice could deserve another attempt if it enabled something uniquely compelling.

The same reasoning shaped the multimodal roadmap. The guest saw a narrow range of creators’ tastes as a more pressing constraint than missing audio or image generation. User-generated content (UGC)—then largely prompts, names, and images—needed to extend into user-controlled model training. A creator business earning or spending 100 million annually on Chai content, with currency unspecified, served as an ambition for that ecosystem, not an existing result.

Chaiverse gives model builders users and feedback

Chai’s proposed defensibility lies in a feedback loop among creators, users, and recommendations. Compelling content attracts users; their activity improves recommendations; better distribution attracts creators and teaches them what works. YouTube thumbnails and opening sequences illustrate the guest’s point: creators accumulate knowledge specific to a platform, which may not transfer readily to another distributor.

[Chaiverse](https://www.chaiverse.com/) extends participation into the underlying models. Chai is the consumer app; Chaiverse is described as a prototype developer platform that hosts community-built models. It offers distribution and feedback to builders already investing in datasets, training, and evaluation.

The submission loop connects hosting to human judgment: 1. Host the model. Chaiverse starts a GPU, downloads the submission, and serves it. 2. Route user traffic. Users compare two completions and choose the better one, using an LMSYS-style pairwise comparison. 3. Rank entertainment preference. The guest estimates that roughly 5,000 completions provide an accurate signal. 4. Return feedback. Developers receive at least a score; additional data must balance usefulness with user privacy. An earlier overview gives the count as 5,000 users rather than completions. The estimate comes without validation details or uncertainty bounds, so it is not a general sample-size prescription.

The guest describes discarding roughly the bottom 80% of submissions and finding distinct strengths among the rest: descriptiveness, personality, or logical behavior. Two serving approaches then emerge: - Predict the best model for each request. Chai found this routing approach expensive, with little improvement. - Blend models across requests. In the example, random selection sends half the requests to a smart model and half to a funny model. This is request-level selection, with a reported qualitative benefit but no numerical comparison.

Elo scores—ratings derived from model comparisons—give contributors an iteration signal. Chaiverse tracks them across submissions, and the guest describes a recurring pattern of unsuccessful attempts followed by a breakthrough. Even a single score lets a builder propose a change, test it, and retain improvements. The platform makes users’ reactions available to people who could otherwise train a model without learning whether anyone enjoys it.

Crowdsourcing knowledge and enabling contributors

The conversation distinguishes crowdsourcing knowledge from crowdsourcing intelligence. The guest argues that Wikipedia accumulates knowledge, while language models act more like simulators producing likely continuations than reasoning engines. A bridge simulation supplies the analogy: construct a situation and predict what happens next. This conceptual position leads to superknowledge—storage and retrieval beyond an individual human’s capacity, made broadly accessible—without assuming greater intelligence. The related term appears in Latent Space’s conversation with Exa.

The ambition is long-term: one speaker places consumer AI around year four of a 20-year journey, with entertainment as a starting point and broader participation as a way to compound its value. Wikipedia also supplies an incentive argument: people can contribute for the satisfaction of making something correct.

Chaiverse reportedly paid creators more than $100,000 without materially changing submission rates. The useful effect was funding experiments contributors already wanted to run, such as fine-tuning Llama 70B overnight on eight H100 GPUs. A contributor recalled as approximately 17 years old reportedly spent a roughly $1,000 payment on a physical GPU for further training.

Asked whether Chai should hire that contributor, the respondent returns to the platform premise: build systems for creators. A platform cannot hire every promising participant, and today’s strongest creator may not be next year’s.

What a global preference score can miss

Chai treats Elo as its primary evaluation target because it reflects human preference. The guest argues that fixed evaluations capture a slice of a task and eventually saturate, while people can keep judging new outputs. Targeted tests had mainly helped resolve specific blockers; one safety evaluation was reportedly retired after about a month because the team considered the issue addressed.

The discussion identifies two limitations of preference rankings: - Style can win without deeper improvement. Profanity making a model seem funnier is the example, prompting a discussion of controlling for style in comparisons. - A global score can obscure distinct audiences. An interviewer asks whether role-play, therapy, and sexually explicit interactions warrant separate evaluations.

The response defends global optimization because users’ preferences appear substantially correlated. Its condition for useful segmentation is preference reversals, not merely different enthusiasm: personalization has more room to help when one group likes what another dislikes than when both rank content in the same order. The guest argues that AI content was not yet varied enough for YouTube-like personalization. Richer creation tools came first—a tension underscored by an interviewer who found Chai surprisingly thin.

Make a spaceship drama easy to create

A spaceship role-play makes the creation problem concrete. An enthusiast can give Llama 70B a detailed scenario specifying crew personalities and a new cadet’s ambitions. Chai aims to produce an equally good or better experience from a brief request for a spaceship, three crewmates, drama, and fighting. Steerability means responding to creator and user intent without requiring elaborate prompts. This is a product aspiration, not a measured comparison.

TikTok’s music and effects provide the analogy: ordinary people should be able to make entertaining content without exceptional writing skill. The guest also acknowledges that open-source role-play enthusiasts want something different from typical consumers. Their proposed role is to fine-tune models that create distinctive effects, while Chai simplifies creation for the broader audience.

Two invitations follow: - Chai Grant supports open-source builders. Described as no-strings-attached cash, it aims to support a community whose packages Chai uses and to meet talented contributors. The guest encourages applications from people proud of a GitHub project. - The team recruits for an intense work environment. At roughly 15 people, Chai explicitly seeks applicants who want that culture and discourages those who would dislike it.

A workplace anecdote makes that expectation concrete: the guest says taking Friday off is acceptable, but responsibilities may require finishing work Thursday or Saturday. The employee mentioned was reportedly gone four weeks later; the account does not establish the reason for the departure.

Sixteen completions, one delivered reply

Before closing, the interview returns to inference. The respondent affirms that Chai spent $10 million on compute in the previous year and expected spending might triple. After using vLLM, Chai evaluated MK1’s inference engine, which reportedly ran much faster for its workload. The guest connects the benefit to hardware expertise and support for Chai’s heavy rejection sampling; no numerical speedup or benchmark configuration is supplied.

The serving procedure is to generate 16 complete responses, score them, and deliver one. A separate reward model evaluates the candidates. Because this workflow needs the full responses before selection, Chai waits and displays the chosen reply all at once; the guest says the product has never streamed replies.

The conversation presents two uses of the latency budget: - Stream early, finish later. Showing the first token quickly can make a longer total generation time—and potentially a larger model—acceptable. - Wait to select a complete response. Generate and evaluate alternatives before displaying the chosen reply. The discussion’s four- versus ten-second example is hypothetical. The practical choice is how to spend time before committing to an answer.

A proposed reward-model setup learns from whether users reply. The guest suggests taking a corpus—50 million messages in the example—and training a scorer on which messages received responses. It would favor completions predicted to elicit engagement and withhold those predicted to end the exchange. Conversational continuation is the proposed reward, rather than a general measure of response quality; this is a possible training setup, not a reported experiment or disclosure of Chai’s exact dataset.

An interviewer draws a parallel between routing among model experts and choosing among generated paths. The guest endorses combining generation with tree search—exploring multiple possible paths—and invokes AlphaGo. Applying that explanation to o1 and o3 prompts an objection: another participant notes that their use of tree search has not been explicitly established. It remains a contested hypothesis here. The broader closing position is that AI’s ability to make things is what excites consumers.

The final invitation is to builders with startup experience interested in the Palo Alto team. Chai’s hiring philosophy favors tangible evidence of initiative: a useful low-level software project or entrepreneurial achievement can matter more than a prestigious credential alone. Exceptional prior achievement is not presented as mandatory; the strongest signal is having built or accomplished something concrete.