Simulation’s scaling bet: learning how people actually behave

An assistant can order dinner and still fail

An assistant orders Hawaiian pizza for someone who dislikes pineapple. The transaction succeeds; the assistance fails. This early example in Latent Space’s conversation about Simile captures a motivation for human simulation: an agent acting on someone’s behalf needs to understand that person’s preferences. Modeling the person could be a prerequisite for useful personalized automation.

The research grew from a creative ambition. The guest pursued art professionally before turning to computation as a medium for making something new. After starting a Stanford PhD in 2020, the guest joined the group led by Percy Liang that wrote about the opportunities and risks of foundation models: broadly adaptable models that could support many downstream tasks. Their appeal went beyond classification and generation to new kinds of interaction.

The team played a time machine game: look back from ten years ahead and ask which application would have mattered most. Their answer was to recreate the social world. The initial hypothesis was that web text contained traces of human behavior that suitable prompts could elicit as realistic patterns. Social Simulacra, a precursor to Generative Agents, began pursuing that possibility.

Where text memory ends and training begins

For Generative Agents’ memory, the team considered knowledge graphs and bespoke models before choosing text. Language models already understood and reasoned over language; plain-text or Markdown memories used that capability directly. The simple representation left a difficult scaling problem: retrieving relevant information and making sense of a large accumulated history.

The guest sees increasingly tailored responses from ChatGPT and Claude, but argues that deeper personalization also requires changing model parameters. That creates a continuing data problem: how to obtain useful information about people and sustain their participation as models learn more about them.

The proposed boundary is prompting for known behavioral rules, training to learn missing ones. If a model already has reliable behavioral patterns and base statistics, a prompt may suffice to describe an environment and elicit a response. The guest argues that web-trained models still lack important patterns: much of their data records what people choose to express about themselves, with less coverage of what they actually do.

Three kinds of evidence about a person

A behavioral foundation model, intended to predict human responses across situations, draws on three complementary kinds of data: - Qualitative interviews supply life history and personal context, including experiences whose relevance may be difficult to anticipate. - Observed behavior, such as transactions, supplies base statistics about what people do. - Randomized controlled trials provide evidence about causal mechanisms by testing how behavior changes under different experimental conditions. A person’s explanation of a choice and a record of that choice offer different information. Interviews add context; they are not themselves behavioral observations.

Causal evidence matters because decision makers want to change outcomes. A forecast that sales will fall in two quarters leaves the practical question unanswered: what should the company do now? Yet naturally occurring situations rarely differ in just one relevant variable. Controlled experiments help supply comparisons that ordinary observations cannot easily provide.

The team reports recruiting participants into physical or virtual labs with consent covering data sharing and representation. Experiments aim to give decisions real consequences. In an illustrative online-store study, participants receive the items they purchase, making their choices consequential rather than merely statements about what they might buy. Firm partnerships and customer-provided behavioral data supplement the experiments.

A population, an environment, and a decision

Customer work starts by defining a population: perhaps the general US population for a consumer packaged goods company, or Californians in their twenties and thirties for a narrower market. The team recruits people with consent and incentives, collects data, and creates models of them. Customers then provide a population filter and an environment in which to query those models.

The environment can be a survey, behavioral experiment, or A/B test. Starting applications include concept testing—comparing messages, products, or ideas—alongside focus groups and modeling public-company earnings calls. The team reports a strategic partnership with Gallup, but says it had not yet worked deeply in politics and wanted clearer guardrails before entering that market.

An interviewer challenges the decision value through a fictional example from The West Wing: polling reactions to a governor’s medical disclosure to plan the president’s own disclosure. If the news is obviously damaging, would knowing the exact approval drop change the decision? The response is that direction alone may be insufficient. Magnitude and severity can be difficult to anticipate, even when an outcome looks obvious in hindsight.

The larger ambition is to search for a sequence of interventions that reaches a goal, including counterintuitive first steps. Asimov’s Foundation supplies the fictional analogy: the scientists’ exile helps set a path toward shortening a predicted period of unrest.

A hypothetical automaker makes the commercial consequence clearer. A campaign could increase electric vehicle sales while damaging perceptions of the company’s other cars and reducing total sales. Optimizing one product’s sales can undermine the company’s goal. Simulation is proposed as a way to explore these broader effects across multiple steps. An interviewer connects this to Shopify’s Sim Gym, described as seeking interventions along shopping journeys to improve conversion.

What the 85% result measures

The engineering question is whether a specialized simulator predicts people better than a general large language model (LLM) given the same goal—and whether errors compound across a simulated sequence. In response, the guest describes a thousand-person generative-agent study published at the end of 2024.

The reported procedure was: 1. Recruit 1,000 people, described as representatively sampled from the US, into a virtual lab. 2. Collect two hours of broad information per person, emphasizing interviews using an American Voices Project script and including behavioral data. 3. Create digital twins: agents built from each participant’s data to predict that individual’s responses. 4. Bring participants back after two weeks for surveys, experiments, and behavioral studies, and compare their responses with the agents’ predictions. The evaluation included behavioral economics games, a personality test, the General Social Survey, and published randomized controlled trials.

The agents reportedly reproduced participants’ behaviors and attitudes 85% as accurately as participants reproduced their own responses. The reference point is human self-consistency, not perfect prediction. This is not an unqualified 85% accuracy score; separate results for the individual evaluation tasks are not given.

The guest argues that simulation requires a different training objective from maximizing reasoning ability. A faithful model must reproduce a person’s characteristic mistakes, biases, and preferences, even when a more rational assistant would choose differently. Later claims of weaker frontier-model performance on niche populations lack a named metric and evaluation protocol, so they cannot be compared directly with the study’s result relative to human self-consistency.

Learning preferences that efficiency misses

A follow-up project explored post-training on randomized controlled trials. Its data source grew from preregistration: recording study populations and hypotheses before running experiments, so the original hypotheses cannot be changed retroactively. The guest connects this practice to concerns about social-science findings failing to replicate and selective publication of statistically significant results.

The team drew studies and data from a platform described as holding tens of thousands of experiments and hypotheses. It reports substantial improvement in behavioral prediction, without specifying an effect size. That research model was not being served commercially. In both research and company work, the team describes two distinct models—one for population-level prediction, another for individual-level prediction—that receive a population or person description together with a stimulus, such as a question or experimental condition.

An interviewer raises a car-wash puzzle: recommending a short walk over a longer drive overlooks the need to bring the car. The response shifts to personal motivation. While living roughly forty minutes’ walk from Stanford, the responding speaker preferred walking home because it created time to think. A model recommending an Uber or bus for efficiency could miss the reason for choosing the walk. Behavioral fidelity requires learning what the person values about an action.

Different datasets reveal different parts of that motivation. Transactions record purchases; social platforms expose behavior in particular contexts. Asked to choose among platforms, the guest favors Facebook’s interactions among friends as a view of everyday life, while recognizing LinkedIn’s professional presentation and Twitter’s curated personas as real but more context-specific behavior.

An interviewer offers an alternative to collecting rich personal data: generate many personas by combining professions and backgrounds, citing Tencent’s billion-persona work. The guest admires its scale but argues that the approach depends on statistics already embedded in the base model. Where those statistics are reliable, persona prompting can help. Where detailed knowledge is missing, more persona combinations do not supply new evidence. The team’s bet remains targeted data collection about real lives.

Scaling fidelity, coverage, and interaction

The team reports early signs of a simulation scaling law: more human data and compute yield predictable improvements in simulating and predicting people. This is an early qualitative claim; no quantitative curve, evaluation metric, model size, or compute budget accompanies it.

The longer-term ambition extends from better models to agents interacting in rich environments. Simulating eight billion people is presented as a five-to-ten-year aspiration, with potential applications to climate coordination among actors with competing incentives, warning signs of democratic collapse, and the emergence of monetary systems.

Thomas Schelling’s segregation model illustrates the appeal of agent-based modeling: simulate individual actions and examine the collective pattern they produce. In the account given, red and blue agents occupy a grid. An agent moves randomly when the share of differently colored neighbors exceeds its tolerance. Mild preferences for same-color neighbors can produce strong segregation over time—an example of emergence within a model, rather than direct evidence of what causes real-world segregation. Generative agents promise richer representations than these simple rules, if they achieve sufficient fidelity.

At the time of the conversation, the team reports useful customer insights from simulations of thousands or tens of thousands of people, with an operating population in the tens to hundreds of thousands. It also reports collecting data from tens of thousands of people weekly and panel partnerships reaching tens of millions globally. Recruitment reach is distinct from the population being simulated. The main reason to expand coverage is to let customers filter down to specific subpopulations.

Agents are intended to be reusable across studies and domains by capturing underlying personal traits. Recent activities will change; the model aims to retain characteristics that persist. The guest offers risk tolerance as one such characteristic, though its stability is an assumption in this discussion rather than a demonstrated result.

What makes a simulation worth its cost

Agents can talk to one another in multi-agent simulations, with [Smallville](https://arxiv.org/pdf/2304.03442) offered as an example. Whether those interactions matter depends on the task: even a solitary purchase can be influenced by what friends buy and discuss. An interviewer raises the potential combinatorial cost of thousands of agents interacting. The response compares it with the expense and practical difficulty of recruiting people for real-world studies, without supplying measured simulation costs.

The proposed economic value depends on the decision at stake. Spending millions on simulation could be worthwhile if it improves a process costing hundreds of millions. The guest imagines future simulations consuming an entire data center and costing as much as foundation-model training, justified by their societal value. These are prospective arguments, not reported customer returns or current prices.

Enterprise deployments already generate substantial inference demand, according to the team, making efficiency an ongoing concern. Product testing also expands the input beyond text. An early customer needed reasoning over images; agents are described as able to receive a website URL and use the site. Traversing Figma mockups is discussed as a further possibility.

Much demand begins with work historically done by human panels—groups recruited to participate in research. The team rejects wholesale replacement of those panels because its representations remain grounded in people. The broader opportunity is to consult users more often: cost, time, and participant availability prevent organizations from asking about every decision. Reusable simulations could keep stakeholder preferences represented between direct studies.

The guest estimates market research at $100 billion, but treats that as a starting market rather than a calculation of simulation’s full opportunity. The stated priority is improving decisions made about and for people. Customer value and business upside matter; valuation is described as a superficial measure of progress toward that goal.

Representation still needs reality

The guest compares simulation’s maturity to the GPT-3.5/GPT-4 stage of language models: useful in selected applications, with substantial advances in data, algorithms, and scale still ahead. An analogy to painting gives that ambition a different perspective. Drawing on professional training in figurative oil painting and realism studios, the speaker describes an imperfect representation that can nevertheless reveal something important about its subject.

An artistic study of a rural town—photographing people simply living their lives—connects that interest to simulation. The speaker uses a fractal analogy: understanding can come from exploring society broadly or examining ordinary life closely. Mundane behaviors, considered together, may reveal something about both individuals and their society. An interviewer recommends Working for its in-depth accounts of seemingly ordinary lives.

Asked what to simulate ten years ahead, the respondent prioritizes whatever societal questions prove most consequential, offering whether society needs universal basic income as one possibility. The attraction is rapid repeated experimentation. Another participant supplies the essential counterweight: real-world experiments check whether expected results are right. Faster exploration still needs that contact with reality.

The closing philosophical question is whether reality itself is simulated. The respondent leaves the question unresolved: lived experience remains real either way. Another participant suggests simulated realities might vastly outnumber nonsimulated ones, while acknowledging computational expense as a counterargument. This remains an intuition, not a probability calculation.

Science fiction supplies a final image of artificial general intelligence and simulation as twin technologies of advanced civilizations. It expresses the guest’s confidence in simulation’s potential, rather than a technical forecast.

A research lab building a product

At the time of the conversation, Simile reported roughly 60 employees, a headquarters in San Francisco’s Mission Rock, and a smaller, recently opened New York office. Its operating model combines continuing AI research with engineers, product staff, and designers building tools for real-world decisions.

Approximately 15–20% of the team were described as former lab mates from Michael Bernstein’s and Percy Liang’s labs, many returning after work at OpenAI or Google Gemini. The company also reported team members from Figma, Notion, and Harvey, and broad hiring across research, product engineering, and infrastructure.