Elicit: Making AI Research a Process You Can Inspect

Evaluating work you cannot read in full

In Ought’s experiments around 2018–2019, one person received a limited amount of text while another received the full set. The less-informed participant had to evaluate the better-informed participant’s work. As Jungwon explains, this simulated a practical problem for advanced AI: a system can read more than its evaluator, yet someone still needs to judge its conclusions.

Elicit applies that oversight problem to scientific research. The AI research assistant finds, summarizes, and extracts information from papers to help users understand what is already known. In the April 2024 conversation, its central approach is process supervision: identify the steps an expert would take, break them into smaller tasks, and train systems to perform each reliably. Intermediate results give users and developers places to check the work and locate failures.

Andreas’s path began with programming languages for AI during his MIT PhD. He wanted better ways to express and learn models of the world, ultimately helping people reason and make decisions. After a Stanford postdoc, he found academic hiring constraints and paper-to-paper timelines poorly suited to building useful tools. He judged a startup premature and founded Ought as a nonprofit research lab.

Jungwon arrived through an interest in scalable mental-health support. In 2015, she felt neither she nor the technology was ready for a company; after working in fintech, she decided in 2019 to build something herself. Ought’s approach to breaking overwhelming thinking into manageable pieces resonated with her. A close friend introduced the two, and they explicitly evaluated complementary skills and shared values. Their questions and responses filled a Google Doc Andreas recalls as roughly fifty pages.

The team decided to build a product at the start of 2021, giving its research a direct implementation path. The strategy was to build software around powerful models that made their work transparent and controllable, anticipating capabilities a year or two ahead. Early experiments used humans, followed by generative models including GPT-2 and TNLG; T5-based models remained in use at the interview.

Turning papers into workable evidence

A representative Elicit assignment resembles work given to a research assistant: find papers about a disease, check whether their study populations are sufficiently large, and organize their experiments, interventions, and outcomes. The team saw these repeated operations as work language models could perform at a scale individual assistants could not reach.

Systematic reviews supplied a model: comprehensively find relevant papers, screen them against explicit criteria, and extract comparable details. The team described a manual effort that might involve five people working for more than a year, starting with around 10,000 papers and narrowing to hundreds or perhaps fifty. Two people might extract details, with a third reviewing. These figures describe the manual workflow, not Elicit’s performance. Its appeal for automation was structural: the steps were already designed to be reproducible.

The broader goal was to give text research some of data science’s flexibility. Language models could separate PDFs into claims, evidence, and insights that researchers could compare and recombine. Computational notebooks such as Jupyter, DeepNote, and Colab offered a pattern for extending an analysis beyond the first answer. Personal uses—including parents investigating a child’s rare disease—were already part of Elicit’s audience, but the product emphasis was on power users.

The forecasting detour and the wall of bad answers

Elicit’s first product was a forecasting assistant. The team had considered literature review but expected others to build it, so it chose geopolitical forecasting, including questions about whether China would invade Taiwan. Initially, it helped turn fuzzy beliefs into probability distributions. Iteration exposed a different bottleneck: people needed better research before more precise probabilities would help.

That realization, together with GPT-3’s general capabilities, led to a broader research assistant. The team embedded with researchers and prototyped data labeling, expert discovery, and research-question formulation. Meanwhile, responses to a vague signup page kept asking for literature-review help. Repeated demand across domains overcame the team’s reluctance.

To keep the open-ended problem manageable, the team set targets of shipping a feature every week and doubling users every month. Andreas built the first literature prototype over a weekend. A question such as how creatine affects cognition produced claims loosely based on papers, but both the claims and papers were often irrelevant. The team printed bad results and put them on the wall to keep failures visible.

The output began as a one-sentence abstract summary. Learning about systematic reviews pushed Elicit toward extracting more fields for analysis. Search evolved too: one participant tentatively recalled using the Semantic Scholar API or something similar initially. The team later built its own search after finding the initial retrieval insufficiently sensitive to meaning, and reported a substantial improvement without retrieval measurements.

The commercial thesis rested on the value of discoveries and avoiding unproductive experiments. Some seed investors reportedly recommended a legal assistant, arguing that researchers lacked money. Elicit’s counterargument was that better research and development could create substantial value, even though the route from literature tools to that outcome remained uncertain.

Better models expanded this direction rather than repeatedly replacing it. The team described GPT-3 as a qualitative shift in feasibility and GPT-4 as enabling more accurate answers and harder tasks within an established product. One extension moved beyond tables of papers toward lists of arguments, effects, interventions, or techniques: extract information, then group it around the researcher’s question while retaining its connection to the literature.

Training for what the text actually says

Abstract summarization made speed, cost, and faithfulness immediate engineering concerns. Elicit generated summaries on demand, relative to each query; a few searches could require hundreds. The desired behavior was precise: report what the abstract says about the question. Earlier models often tried to answer the question directly, introducing information beyond the supplied text.

Charlie, hired after his freshman year at the University of Warwick, implemented an approach inspired by constitutional AI. He wrote a “constitution”—a set of desired summary properties, including concision and support for every statement in the abstract. A model trained on those criteria was then used to fine-tune an open-source summarizer. The team recalled an implementation in roughly four days and an in-app rollout about a week later. It also credited him with subsequent improvements that reduced costs to a tenth of their previous level, though the cost scope and measurement period were unspecified.

Ongoing model-quality monitoring was less settled than conventional operations. Latency and uptime could use ordinary software practices, but the team acknowledged it lacked a good solution for continuous quality monitoring. Its main defense against unsupported output was internal benchmarks representing user queries and tasks, backed by carefully vetted datasets. New model releases prompted comparisons against production every couple of months. Actual user inputs could differ from those datasets—a distribution shift—but the team placed greater emphasis on getting training and evaluation data right.

Model choice followed the task. The team liked Claude Haiku’s cost–accuracy tradeoff, while another participant reported disappointing summarization without examples and relayed advice to provide ten. Those were experiences, not comparative benchmark results. GPT-4 Vision enabled a more concrete capability: processing data in tables. Open and closed models handled roughly similar query volumes, but closed models accounted for more cost and compute because they served tasks the available open models could not adequately perform.

Notebooks make the process reusable

Elicit notebooks were the team’s fourth attempt at open-ended research workflows. An early effort, probably in early 2021, let GPT-3 brainstorm questions and decompose each into subquestions; the branching structure was overwhelming. A late-2022 attempt sought generic abstractions for many workflows but failed to narrow the problem enough. Around early 2023, computational notebooks offered a concrete pattern: flexible analyses assembled from dependable building blocks, with each step a suitable unit of model work.

The key distinction from chat was a process that can be rerun. The proposed workflow was to analyze a few papers, check the conclusions, debug the steps, and then apply the same process to 10,000 papers. That scale was a product vision, not a reported validation. The useful analogy was a data analysis developed on a small dataset and reused on a larger one.

The intended building blocks expressed research operations: paper search, extraction, and concept lists. Detailed model reasoning would remain inspectable one level below them without always occupying the main interface. Shared templates could package frequent action sequences into reusable operations, much as programming abstractions package repeated work.

The team envisioned a gradual path from notebooks to agents: 1. A human chooses and adds action steps. 2. A model predicts which action the human would choose. 3. Predictions reliable enough for the task become automatic actions. This was a proposed progression toward autonomy, not a claim that the required reliability had already been achieved.

Make checking easier and budget it by task

As models perform more work, Elicit expects people to do more evaluation. Its interface already linked generated answers to citations and highlighted relevant passages, letting users inspect context and check whether the text supported the answer. Faster spot-checking was a planned extension of that design.

Uncertainty flags helped direct that attention. For an extraction such as a study’s sample size, Elicit asked models to assess confidence and flagged uncertain results. An interviewer questioned the reliability of generated confidence; the team reported fairly calibrated estimates that varied by model, without quantitative calibration results. Sometimes a separate model reviewed the answering model’s output and reasoning, which the team said worked better in some cases.

Those checks require more model calls, so the appropriate budget depends on the project. A casual question about creatine warrants less expenditure than a systematic review replacing substantial research-assistant labor. The conversation used $100,000 as an illustrative human-equivalent project cost, not an Elicit bill. The ambition was to convert willingness to spend into better-checked answers, with more control over computational effort than model selection or prompting alone provided.

At the time, credit-based pricing reflected increasing amounts of work: - Abstract summaries used an open-source model. - Additional columns added extraction and uncertainty features. - High-accuracy columns also parsed tables. The team had considered awarding credits for substantive feedback but had not implemented the exchange. It still needed a way to identify contributors likely to provide careful evaluations.

Long context moves the inspection problem

Elicit’s search pipeline first retrieved roughly 400 relevant papers using embedding-based semantic search, then ranked them. Longer context windows—the amount of material a model can consider at once—could support listwise reranking, in which the model compares candidates together. This makes relative relevance and diversity part of the decision, potentially avoiding ten nearly identical papers at the top. The team considered the additional cost and latency more acceptable for power users seeking better results.

Longer context also changes where decomposition happens. A pipeline that retrieves a passage and answers from it already has a passage to show the user. When a model receives the whole paper, the system must locate supporting passages afterward. More context can move work from generating the answer to making it checkable.

A deeper problem is causal source attribution: establishing that removing a passage would change the answer. One proposed approximation was to remove a chunk, rerun the question, and compare results. That would be expensive; obtaining the same counterfactual information directly from the model remained an aspiration.

Retrieval remained necessary because Elicit’s workloads exceeded context limits. It also provided identifiable failure points: retrieving the wrong paper, missing the relevant paragraph, or answering incorrectly afterward. Putting everything into context can simplify model replacement, but a replacement that silently becomes worse at finding relevant information is harder to debug.

What the papers say—and what else to investigate

An interviewer described the downside of strict document grounding. Given AMD MI300 and Nvidia Blackwell specifications, a document-chat tool reportedly declined to answer an NVLink compatibility question because the documents did not say. GPT-4 answered when asked without those documents. The anecdote illustrates a task-design choice: should the system report only what its sources establish, or pursue an answer beyond them?

For a researcher describing other papers in a background section, the team favored strict fidelity. For broader inquiry, it proposed a middle ground: turn an unsupported hunch into a search query, retrieve material that substantiates it, and then give an evidence-backed answer. Elicit’s focus at the interview was what the papers say; supporting the broader inquiry was a future ambition.

Users could make extraction tasks precise through the table interface. Elicit offered more than thirty predefined fields, including methods, findings, and participant counts, and exposed their prompts for editing. A population question could be changed from people to rats. Custom instructions could also produce categories such as prospective, retrospective, or case study, turning a methodology column into something users could filter rather than only read.

One unexpected use came from a medical-affairs employee at a genomic sequencing company, who reportedly entered test results as queries to find papers that could help doctors interpret them. This prompted a proposed evaluation comparing infectious-disease specialists’ written interpretations with Elicit’s answers. The intended benefit was evidence-backed interpretation where specialists were scarce; no evaluation results were reported.

Reliable research from unreliable components

Elicit’s stated values were systematic, transparent, and unbounded. A proposed longevity-research application illustrated how these could extend into planning: organize interventions across scientific subdomains, compare candidate experiments by cost and expected return, and examine evidence that they would produce new information.

The team wanted models to progress from novice-level work toward expert performance through explicit demonstrations of competence. Even systems that surpassed human researchers would still need inspectable processes for auditing and spot-checking. One speaker identified weak world models—representations of a domain’s underlying structures and relationships—as a barrier to autonomous discovery. In that assessment, models were better at conclusions near existing human conclusions than at making surprising connections.

At the interview, Elicit had twelve people, about half in the Bay Area and the rest across the US and Europe. The transition from nonprofit to public benefit corporation was operationally uneventful because the team had already been shipping weekly. It framed the structure as a commitment to earning revenue through its mission: helping users discover truth and conduct research rigorously.

Growth plans prioritized engineering, alongside design, product marketing, and go-to-market roles. Elicit sought senior software engineers without requiring specific AI expertise: the work involved orchestrating complex tasks, scaling notebooks, designing abstractions, and keeping applications reliable when individual components failed. Quickly streaming work back to users was a specific requirement. The team had briefly investigated existing orchestration frameworks but left their suitability open.

The closing advice returned to the choice of application: be deliberate about which AI systems to build and support, and take their different effects on the world seriously.