Hiring AI Engineers for the Work Between Models and Users

Build a dependable application around unpredictable models

Ask a language model for JSON and it may return something that is almost JSON. Response times can vary dramatically, too. James describes Elicit regularly seeing tenfold variation in P90—the 90th-percentile response latency—over half-hour to hour-long periods, without specifying a model, provider, or workload. The engineering task is to make the application responsive and reliable despite unpredictable timing, formats, and meaning.

Elicit applies language models to scientific work and literature search. Adam sees it as a tool for thought, motivated by the possibility of accelerating discoveries such as new medicines. His contribution included helping translate the team’s expertise into accessible writing, including the hiring article discussed in this June 2024 conversation.

James arrived after roughly 15 years in technology leadership. Around 2019, he began about a year of reading and practical study, then considered an interpretability and transparency startup. When he approached Andreas about advising it, Andreas invited him to lead engineering at Elicit instead. Much of his software engineering experience transferred: building AI applications involved familiar work with a few consequential differences.

Elicit’s role emerged from staffing needs rather than a new title. James identifies three hiring priorities: - Software engineering skill: build and maintain the application around the model. - Curiosity about machine learning (ML): explore language models and learn what they can do. - A faults-first mindset: design for things going wrong instead of leaving failure handling until the end.

Failure handling matters from day one

Adam finds the combination distinctive: a developer focused on an end user’s task also needs skills associated with distributed systems. At Elicit, this means application-level retries, fallbacks, timeouts, careful error handling, and parallel work to compensate for slow model responses and substantial input/output activity. These concerns arrive before large scale, making experience with demanding backends useful even in a small AI application.

Two implementation choices make that discipline concrete: - Shared data shapes: Python definitions are the source of truth; TypeScript types are generated through OpenAPI definitions. This helps engineers reason across backend and frontend about structured data obtained from model responses. - Explicit error cases: the team reports using checked exceptions in Python so type checking helps expose unhandled failures. The implementation is unspecified, but its purpose is to make forgotten edge cases visible.

Interviews must exercise the same habits. Solving the happy path and promising to handle edge cases later provides weak evidence when that path is narrow. Elicit uses two technical exercises: - Coding in an existing application: add features and fix bugs in a codebase where success requires handling errors and edge cases. - System design under failure: explain how the application continues when a node dies, a network slows, or a database exhausts capacity. Neither exercise needs to center on language models. Both put defensive reasoning inside the work being assessed.

A fallback can make the product worse

Recovery mechanisms have product and business consequences. A retry may rescue a request while increasing inference or infrastructure costs. A fallback may preserve availability while lowering answer quality. The engineer must judge whether the recovered experience is worth its cost.

A recalled search-provider slowdown or outage made this tradeoff tangible at Elicit. A fallback continued returning results, but they were worse. New users would not know they were seeing degraded behavior. The team replaced the fallback with an error message: for this search feature, an explicit failure was preferable to silently worse results. The right balance depended on the application, user, and feature.

A model-provider fallback could be acceptable if task performance remained sufficiently close. That was a conditional example, not a benchmark result. Elicit maintained fallbacks for selected crucial functions rather than every task. Two distinct maintenance concerns remained: - Protocol compatibility: increasingly similar Anthropic and OpenAI APIs had reduced integration work at the time. - Behavioral compatibility: comparable performance still required provider-specific prompts, and infrequently used backup paths could become stale.

Elicit’s regular model switching helped make maintained fallbacks plausible. Behind one interface were several dozen main paths, including search, data extraction, and chat with papers. The team selected models and prompts for each path, testing both major providers and models on its own infrastructure. Choices reportedly changed weekly for reasons including cost and performance. An actively evaluated alternative is more credible than a neglected backup.

Standardize around a specific problem

An internal AI gateway—a central point for controlling model access—may suit a large organization with compliance requirements. A hypothetical requirement to keep data in the EU illustrates its value: centralized controls could prevent an engineer from choosing an inappropriate regional endpoint. Elicit’s small team, however, wanted direct control over models and prompts.

Data security, centralized billing, and reusable prompts solve different problems. Platform fit depends on which problem needs solving. For Elicit, differentiation lay in selecting and combining models around users’ needs. Abstractions that restricted those choices could obstruct experimentation and rapid iteration.

The interviewer identifies a gap between serving models and supporting complete application development. The comparison with Rails concerns when patterns become ready to standardize: years of database-backed application development had exposed recurring needs that a framework could encode, making development and onboarding easier. The succession of React state-management libraries illustrates the opposite pressure—abstractions changing while practical needs are still being discovered. One participant, qualifying their view as a relative newcomer’s impression, considered AI development in 2024 too unsettled for extensive standardization, while welcoming framework experiments.

One speaker offers an informal prediction: framing the ML problem may remain difficult longer than precise prompt phrasing. Operational uncertainty may also persist. A model moves an untrusted boundary into the application’s foundations, requiring defensive reasoning about internal behavior as well as external input. The interviewer distinguishes code coordinating an application from a large language model (LLM) coordinating it; in the latter, nondeterminism and prompt injection affect the central orchestrator itself.

Curiosity should change what you build

Elicit evaluates a new model on existing tasks, then explores capabilities that might support new uses. Observable behavior matters even when developers disclose little about the model’s internals. Reasoning, self-assessment, and confidence estimates are among the possibilities worth investigating—not improvements established by the conversation.

The evaluation framework measures accuracy, confidence, performance, and cost. Product judgment then takes two forms: - Improve an existing task: a slightly less accurate model might be worthwhile if much faster and cheaper; greater accuracy might justify more latency and expense. - Enable a new feature: a capability change might make a previously impractical user task feasible, requiring more than a model replacement. These are possible tradeoffs. Choosing among them requires understanding what users are trying to accomplish.

An Elicit team member describes model-release discussions as focused on specific applications and new product possibilities, rather than headline parameter counts or context-window sizes. Curiosity becomes observable through action: reading papers, asking useful questions, and building side projects show interest translated into learning.

The three-tier Elicit ML reading list provides structure for that learning. Elicit nevertheless reported no ML-focused technical interview for this role. It assessed software engineering directly and explored ML curiosity and conceptual familiarity through conversation. Candidates were expected to develop a view of the product and ask interesting questions as interviews progressed; exhaustive familiarity at the outset was unnecessary.

Give the model room, then handle what returns

An experienced engineer describes joining Elicit as an exercise in unlearning. Familiar software patterns offered well-defined operations behind validated inputs. An opaque model introduced uncertainty deeper inside the application, where further debugging eventually reached a boundary. An ML-first approach meant allowing the model room to solve a problem without prescribing every part of the solution.

An early citation-generation task exposed the tension. The application needed citation markers in generated text to correspond to footnotes. The engineer wanted structured output to guarantee those relationships. Andreas proposed a few examples followed by a regular expression to handle the response. The engineer resisted because the model could return anything. The anecdote describes that disagreement without establishing an implemented solution or reliable citation results.

Both approaches have limits: - Too much freedom: an overambitious request can produce repetition, loops, or irrelevant output. - Too much constraint: aggressive decomposition can prevent the model from using capabilities the engineer has not anticipated. The proposed balance is exploration coupled with defensive handling downstream. Evaluation must support that balance: a small sample can miss hallucinations.

Adam sees this as engineering’s familiar combination of creativity and discipline applied to a new medium. The desired candidate explores possibilities and makes systems reliable and repeatable. Infrastructure experience and willingness to relinquish control can pull in different directions, but career stage does not determine the balance. Senior and junior engineers can both develop the side they have practiced less.

Find evidence of interest and make the work visible

At the time of the conversation, professional language-model experience was optional. Conventional engineering experience could be paired with side projects, hackathons, or writing. Model APIs let candidates demonstrate interesting application work without training models themselves.

Sourcing methods reflected different kinds of evidence: - Experienced engineers: LinkedIn and direct outreach could find people with clear software expertise but no visible ML interest. Explain how their existing skills would contribute. - Earlier-career builders: blog posts, challenges, and community activity helped attract people with smaller professional footprints. The interviewer also recommended San Francisco hackathons as a place to find self-directed builders. - Mission-aligned candidates: specialized communities and job channels narrowed the search on both sides. Elicit reported hiring through the 80,000 Hours list and participating in AI safety and effective-altruism communities.

Adam recommends pairing public visibility with outbound recruiting. Technical posts, open-source releases, job descriptions, and events should reveal what the work actually involves. Small companies generally cannot rely on inbound interest alone, though a strong brand or overlap between users and potential hires can help. Public technical work also gives later outreach context: someone who already likes a project has a reason to engage.

Those relationships should extend beyond a vacancy. Following interesting work, attending meetups and conferences, and participating in open source create opportunities to learn from people facing similar problems. When someone signals interest in a new role, an existing connection creates a natural opening. Recruiting grows from sustained participation rather than making every interaction a hiring transaction.

Make the interview a sample of the job

The closing advice includes an appeal to seek out highly capable young people. One participant was struck by the maturity and self-direction of early-twenties candidates compared with their own younger self—a personal impression supporting broader recruiting, rather than a measured generational claim.

Elicit’s central interview principle is to simulate working together. Candidates entering the formal process are already expected to have demonstrated unusual capability. Realistic tasks then provide evidence in both directions: the employer sees how someone approaches work and collaborates, while the candidate encounters the teammates, tools, and environment they would join.

That fits an empirical role in an emerging field: applying knowledge in context is especially informative when there is no established hiring playbook. The associated hiring piece shares an interview loop, reading list, and AI engineer job description, offering concrete starting points for other teams.

The interviewer closes with the labor market behind the role: model-lab incentives, company demand, engineer supply, and interest in the work all contribute to its emergence. They decline credit for coining the term. Giving the work a recognizable name serves a practical purpose—helping companies create roles and helping people find one another.