Shreya Shankar

Shreya Shankar is a computer science researcher whose work addresses how to make AI-powered software reliable: managing its data, improving document-processing pipelines, and aligning automated evaluators with human judgment. Her collaborative contributions include EvalGen, DocETL, and the Data Agent Benchmark (DAB). At UC Berkeley, she worked with Aditya Parameswaran and built the DocETL stack for LLM-powered data processing.[3][4][5][6]

Shankar coauthored Who Validates the Validators?, submitted in April 2024, with J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, and Ian Arawjo. Their EvalGen interface helps users generate evaluation criteria and implement assertions. It addresses a recursive problem: an LLM-generated evaluator can inherit the weaknesses of the models it is supposed to assess, so it also needs human validation.[4]

EvalGen generates candidate implementations—Python functions or LLM grader prompts—and asks humans to grade a subset of model outputs. It then uses those judgments to select implementations that better align with the user's grades. The team's qualitative study identified criteria drift: users need criteria to judge outputs, but inspecting and grading outputs also helps them define those criteria. Some requirements depend on the particular outputs people encounter, complicating approaches that assume evaluation criteria can always be fixed in advance.[4]

Shankar coauthored DocETL with Tristan Chambers, Tarak Shah, Aditya G. Parameswaran, and Eugene Wu. First submitted in October 2024, the paper describes a system for optimizing complex document-processing pipelines. Its motivating example is exhaustive extraction of clauses such as force majeure or indemnification from lengthy legal documents: even an optimized prompt may miss instances, making it useful to decompose the data, the task, or both.[5]

DocETL gives users a declarative interface for specifying pipelines and uses agents to rewrite them. Its design combines logical rewrites tailored to LLM tasks, task-specific validation prompts for evaluating candidate plans, and an optimization algorithm that searches for promising plans while accounting for the latency of agent-based generation and evaluation. The execution plan itself becomes something to improve, alongside the prompts used within it.[5]

In March 2026, Shankar coauthored Can AI Agents Answer Your Data Questions? with Ruiying Ma, Ruiqi Chen, Yiming Lin, Sepanta Zeighami, Rajoshi Ghosh, Abhinav Gupta, Anushrut Gupta, Tanmai Gopal, and Aditya G. Parameswaran. Their Data Agent Benchmark evaluates the full pipeline of integrating, transforming, and analyzing data across heterogeneous database systems, where references can be inconsistent and relevant information can be buried in unstructured text.[6]

DAB contains 54 queries across 12 datasets, nine domains, and four database management systems, grounded in a formative study of enterprise workloads across six industries. This extends evaluation beyond isolated natural-language-to-SQL translation or questions over small tables supplied in context. In the paper's experiments with five frontier LLMs, the authors reported that the best-performing model, Gemini-3-Pro, achieved 38% pass@1 accuracy.[6]

These projects followed Shankar's earlier work on production machine learning. In a March 2023 interview, she described the development-versus-production mismatches and data-management questions she was investigating during her Berkeley PhD.[1]

Shankar studied computer science as an undergraduate at Stanford and worked as an ML engineer before her Berkeley research. Her career included Facebook, Google Brain, and becoming the first ML engineer at Viaduct. In the March 2023 interview, she described her PhD as databases research focused on data management for AI- and ML-powered software.[3][1]

Shankar described an interview study whose iterative transcript coding surfaced velocity, validation, and versioning as themes in ML development. Researchers revised their questions and coding after successive groups of interviews. She also said she liked examining the repositories, issues, and evolution of tools people actually used to understand their practical problems.[1]

Her research sought a unified development and production experience. She pointed to differences at several levels: developers wanted interactive experimentation and notebooks, while production features might arrive on schedules with lag or staleness that training did not simulate. She recalled finding leakage assumptions while examining colleagues' code at an unnamed previous company. Even using a summary of the full dataset to design features before a train–test split could introduce leakage; she argued for system guardrails that would make exploration safer.[1]

Shankar proposed treating models as regularly recomputed data transformations: “Models are views that are frequently recomputed over your data to use database speak”. In the system she was then developing, the execution engine would monitor data and refit computations when distributions changed, making monitoring and retraining part of pipeline behavior. This was an in-progress research proposal, with the goal of making development-time performance reflect deployment behavior.[1]

Her practical advice included versioning code and data around every experimental run and validating pipeline inputs for completeness, schema consistency, and changing values or vocabulary. She also emphasized that ML validation could generate too many false-positive alerts: some data corruption is tolerable if predictions remain useful. For model selection, she recommended first considering whether a team could host, serve, and maintain its own model, then weighing API latency and cost.[1]

In that same interview, Shankar explained why she was not training models in her academic setting: compute credits alone did not provide the operational support she remembered from Google Brain, and infrastructure problems would consume too much of her time. Her broader concern was deployment: “I consistently think people underestimate the data management challenges.” She expected organizations to struggle to turn impressive AI prototypes into usable production systems.[1]