From MLOps to LLMOps: The Data Work Behind the Model

A Gantt chart still leaves room for pizza

Shreya Shankar planned dinner for roughly 25 guests with a Gantt chart. Scheduling was only part of the challenge: the menu had to accommodate different diets, and the children were unlikely to eat tofu. Pizza ordered for them proved popular with the adults, too.

Her technical work addresses another gap between planning and execution: how to develop machine learning software under the conditions it will encounter in production. In the March 2023 Latent Space conversation, Shankar describes her Berkeley PhD as database research focused on data management for AI-powered software. Her goal is a unified development and production experience, where both environments incorporate the same execution assumptions.

Why experimentation pulls development away from production

Machine learning operations, or MLOps, concerns the work of bringing experimental models into production and maintaining them. The interviewer introduces three organizing concerns from the paper under discussion: velocity, validation, and versioning. Shankar’s team developed these themes through practitioner interviews, annotating transcripts and revising questions and tags after batches of three or four interviews. For participants discussing development, the leading concern was the ability to experiment quickly.

Maturity did not remove that uncertainty. Practitioners at Microsoft and Google, already operating and frequently training production models, still described highly exploratory development. Shankar likens it to throwing 100 darts for five to stick—a metaphor for experimentation, not a measured success rate.

The demand for speed separates development from production at several layers: - Hardware access: developers want direct access to graphics processing units (GPUs), without submitting each experiment to a Kubernetes cluster and waiting. - Feature execution: training may generate features—the inputs used by a model—immediately, while production generates them on a weekly schedule. - Interaction: notebooks support visualization and inspection of state, while production jobs typically use different interfaces. Tools such as Papermill address parts of this separation, but Shankar argues that differences throughout the stack help explain bugs that appear after deployment.

Feature freshness makes the execution problem concrete. Training can assume every input is available and current; production may refresh it hourly, daily, or weekly. Development needs to account for that lag while preserving interactive experimentation. Sharing an interface alone cannot make the underlying assumptions agree.

Exploration needs guardrails against leakage

Data leakage occurs when information that should be held out of development influences the model. It can begin before training: a developer loads data from Snowflake, inspects a dataframe summary, uses those observations to design features, and only then separates training data from test data. The eventual test set has already influenced the feature choices.

The consequences vary. Using the target label—the answer the model should predict—as an input is especially severe; letting whole-dataset summaries shape feature design is subtler. An informal review of colleagues’ code illustrates how readily such assumptions arise without guardrails. The proposed direction is exploratory data analysis without leakage, preserving the ability to inspect and understand data while protecting held-out information.

These questions sit within two broad, evolving Berkeley research agendas. Sky Computing Lab projects seek to run jobs across clouds according to cost and resource availability. The EPIC Data Lab direction emphasizes low-code and no-code data management, interfaces, and execution models that help people reason about their systems.

Treat the model as a recurring computation

Dynamic-data benchmarks and streaming datasets were gaining attention, yet the discussion identifies a persistent habit: treating the trained model as a static artifact. The proposed alternative is models as database-like views—computations over data that are refreshed as the data changes. The objective is an evolving prediction process, which need not require retraining a large language model every day.

The research system described would make recurrence part of the execution engine’s understanding of train or fit. It would monitor underlying data distributions and refit computations when it detected changes. Testing a pipeline before deployment would therefore include monitoring and retraining as part of its behavior.

The success criterion was agreement between development-time and deployed performance. This remained an aspiration: the project was unfinished and had not been submitted. The reference to avoiding an expected 10% drop was not a measured result or a specified benchmark.

Preserve the trial, then validate what matters

Experiment versioning should preserve the code, data, and other state associated with an individual trial. A result intended for publication needs a recoverable reference, even if committing code for every training run feels burdensome. The discussion also acknowledges that a preexisting commitment to versioning may have influenced the interview study’s framing.

For datasets small enough to copy conveniently, the starting point is simple: save the training set whenever the script runs, perhaps in S3 storage. DVC, Weights & Biases, and MLflow are mentioned as tools with data-versioning capabilities, with uncertainty about how well those capabilities worked at the time.

Validation addresses a different failure: performance drops, but the team cannot explain why because it has insufficient checks on pipeline inputs. Suggested baseline checks include: - Data completeness and schema consistency. - Changes in the most frequently occurring value. - Vocabulary changes in language-model data. As feature counts and datasets grow, however, these checks can generate too many false-positive alerts.

The deeper question is whether an input anomaly harms predictions. A model may tolerate some corrupted data, making an alert a false positive when judged by its effect on model outputs. The research approach described learns from performance drops to improve validation precision, combining older database techniques with adaptations for machine learning. The paper was being resubmitted after rejection; no quantitative improvement was reported in the conversation.

An API leaves a stateful application to build

Model selection begins with operational capacity: can the team host, serve, and maintain a model over time? Producing a customized model may leave the organization with an artifact it cannot support. For some teams, a model API is the better long-term choice, provided its latency and cost are acceptable. Interview participants reportedly used in-memory caching to avoid repeated calls.

Large language model operations, or LLMOps, still requires managing application state. Prompts evolve during experimentation, and their contents can depend on changing database results. One pattern retrieves recent, similar records and inserts them into a prompt. A history of prompt text alone does not manage the retrieval and construction process that produced it.

Production fixes add another component. Practitioners often address unwanted outputs with custom filters that prohibit particular outputs or behaviors. The application becomes a combination of a model API, a database-context service, and filters. LLMOps inherits MLOps’ integration problems: state spans the pipeline, behavior needs monitoring, and developers need a fast feedback loop across the assembled system.

Shadow evaluation benchmarks another model against the production model without requiring direct user feedback. Predictive analytics and predictive maintenance are offered as settings where it worked well. Outputs that consumers immediately interact with complicate that comparison; adapting shadowing to those settings remained unresolved. A/B testing and staged deployment are also named as evaluation and rollout options.

Find research questions in working software

Even weekly university talks and direct access to authors do not make keeping up easy. The respondent describes abandoning an ever-growing bookmark list and accepting missed papers, trusting that essential work will eventually become necessary to read.

Actively used tool repositories offer another way to find consequential questions. Their issues and evolution reveal practical problems and help establish their significance. The recommendation concerns software tools, rather than specifically repositories accompanying papers. Social-media complaints can surface problems without showing how substantial they are.

The methodological connection is grounded theory: repeatedly observe a domain and revise assumptions until further observation stops changing them. That process becomes useful again when the domain shifts. Teams previously making predictions from tabular data were beginning to incorporate language data, including customer feedback, through large language models. Whether they were addressing inadequate earlier models, responding to pressure to keep up, or pursuing other benefits remained a question for practitioner interviews.

Cloud credits do not buy a supported research environment

Academic research raises a disagreement about resources and incentives. The respondent argues that PhD students lack the compute and data available to large companies, strongly discouraging someone focused on language-model training from pursuing that goal at Berkeley. A host counters with EleutherAI’s reported independent reproductions of GPT-3. The respondent agrees that independent reproduction is possible.

The remaining disagreement concerns what a PhD program rewards. The respondent argues that students need publications, while reproduction may not yield a paper. Stanford’s reproduction work is acknowledged, with substantial efforts seen as more plausible when they begin a multiyear research agenda. The host emphasizes their educational value and proposes reproduction as a first-year rite of passage.

Compute access and operational support are different resources. Cloud credits leave researchers responsible for collecting data and keeping experiments running. The respondent recalls experiments at Google Brain roughly five or six years earlier running without infrastructure problems in their experience. A cluster supported by site reliability engineers provided an advantage that credits alone could not reproduce; otherwise, infrastructure work could consume the time intended for research.

For new graduates seeking impact, the recommendation is to consider data management research or industry platform work on provenance and training efficiency. These are presented as durable problems. That preference coexists with enthusiasm for OpenAI’s model research and the expectation that additional compute and data would make its models more interesting.

Exposure makes careers imaginable

She++ introduces high school students from underrepresented minorities to coding. One participant recalls organizing visits to Stanford and Silicon Valley while in college, and seeing nearly all the participating students move from uncertainty toward intending to major or minor in computer science. The recollection concerns intentions, not subsequent enrollment outcomes.

Exposure matters because people must first know a career exists. One speaker recalls growing up in a small Texas town without knowing what a programmer was; a mentee whose father was a venture capitalist already knew venture capital was an option. Berkeley mentoring and summer undergraduate research programs are discussed as further ways to introduce underrepresented students to opportunities.

The prototype is only the beginning

The lightning round starts with a concrete pleasure: managed Stable Diffusion services, including Dreamstudio, make preparing talks more enjoyable. One speaker uses them for illustrations—pictures, not graphs—and reports generating a bridge between a sandbox and a city to represent the development–production gap. Service choice depends on available credits.

The forecast for the year following this 2023 conversation is harder: organizations would struggle to turn impressive language-model prototypes into production systems because they underestimated data management. A data management platform is defined broadly as a system that executes or orchestrates data transformations across a pipeline, including feature stores and monitoring. A model API call leaves that surrounding work to be done.

A product request makes the integration problem tangible. While building a personal Jupyter Notebook project and learning Next with Next UI, one speaker generated a pleasing color palette but struggled to apply it consistently. The desired tool would read the application, generate and apply a palette across frameworks, and support changes over time—for about $5 a month. Another participant suggests a design system, while acknowledging that knowing the available frontend tools does not make choosing among them easy.

Shankar closes with enthusiasm for AI’s wider reach, comparing the excitement to the arrival of Google and YouTube. Her grandmother, who did not know what a database was, had seen a video about ChatGPT and asked over WhatsApp whether it was a fad. Shankar sent her access to try it, though she did not know whether she had used it.

For readers continuing the conversation, Shankar pointed to Twitter for her online presence and email for direct contact. At the time, she had closed her direct messages because of their volume and said she would respond to email when she could be helpful.