Notebooks as Chat++, RAG as a Recommendation System

Start with the warehouse

As Blue Bottle’s first data hire, Bryan Bischof began by getting the data warehouse in order. He then built and expanded a team whose work centered on demand forecasting, alongside recommendation systems, website optimization, and analytics. Answering questions about coffee and pastries required usable data infrastructure and an understanding of the business.

That combination runs through his November 2023 conversation about [Hex Magic](https://hex.tech/product/magic-ai/), a suite of AI capabilities embedded in Hex’s analytics platform. Bischof was then Hex’s head of AI, leading Magic, and had just completed a recommendation-systems book using Python and JAX with Hector Yee. Magic’s goal was to make existing users more effective within their analytical workflows.

When Bischof joined, before GPT-4 and the newer chat API, Magic could already generate SQL and Python from natural-language requests and explain existing code. He found that even these bounded capabilities substantially accelerated his own product analytics, though he gave no measured speedup. Moving from GPT-3.5 Turbo to GPT-4, then adding function calling and a better understanding of agent workflows, expanded the tasks the team could tackle.

Let the model write SQL; let the application build the cell

Hex gives the model and application code different responsibilities. The model generates content in a language it handles well; application code converts it into Hex’s representation. For a SQL cell, the model writes SQL while Hex supplies the notebook wrapper. Magic’s chart generation similarly avoids requiring the model to produce Vega chart specifications directly.

Bischof therefore rejected fine-tuning a model specifically to understand Hex cells, favoring preprocessing and postprocessing. Earlier work with GPT-2 and GPT-3 at Stitch Fix had shown him the value of reshaping a task before adapting a model. Clothing descriptions, for example, might seem to require image input, but existing structured attributes could supply enough information to generate reasonable text.

At the time, Hex used GPT-3.5 Turbo and GPT-4 in production, and Bischof considered GPT-4 effectively necessary for many applications. His argument for a general model went beyond code quality: Hex’s own analytics concern projects called Hexes, while Blue Bottle’s concern coffee, pastries, and demand forecasting. He doubted that a model fine-tuned only on Kaggle notebooks would cover that breadth. Business analysis requires more than notebook syntax.

A cell is one thought; an airlock is a proposed workflow

A notebook cell provides a natural unit of AI assistance: one cell, one analytical idea. Removing a prefix from every string in a DataFrame column may take only two lines of pandas, but it is also a complete task a user can delegate. Asking Magic to perform that operation is described as faster than writing it manually—a personal workflow judgment. Keeping cells small helps preserve these boundaries; a cell that cannot fit on one page is treated as a code smell.

Larger analyses need cells that use one another’s outputs. Hex represents their execution with a directed acyclic graph (DAG), tracking dependencies between cells to support reactive notebook behavior. Its internally named airlock extends AI assistance to a short sequence of interdependent cells. Users preview the generated workflow, then accept or reject the group together.

This is the idea behind Chat++: a notebook expresses a sequence of thoughts through executable cells, optionally joined by explanatory prose. AI can participate at those existing boundaries, and the conversation produces work directly inside the analysis environment.

Magic initially focused on users already comfortable with Hex. Customer enthusiasm for letting less technical stakeholders analyze data themselves introduced a harder requirement: data teams needed visibility into that work and ways to prevent false conclusions. Hex’s data manager let users add relevant warehouse metadata for Magic to use. Further governance capabilities were still forthcoming.

From raw capability to useful applications

An analogy credited to Adam Azzam of Prefect describes AI as an iron mine: extracting something useful takes substantial work. One participant worries that infrastructure vendors—including tools for people building AI tools—have outpaced valuable applications, and can recall perhaps only Copilot as an especially impressive integration at a medium or large company. Another argues that promising applications may serve markets infrastructure builders overlook. These are competing personal assessments of where value is emerging.

A personal Memex, or memory extender, offers a concrete application to pursue. It would index information from consumed podcasts, films, and books, then retrieve a forgotten detail during a later conversation—such as the name of someone associated with an idea. Its central challenge is quick access at the moment of need. Better retrieval systems, simpler retrieval tooling, and natural-language interfaces make the proposal feel more feasible.

Wearable access could bring that assistance into everyday conversations. Enthusiasm for trying an always-recording AI pendant comes with acknowledged privacy challenges and the expectation that a satisfying memory assistant might still take years to arrive.

Experimental engineering starts with an objective

Building large language model (LLM) applications combines sound software engineering with experimental machine-learning practice. Evaluations start by defining success: specify the objective, decide how to measure it, and connect that measure to the problem being solved. Hex’s evaluation work began in February, drawing on established experience evaluating machine-learning systems.

The engineering team included a dedicated experimentalist who formed hypotheses about model behavior, ran experiments, and extended evaluations to test them. Supporting tools needed to be composable: engineers should be able to change parts of the system without paying a large integration cost on every iteration.

RAG recommends resources to an LLM

A request to analyze customer trends exposes the difference between knowing SQL and knowing a warehouse. A model can invent plausible tables and columns, yet lack the actual joins, data caveats, and required WHERE conditions. Plausible SQL can still describe the wrong data system.

Retrieval-augmented generation (RAG) supplies relevant information to a model for the task at hand. For Magic, that means retrieving warehouse context: which data objects exist, how they connect, and what conditions govern their use. The retrieval system can be built to understand a customer’s warehouse structure even when the language model does not.

This leads to RAG as a recommendation system whose consumer is an LLM. The system recommends resources that help the model complete a task. That framing broadens retrieval design beyond splitting documents into chunks and combining partial answers; it also guided the retrieval improvements described for Magic.

Learn from concentrated experience

Technical books can distill understanding earned through years of work. Will Kurt’s Bayesian Statistics the Fun Way supplies an example: a reader already familiar with Bayesian statistics describes learning substantially from a book aimed at less experienced readers. Twitter remains useful for early signals, despite its noise. The preference is for better sources, rather than more channels.

That preference leaves room for tentative ideas and blogging. The encouragement is to publish even naive approaches; the stronger objection is to short technical pieces whose primary purpose is advertising.

Turn recurring failures into separate evaluations

Objective evaluation requires making good and bad behavior explicit, even when overall quality seems nuanced. During Magic’s early internal testing, one engineer describes manually reviewing every internal request for opportunities to create evaluations. That labor exposed recurring ways the model misbehaved.

Task success and individual failure modes need separate checks. For generated SQL and code, useful questions include: - Table references: Does the query use tables in the intended database? - Answer correctness: Does it return the right answer? - Code completeness: Is a required closing brace missing? - Scope conflicts: Does the code redefine an existing variable or assign a cell output to a name already in scope? These checks distinguish problems that a single success score obscures. Valid table references alone do not establish a correct answer.

Some failures also become targets for defensive postprocessing. A notebook output-name collision, for example, can be detected and fixed in surrounding code. Repeated observation, evaluation, and repair build robustness over time. Hex’s goal of general availability required dependable behavior beyond a collection of successful demonstrations.

Retrieval, ranking, and serving offer different controls

An application team without access to model weights still has substantial room to improve behavior. The recommendation-system analogy, introduced with a reference to Eugene Yan’s writing on discovery-system design, separates three stages: 1. Retrieve useful candidates, using representations that encode relevant features. 2. Rank those candidates for the objective. 3. Serve a result that respects diversity, relevance, hard rules, and availability.

The Stitch Fix example explains why serving needs its own logic. Five shirts with the same shape, silhouette, and cut may rank well individually while forming a poor selection. The final recommendations must account for variety, the request that brought the user there, and stock availability.

For an LLM application, the corresponding controls are retrieved context and downstream handling. RAG can adapt behavior through information supplied to the model without retraining its weights. Recommendation systems provide a precedent: the argument here is that surrounding components often change more frequently than the core ranking model.

Application design must also distinguish where variability is useful and where precision is mandatory. Planning and descriptions of intended actions may allow flexibility. Outputs that must match a schema need explicit constraints, with function calling offered as a way to impose structure. Making these boundaries explicit is described as successful, though no quantitative comparison is given.

The product counterpart is AI in the loop: the user invokes AI for a task, then returns to their workflow. This guiding goal for Magic keeps assistance available where work happens and leaves the user in control.

Experiment beyond the notebook

The closing lightning round begins with a different kind of exploration: a participant describes walking every part of every road in Berkeley, tracking visited places with an app, and seeking new trails. Kensington earns a recommendation for its beauty and quirkiness; curiosity about Berkeley’s own reputation helped motivate the walking project.

Bischof also revisits a mistaken prediction. When the Hugging Face CEO spoke at his Stitch Fix seminar about LLMs defining the next decade of technological progress, he was skeptical. He expected reinforcement learning to produce the larger breakthrough. By this conversation, he considered his dismissal of LLMs a mistake.

His interest in the Memex extends into a broader wish for AI in everyday hardware. One playful proposal is a Big Mouth Billy Bass as a Babel fish: connect the talking fish to Whisper as part of a system that hears one language and speaks another. The idea is an invitation to experiment, rather than a demonstrated prototype or complete translation architecture.

Bischof’s closing recommendation is to learn through use and help others do the same. Within the first fifteen minutes of his semester’s first class, he introduced GPT, discussed appropriate uses, and demonstrated both good and poor data-analysis performance. He advocates extensive experimentation while retaining respect for established machine-learning and data-science ideas.

That invitation reaches beyond engineers. His wife, a technical product manager, used ChatGPT almost daily for communication and learning unfamiliar concepts; his mother and sister had recently begun using it too. He urges listeners to help people in their own lives become familiar with these tools, preparing for a future in which AI assistance is commonplace.