Notion’s Custom Agents: Rebuilding the System Around the Model

The agent’s audience and its access are different things

A custom agent is shared in a Slack channel with one group of people. It can read documents available to another, only partly overlapping group. What has the administrator authorized? Making that relationship understandable required several iterations of the permissions interface for Notion Custom Agents, which use Notion’s tools to perform work in the background.

The product described in the Latent Space conversation followed roughly four or five rebuilds. Work began after Notion gained access to GPT-4 in late 2022. Before native function calling—the ability to request actions through tools—the team built its own tool-calling framework and explored fine-tuning with frontier labs and Fireworks. Multi-turn experiments showed promise, but model capability and short context windows kept them from becoming dependable products.

A guest credits a later Sonnet model with making the previously shipped agent practical. Custom Agents took longer because background execution required substantially higher reliability. Model progress opened the possibility; permissions and product behavior still needed work.

Notion balances those uncertainties through a portfolio: maintain shipped features, develop capabilities that already work, and reserve room for speculative projects. That requires two complementary judgments: distinguish a genuine model limit from missing information or inadequate infrastructure, and prepare products for capabilities that are improving. Early tool-calling experiments and repeated transcription attempts tested both judgments.

Start with a workflow, then justify the infrastructure

A guest uses the Agent Lab thesis in recruiting to explain application development on external models. Their analogy is Datadog and AWS: dependence on infrastructure leaves room for expertise in customer needs. Notion locates that expertise in collaboration, translating varied requests into reusable building blocks.

Sarah Sachs grounds that strategy in a Friday review of custom-agent transcripts around the 99th percentile of token consumption. The team investigates poor performance and creates follow-up tasks around workflows it expects to work reliably, such as email triage. PDF export supplies another concrete test: the need to export a document could justify a computer sandbox with filesystem access and code execution. Starting with interesting tools alone, Sachs argues, leaves the team without a basis for prioritization.

Sachs describes leadership as clarifying objectives and giving people support and a way to prioritize their ideas. Many of the team’s best ideas begin as prototypes prompted by user problems. Repeated rebuilds of the agent harness—the surrounding system that manages an agent’s tools and execution—make willingness to delete one’s own code essential. In the Simon Vortex, Notion’s fast-moving prototyping environment, senior engineers move between experiments and production work; reporting lines need not determine project assignments.

Company-wide hackathons give people outside the immediate project time to learn these tools. One exercise allocated a day and a half to building an agentic tool-calling loop from scratch. Experiments acquire rollout plans and assigned data scientists as they mature, but security review enters early: Sachs says early involvement improves the product and avoids later delays.

A working prototype still needs a platform

Notion’s image-generation feature began with Jimmy on the database collections team wanting generated covers and images inside Notion. The AI team supplied Gemini access, usage tracking, endpoint integration, and evaluation support. The prototype became a full project, but production required more than a successful generation: attachment handling and a completion system expecting tokens had to accommodate images, alongside vendor contracts, compliance requirements, and evaluations.

The ownership rule is that the team owning a product capability also owns its agent-facing tool. The editor team responsible for offline collaboration handles agents editing competing blocks. The database query engine team owns performant query execution for agents. The goal is for the whole product organization to build for human and agent users.

Designers use Design Playground, a separate GitHub repository containing reusable interface components and an agent, to deliver working prototypes through URLs. Engineers’ prototypes are functioning features behind flags. Staff across recruiting, procurement, and IT use these experiments in their daily work, providing rapid feedback—and sometimes changing flags to work around broken behavior.

Easy prototyping makes prioritization more consequential. One view emphasizes the danger of accumulating shallow features; another holds that engineers usually propose sensible additions, leaving sequencing and toggles as the main challenge. The experimental chat interface reportedly had roughly 100 checkboxes. Feature teams must maintain what they ship, supported by a central evaluation framework rather than a small group inheriting every prototype.

Three kinds of evaluation, three different decisions

Evaluations test the behaviors that matter to Notion’s workloads. The team reports slower responses from some providers during working hours, a latency observation distinct from hidden quality degradation. It also reports occasional quality differences between nominally identical models served directly or through Bedrock and Azure. Quantization was suggested as a possible cause, without measurements establishing that explanation.

A quality regression can be acceptable if it buys lower latency without harming behavior the product needs. Prerelease partnerships bring similar decisions: Notion tests multiple snapshots and supplies enterprise-work feedback, sometimes influencing which version a lab releases and sometimes adapting to a release different from its preferred snapshot.

Notion separates evaluations by purpose: - Regression evaluations run in continuous integration, with pass thresholds that allow random variation in model outputs. - Launch evaluations form a report card for defined categories of user journeys, targeting roughly 80–90% success within those categories. - Headroom evaluations deliberately target roughly a 30% pass rate, preserving difficult tasks that can reveal future capability gains. These percentages describe different test populations. The headroom target is not a launch standard. Feature teams own their evaluations; many run through opt-in integration or nightly jobs, with a custom agent directing teams to major failures.

Notion’s Last Exam emerged after earlier evaluations saturated and mostly revealed regressions. A data scientist, a model behavior engineer, and a full-time evaluations engineer were dedicated to the harder tasks. The team reports developing them with Anthropic and OpenAI over the two or three months preceding the conversation.

Model behavior engineering grew from data specialists judging outputs in Google Sheets into a career combining testing, data science, product judgment, and prompt engineering. Its practitioners define successful journeys, compare models, and diagnose failures. Coding agents now help them build evaluations and model-based judges, but substantial supervision remains. The team argues that the necessary judgment and taste do not require a software-engineering background.

The software factory needs an outer system

Notion is making its evaluation system usable as an agent harness. An agent should be able to download a dataset, run an evaluation, investigate failures, and implement a fix while a human observes the surrounding process. Command-line tools make that workflow available to different coding agents. A guest reports strong early results, without quantitative measurements.

The exploratory software factory extends that idea to agents developing, debugging, reviewing, merging, and maintaining software. Human work shifts toward maintaining the surrounding system: monitoring failures, managing approvals, and designing learning or memory mechanisms. Delegation resembles management, but agent workflows can also be modeled as explicit technical states, including what happens when a pull request is blocked.

The proposed design has three central components: - A human-readable, immutable specification layer, potentially committed Markdown files or a database of pages. - A self-verification loop backed by strong testing. - Explicit workflows for bug intake, agent assignment, pull-request creation, review, and merging. The aim is to minimize human intervention while preserving required system properties. This remains an exploration rather than an established autonomous production pipeline.

Compose agents through records people already use

A tenant-application workflow for Kernel Labs makes the product concrete. Previously, an email prompted manual applicant research and an exchange of replies. A participant describes spending roughly 15 minutes setting up an agent that creates a Notion database row per applicant, enriches profiles through web search, and records intended move-in dates. The information lands where the participant already intended to keep it.

Notion’s internal bug-triage agent watches Slack, consults routing instructions, creates a database task, and posts to the responsible team’s channel. A participant describes a major operational improvement from replacing that manual process, while qualifying that most, rather than necessarily all, issues avoid falling through the cracks.

Two patterns compose these workflows: - Shared records: one agent writes to a database that another watches. - Direct invocation: an agent receives permission to invoke another agent. Direct invocation was unreleased at recording time, with release tentatively expected the following week. Participants believed a recursion limit existed but could not specify its value or behavior.

One internal go-to-market user built more than 30 agents for customer research, feedback triage, and related work, then received over 70 daily notifications about blocked tasks. Adding a manager agent that could observe and invoke the others reportedly reduced notifications to about five per day in that deployment. Individual agents could file problems in a shared issue database; the manager could read them and help investigate.

Agent memory uses the same building blocks: editable pages, databases, and subpages that humans and agents can both update. Notion first makes a workflow possible through existing components, then considers dedicated packaging if the pattern proves broadly useful.

Command lines, MCP, and custom integrations

Users can connect Gmail without adopting Notion Mail. Native integration nevertheless gives Notion’s mail team direct responsibility for tools optimized around latency, performance, and quality; external integration uses the available programming interfaces or MCP tools. The team tends to build natively first and generalize afterward.

A guest favors command-line interfaces (CLIs) because a terminal lets agents navigate long outputs and discover functionality through help commands and files. This is progressive disclosure: reveal relevant capabilities as needed instead of loading every tool definition up front. A coding runtime also lets an agent investigate and repair tools where it uses them.

MCP, the open tool-access protocol discussed here, suits narrow agents that do not need a coding runtime. The guest’s permissioning argument is that an agent can be confined to exposed tools. Progressive disclosure can also be implemented around MCP; presenting every tool at once is a harness choice, not an inherent protocol requirement. Notion intends to keep supporting its MCP while customers use it.

For deterministic recurring work, the team favors generating reusable code that calls a CLI. The proposed benefit is paying for code generation once instead of repeatedly asking a model to mediate the same interactions, especially outside the cache window. Under usage-based pricing, that repeated model work raises the customer’s cost of adoption; the discussion gives no measured savings for the alternative.

Quality can justify custom integration regardless of protocol convenience. Notion sometimes bypasses provider-supplied MCP search for Slack, Linear, and Jira to control search behavior and the agent’s subsequent actions. Triggers also require separate implementation in the architecture described. Internally, Notion abstracts tools, agents, completion calls, tasks, and chat types; MCP is one implementation of tool integration.

Give the model an interface it understands

The rebuild history begins with a coding agent given JavaScript interfaces in late 2022. Weak code generation led to a custom XML tool-calling format. It mapped losslessly to Notion blocks and supported page mutations, but the model did not know the format and needed it explained in the prompt. A representation convenient for Notion’s implementation proved awkward for the model.

The replacement was Notion-flavored Markdown: ordinary Markdown at the core, extended where necessary, with fully lossless conversion deliberately abandoned. Database queries followed the same lesson. A restrictive JSON query format gave way to a SQLite-style interface that models handled better. This agent-facing redesign drew on a cluster of SQLite databases already being developed for Notion queries before agents; it was not a claim that all underlying storage had migrated.

The prompting architecture evolved from one prompt containing examples, to tools with examples, to a larger tool inventory without them. Moving toward goal-oriented tool definitions and feedback reduced a coordination bottleneck: contributors had been editing one string, competing over example selection and ordering, with roughly five or six people required to approve changes.

Evaluations made it possible for individual teams to own tool definitions. The centralized AI group then had to become a platform team—a difficult transition that one speaker calls the largest contributor to scaling agent development beyond coding tools. Integration failures remained: duplicate tool names once caused an agent crash and a severity incident.

Tool count became the next bottleneck. A niche tool could attract too many calls and degrade the whole agent, while definitions added token overhead. Notion was implementing progressive disclosure to address both selection and cost. At recording time, the latest agent version contained more than 100 Notion tools, with rollout to Custom Agents still expected in approximately one or two weeks.

Self-configuration stops at the permission boundary

Notion treats Custom Agents as a power tool whose operators should be able to inspect its capabilities. Users can ask about available tools and learn how to prompt their use. The team distinguishes easier setup from hiding complexity: concealing too much can reduce interpretability and capability. It aligned around users willing to learn a deeper system.

The agent writes its own system prompt using configuration and debugging tools, guided by development instructions that include end-to-end testing. Users can ask why it failed and request an instruction update. One participant reports repairing a misconfiguration through a fix button; automatic self-healing remained a roadmap item.

Custom agents start with no permissions and require explicit grants. Reading email and sending email are separate capabilities. An agent cannot edit its own permissions during background work; repair instead enters an admin-like mode or synchronous chat where the user observes the work and confirms changes.

The redesign called flippy made chat the primary interface for both configuration and use. Settings became a side panel that previews changes and permits manual edits. The shift delayed launch by roughly a month and disrupted early adopters’ familiar setup flow. Three engineers from three teams built the launch-blocking redesign, demonstrating the flexible staffing model in practice.

Choose how much intelligence the work needs

Notion credits combine costs that a token count cannot capture: serving fine-tuned open-source models, web search, different models and serving tiers, and potentially sandbox hosting. Priority versus asynchronous processing and variable cache rates further complicate the mapping. Credit packs also support enterprise volume discounts.

Usage and value can diverge: updating a recipe document and answering an investor email might consume similar tokens while producing very different value. The response to that challenge rejects value-based pricing for the product at the time, citing market readiness and difficulty classifying task complexity. An initial per-run design accumulated distinctions that ultimately mapped back to token throughput.

Usage pricing lets customers buy expensive capabilities that existing margin constraints would otherwise prevent Notion from offering. Planned agentic database autofill illustrates the issue: running Opus for every cell could be prohibitively expensive. Notion provides expense indicators and nudges when users choose powerful models for routine work. For asynchronous agents, the team finds speed a weak incentive to choose a smaller model such as Haiku; price makes the tradeoff tangible.

Auto model selection aims to match tasks to models and reduce the burden of following releases. A guest says it was not being used to increase margins, while acknowledging users’ suspicion that Auto simply chooses the cheapest option. Notion also wants recurring work to become code when appropriate—even if an agent thereby automates itself out of the workflow.

The guests see a missing middle among intelligence, price, and latency: frontier models add capability users may not need, while smaller models have not caught up sufficiently. Notion is preserving model-switching options and investing in open-source alternatives, including MiniMax and collaboration with open-source labs on Notion’s Last Exam tasks. The aim is a broader range of useful tradeoffs.

Training must improve the whole loop

Asked about building a foundation model, a guest questions both the budget and whether it should be Notion’s core competency. Company-specific fine-tuning holds more interest for another participant: understanding an organization’s people and current context might improve quality. Notion already uses selected fine-tuning and reinforcement fine-tuning when a data scientist and model behavior engineer identify a specific capability gap, without necessarily using user data.

Long-running coding agents have revived a habit from model training: starting enough work before bed to keep computation busy overnight. One participant describes that as a personal goal. Another anecdote recounts a single coding-agent thread running almost continuously for roughly 17 days, with perhaps 100 context compactions before encountering a harness bug. That describes thread longevity, not unattended completion of a 17-day task.

Fine-tuning tool use has a practical drawback: Notion changes tools daily. Retraining and releasing a model with each change would slow iteration, and frontier models could acquire the targeted capability before a specialized model paid off. Enterprise-specific training would also require rethinking opt-in around zero-data-retention offerings.

The recommendation is to optimize the model–harness–tool loop, treating training as one implementation choice. A failure may come from a tool bug rather than inadequate model capability. Reliable tools, a robust harness, fast iteration, and verification all belong in the same improvement process.

Agentic find changes the search objective

Retrieval is an exception to the team’s caution about training. On Notion’s AI-enabled Business and Enterprise plans, agents reportedly generate the majority of search traffic. Their queries differ from human queries, and they need different results from Elasticsearch and vector indices.

For a person deciding what to click, ordering among the first few results matters greatly. An agent consuming a larger set may benefit more from having relevant material anywhere in that set. The discussion contrasts positions one through six with inclusion in the top 100. Top-K retrieval—returning the K highest-ranked results—still involves ranking; the change is in which ranking improvements matter most.

Snippet selection, query formulation, and indexing therefore need to work together. Meeting Notes and the agent setup generator also challenge assumptions built around nested blocks. Better ranking and parallel searches can serve the same outcome. As a separate example, a participant describes their own eight-query fan-out, using a model to diversify queries and cover more of the search space.

Notion calls the combined effort agentic find. The same people work across query generation, ranking, and retrieval; choosing a vector embedding model no longer receives much optimization effort. The unit of improvement is the complete search journey: what the agent asks, what the system finds, and what information the agent receives.

Meeting Notes turns conversation into operational context

Meeting Notes initially raised concerns about workflow changes and user friction. A guest now describes it as a major growth lever with strong adoption and retention, without numerical results. One personal use is drafting a self-review primarily from conversations with a manager: those conversations help identify which work mattered.

Meeting transcripts substantially expand the material search and agents must handle. Passing them into context prompted work on compaction and longer content. Notion treats Meeting Notes as an agent capability layered on transcription, with data capture as its central purpose.

An internal standup workflow connects that context to action: 1. A custom agent summarizes Slack and GitHub activity and prepares a meeting-note pre-read. 2. The team discusses the pre-read and decides what should happen next. 3. Another custom agent, connected to calendar events and triggers, files tasks and sends the agreed Slack follow-ups. The reported benefit is a meeting spent discussing problems with hands off the keyboard, while agents handle the surrounding bookkeeping.

Summarization has become agentic too. Meeting Notes can @mention people referenced in a discussion, prompting follow-up conversations. To resolve which person a name means, the team describes using generated profiles and a people-to-people similarity cache. Mistaken identity remains possible, and the speaker was unsure whether the interface showing that resolution process had shipped. Updating the relevant task database during a meeting was a future direction.

Asked about meeting-capture hardware, one guest says Notion probably would not build it; another welcomes partnerships. Early wearable discussions include using Notion Agent to assemble context from customer records when someone enters a conference. Simple interfaces suit capture but provide fewer controls for sophisticated agent workflows. Notion’s closing priority is to make that captured context useful to a company: a place where people collaborate and meeting notes live.