Serverless AI infrastructure starts below the function decorator

From a foosball-room cluster to a compute platform

Early Spotify data work ran on rudimentary infrastructure, including a Hadoop cluster on the foosball-room floor. Recommendations, reporting, A/B testing and business analytics all needed tools that barely existed. The guest in this February 2024 Latent Space conversation recalls joining the roughly 30-person company for a six-month music-recommendation thesis project, then returning full-time after graduation.

Over seven years, those infrastructure gaps became projects. [Luigi](https://github.com/spotify/luigi) was a workflow engine that preceded Airflow. [Annoy](https://github.com/spotify/annoy) supported vector lookups through a C++ library with Python bindings, memory-mapping large files and searching through recursive splits of vector space. Its creator considers the search strategy adequate for its time; a consequential limitation was that it did not support inserts or updates.

The guest expected vectors to become important around Annoy’s early adoption in 2012, but remembers waiting roughly nine years for the vector-database startup wave. He is glad he did not start a company in that market then. Better provided a different education: he joined as CTO before a technology team existed, eventually managed roughly 300–400 people, and spent much of his time recruiting. That experience made management one of the less daunting parts of founding Modal in 2021, after leaving Better and taking a short pandemic break.

Make the write–run loop fast

[Modal](https://modal.com/) grew from a recurring problem: data teams stitched together poorly integrated tools and built their own Kubernetes wrappers to run code. Workflow scheduling initially looked attractive because it connected many data products. But building and monetizing a scheduler in isolation proved difficult, shifting attention to the execution layer beneath it.

The design priority became feedback-loop speed: how quickly can a developer write code, run it and learn from the result? Building a Docker image, pushing it to the cloud and executing it could turn that loop into minutes of waiting. Modal aimed to preserve the responsiveness of local development while supplying cloud compute.

Data teams brought three requirements that infrastructure designed for backend services often handled poorly: - Bursty fan-out: applying a function across a large dataset with many workers at once. - Specialized hardware: getting GPU access without a separate platform-team project. - Conflicting environments: running different package stacks without forcing everyone into one shared container. The guest recalls platform teams declining GPU requests because of the complexity, while shared containers broke under incompatible dependencies. These workloads needed a toolchain designed around their execution patterns.

Cold starts become a filesystem problem

A container cold start is the work needed to bring a new container into execution. Modal’s target went beyond starting one fixed environment: developers should be able to move laptop code into custom cloud containers and launch hundreds or thousands of them quickly.

The early implementation used a low-level container runner directly, pointed at a custom root filesystem whose files were exposed over the network. This avoided requiring the complete image to pass through a build-upload-download sequence before execution could begin.

The opportunity was sparse access and reuse. A several-gigabyte image might contain many files a workload never reads, while the libraries it does read overlap with those used by other workloads and can be cached. The guest estimated that 99% of an image might go unused—an informal figure, not a measured proportion across workloads.

The guest reports cloud launches in roughly one or two seconds from this early implementation, without specifying image size, cache state or concurrency. The resulting architecture had two connected layers: a Rust container runtime and filesystem, and a Python software development kit, or SDK, for defining applications.

Let application code describe its execution

The Python layer removes another kind of friction: plumbing between containers. Infrastructure as code makes infrastructure definitions programmable; Modal extends that idea by putting application code and its execution requirements together. Functions can run in different container environments, call one another and start more containers, with orchestration expressed in Python.

A concrete example is [attaching a GPU to a function](https://modal.com/docs/guide/gpu): choose an A100, A10 or T4 through a decorator, then run the code without separately launching a cloud instance. At the time of the conversation, the SDK’s syntax was still changing frequently.

This is the idea behind a self-provisioning runtime: application definitions also specify the infrastructure needed to execute them. A host describes the coordination problem in an earlier AWS workflow, where infrastructure code compiled to CloudFormation but configuration still had to be carried into application code and kept aligned. The participants criticize that developer experience while acknowledging AWS’s obligations to preserve compatibility and support a broad range of uses.

Python was a focused entry point because of its prevalence among data teams; the underlying container scheduler and filesystem were not tied to it. JavaScript was an obvious candidate for future support, without a delivery commitment. A host compares the abstraction to automatic memory management: accepting some loss of control can buy useful defaults and greater productivity.

Inference turns fast startup into better utilization

Modal initially pitched a general-purpose compute platform—a better Kubernetes for data teams. After roughly a year and a half with few users or revenue, the team began looking for concrete applications around the arrival of Stable Diffusion. Model inference became its strongest source of usage and growth, despite not being the original focus.

The fit was a particular execution pattern: small input, substantial GPU computation, small output. A prompt could trigger much more computation than data transfer. Serverless execution—letting the platform start and stop capacity as work arrives—suited this pattern because running GPU capacity could follow traffic closely.

Fast starts and stops improve the latency–utilization tradeoff: capacity can arrive quickly when needed and stop consuming billable time when demand subsides. Modal reported running on AWS, GCP and Oracle, charging a slightly higher GPU-hour price than those clouds while potentially lowering the total bill by reducing billed GPU-hours. Its custom scheduler and filesystem supported that rapid adjustment.

Modal also reported implementing CPU and memory checkpointing: snapshotting a running container, including CPU registers, so it could be restored in an initialized state. GPU checkpointing was still exploratory. The guest claimed tenfold better inference economics, but supplied no workload, comparison baseline or pricing assumptions; the claim does not establish a general savings rate.

Training exposes the boundary. A long-running job that keeps provisioned machines busy gains less from rapid scale-down. Bursty fine-tuning can still fit: Ramp is cited as fine-tuning 100 models in parallel, with short jobs followed by selecting the best model through hyperparameter tuning. The suggested 10–20-minute duration is approximate. Very large foundation-model training was a weaker fit because of its intensive data-transfer requirements; Modal did not yet support the InfiniBand connectivity discussed for that workload.

Custom workloads and the graduation problem

Modal’s contract was to execute arbitrary user code or containers. That limited how aggressively it could optimize application internals: recommending model-specific optimizations was compatible with the contract, while automatically rewriting users’ containers was harder to justify.

The workloads extended beyond model serving: - Hundreds or thousands of Chromium containers rendering web pages. - Protein folding and sequence alignment. - Large mixed-integer optimization problems using Gurobi. Parallel video transcoding with FFmpeg was another suggested application. AI remained Modal’s largest use case by far, but the execution layer served a broader range of custom work.

Asked about Replicate, the guest distinguishes an off-the-shelf model for a frontend engineer adding an AI feature from Modal’s focus on custom models, workflows and large workloads. Suno illustrates the latter: it is cited as running production AI music infrastructure with its own code, model and weights. The guest also claims lower prices, without a matched-workload comparison.

The ambition is a second layer of cloud infrastructure: a shared platform running customers’ code above public clouds, competing through developer experience. Snowflake supplies the analogy. The unresolved business risk is the graduation problem—successful customers growing large enough to move directly to a cloud provider. The guest argues that an enduring infrastructure business must keep serving customers as they become enterprises. Better utilization is his proposed reason to stay, though he sees stronger ownership economics for large training clusters.

ErikBot and controlled fan-out

[ErikBot](https://modal.com/docs/examples/slack-finetune) demonstrates how an application can connect several execution stages. The prototype downloads Slack history, fine-tunes a model to imitate a person, then dynamically creates an inference endpoint for chatting with the imitation. The guest describes a self-contained Modal application handling the workflow in a few hundred lines.

The imitation’s quality was less convincing. Its subject felt it captured their language but often produced plausible sentences without a meaningful point. A host reports a similar experience with podcast automation: useful transcripts and chapters, but summaries far below the team’s own writing. The participants agree on the shortcomings while differing in their confidence about future improvement. Fluency and substance are separate judgments.

Modal’s language-model strategy emphasized surrounding work—fine-tuning, custom inference and batch embeddings, which produce vectors for collections of content. Ramp and other customers reportedly used it for embeddings. The guest described an example being prepared for publication that processed Wikipedia in 15 minutes, without specifying the model, corpus snapshot, hardware allocation or cost.

The reported fan-out capacity was at least 100 GPUs in a few seconds and thousands in a couple of minutes, without benchmark conditions. A host’s coding-agent example shows why limiting parallelism matters too: the application moved from laptop development to cloud execution while declaring a concurrency limit for calls to the rate-limited GPT-3 API. Modal handled queueing underneath, without requiring the developer to manage an exposed queue system.

Sandboxes expose the runtime to other platforms

Modal Sandbox began with code-generation customers asking how to execute generated programs safely. It used gVisor for container isolation and allowed callers to run arbitrary code, monitor output and retrieve results. Its proposed direction expanded toward exposing the container infrastructure separately from the Python SDK.

The emerging customer was itself a platform: accept a user’s code, define a job programmatically and launch a container on that user’s behalf. Some companies already offered Kubernetes or Lambda as execution backends and wanted to add Modal. The requirement extended beyond isolation to programmable execution for other platforms’ users.

A host asks what changes when the developer operating the sandbox is a large language model, or LLM. Error streams, logs and feedback might need different interfaces; these remain open design questions. The response endorses network restrictions when a model chooses packages and programs to run, particularly to limit malicious installations and secret theft. Sandbox is described as relatively safe, without an absolute guarantee.

Price wars and work beyond inference

The guest sees more room for a premium in custom workflows than in interchangeable endpoints serving an unchanged open-source model. He considers switching costs for those endpoints effectively zero: a cheaper provider can offer the same model. Modal instead seeks differentiation in fine-tuning, guided outputs, and custom image, audio and video workflows. Alternative GPU prices still constrain its premium.

The price-war discussion separates customer benefits from investor returns. The guest compares GPU investment with the late-1990s fiber buildout: excess investment could leave useful infrastructure even if returns disappoint. Another participant raises the risk of building product features around subsidized inference prices, then facing either higher costs or unhappy customers when features disappear. The response distinguishes provider margins from customer prices: falling GPU costs could allow margins to recover without comparable price increases. These are possible outcomes, not settled forecasts.

The technical roadmap moved toward workloads with more input/output, or I/O. Early direct TCP tunnels let callers feed video through a raw socket into a container and receive video back. The guest sees potential for real-time applications, but describes an unfinished interface without latency or throughput measurements. Higher-level dataset operations—filters, group-bys and joins above map and fan-out—were more distant ideas, potentially years away.

Elastic execution still requires capacity commitments

Beneath the application abstraction, Modal still had to obtain machines. The guest praises Oracle’s GPU offering for value, citing bare-metal machines with two terabytes of RAM and fast storage. AWS and GCP were also suppliers; Azure and smaller providers were possible additions. Integration depended on programmatically launching and terminating machines, with cloud APIs and Terraform networking support easing the work.

The preferred financial model made compute costs track revenue closely: a customer’s dollar might immediately incur 70–80 cents of compute cost. That was an illustrative target, not a disclosure of actual margins. In practice, spot and on-demand instances were insufficient, and Modal added significant reserved capacity.

Reservations were a step toward hardware ownership, but full ownership would bring switches, cooling and power supplies into the company’s responsibilities. A host argues that extending hardware life from an assumed two or three years to seven changes the economics; the guest questions whether that confidence is warranted. He invokes the Waste Management accounting scandal as a cautionary anecdote about optimistic depreciation assumptions. Hardware ownership remained possible, but was not a near-term plan.

Choose problems worth years of attention

The closing discussion turns to programming as a craft. The guest argues that AI could increase demand for engineers by making more software economical to build, drawing on his belief in substantial unmet demand. Competitive programming was his own route into deep practice, puzzle-solving, competition and international travel as a teenager. He sees some competitors translating those skills into products, while others remain absorbed in puzzles.

At Modal, there was a concrete connection between algorithmic skill and production work. The company reported using mixed-integer programming to continually optimize cloud resource allocation, alongside difficult scheduling and container bin-packing problems. The guest presents competitive-programming experience as useful for these particular challenges, not a universal hiring rule or a substitute for knowing the rest of the stack.

When a host questions how well specialized contest skills transfer, the response points to sustained attention: competitive programming may select for people willing to go deep and practice hard. The guest draws a parallel with musicians, whose habits of focus can matter even when the immediate tasks differ.

A brief personal detour returns to infrastructure in everyday life. The guest tentatively connects Sweden’s early connectivity with computer-purchase tax rebates and infrastructure investment, recalling subway phone use there in 1995. He also says he has never owned a television or held a driver’s license.

The final advice returns to the engineering beneath the function decorator: better developer tools can require years of deep infrastructure work. The guest argues that thin Kubernetes wrappers offer too little differentiation; founders must be willing to spend five, ten or twenty years building the technology their product needs. Choose a subject that can sustain that attention. The conversation closes with an invitation to try Modal, which had reached general availability after its earlier waitlist.