Modal’s Agent Infrastructure: From Bursty Jobs to Programmable Compute

The first problem was running the workflow

A workflow needs a custom container image, sometimes an accelerator, and enough compute to handle a sudden burst of jobs. Managing that environment can become a substantial part of the work. Akshat traces [Modal](https://modal.com)’s origins to this problem: workflow orchestration depended on Kubernetes, which he regarded as cumbersome to manage and poorly matched to rapidly changing resource requirements.

The initial plan was a serverless runtime that would provision execution resources for jobs, with specialized products built on top. Job queues and bursty processing supplied immediate uses; data engineering was a possible first vertical. Inference was already in view through computer vision and XGBoost. Akshat says Modal added GPUs a year before ChatGPT launched, without anticipating how consequential they would become.

That origin explains the platform’s scope in the Latent Space conversation: inference, training, batch processing and sandbox workloads share a need for specialized compute and substantial changes in capacity. Modal leaves always-on web applications to other platforms and concentrates on workloads with more variable execution requirements.

Infrastructure beside the code

Decorators alongside application code make infrastructure configuration programmable. Hardware and scaling requirements live near the functions they affect, reducing YAML and allowing developers to manipulate configuration in the application’s language. This was central to Modal’s original design.

The portability tradeoff remains: Modal supplies a provider-specific configuration language. Akshat’s answer is that customers retain their application code while Modal’s interface specifies hardware and scaling. That preserves control over application logic, without making infrastructure migration cost-free.

The same design underpins agent experience—how effectively an agent can configure, run and investigate an application. Akshat says Modal shifted its software development kit team toward this goal: an agent can change a few decorators instead of navigating hundreds of infrastructure files, then observe those changes in the self-provisioning runtime. Customers report faster agent work, though he gives no measured comparison.

As people inspect less generated code, observability becomes more important: they need to understand what the running system actually does. Modal exposes operational information through its command-line interface, or CLI, so agents can investigate. Humans still need dashboards to interpret outcomes and make judgment calls.

From an idle demo to regional GPU bursts

Early agent experiments exposed both the runtime’s usefulness and the models’ limits. A developer-agent project used Modal functions for bursty code execution, producing a traffic spike after gaining attention on Hacker News. Modal dates its sandbox implementation to May 2023, following customer requests; an early example put a developer agent into a loop. A participant recalls models diverging after roughly ten iterations, attributing this to weak tool calling and insufficient post-training for looping and self-correction.

A hackathon anecdote captures the smaller-scale appeal: deploy a model from a Hugging Face link on a GPU, let it scale down while idle, and bring it back for a demo days later. Production changes the scale of the same problem. Modal’s example is expanding from 1,000 to 1,500 GPUs within a particular region, quickly enough to meet demand.

Elastic inference, which adjusts serving capacity as demand changes, is described as Modal’s largest use case. Its early traction came from custom models: Suno for audio, Runway for video, and robotics and computational biology companies that train elsewhere. Daily traffic cycles and launches affect multiple models across regions, whose peaks arrive at different times. Capacity must grow in the region where it is needed.

GPU snapshotting captures GPU state for reuse on subsequent cold starts, which Modal says substantially reduces startup time. Other bursts come from on-demand training, reinforcement learning and preprocessing, including thousands of GPUs performing encoding before a training run. Continual learning is more exploratory: Engram uses existing Modal primitives for experiments, while a dedicated offering would depend on broader demand.

Making inference faster inside the runtime

Modal’s expansion into large language model inference adds model-performance engineering to deployment and autoscaling. One focus is D-Flash, a speculator that predicts blocks of tokens. Modal says it has open-sourced its work and claims performance comparable to proprietary providers, without specifying a comparison workload or hardware configuration.

Speculative decoding separates proposing tokens from verifying them: 1. A smaller draft model predicts tokens ahead of the larger model. 2. The larger model verifies those proposals together. 3. Accepted predictions advance generation by multiple tokens. Single-token generation can be limited by memory bandwidth. Batched verification uses compute more efficiently, making accepted draft length—the number of proposed tokens that survive verification—a major determinant of speedup.

The participants suggest 2–4× generation speedups, as conversational estimates rather than benchmark results. They distinguish generation quality from compute cost: the second model adds work, while they argue that the larger model’s verification prevents a quality drop. The exchange includes neither a quantified cost comparison nor a quality evaluation.

Auto Endpoints packages optimizations such as D-Flash into endpoints created through a UI or CLI. Customers receive the code and can customize or fine-tune within the broader Modal environment. A forthcoming workflow would shadow traffic and adapt draft models as customer data changes, reducing reliance on customer-specific engineering support; its implementation remains only sketched out.

Modal says it contributes inference improvements upstream and works closely with SGLang, so those improvements remain usable outside its platform. Its service proposition combines performance expertise with elasticity and scaling to zero. Production also requires controlling tail latency—the slow end of request response times—and ensuring requests are delivered at least once.

Keeping compute and agent state close

Modal organizes its scope around two lifecycles: - data preparation, training and inference for models; - sandboxes and persistent storage for background agents. Ramp Inspect illustrates the latter. Modal credits snapshotting and fast scaling with making the agent feel responsive, without supplying latency measurements.

An interviewer argues that agents make placement more consequential by repeatedly alternating between GPU inference and CPU-heavy operations. Modal reports customers requesting specific locations for CPUs and GPUs to satisfy latency or data-locality requirements, including placing sandboxes close to the models they call.

Modal says its capacity pool spans 17 cloud providers, without owning data centers. Buying capacity lets it concentrate investment on software, but requires operating across providers with different reliability characteristics. Modal claims its reliability layer shields workloads from failures such as a GPU disconnecting from its bus; it does not describe the recovery mechanism or provide a failure benchmark.

Sandbox networking and multi-node training

A Modal sandbox can run multiple containers through sidecar support, enabling Docker Compose–style workloads. Its networking capabilities serve several purposes: - A proxy can log activity for reinforcement learning, control outbound access to particular domains or inject credentials. - A tunnel can expose a sandbox to the public internet with an HTTP authentication layer. - A private IPv6 overlay can let enabled containers in the same workspace address one another across nodes.

The private overlay originated in serverless multi-node training. A decorator provisions a GPU cluster with RDMA networking, which bypasses the TCP networking stack to accelerate transfers between nodes. The overlay itself uses TCP and initially handled the key exchange required before establishing RDMA. Users discovered and repurposed that TCP layer, reportedly before it was documented.

The overlay is private but unencrypted. Programs using eBPF allow or reject TCP connections according to authorization. Its resemblance to a private network does not imply encrypted transport.

Moving model weights from training GPUs to inference GPUs during reinforcement learning makes memory movement and scheduling central systems concerns. Modal targets smaller post-training runs on medium-sized models, rather than large-scale pretraining. Researchers’ small experiments can benefit from this elasticity even when their organization already owns a large cluster.

Agents that tune infrastructure and investigate failures

An internal repository called auto inference makes automated experimentation concrete. Modal reports using an agent to try configurations, run NVIDIA Nsight Profiler and change GPU choices from H200 to B200. Both training and inference teams use the approach to automate parts of customer-facing performance engineering. That team is described as applied researchers; solution architects handle more of the presales work.

The automated research workflows described are primarily parameter sweeps guided by model intuition: trying configurations with the model helping choose what to test. The respondent has seen parameter tuning rather than architecture discovery, and claims greater efficiency than conventional sweepers without presenting a benchmark.

Failure recovery remains difficult. An agent must 1. find the relevant logs, 2. reason about the cause 3. and change the appropriate part of the system. Modal says this can require explicit guidance or an agent skill. Its benchmark effort identifies tasks agents cannot reliably complete and uses those failures to improve that guidance; public release is under consideration.

Trading turnaround time for cheaper compute

Elasticity depends on capacity planning underneath. Modal’s compute strategy team forecasts demand, chooses between one-year and three-year reservations, and makes supply-chain bets. Its planning accounts for capacity that can serve different GPU types and regions, connecting procurement decisions to scheduling and customer pricing.

A proposed cheaper batch tier would trade delayed completion for lower prices. Modal suggests results in roughly the next 24 hours, without committing to a delivery window or discount. Service-level agreements could bound turnaround. Reported demand comes largely from non-language-model workloads, including computational biology; evaluations and synthetic-data preparation are also candidates.

At the other end of the latency spectrum, Modal is working on regional routing with fallbacks to place GPUs near users of real-time audio and video applications. It also expects more companies to post-train and deploy their own models, shaping plans for training tools and optimized endpoints. Additional production-agent infrastructure is being developed with customers as requirements emerge.

Hard boundaries, flexible agent placement

Permissions introduce a different constraint. An interviewer questions whether a language model should mediate permissions at the sandbox boundary, where hard limits protect against exfiltration. The response favors hard guardrails paired with softer, potentially model-mediated guardrails. The interviewer’s hypothetical model acting as an operating-system kernel is a thought experiment, not an endorsed design.

Modal treats managed agents as a useful starting point. Production applications may need finer control over file persistence, snapshot and restore, networking and GPUs—the role it wants Modal Sandbox to fill. It cites Ramp’s external-facing accounting agent as an example and reports partnering with Anthropic and working with other unnamed foundation-model labs.

The agent harness can connect to a sandbox from a managed cloud service or run inside the sandbox itself. Modal leaves that choice open and concentrates on the infrastructure layer. The platform is described as cloud-only, with no on-premises or air-gapped version.

Editable code for differentiated products

Modal’s ambitions extend beyond language models to drug discovery, computational biology and robotics. Its model examples are editable starter code that companies can incorporate into larger applications. Modal views parts of the hosted-model API market as less sticky and targets products that need deeper flexibility.

Customization does not guarantee differentiation; the respondent acknowledges a selection effect. Ramp reportedly trained a tokenizer and experimented with swapping it into Llama, but the speaker declines to call that experiment successful. Suno supplies a clearer reason for code-level control: Modal says it handles all Suno inference, but not training, because Suno’s custom architecture requires its own code and tuning.

Video suggests another direction. An interviewer relays a prediction from an earlier guest, Ethan: an agent orchestrating video models could bridge the gap between six- or ten-second clips and a six-minute production. Modal reports companies using GPU sandboxes for video-manipulation agents that can run FFmpeg. The participants distinguish editing from fuller production workflows, suggesting Adobe tools and citing Luma Agent. The six-minute production remains a proposed use, not a demonstrated result.

Agent experience includes the operational loop

Modal expects coding agents to increase continuous integration (CI) runs, amplifying time spent packaging artifacts and preparing dependencies. Memory snapshot and restore could reduce that repeated setup alongside build systems, though no speedup is measured here. Runtime sandboxes also need different configuration controls from build-time environments, particularly for images and attached storage.

The interface has expanded beyond Python to Go and TypeScript, while Modal’s runtime is written in Rust and remains independent of the client language. Modal reports Python dominating inference and training; agent workloads use TypeScript more heavily because many do not perform machine learning themselves.

The closing definition of agent experience reaches beyond CLI ergonomics to how quickly a user can start a service, iterate and reach production. Modal Bench identifies where agents struggle and informs product changes. When an agent reaches for a nonexistent capability that makes sense, the team treats that attempt as product feedback.

The final example is practical: make UI logs and metrics available through the CLI too. An agent that provisions compute also needs to inspect its execution, diagnose failures and choose an informed next action. That closes the loop begun by putting infrastructure configuration beside the code.