From Easy GPU Access to Disaggregated Inference — Part 1 of 3

Agent capabilities need enforcement boundaries

An agent that can read files, access the internet, and write and execute arbitrary code combines three powerful capabilities. The opening of this Latent Space conversation proposes a security heuristic: allow only two of those three capabilities together. Combining all three creates opportunities for malware injection or other harmful behavior. The engineering question is where to enforce limits on the agent’s actions; the heuristic itself is no guarantee of safety.

Brev puts the requested GPU at the center

Brev faced a more physical visibility problem at GTC: late registration left its booth in a distant corner. Nader recalls bringing palm trees, surfboards, and an investor’s dog to attract visitors. The decorations rose above the booth walls, making it visible from far away. He credits Evan Conrad of SF Compute with encouraging the team to embrace being a tiny startup; Brev’s shaka logo and booth followed from that advice.

Brev simplifies access to GPU machines by connecting multiple GPU sources and minimizing the steps needed to connect over SSH. Its interface foregrounds the hardware users request. Someone asking for an A100 sees a prominent A100 label and GPU graphic, rather than a provider-specific instance code buried in a dropdown. The user’s language determines the interface’s hierarchy.

Nader built the animated GPU graphics from Figma designs exported as SVGs. JavaScript changed the underlying SVG when a chip’s selection state changed, and style transitions created the animation. The artwork also became physical GPU gift cards: a San Francisco printer made an inverse mold of the A100 design and pressed foil into paper. A host, speaking as a small Brev investor, remembers initially questioning the time spent on these marketing efforts.

Launchables package GPU software for one-click deployment. The team works with partners, customers, and independent software vendors to make applications quick to try. An acquisition participant describes joining NVIDIA as gaining more resources for Brev’s existing mission, and argues that alignment between the products’ underlying goals helped make the acquisition a good fit.

Easy experimentation also brings the opening security problem back into view. The reported internal guidance was to run experimental software in an isolated cloud virtual machine off the corporate network through Brev, while secure deployment practices were still being worked out. Isolation established a network boundary; it did not make the experimental software itself safe.

DGX Spark: make the first connection easy

AI’s expanding audience changes what developer tools must accomplish. Some users have no GPU programming experience and simply want an application to run. Designing for that audience means reconsidering the first steps a product asks people to take.

A participant involved in DGX Spark’s user experience recalls engineers proposing that a user’s first action would be connecting two Sparks into a Kubernetes cluster. Their counterproposal was easy SSH access: prioritize users with less tolerance for setup friction. Someone deliberately building a cluster can tolerate more configuration than someone trying to run a first application.

Two tools apply that priority to connecting from a laptop: - NVIDIA Sync simplifies SSH connections to Spark. - Brev’s command-line interface wraps SSH to simplify access. Its Spark integration lets users run a registration command on the machine so it appears in their Brev account.

The intended experience is to leave a Spark at home, take a laptop to a café, and continue using the machine like another cloud node. At the time of the conversation, the integration had launched at CES in early access, with issues still being resolved. Readiness by GTC was an expectation.

“Speed of light” makes delivery constraints explicit

SOL, or speed of light, is NVIDIA’s first-principles approach to a delivery timeline: establish the theoretical minimum, then explain the constraints separating it from reality. Participants describe it as a way to preserve scrutiny of root causes as an organization adds layers, and say frontline engineers can invoke it too.

A laptop purchase makes the reasoning concrete: 1. Establish the fastest baseline. Against a quoted five-day delivery, someone could walk into Best Buy and buy a laptop immediately. 2. Add the actual constraints. Supplying many employees introduces purchasing and scale problems that a single retail purchase avoids. 3. Explain the gap. Build the delivery estimate from those constraints so the reasons for elapsed time are inspectable.

Repeated demands for SOL can collide with planning and stability. The response is that stability belongs among the constraints: if a faster implementation would be unstable, explain why. Spark registration at CES illustrates staged delivery. The team reached an early-access experience that people could try, demonstrated it in a booth and Jensen’s keynote, and still had networking and security work to complete before broader delivery.

The same reasoning applies to each subsequent component, including operational work. Its hardware interpretation asks how fast a program could run if the accelerator operated at full speed without other constraints. That ideal supplies a reference point for practically achievable utilization, rather than a promise that a workload will reach it.

From recommenders to future workloads

Kyle joined NVIDIA directly after college, six years before the conversation—or seven counting his internship. His internship involved heavyweight object detection for autonomous vehicles; afterward, he moved into recommenders. He describes recommendation systems as matching audience members with items using tables of audience and item attributes. His team worked to move this historically CPU-oriented workflow onto GPUs.

The discussion continues into graph neural networks, which interested one participant because they could represent relationships among people, items, and concepts. That led to roughly two years of work. Participants describe an organization where engineers can propose projects to senior leaders outside their reporting chain and recruit collaborators much as they would assemble a pickup basketball team.

Exploring workloads early also serves hardware planning: a participant puts chip-design lead time at three to five years before market arrival. Understanding what people may need to run is therefore a problem that precedes the finished chip.

The email-versus-Slack disagreement exposes different preferences for organizing that collaboration. One participant favors email because active threads resurface, subjects organize ongoing work, and conversations can expand or fork; another prefers Slack. Neither establishes a company-size threshold for choosing between them. The practical advice is to give collaborators something they can try: initial progress helps an initiative advance.

A participant also describes NVIDIA’s willingness to explore markets without current revenue when learning about them may matter later. Autonomous driving is offered, then acknowledged as a poor example of a zero-revenue market. The narrower organizational point is that teams need not immediately generate revenue to justify useful technical exploration.

Recommendation work returns as the conversation approaches Dynamo. A participant recalls Amazon Ads presenting its use of Dynamo for generative recommendation, with large language models now applied to tasks they had worked on about five years earlier. The example connects an established workload to newer inference infrastructure, without a reported deployment scale or performance result.

Dynamo optimizes across model replicas

Test-time scaling spends more inference on a result through reasoning, repeated queries, or agent workflows that call tools and use skills. These workflows motivate Dynamo, a data-center-scale inference layer above vLLM, SGLang, and TensorRT-LLM. It addresses optimization across model replicas once adding resources to an individual replica encounters limits.

The two scaling choices have different consequences: - Scale up: assign more GPUs or CPUs to one model representation. Hardware and algorithmic limits constrain the benefit. - Scale out: duplicate that representation into multiple replicas to handle load, creating opportunities to optimize across the serving fleet.

Most DGX H100 systems are described as having an eight-GPU NVLink domain. Extending a model beyond that domain introduces communication over InfiniBand, which the speaker describes as slower than NVLink. Crossing a communication boundary changes the scaling tradeoff; the size of the bandwidth difference depends on the hardware generation.

Dynamo aims to maximize reuse of caches distributed across machines and support separate inference stages within a modular framework. These are KV caches, described here as vectors representing an input sequence that the model uses when generating new tokens. The stage separation, called disaggregation, preceded Dynamo in academic work; the speaker’s narrower claim is that Dynamo was among the early frameworks to support it.

Choosing how to distribute and replicate a model still requires workload-specific experimentation. The right arrangement depends on the model, the application, and the team’s willingness to keep tuning; there is no universal serving recipe.

Optimize the whole agent workflow

Inference design has three requirements: - Quality: can the application complete its task accurately enough? - Cost: can it afford the complete workflow, including an agent’s multiple turns and calls? - Latency: can that workflow finish fast enough for the application?

Increasing a thinking budget can improve quality while producing more tokens and increasing cost and latency. Model selection and serving configuration are therefore coupled. A participant suggests testing whether a smaller model with more test-time scaling or supporting workflow logic can match a larger model’s quality; no benchmark establishes when that substitution works.

The decision rule is to find the lowest-cost configuration that meets the application’s constraints. Establish the model, call count, input lengths, workflow, and latency requirement, then experiment across common serving configurations. The application’s required behavior becomes the basis for choosing infrastructure.

Retries add another choice: carry failure context into the next attempt so the model has more information to work with. Participants connect that idea to using past failures as a learning signal in self-distillation and reinforcement learning. Their recollection of a paper about repeating prompts supplies neither a precise procedure nor a usable quantitative result.

Reading habits and reusable model releases

Paper clubs offer social pressure to sustain a reading habit, and a participant reports a large inference reading group at NVIDIA. There is disagreement about whether inference advances arrive frequently enough to sustain a dedicated podcast: one participant doubts it, while another points to the volume of technical posts and new speculative-decoding approaches.

Nemotron combines a model release with reusable training materials. It is described as co-designed for its hardware, with pretraining and post-training datasets, training recipes, and the model itself released together. Users can: - Adopt the trained model to benefit directly from the completed training work. - Reuse datasets or recipes in an independent model effort. ServiceNow is cited as having trained its own model using the released dataset. The participants remain uncertain about whether a base checkpoint was included.

Separate prefill and decode, then scale each pool

Dynamo provides runtime controls for navigating cost, quality, and latency. One is disaggregation, which separates two phases that a conventional inference engine alternates between: - Prefill: read the input sequence and generate its KV cache. - Decode: use that cache to generate new tokens.

Separating the phases reduces scheduling interference. In step-synchronous scheduling, the current step must finish before the next is scheduled. A long prefill can therefore block decode steps. Running the phases in separate pools removes that particular source of blocking and lets each pool receive resources suited to its work.

The resource requirements differ: prefill is typically compute-bound, with computation limiting its speed, especially for sufficiently long inputs. Decode is usually memory-bound, with memory access the limiting factor. The balance can change with model architecture. Those different bottlenecks motivate assigning the phases to different hardware.

An Exo Labs demonstration makes the split concrete: prefill runs on a DGX Spark and decode on a Mac. A participant calls it faster, but supplies no configuration or comparison baseline. The example illustrates how separate stages can use different kinds of machines.

Grove, Dynamo’s Kubernetes component for specialized scaling, manages stages whose capacity requirements change relative to one another. The speaker contrasts it with LeaderWorkerSet, which they say struggles to represent multi-node replicas with paired stages and a changing capacity ratio. An influx of long queries can increase prefill demand disproportionately, making independent scaling of prefill and decode useful.

Part 1 closes with a refinement: decode demand depends on both input and output length. Output length determines how many generation steps are needed, while the tokens already in context affect the work per step. Long output alone therefore does not describe the decode capacity a workload needs.