Mission-Critical Inference: From DeepSeek V3 to Production

First, make the model fit

An eight-H100 configuration with 640 GB of aggregate memory could not hold Baseten’s described DeepSeek V3 deployment. Even with lower-precision FP8 weights, the model needed additional room for its KV cache—the inference state retained during serving. The alternatives were H200s or multiple H100 nodes. Before optimizing latency, the team had to find hardware that could accommodate the deployment.

V3’s scale explains the challenge. The January 2025 Latent Space conversation describes a 671-billion-parameter mixture-of-experts model with 256 experts. Its training combined native FP8 mixed precision, multi-head latent attention inherited from V2, a multi-token prediction objective and 15 trillion tokens, including synthetic reasoning data distilled from DeepSeek-R1. In the January LM Arena snapshot, it ranked seventh overall and highest among open-weights models.

Baseten’s early deployment combined H200 capacity, collaboration with DeepSeek and support for [SGLang](https://github.com/sgl-project/sglang), an inference framework. Each H200 provided 141 GB of memory and 4.8 TB/s of memory bandwidth; eight in one node accommodated the FP8 deployment and KV cache. This was an unusual requirement in Baseten’s customer base, where Llama 70B was common and Llama 405B had relatively few users.

Memory was only the first obstacle. The released FP8 weights required compatible kernels and quantization support, and TensorRT-LLM did not then support the particular FP8 path V3 needed. Additional implementation work followed. Testing was slow too: long model-loading times stretched debugging and benchmarking cycles. The immediate difficulty was the wait between iterations.

Customers buy latency, capacity and control

Amir Haghighat describes interest in V3 coming largely from Claude users, rather than people upgrading from another open model. His account includes both production deployments and prospective customers. Their reasons overlapped: rate limits, price, latency, time to first token and control over a model that an API provider might otherwise change underneath them.

Larger customers generally specified latency, throughput and cost requirements, Haghighat says. Hardware selection followed those requirements. A GPU mattered because it could deliver the required delay before the first output token, sustained throughput or cost per million tokens or images.

Baseten consequently chose not to offer shared endpoints for popular open models. Its target customers brought custom models or workflows and needed dedicated inference resources. A noisy neighbor—another customer consuming shared capacity—could not be allowed to compromise their latency. Regional placement and compliance requirements also shaped deployments.

Quality was another constraint. Baseten says it did not silently quantize customer models, changing their numerical precision to improve serving efficiency. It supplied tooling and worked with customer engineers on evaluations to check the resulting quality. It contrasted this with the speculative-decoding methods it used, which it described as preserving output, although the particular methods and guarantee conditions were not specified.

Lower precision changes the implementation work

V3 required block-wise FP8 kernels. The CUDA and cuBLAS facilities discussed did not provide the needed support, so the serving team needed custom kernels, with Triton named as one implementation option. Lower-precision weights reduced neither the need for compatible execution code nor the work of integrating it.

The outlook for mixture-of-experts, or MoE, models was less settled. One participant expected MoE inference optimization to become essential, based on their understanding of internal models at other companies. A host recalled expecting wider adoption after Mixtral, only to see dense models such as Llama 405B remain prominent. V3 renewed enthusiasm for fine-grained experts without resolving how quickly other model developers would follow.

Baseten’s commercial model reflected its dedicated deployments: resource-consumption pricing, rather than per-token pricing. - In the hosted offering, customers paid for hardware resources across public clouds and regions. - In the bring-your-own-cloud offering, Baseten managed customer resources under a differently priced consumption model. Customers could combine existing cloud commitments and optionally overflow into Baseten’s cloud when those resources ran out.

From two serving hooks to an AI phone call

Truss, Baseten’s open-source model packaging and deployment library, sat above the inference runtime. Its deep TensorRT-LLM integration grew from early access, ongoing contributions and customer latency and throughput requirements. Baseten also built its own version of the Triton inference server for performance and reliability.

Truss supported multiple runtimes: the DeepSeek deployment used SGLang, while other customers used vLLM or TensorRT-LLM. Baseten found SGLang substantially easier to develop against, even after investing in tooling around TensorRT-LLM. A reported plan to make TensorRT-LLM more modular offered the prospect of other frameworks reusing selected components; that integration was still a future direction.

The original Truss interface needed two functions: one to load a model and one to run inference. That made a single model straightforward to serve, with horizontal scaling handled separately. Custom architectures required more. Baseten reported serving six or seven foundation-model companies that pretrained their own models, prompting deeper integrations and changes to the original abstractions.

AI phone calls exposed a different limit. Their inference sequence was: 1. Transcribe the human’s speech. 2. Make several language-model calls to decide what to say. 3. Synthesize the response as speech. When the client orchestrated separate deployments, each stage added a network round trip and a wait for the preceding response. Faster individual models could not eliminate those accumulated delays.

Baseten’s chains feature replaced that orchestration with one external call and internal streaming between models. Each model retained its own hardware and autoscaling behavior. Baseten credited this architecture with phone-call latency below 400 milliseconds, without specifying the measurement boundary, workload, hardware or percentile. The described low-latency setup depended on hosting every model on Baseten; an external step reintroduced network latency.

A fast replica is only one layer

Customers evaluated the resulting service, not just its runtime. They wanted preserved output quality, sustained time to first token and tail latency that held during traffic bursts—including P95 and P99 behavior among slower requests. They also brought requirements for data security, healthcare compliance and keeping data within particular geographies.

SGLang’s DeepSeek support addressed model execution through attention optimizations and a block-wise FP8 kernel. But Baseten framed production serving around three capabilities, each necessary and insufficient alone: - Model-level performance: efficient attention, kernels and decoding. - Horizontal scaling: adding model replicas quickly enough to meet demand. - Workflow support: combining multiple inference steps with low latency. Choosing a runtime addressed only part of the first capability.

Once traffic exceeded one replica’s capacity, the question became how quickly it could become five, ten or a hundred replicas. Baseten found a Kubernetes autoscaler alone insufficient for its workloads and rebuilt infrastructure components. Available hardware imposed another limit: a hypothetical deployment of 200 replicas, each consuming multiple H100s or a full node, could exhaust GPU capacity in one region.

Baseten therefore distributed replicas of a single model across regions and clouds. Its deployment example included 50 replicas in GCP East, 80 in AWS West and additional replicas on Oracle in London. A serving library does not supply this surrounding production system.

SGLang: reuse work, then route requests to it

SGLang reportedly began around August 2023 as a frontend for expressing programs composed of language-model calls. It added Radix prefix caching in January 2024 and constrained decoding with jump-forward support in February. Around June or July, it shifted toward a full inference engine, aiming to combine strong performance with code developers could extend and maintain.

Later 2024 releases added DeepSeek attention optimizations, a scheduler described as having zero CPU overhead and a router for cache-aware load balancing. These addressed both execution within a replica and the choice of which replica should receive a request.

Prefix caching reuses cached work for shared prompt beginnings. Radix caching was described as using a block size of one, compared with blocks of 32 in other frameworks, allowing finer-grained reuse and potentially more cache hits. Baseten’s workloads offered a favorable pattern: each customer repeatedly used a finite set of prompts or prefixes. The reported benefits were substantial but unquantified.

With dozens of replicas, however, each held its own cache state. Random assignment could send a request to a replica lacking the useful prefix. Baseten moved to KV-cache-aware routing, considering cached prefixes alongside queue lengths and replica locations, and reported noticeably better latency. Cache reuse had to be balanced with queueing and geography.

Structured output can skip decoding work

For structured output, a JSON schema could be converted into a finite-state machine: rules describing permitted output paths. When those rules determined upcoming tokens, jump-forward decoding could skip individual generation steps. The example replaced four decoding steps with one operation; it illustrated the mechanism, rather than a measured fourfold speedup.

The tradeoff was integration cost. Jump-forward decoding proved difficult to maintain alongside other optimizations, so it was disabled by default and could be enabled through arguments. A shortcut in generation could complicate the engine as a whole.

XGrammar and Outlines were discussed as constrained-decoding tools. One participant preferred XGrammar and reported its integration into the then-latest TensorRT release, although the performance comparison came without benchmark conditions. SGLang’s frontend served a separate role: expressing control flow across model calls, such as using one call’s result before continuing to another. The exchange about API speculative execution explained that programming interface without detailing the speculation mechanism.

The framework recommendations were practical judgments from the time: - TensorRT-LLM for performance, especially latency-sensitive serving. - SGLang for developers adding their own features and optimizations. - vLLM for broad community support. SGLang coordinated development through a pinned roadmap and biweekly community meetings about progress and priorities, alongside meetups and hackathon presentations.

Speculative decoding still needs training

By the conversation, SGLang supported EAGLE speculative decoding. A participant contrasted that with TensorRT-LLM’s then-current support for EAGLE 1 rather than EAGLE 2, but supplied no measurements for the accompanying performance claim. The consequential qualification was that framework support alone does not deliver the gain.

Speculative decoding can use a draft model whose proposals are checked against a target model. The draft—or additional prediction heads—must be trained to mimic the target closely enough to achieve a high acceptance rate. Training data and imitation quality therefore affect throughput. A runtime flag does not perform that training work.

Customer fine-tuning also remained useful for domain-specific tasks. Reported examples included adapting Whisper to medical jargon and training language models for medical-document extraction with data gathered through human-in-the-loop processes. The extraction models were described as competitive with leading alternatives, without evaluation details. These examples established uses of fine-tuning; their particular training methods were not identified.

One participant reported no decline in fine-tuning demand, while considering whether stronger reasoning models might eventually handle some tasks from a few examples in a prompt. The cost-effectiveness of that substitution remained open. Another doubted fine-tuning would disappear and saw the ease of editing prompts as a stronger reason to avoid it.

The product strategy was to solve today’s customer problems even if the market changed within two years. Working with those customers would expose emerging needs and guide future development. Interest in SGLang was also arriving from application teams: after V3 support shipped, a Cursor employee reportedly contacted the team with implementation questions.

When inference fails, the product fails

Mission-critical inference means that an inference slowdown or outage directly slows or disables the customer’s main product. That makes crash recovery, isolation from other customers’ traffic and rapid capacity expansion product requirements. Someone must train speculative draft models or additional heads, recover failed servers without disrupting traffic and add replicas fast enough to preserve latency requirements.

The AI phone call brings the three capabilities together: multiple models must execute efficiently, have enough capacity and communicate reliably with little delay. Geography affects both compliance and latency, especially in applications where another 50–100 milliseconds matters. The surrounding system must make the engine’s execution capabilities dependable across the entire workflow.

A closing proposal was to publish these operational requirements as a manifesto separate from Baseten’s brand, using Heroku’s Twelve-Factor App as a model. The aim was to make the expectations explicit enough for the wider market to adopt.

The episode ended with a community memory: Baseten’s early support for Latent Space events and Amir bringing 100 croissants to the 2023 hackathon. Participants remembered the event as a turning point for their community, recalling Joseph from Roboflow in a prompt battle, Harrison judging and Jerry from Llama Index attending.