Inference Engineering: From a 200,000-Token Request to the GPU — Part 1 of 2

A 200,000-token request starts with work you can skip

Send Baseten a 200,000-token query, and the first optimization is to check whether some of that input has already been processed. Cache-aware routing seeks a serving replica with available input-processing capacity and reusable cached input. Reusing that computation reduces the work—and potentially the cost—of handling the request. This is the opening technical example in The Inference Engineering Masterclass.

The remaining work has two stages. Prefill processes the input, builds a cache of values that can be reused during generation, and produces the first token. Decode generates subsequent tokens iteratively. On certain models, Baseten separates these stages across GPU pools: prefill workers build the cache, called the KV cache, then hand it off to decoding workers. Cache placement and transfer therefore become part of serving a request.

Decoding can use a smaller model to propose tokens for the original model to verify. Speculative decoding replaces some expensive sequential generation with cheaper proposals: in the example, the draft model makes three fast passes, each proposing a token, and the original model evaluates all three proposals in one forward pass. It accepts or rejects them rather than handing control of the output to the draft model.

Acceptance depends on the workload. A coding-oriented speculator is expected to predict coding traffic well; summarizing every Harry Potter book could run more slowly with the same speculator. The benefit depends on how much proposed work survives verification.

Dedicated capacity buys control—if you can keep it busy

Public model APIs charge per token; dedicated deployments charge for capacity. A reported adoption pattern is to experiment through a public API, then move a successful use case to dedicated infrastructure for reliability and customization. Hourly capacity can be cheaper at sustained high volume, provided it stays utilized. Millions of tokens per hour is an illustration here, not a measured break-even threshold.

Dedicated serving exposes several controls: - Traffic-specific speculation: train a draft model for the customer's requests to improve acceptance rates. - Latency versus throughput: choose batch sizes and parallelism strategies for the application's priority. - Precision: retain higher numerical precision if quantization—compressing model values into fewer bits—fails the customer's benchmarks. - Isolation: avoid competing with another customer's large benchmarking run on a shared endpoint.

A shared speculator must accommodate heterogeneous requests; a dedicated one can target a narrower workload. The Harry Potter example illustrates that specialization, but the suggestion that training on those books would guarantee perfect acceptance remains unvalidated. Higher acceptance is the intended benefit; perfect acceptance is not established.

Valid JSON does not make a correct tool call

A consequential tool-calling failure occurs when the model emits a request but fails to terminate it correctly. It continues decoding and invents the tool's result before receiving one. Specialized tool behavior is described as sensitive both to post-training quality and to subsequent quantization.

Three responsibilities remain distinct: - Output structure: a state machine constrains generation to the specified format. - Tool-use decisions: the model must still choose an appropriate action; valid structure does not prevent it from omitting a needed call. - Execution: an external system interprets the model's formatted proposal and performs the action.

An agent loop can sometimes recover by retrying an incorrect output or failed invocation. Recovery may become harder when replacing a larger model with a smaller one whose output quality differs. A normal dedicated deployment is described as sufficient for generating tool calls; the requirements for safely executing them are a separate question.

JSON creates another problem during streaming: incomplete structures can be difficult to parse and validate as they arrive. One participant expects short, predictable tool calls to decode quickly with speculation, reducing streaming's value. That is a hypothesis, and structured writing supplies a counterexample: fields may contain entire paragraphs, separating facts, opinions, summaries, dates, entities and sources. Structured output can be long-form content. Existing JSON tool interfaces also make changing formats an integration decision.

A first token is only the beginning of model support

Basic execution and a production-ready API require different amounts of work. Open-source engine maintainers may receive weights early, or model developers may contribute support patches. A provider still has to integrate each release into its own combination of open-source and proprietary systems, even when the architecture resembles a model it already serves.

The deployment work includes: - Quantization and calibration: convert weights to the desired numerical format—NVFP4 for Blackwell compatibility in this setup—and check that capability is preserved. - Speculator training: run representative coding and agent prompts through the target model, collecting its internal representations, or hidden states, as training data. This workflow requires access to the target weights. - Infrastructure and runtime integration: provision capacity, load components and test the system. Novel architectures add runtime work.

For general speculators, a speaker says they do not know exactly what customers submit to the API. Public datasets representing popular coding and agent workloads stand in for that direct knowledge. This makes the shared speculator an approximation of likely traffic, while dedicated training can target a more specific distribution.

Training can make new components useful—and serving faster

An experimental model combines Kimi vision, GLM language-model weights and DeepSeek attention. The vision encoder converts images into latent representations. A projector, an adapter with a few million parameters, maps those representations into the language model. The encoder and language model remain frozen while only the projector is trained, with the aim of adding vision without damaging other capabilities.

Training begins with image descriptions, such as describing a mountain photograph, then moves to multiple questions per image about objects, people and positions. The speaker reports that this improves the model's use of visual information, without giving a controlled measurement. One illustrative failure is identifying Stephen Hawking as Albert Einstein: the speaker interprets this as retaining a scientist-related association, but a related answer is still an identification error.

For text-only requests, the system skips the image encoder and is claimed to preserve the base model's behavior. Vision encoders are described as a relatively minor serving component, although their lower standardization leaves less complete runtime support. The experiment demonstrates the intended modular setup; its achieved vision quality is not established here.

Component replacement also appears in decoding optimization. A full-attention layer reportedly creates a bottleneck and a large KV cache during three-token draft generation. The remedy is to substitute a layer using GQA, whose internals are not explained here, then train it to recover the same acceptance rate. No measured speedup accompanies the example. The consequential claim is that fast inference can depend on training replacement components well enough to preserve useful behavior.

Repeated tokens can expose a serving bug

Production support continues after launch. A reported GLM deployment repeatedly emitted the same token for particular prompts and temperatures, exposing cases internal testing had missed. One mitigation stops generation after four or more identical tokens, then requests a retry or reprocesses the request. Repetition detection is a heuristic: legitimate output, such as dashes in a table, needs exceptions. The exact exclusions remain uncertain.

The repeated token is often reported as S, and the failure can occur around temperature 0.9. Holding the weights fixed while changing the serving stack helps isolate the cause. The speaker reports cases where updating the stack or switching from SGLang to vLLM removed the repetition with identical weights, pointing toward the serving implementation in those cases.

Hardware timing can expose a software error. One reported race condition appears only on a cluster with slower node-to-node transfers: changed KV-cache transfer timing triggers the defect. Moving the model to another cluster avoids it operationally, but does not repair the synchronization.

Kernel optimizations can overlap execution by starting one kernel before the preceding kernel finishes. Missing barriers or incorrect synchronization can then allow a thread to read a value before another thread writes it. These timing-sensitive failures are difficult to test in highly optimized kernels.

Quantization targets fidelity to the original model

Inference quality means fidelity to a reference implementation: serve the model faster while preserving its intended behavior. KV caching avoids recalculating values, and speculative decoding rejects incorrect drafts. Quantization is the main lossy optimization identified here. Compressing values from, for example, 16 bits to 4 bits discards information, so quality depends on the numerical format, which layers are compressed and how calibration preserves important outliers.

Other performance decisions change the supported workload. A shorter-context endpoint can sit alongside a full-context endpoint and offer better performance to users who do not need the maximum input length. Provider fidelity also matters to model developers: users may blame the model for poor outputs caused by a serving implementation. The Kimi vendor verifier is cited as a benchmark for assessing quality across providers.

Tiny benchmark improvements after quantization may be noise rather than better model quality. A more substantive possibility is that quantization errors cancel across layers. Quantizing layers 1, 5 and 10 could preserve behavior better than quantizing only layers 1 and 2 if their errors oppose one another. This compares two compressed versions; it does not establish improvement over the full-precision model.

A speaker reports research that predicts which layers' errors cancel and selects those layers for quantization. Fidelity is evaluated by comparing the quantized model's output-token probability distributions with those of the original full-precision model. The aim is to preserve the original probabilities, beyond matching an aggregate benchmark score.

The reported outcome combines better fidelity with more quantized layers and a claimed 20% throughput gain over another quantization. The comparator and test conditions are unspecified, and earlier descriptions of the percentage vary. The mechanism is the useful distinction: additional quantization need not worsen fidelity monotonically when layer errors interact.

Speed comparisons need a workload and a baseline

“Tokens per second” hides two different measurements: - Aggregate throughput: total output tokens produced by the GPU system per second. - Per-request generation speed: how quickly an individual response arrives. Intertoken latency measures the time between generated tokens. Comparisons also depend on hardware, system load, prompt characteristics, and input and output lengths.

An illustrative starting point is a relatively unoptimized API serving a trillion-parameter model at 30–50 tokens per second under what is described as a reasonable traffic profile. The ambition is roughly 10× faster through accumulated optimizations. The discussed 300–400-token-per-second range requires top-end hardware, an optimized speculator, quantization, high cache reuse, a reasonably small batch and parallelism tuned for latency. Fast individual responses can come at the expense of throughput.

The speakers call 10× aggressive and suggest 4–6× provider differences are more common. Holding hardware and GPU count fixed narrows their estimate to about 2–4× from inference optimization. Moving from one node of H100s to four nodes of B200s illustrates why a headline gain cannot automatically be credited to software.

Their rough accounting separates the contributors: - Lower precision: about 30–40% from 16 to 8 bits, multiplied by another 30–40% from 8 to 4 bits. - Speculation: roughly 2×. - Prefill/decode disaggregation: roughly another 2×, given enough hardware and traffic. - Runtime and kernels: a further double-digit percentage from improved implementations. These are explanatory estimates, not controlled measurements whose factors can be assumed to multiply on every deployment.

For self-hosters, the practical advice is to reuse published quantized checkpoints and trained speculators when available. Creating each from scratch is estimated at hours to days for an experienced practitioner. The similar estimate for disaggregation covers marginal implementation after the system exists; getting it working for the first time is described as very difficult.

NVIDIA Dynamo is described as an open-source toolkit for cache-aware routing, cache offloading, prefill/decode disaggregation and data movement across a cluster. Installing it alone does not establish an optimized deployment. Its replaceable defaults prompt a disagreement about benchmarking: one participant proposes Dynamo as a baseline, while another argues for comparing configurations actually deployed in practice.

Speculating on the speculator adds another cost

The Inference Engineering book aims to build vocabulary and intuition as techniques evolve. Medusa explains the history of speculative decoding rather than serving as a recommendation of the latest method; Eagle is described as still commonly used at the time of the conversation. Speculation is singled out as advancing faster than the other areas covered.

Recursive speculative decoding applies speculation to the draft model itself. If a draft model spends three sequential passes proposing tokens for a larger model, an even smaller model could accelerate those proposals. This raises an unresolved question: if the smallest model predicts the intermediate model well enough, could it draft directly for the target?

Draft tokens also consume resources. In the described setup, draft and target models compete for hardware, and every additional level adds coordination and inference-engine complexity. A recursive approach is reported to show nontrivial speedups, but without numerical results or workload conditions. Its practical value depends on whether the saved work exceeds the cost of generating and managing more proposals.

Local inference changes the binding constraint

Local and data-center inference begin with different priorities: - Local inference: fit the model onto constrained hardware, then preserve as much capability as possible. - Data-center inference: load the model, then make serving faster. Both care about quality and speed. The local community is credited with expertise in quantization, pruning, distillation and layer removal to fit models onto laptops. Its process and openness offer lessons even when different resources make a technique unsuitable for direct transfer.

TurboQuant illustrates a hardware-dependent tradeoff. A speaker says it attracted local implementations because of memory-bandwidth constraints, but reports that on a B200 its quantization and dequantization overhead exceeded the time saved on memory transfers. Without workload details or measured timings, this is not a universal B200 result. The decision turns on whether reduced data movement saves more time than conversion adds.

The final topic is distributing models across machines. Exo Labs' heterogeneous hardware work is discussed through examples including DGX Sparks connected over Ethernet and stacks of Mac Minis. The interconnect is a central constraint. In an eight-GPU example, tensor parallelism shards the model across the GPUs; the speaker argues that it fits poorly when local machines lack a high-bandwidth connection such as NVLink. Pipeline parallelism is raised as an alternative, but Part 1 ends before that comparison is developed.