Inference Engineering, Part 2 of 2: Parallelism, Video, and Learning Loops

A sparse model can still exercise every expert

A mixture-of-experts (MoE) model activates only a subset of its parameters for each token. With a batch size of one, that active parameter count matters greatly. A batched API server faces a different workload: requests can use different experts. One engineer’s operating assumption is that the batch will exercise the whole model, even though no individual token does. This early distinction in Part 2 of the inference engineering conversation explains why serving decisions depend on the workload, not just the model’s active parameter count.

Three forms of parallelism distribute that work differently: - Expert parallelism places whole experts on GPUs, potentially several per GPU, and replicates the small router. It requires an MoE architecture and is presented primarily as a way to increase throughput. - Tensor parallelism splits computation across GPUs and combines partial results through collective operations such as all-gather and all-reduce. It works with dense and MoE models, often helps latency, and makes interconnect performance consequential. - Pipeline parallelism places different layers on different hardware. For a hypothetical trillion-parameter deployment spanning multiple H100 nodes, an engineer recommends pipelines between nodes to accommodate slower inter-node communication, with tensor and expert parallelism within each node. The throughput-versus-latency distinction is a generalization with exceptions. Deployments often combine these techniques.

Hardware can influence the architecture before serving begins. AI21’s Jamba is cited as an example of choosing hardware and designing model dimensions around it. For an existing model, however, the proposed tuning procedure is empirical: compare configurations against the same real traffic. 1. Enumerate feasible tensor- and expert-parallelism configurations. 2. Shadow the same product traffic through each configuration. 3. Compare observed token throughput and generation speed, then select the best configuration for that workload. Kernel tuning follows the same pattern: after designing the kernel, sweep thread counts, shared-memory usage, and other configuration choices. No single configuration consistently wins.

NVIDIA Nemotron models offer another example of hardware alignment: a participant describes them as running particularly well on Blackwell, without giving benchmark results. The tradeoff is portability. The same participant suggests that most open-model labs favor compatibility across many chips over targeting one accelerator.

Fusion cannot erase a communication dependency

A megakernel fuses operations to reduce kernel launches and data movement. But fusion cannot remove a dependency on another GPU’s result. In the tensor-parallel example, a subsequent softmax needs information from partial results on both GPUs. Combining local operations into a larger kernel does not make the remote information available: the GPUs still have to communicate.

One engineer also questions whether megakernels repay their implementation complexity. Modular kernels let engineers optimize individual components; a large fused kernel is difficult to optimize throughout. Drawing on unnamed companies they have worked with or heard about, the speaker reports that modular implementations often displace megakernels in production. This is an experience-based assessment, not a benchmark establishing a universal winner.

Hardware upgrades bring a related software problem: installation does not mean inference readiness. One participant recalls early vLLM targeting Ampere, then requiring updates for Hopper and Blackwell. Each generation made the transition more urgent and more complicated, with usable inference software potentially lagging the arrival of hardware.

Looking ahead to Rubin, a participant expects CPU–GPU and GPU–GPU connections to matter increasingly alongside memory bandwidth. KV cache occupies context-dependent inference storage distinct from model weights; longer contexts compete with weights for available memory. This part does not explain what the cache contains. Dynamo is framed around moving KV cache to where it is needed, when it is needed. That makes cache offloading, cache-aware routing, and separating serving work across hardware increasingly valuable. The forecast expands inference engineering from kernel optimization into infrastructure; another participant goes further, predicting that programmers will increasingly orchestrate tiles of data while hardware handles execution details.

Custom silicon depends on workload longevity

The hardware debate turns on specialization versus fixed weights. A GPU can gain specialized instructions and data-movement facilities while retaining programmability. The participant comparing Rubin to an application-specific integrated circuit (ASIC) explicitly acknowledges that it remains a general GPU programmable with CUDA. Building a chip around fixed model weights is a stronger commitment: fine-tuning, quantization, and new checkpoints can conflict with that choice.

Two economic arguments pull in different directions: - Programmable GPUs absorb specialization. As NVIDIA adds facilities tailored to model workloads, independent accelerator companies may have less room to differentiate. Changing architectures also threaten fixed hardware assumptions. - Model labs can justify custom silicon at sufficient scale. A participant recounts Martin Casado’s hypothetical: against a $500 billion or $1 trillion training run, spending $50 billion on an ASIC could make sense if it delivered more than a 10% efficiency improvement. These are illustrative economics, not an actual training run or measured accelerator result.

Another participant argues that gains of 10×–1,000× would require changes beyond rearranging work within existing GPU designs, particularly memory allocation and communication between chips. They leave open how much Rubin will address. That ambition also prompts a defense of incremental kernel work: modest serving efficiencies can still matter for expensive models deployed over long periods.

Model longevity changes the investment calculation. One participant worries that annual architecture changes could force repeated chip investments. Others dispute the premise that older models quickly lose value. An engineer still sees Llama 3 workloads; a hypothetical enterprise batch job explains why. Once stakeholders approve a model and its daily results are predictable, replacing it creates organizational work. Open models also let users preserve a deployment without depending on a provider to maintain a hosted endpoint.

The disagreement extends to capability. One participant argues that a sufficiently capable model with search, tools, and code execution could remain useful for a long time; another questions whether that assumes users do not need further intelligence gains. Newer models becoming cheaper and better does not, by itself, make older models unusable.

Fitting weights is only part of fitting inference. For a model described as having 2.8 trillion parameters, a participant estimates about 1.4 TB of weights in the low-precision NVFP4 format and says eight GB300 GPUs with 288 GB each can accommodate those weights. Context-dependent cache storage must also fit, so large models and long contexts compete for memory. The same discussion cautions against assuming access to NVL72 systems when eight-GPU configurations are much more common.

Serving hardware therefore influences model design through both capacity and architectural fit. A participant says DeepSeek R1’s 671 billion parameters pushed their team to adopt Blackwell quickly. Available memory sets a practical model-size ceiling, while architecture and native quantization affect how well a model uses the target accelerator.

Video fits on a GPU but strains attention

Ali describes a video-diffusion serving setup in which each request occupies one GPU, without batching or model sharding. He cites Wan2-2, described as a 20-billion-parameter model, as much smaller than the large language models just discussed. This placement pattern applies to the setup he describes, rather than all video serving.

Quality can outweigh a large cost advantage. A participant judges open and closed language models to be approaching parity, while open video models still lag Kling and Veo substantially. In a hypothetical buying decision, a higher-quality $1,000 movie remains preferable to a $10 alternative. The proposed consequence is a feedback loop: weak demand for open video models reduces investment and checkpoint releases. These are qualitative judgments, not benchmark or purchasing results.

Video also creates a large attention workload even when its weights fit comfortably on one GPU. The example uses roughly five seconds of low-resolution video at 16 frames per second. Compression into a latent representation still leaves tens of thousands of tokens. Full attention scales quadratically with sequence length because every token attends to every other token, making longer clips increasingly expensive.

Three approaches expose the compute–quality tradeoff: - Retain full attention. All tokens can interact, but long sequences require much more compute. - Use sparse attention. Each token attends only to selected tokens—12.5% in the example. Selection can exploit spatial and temporal relationships: a crown attends to a face and to the head in adjacent frames. The speaker reports substantial quality degradation in that case; training can mitigate it without eliminating the tradeoff. - Generate autoregressively. Produce the sequence progressively, potentially streaming frames while the viewer watches and reusing language-model serving optimizations. The speaker favors this direction for long video, while judging available open autoregressive video models insufficiently good and the necessary training techniques unsettled.

Streaming gives up the ability to revise earlier frames

A practical way to extend a clip is to feed its final frame and a text prompt into an image-and-text-to-video model, generate another five seconds, and repeat. This approximates clip extension without knowing proprietary systems’ stitching techniques. But each generated output becomes the next input, allowing errors to compound. One speaker describes successive clips becoming darker until the screen turns black around 25 seconds; their team abandoned a planned demo because the output was too poor. This is a reported failure case, not a systematic evaluation.

The deeper distinction is revision. In the diffusion-video setup described, bidirectional attention lets earlier and later frames constrain each other while the model iterates over the full sequence. A naive forward-only generator cannot repair a frame it has already emitted. Video also needs distant regions within one frame to remain coherent. Another participant therefore forecasts a combination of autoregressive and diffusion components, using progressive generation and whole-sequence refinement together.

Orpheus illustrates a more direct streaming path for speech: extend a language model’s vocabulary with tokens representing audio waveforms, generate those tokens autoregressively, and construct streamed speech from them. The respondent explicitly declines to generalize this explanation to music.

For text diffusion, the proposed opportunity is an interface for whole-output refinement, beyond cheap, fast chat responses. Two applications make the distinction concrete: - Poetry: revise an earlier line to support a later rhyme and maintain constraints across the poem. Better syllable counting is a proposed benefit, not a demonstrated result. - Storyboarding: let a movie’s ending and beginning constrain each other, then generate individual scenes with a video model. Limited adoption does not settle the architectural question: participants argue that smaller training scale may confound comparisons with leading autoregressive models.

Training and inference depend on each other

The boundary between training and serving is becoming less distinct in two ways: - Inference for training: training consumes generated outputs, or rollouts. Slow inference delays them. Avoiding rollouts that are too far off-policy can force training to wait. - Training for inference: efficient serving can require trained speculative-decoding components or additional training to recover quality after quantization reduces numerical precision.

The quantization example makes the second dependency concrete. Keep low-precision and full-precision versions of a model, then train the low-precision version to match the full-precision version’s outputs, using their logits. Post-training quantization sometimes works directly; when it degrades quality too far, recovery becomes a training-pipeline responsibility for the inference team.

A proposed product loop extends that relationship: collect product traces, post-train the model, deploy an update, A/B test it, and use the feedback for the next iteration. A participant forecasts broader production adoption within months to a couple of years. Separately, an engineer reports already using live inference feedback to adjust serving configuration and speculative generation behavior, claiming an advantage over static configuration without supplying measurements.

Configuration assistance is already changing routine work: one participant says preparing an arbitrary Hugging Face model to run used to take substantial effort and can now be handled in one model response. They distinguish that assistance from a model optimizing its own inference. An engineer then describes an internal example of the latter: 1. Run an inference forward pass and collect a profiling trace. 2. Identify bottleneck kernels and write replacements. 3. Profile again and upload an image the team can pull to repeat the cycle. Some kernels serving the coding model were reportedly written by that same model. Human oversight remains necessary: the engineer reports poor decisions and reward hacking, and gives no measured speedup.

Moving state and preserving knowledge

The closing systems discussion returns to communication. A voice agent may compose three to five models, making the connections between them another optimization target. One participant anticipates further orders of magnitude in demand, leaving long-tail reliability and the capacity to produce another 10–100 times as many tokens as continuing infrastructure challenges.

Cache transfer can bottleneck decode. In the serving path an engineer describes, KV cache moves to another node’s memory and then into its GPU. The speaker imagines much faster network interfaces avoiding intermediate staging and yielding nearly a hundredfold decode improvement in that inter-node scenario. The estimate is theoretical: they explicitly lack details about the engineering challenges of making those interfaces fast enough.

The final technical question asks how serving changes if a model learns from everything it processes. Two proposed routes put persistent knowledge in different places: - Update model weights. Incorporate new knowledge into the model and have serving systems fetch the updated weights. - Retain and compact KV memory. Preserve knowledge in the cache while leaving weights unchanged, adding cache updates or compaction to the serving workflow.

The objection to weight editing concerns whether a changed fact propagates into reasoning. In a hypothetical example, a model is told that Waterloo is the best university. Would that edit also change its advice about hiring an intern from Waterloo versus MIT? The speaker argues that changing direct recall need not change downstream reasoning. The example illustrates a concern about knowledge editing, rather than establishing a general experimental result.

An engineer consequently favors KV-based memory, but strengthens the conditions while explaining the proposal: compaction alone is insufficient. It would require almost indefinitely extensible memory that compacts without losing knowledge. Under those conditions, the speaker argues, it could support continual learning with relatively little change to serving architecture. That is an architectural expectation, not a demonstrated solution or latency result.

The conversation closes with a wish to return in a year and examine which views proved wrong—and anticipation of the community’s reaction to the megakernel skepticism.