LLMs everywhere: the compiler work behind local inference

XGBoost began with an experiment too slow to run

Around AlexNet’s arrival, Tianqi Chen and his advisor Carlos Guestrin wanted to test whether alternatives to deep learning could achieve comparable performance given enough data. Existing gradient-boosting libraries were too slow for the experiment. XGBoost grew out of their need for a faster implementation.

Chen initially thought a command-line interface would suffice. A friend persuaded him to add Python bindings, which he later considered an especially good decision. Python and R interfaces helped the project grow into a community effort, with contributors adding capabilities such as visualization. Chen brought lessons from earlier matrix-factorization library work; MXNet and TVM, he says, were more collaborative from the outset.

His later architecture work also took shape in physical sketchbooks, filled at roughly one per year. They let him explore new directions and look back at how designs became code. The habit began after XGBoost, but captured many TVM design details; he had not published the books.

The original hypothesis needs a qualification: the disappointing comparison concerned image classification, where Chen considers deep learning the clear choice. For tabular data, he values trees’ rule-based decisions, insensitivity to input scale, automatic feature composition, and some interpretability and control. TVM itself used XGBoost to build program cost models—tree-based learning supporting a machine-learning compiler. Chen sees room for a diverse toolkit, including possible ensembles of neural and tree-based models.

Make trying a model less expensive

Another early experiment exposes the cost of low-level implementation. A master’s-project account in the conversation describes roughly six months spent handcrafting CUDA kernels for a convolutional restricted Boltzmann machine intended for ImageNet classification, before AlexNet. The model ultimately performed poorly. Simply getting an idea to run had consumed months of engineering.

[Apache TVM](https://tvm.apache.org/) addresses that burden through machine learning compilation: translating a supported model representation into optimized code for a target device. It accepts computational graphs or lower-level loop representations, transforms them, and generates code for GPU and non-GPU targets. Model formats such as ONNX and tracing through PyTorch FX provide entry points, but the model still has to map into a representation TVM supports.

TVM Unity was the second iteration described in the conversation, designed to make new models and optimizations easier to bring in. The [Machine Learning Compilation](https://mlc.ai/) effort, or MLC, applies that work through tutorials and projects such as [MLC LLM](https://github.com/mlc-ai/mlc-llm). Its goal is universal deployment, with phones, Apple laptops, CUDA systems, and AMD GPUs among the targets.

Portability does not imply performance parity. One reported single-batch result put a recent AMD consumer GPU at about 80% of NVIDIA’s leading consumer card. The comparison acknowledged the gap, but did not specify a model, performance measure, or benchmark procedure, and the exact NVIDIA card is unclear. It is a limited result, not a general hardware ranking.

Represent the model, then transform it

TVM Script represents model computations at graph and operational levels. In the workflow described, vision models and Stable Diffusion could usually enter through PyTorch tracing, though that path was still being made more robust. Language models used directly built constructors mapped from Hugging Face models, with the goal of accepting a configuration and applying optimizations.

Once a model is inside the compiler, developers can control intermediate transformations through a Python API. Optimization therefore extends beyond edits to the original model source. Compiled functions that take tensors as inputs and return tensors can then be loaded into supported runtimes, including JavaScript, Python, C++, and Java. The optimization language need not be the application language.

TVM builds on an existing compiler tradition. It initially drew substantial inspiration from Halide, an image-processing compiler, and developed within a broader machine-learning compilation community. Its approach brings compiler optimization, machine learning, and systems knowledge together.

Where the performance comes from

A low-level runtime such as CUDA is only part of the performance story. Libraries such as CUTLASS and cuDNN supply optimized implementations of computations. In a retrospective example, NVIDIA libraries were described as difficult to beat on familiar ResNet benchmarks roughly two years before the conversation, while variants for lane detection left more room for optimization. The qualitative lesson is that engineering effort can concentrate on familiar benchmarks and leave other workloads underserved.

TVM uses existing libraries where useful and generates the required functionality where possible. That code generation enabled execution on Apple M2 and WebGPU without depending on an available optimized library for those targets. Vulkan also provided a route to AMD hardware when ROCm driver support was less mature, trading some performance for portability.

The compiler works at several levels: - Kernel fusion combines an operation, such as a convolution, with following operations so they can execute without separate GPU kernel launches. - Memory planning arranges allocations ahead of execution, rather than relying entirely on allocations as arrays are created. Frameworks already use smart allocators; static planning offers another way to organize that work. - Loop transformations reorganize computation to use shared memory, tensor cores, and memory copies effectively.

Language models make planning harder because tensor dimensions can change. Symbolic shape tracing preserves relationships between dimensions without requiring their values in advance. A compiler can know that one tensor has shape N × 12 and another has shape 2N × 12, even before it knows N. Those relationships can inform fusion and other optimization decisions.

Exposing the compilation framework as a Python package makes these transformations open to contributions. Engineers can inspect a new model’s bottlenecks and improve the relevant optimizations. Llama 2 was cited as an example of this quick response to a new model, without a measured speedup.

Choose precision and deployment independently

The language-model work discussed mostly used INT4 quantization, a four-bit integer representation that reduces memory overhead. This concerned inference, not training. INT4 was described as workable in many inference cases, rather than a universal precision choice; no model-specific accuracy measurements accompanied that assessment.

Bring Your Own Quantization was proposed work to let developers customize quantization while the compiler generated optimized code. It remained in progress, with the aim of supporting more formats as research evolved. Greater sparsity and more extensive activation quantization were also open directions.

The comparison with [GGML](https://github.com/ggerganov/ggml) centers on where optimization happens. GGML was characterized as implementation in C, offering direct control over individual backends while requiring separate engineering for CUDA, AMD, and other targets. MLC aims to put model definitions, kernel optimization, and memory planning in Python, while letting applications use JavaScript, C, C++, Rust, or Python. Mojo entered the discussion only at a high level; the respondent had not examined its technical details.

Shipping usable systems is also a research method. Real users expose constraints and research questions that prototypes can miss. That feedback complements academic work through papers and insights, while giving compiler research concrete problems to solve.

The browser still needs memory and model weights

[WebLLM](https://webllm.mlc.ai/) brings the compilation pipeline into the browser. MLC compiles a model for WebGPU, which lets browser code access native GPUs, and WebLLM supplies the runtime that invokes the generated kernels. The same compilation pipeline serves iPhones, server GPUs, AMD hardware, and MacBooks, with customization for particular targets.

The large-model demonstration began in a MacBook command-line interface using Metal GPU acceleration, then moved to the existing WebGPU backend. Browser execution was described as smooth, without a quantitative benchmark. It required a high-memory M2 Max MacBook, with an estimate of roughly 50 GB of RAM. The transcript alternates between 70 billion and 17 billion parameters, so the memory estimate cannot confidently be tied to either count. This was a MacBook example, not evidence of that model running on an iPhone.

Avoiding an application installation does not eliminate the model download. Transferring the weights remained a practical obstacle, enough to prompt doubt about whether a browser model of the demonstrated size made sense yet. A hybrid chatbot offered a more selective possibility: run speech-to-text on the client and keep other components on a server. Local inference can be a decision about individual components.

Web applications could integrate WebLLM through its npm package. An OpenAI-compatible REST API was also described, running on a native backend at the time, with a WebGPU version available as well. These interfaces aimed to make local models easier to incorporate into existing applications.

WebLLM’s release coincided with Chrome’s official WebGPU shipment, but development had begun earlier through Chrome Canary. A TVM-based WebGPU backend had existed roughly two years before; improving maturity and performance prompted a stronger push toward a language-model-compatible browser runtime in 2023.

Caching downloaded weights could reduce repeat transfers. The discussion also described reuse by another application using the same WebLLM package, though it did not specify the cache-sharing boundaries or required configuration. Cross-application reuse was therefore conditional; at least one initial download remained necessary.

Better APIs and easier integration were treated as useful work throughout the local-model ecosystem. The goal was to collaborate toward deployment that approached a push-button operation while preserving performance where possible.

Running once is only the start of serving

OctoAI addressed the availability, scaling, and integration problems that arise after a model first runs. A service must handle changing request loads and hardware availability, as well as manage where models reside and move them efficiently. At the time of the conversation, difficulty obtaining recent NVIDIA hardware made support for other hardware especially consequential.

Abstracting hardware choices requires multiple backends, model optimization expertise, scaling infrastructure, and model placement to work together. Recent NVIDIA libraries may perform well on recent GPUs, while a cloud service also needs to accommodate other GPUs. Portability supports the broader goal of using available hardware, controlling costs, and distributing work across devices and the cloud.

Serving optimization proceeds through product iteration: work with users, identify a specific opportunity, have engineers address it, and incorporate the improvement into the service. No single optimization solves every workload.

Make room for models that keep changing

Universal deployment also supports a possible open ecosystem of interacting models. A personal model might know someone’s movie preferences, while a larger model supplies broader knowledge. Some models could run locally and others in the cloud. This was a future MLC aimed to help enable, alongside the possibility that access would concentrate around a few large providers.

Long-context problems also motivated interest in architectures beyond transformers, including state-space and recurrent models. Work to bring RWKV to MLC was underway, without a completed integration or performance result. For the compiler, the central requirement was productivity: help engineers bring emerging architectures to available execution environments quickly.

In the closing questions, conversational chatbot capabilities stood out as having arrived sooner than the respondent expected. Continuous and lifelong learning remained an exciting open direction: systems that evolve as knowledge changes. That meant more than continuous training. It included collecting data, training, fine-tuning, and continually curating local context to feed into models.

The closing engineering advice connects the whole discussion: building useful AI applications requires models, system optimization, and data curation together. Understanding their interactions is part of the work, and becomes increasingly important as AI applications develop.