Building an AI engine that leaves room for the next idea

Making the software fit

Before the first iPhone shipped, Chris Lattner recalls, nobody knew whether all its software could fit on such a constrained device. What later seemed obvious was still research. His subsequent work at SiFive, which builds RISC-V technology, exposed him to the other side of that boundary: designing hardware and understanding the tradeoffs software must accommodate.

That boundary is the central problem in this Latent Space conversation with Lattner, published in September 2023. [Modular’s AI engine](https://www.modular.com/blog/the-case-for-a-next-generation-ai-developer-platform) connects existing model-building frameworks to hardware through a programmable execution layer. Mojo, the language that emerged while building it, serves that broader effort.

Lattner’s diagnosis begins with TensorFlow and PyTorch. Their designs grew around AI research and model construction. Supporting many different kinds of hardware posed another problem: each mathematical operation needed an efficient implementation on the target machine.

From handwritten kernels to programmable compilers

The stack starts with interfaces for building layers, such as Keras. Underneath sit operators—convolutions, matrix multiplications, reductions and elementwise operations. Their hardware implementations are kernels, commonly written using infrastructure such as CUDA or Intel’s math libraries. As frameworks accumulated thousands of operators, Lattner argues, supporting a new chip could require implementing thousands of kernels.

Hardware companies need that coverage to make their chips useful. Researchers need kernel specialists to turn new model ideas into efficient execution. Model design and kernel optimization require different expertise, so a shortage of specialists can constrain both hardware adoption and research. At Google, Lattner encountered this while working on its TPU training accelerators. The team’s decision to use compilers preceded his arrival and helped attract him there.

Kernel fusion shows why compilation helps: combine operations so intermediate results need not repeatedly travel through memory. A compiler can optimize new combinations; a library of manually fused kernels covers combinations its authors anticipated. The discussion credits Google’s [XLA](https://github.com/openxla/xla) compiler with enabling enormous training systems, while identifying another bottleneck: recruiting compiler engineers who also understand machine learning. Extending it outside Google’s TPU development environment is described as particularly difficult.

AlexNet supplies the counterpoint. Its breakthrough is often explained through ImageNet data and GPU compute, but the argument here adds programmability: CUDA let researchers invent convolution kernels before TensorFlow existed. Modular aims to combine that freedom with the compiler’s ability to optimize across combinations of operations.

Higher-level representations also help specialists collaborate. Tri Dao’s work across systems, models and applications illustrates why no individual needs to master the entire stack. A declarative graph—a representation of the computation that tools can transform—offers more than convenient syntax. Transformations such as pmap and vmap become possible at that level. The Google account traces the payoff from low-level cluster programming to researchers directing petaflop-scale training from Jupyter notebooks.

Why start with CPUs?

A framework is the interface where developers build and train models; an engine is the underlying layer that drives hardware. Modular describes its engine as plugging into TensorFlow, PyTorch and other systems. A host recalls a launch demonstration in which the engine was swapped while the application kept working—an example of what separating those layers can enable.

The CPU-first decision follows from treating AI as parallel computation across different kinds of processors. Data loading, preprocessing and networking surround the matrix multiplications; postprocessing and output follow them. A stack designed only around an accelerator’s supported operations leaves much of the application elsewhere. CPUs also increasingly offer matrix acceleration, while GPUs become more programmable, making the boundary a spectrum rather than a clean split.

Modular’s preference was to start general, then specialize. CPUs provided accessible development hardware and variation across Intel, AMD and Arm Graviton, including different vector lengths. Substantial inference still ran on CPUs, and the commercial argument was that inference demand grows with a product’s user base. GPU support was forthcoming at recording time. Modular also claimed to outperform vendor CPU software, though no workloads or measurement conditions accompanied that comparison.

Rebuilding the engine meant revisiting choices that become difficult to change in mature systems. Modular describes a lightweight thread pool whose tasks cannot block, alongside handwritten kernels and an auto-fusing compiler. Its goals included escaping static-shape restrictions and accommodating ragged tensors and tabular data. Much of the underlying technology was already proven; the demanding work was understanding how foundational decisions affected everything above them.

Overlap execution and avoid intermediate writes

Generality must still preserve access to specific hardware capabilities. Consider the discussion’s hypothetical company with $200 million invested in a particular A100 configuration: it wants to exploit those chips fully. Modular’s design target is abstraction by default, with access to hardware details when needed.

Asynchrony is one foundational choice. Treat the CPU and GPU as independently executing computers that exchange messages, allowing useful CPU work to overlap accelerator work. The reasoning extends to CPU cores exchanging cache lines and machines communicating across networks. Building around independent execution makes overlap part of the architecture rather than a later addition.

One speaker recalls a Google project that doubled ads throughput by overlapping CPU and TPU computation on an already highly tuned workload. No further measurement conditions are given; the result concerns that project, not a general Modular speedup.

Fusion removes another cost. A matrix multiplication followed by ReLU, a simple maximum operation, can execute as one kernel without the intervening memory traffic. Hand-tuning that pair is manageable. Repeating the process for increasingly complex combinations, such as attention layers, produces a growing collection of fast special cases.

FasterTransformer illustrates the tradeoff. The discussion credits NVIDIA’s library with valuable performance but argues that researchers become constrained by the transformations it supports. A generalized compiler should let them change the computation while retaining performance; matching or exceeding the specialized library remains a goal here, not a demonstrated comparison. Hardware variation adds another reason to compile: even one vendor’s product family changes as new instructions and numerical formats arrive.

The same issue affects adoption of novel hardware. Cerebras is offered as an example of an accelerator requiring a vertical software stack, with a different interface that raises switching costs. Modular’s proposed common foundation would let hardware plug in and developers reuse tools, including quantization tools across phones and supercomputers. The promise is shared tooling, not equivalent device capabilities.

Mojo emerged from building the engine

Modular did not begin with plans for a language. The team first built runtime foundations, framework integration and compiler technology, including automatic fusion, caching, distribution and autotuning. Mojo’s immediate purpose was developer productivity: engineers were writing low-level compiler representations in MLIR by hand and wanted better syntax.

The team considered an embedded domain-specific language, such as Halide, before choosing a full language. The deciding argument concerned tooling: the responding speaker judged the debugging and development experience around Halide and OpenAI Triton too awkward for outsiders. Python supplied the community and accumulated programming habits Mojo wanted to preserve. Becoming a Python superset was an eventual goal, explicitly expected to take time.

Internal use imposed production requirements: Mojo reportedly powered all the engine’s kernels. The discussion contrasts this with Swift’s lack of significant internal users before launch. Offering Mojo independently then justified usability work beyond what its own developers would tolerate. The lexer and parser had begun in October, less than a year before the interview; that dates the language frontend, not the compiler technology beneath it.

Longer-term ambitions included moving much of Modular’s C++ stack into Mojo and supporting safe, parallel disk and network operations for data loading. Swift’s Actors—a memory-safe parallel programming abstraction—offered design lessons for that work, rather than an already implemented Mojo feature.

Tools and libraries determine how a language feels

Swift Playgrounds makes the tooling argument concrete. Apple’s playgrounds could record state and let users scrub through an animation such as a bouncing ball. Teaching children to code on iPad led to gestures for moving blocks, completion using identifiers in scope and spinners for entering numbers. The responding speaker also credits the project with the iPad keyboard’s swipe-down access to alternate characters.

Bret Victor’s interface research inspired the work; the challenge was making those ideas usable outside concept demonstrations. Mojo had no comparable visual experience at recording time. Its team welcomed community work on interfaces but did not plan to build a UI library.

The language-design argument returns to Objective-C’s combination of dynamic objects and C. High-level libraries could manipulate methods and class hierarchies, while performance-sensitive code could drop into C—useful on the constrained first iPhone. Swift sought to span those levels in one language while preserving framework compatibility. Python’s scientific ecosystem has a similar split: Python interfaces over C/C++ execution layers. Mojo aims to join those levels consistently, retaining dynamic typing’s flexibility while allowing static typing where performance requires it.

Rust supplies ideas for memory safety and performance without a garbage collector. Mojo does have a borrow checker; the response explicitly corrects a host’s suggestion otherwise. The intended distinction is how much low-level machinery library users must confront. The speaker attributes part of Rust’s learning difficulty to libraries that routinely expose borrowing and references, and wants Mojo to offer higher-level defaults using related techniques.

Expressiveness matters more than operator count

tinygrad brings the discussion back to scope. A compact compiler focused on matrix multiplications can capture a useful subset of AI elegantly. The critique is that production integration, quantization and computation outside that subset introduce additional demands. The accompanying negative performance assessment is qualitative, without benchmarks.

Operator counts can obscure that distinction. Grouping many elementwise operations under one operation with an enum would shrink XLA’s apparent count without necessarily simplifying its implementation. The more useful question is how much of the application a system can express.

The discussion credits tinygrad with showing that a framework need not reproduce all of PyTorch. It interprets George’s approach as learning through building and rebuilding, while acknowledging that the speaker had not discussed that interpretation with him since tinygrad. Modular’s stated priority is improving existing workflows: most users it talks to do not want to rewrite their code.

Fusion can preserve precision as well as bandwidth

The quantization discussion separates numerical formats from their implementation. Quantization uses finite-precision representations; even float32 has finite precision. The objection concerns hard-coded kernels that make new combinations difficult. The speaker supports experimentation with low-bit formats while criticizing GGML’s kernel libraries for requiring substantial rework to support novel models.

Internal arithmetic can retain more precision than stored values. The examples use float16 inputs producing a float32 intermediate, and activations that must be requantized to eight bits when written to memory. When fused operations keep intermediates in higher-precision CPU or GPU registers, they can avoid both memory transfers and intervening rounding steps.

Fusion therefore affects numerical behavior as well as speed, provided higher internal precision is preserved across operations. The speaker extends this argument to large quantized attention blocks, claiming better performance and accuracy without providing numerical results. The appropriate bit width remains application-specific. The tooling goal is to let developers explore formats such as four-bit and two-bit without implementing the kernels themselves.

Keep custom computation inside the system

Reinforcement learning exposes how far an AI application extends beyond neural-network operators. Inference becomes one activity of an agent, training drives a larger process, and orchestration creates a distributed application. Moving custom work into external Python or deployment C++ can take it outside the declarative graph and beyond the reach of graph transformations.

A system cannot transform computation it cannot represent. Mojo’s engine integration is intended to preserve programmability within the stack, including: - Custom tokenization. - Compressed on-disk formats. - Unusual numerical representations, such as three-bit values.

Serving makes the cost of that boundary tangible. Using SageMaker as an example, the speaker describes preprocessing pushed into a neighboring microservice, adding data movement and network traffic. The architectural criticism is that limited programmability inside the serving layer exports complexity into the application around it.

Flexibility also leaves room for architectures beyond transformers. Hyena appears as one research direction; large recommender models offer another reminder that AI includes workloads with enormous embeddings and dense matrix multiplications. Even transformers continue to change. The position is deliberately open about which architecture will prevail: infrastructure should allow experimentation without trapping researchers in systems optimized too narrowly for attention.

Mojo’s place in the Python family

At recording time, Mojo’s Linux download support had been announced the previous day. Additional platforms were planned, traits were tentatively targeted for year-end to improve library design, and open-sourcing the language remained ahead. Modular wanted a permissive license; contribution rules and licensing details were still being worked out.

Mojo was positioned as a member of the Python family, with both languages evolving and exchanging ideas. Its intended contribution was systems programming capability, including stack representations supported by its compiler. General Python language proposals were directed back to the Python community. Guido reportedly knew about Mojo before launch, spent time with the team and occasionally joined its Discord.

Python’s effort to remove its global interpreter lock raised a related compatibility problem around the C and C++ libraries wrapped by Python. The responding speaker, explicitly uninvolved in the effort, expected years of work on those interfaces but regarded the community’s approach as thoughtful. A Python ecosystem better able to support concurrency would also benefit Mojo’s use of threads.

Make infrastructure the product

Modular’s argument for independence concerns incentives. Infrastructure inside a large technology company serves that company’s hardware, cloud or advertising priorities. The response rejects a host’s suggestion that engineers at those companies do not care about outside developers: individual commitment and organizational incentives can point in different directions.

Competition can complicate support. The speaker reports AI companies unwilling to share models with Google or Meta for debugging. Modular describes neutrality through its business boundaries: it was not building hardware, cloud infrastructure or an LLM to compete with customers. Its engine was delivered roughly as a Docker container for customers’ clouds, on-premises machines or laptops. Hosting was a future convenience rather than a prerequisite for adoption.

The founding account traces the problem through applied autonomy at Tesla, TPU and TensorFlow work at Google, and a move to a hardware startup in 2020 to learn more about commercial strategy and company building. By mid-2021, discussions with Tim returned to a persistent question: why was deployment still difficult after so much progress in making training accessible?

The diagnosis divided the ecosystem into hardware-specific stacks, frameworks with inherited architectures, and operational layers that simplified the interfaces above them. Easier wrappers could not by themselves repair limitations in programmability, performance, hardware support and deployment. Modular chose to build a common foundation those systems could plug into.

Chris initially built the technology and engineering team while Tim developed the product and business side, interviewing roughly 50–100 companies. Those conversations sharpened the goal of fewer components that work better together. A product that adds models and then a mobile offering can otherwise accumulate another specialized system for each requirement. Recruiting the specialists needed to rebuild the foundation shaped Modular’s financing and compensation strategy.

Build the team that can build the product

The leadership advice begins with defining what winning means. Strong contributors can reinforce or cancel one another’s efforts, like vectors pointing in different directions. Personally building an initial foundation also establishes engineering culture: fast iteration, deliberate testing and restraint around dependencies. Slow automated checks can impede everyone’s work.

Capability is only part of the match. Leaders should keep connecting people with problems they care about, give them ownership of decisions and assess contribution beyond lines of code. Customer understanding, product development and team building reinforce one another. The cofounder partnership provided a sounding board and complementary strengths that the speaker had missed in earlier engineering leadership roles.

At interview time, Modular’s hiring priorities reflected its applied focus: kernels, GPU performance, getting models into production, customer engagements and go-to-market work. It was not recruiting researchers.

Where learned and designed software meet

The closing questions return to usability. The responding speaker views ChatGPT’s breakout as primarily an interface achievement: it made AI’s capabilities tangible to the public, while reinforcement learning from human feedback also contributed. In that assessment, GPT-4’s later release drew less attention outside technical circles than the original experience of conversing with ChatGPT.

The unresolved technical question is how trained algorithms and deliberately designed algorithms fit together. A convolutional neural network is offered for detecting cats; conventional loops for a bootloader or operating system. The difficult part is helping developers reason consistently across both approaches throughout a cloud service, data pipeline or complete product. Learned computation is expected to become a normal part of the software toolkit, with that integration problem still open.

The final exchange takes a longer view. After a host remarks on the rapid escalation of public AI-risk discussion following ChatGPT, the response recalls dismissing safety work in 2017 and now regards those researchers as ahead of their time. Geoff Hinton and Yann LeCun are praised for the perspective gained by working on AI long before its broad popularity: experience across a longer horizon can inform what comes next.