Codeium: Making AI Cheap to Run and Easy to Correct

An expensive GPU with too little to do

A team puts its trained model in a container on a GPU. The model runs, but much of the accelerator sits idle. Varun Mohan describes a recurring mismatch: CPU deployments commonly share a node across many containers, while GPU deployments often assign one container to an accelerator without using its full capacity.

Managing large-scale offline deep learning infrastructure at Nuro led Mohan to build Exafunction, which decoupled workloads from GPUs to improve utilization and simplify serving. In this March 2023 conversation, that work becomes the foundation for [Codeium](https://www.codeium.com/): a code assistant designed to deliver useful suggestions at low latency, potentially on every keystroke.

Exafunction addressed two sources of waste. Dynamic multiplexing let multiple clients share a GPU. Operator fusion combined operations so their joint execution could take less time than running them separately. Alessio cites a customer serving 30 clients on one GPU with a reported 97% cost reduction; the workload, hardware baseline and measurement method were unspecified.

Even sophisticated teams primarily want their model to solve a business problem, Mohan argues. Choosing hardware and optimizing its use adds another specialization. His recommendation is to reuse established models—fine-tuning BERT or ResNet, for example—unless the application demands architectural invention. Autonomous driving is his exception: strict response-time and accuracy requirements can justify custom architectures.

Inference on every keystroke

As Exafunction optimized popular architectures, transformers became increasingly important. Companies building with large language models approached it for help reducing high GPT-3 bills. That experience prepared the team to serve its own models. Mohan frames Codeium’s goal as lasting enterprise value: applications whose usefulness was constrained by serving costs or limited control over training.

Code completion makes those constraints immediate. A batch application can ask someone to wait minutes; an assistant embedded in typing must respond quickly enough to help with the work underway. Mohan argues that imperfect output can still be valuable at sufficiently low latency.

The team learned this through GitHub Copilot. Initially skeptical that it could help with their CUDA and C++ work, they watched it generate difficult variadic-template code in a roughly 500-line file. The code compiled—an encouraging result, though not a demonstration of runtime correctness. The assistant handled rules that were mentally demanding to recall even when the underlying logic was manageable.

Useful suggestions could also change how developers worked. One participant describes documenting functions more extensively to give the model better prompts. Greater usefulness encouraged more deliberate prompting, making generated code a larger part of the workflow.

Making free completion viable

The team collected permissively licensed code from the public internet, describing Eleuther’s The Pile as a relatively small subset. Open-source models helped bootstrap internal development. Requirements including fill-in-the-middle completion, or filling a gap within existing code, led to shipping a model of their own. Development took multiple months and drew on infrastructure and training experience already in place.

Autoregressive generation produces tokens successively, so longer completions take longer to generate. Batching requests helps use the GPU’s compute capacity, but the serving system must batch without sacrificing responsiveness. The team considered then-available open-source serving frameworks insufficiently specialized for this transformer workload; its existing infrastructure was a substantial advantage.

Free completion emerged from those economics and a broader product strategy. Completion would attract a loyal user base for later products, with a pro tier and enterprise sales envisioned. Giving everything away was not the original plan; aggressive optimization made free completion seem feasible. At the time of the conversation, the team reported probably more than 10,000 users and thousands of daily active users.

The advice to other founders was to start with an existing API and work on the quality–latency trade-off, unless substantial specialized training was necessary. Codeium’s decision to train its own model rested on existing expertise and confidence that the product was feasible. Speed alone would not persuade users to switch if the completions were poor.

Correctability determines how much to generate

A useful assistant needs a third property alongside quality and latency: correctability, the ease of salvaging useful parts of an incorrect generation. A local completion is relatively easy to inspect and edit. A generated pull request spanning many files creates a larger review and repair burden.

In the conversation’s hypothetical ten-file change, repairing two files can leave eight others inconsistent with those repairs. The user must then reconcile the whole change. The proposed design response is to deliver smaller units of useful work that preserve trust. Entire pull requests remain a future possibility, potentially requiring models trained on diffs, commits and code improvements rather than generic code alone.

Completion length also depends on the developer’s intent. Drawing on an unnamed Copilot study, the discussion distinguishes two modes: - Exploration: the developer wants enough generated code to discover an unfamiliar API or possible implementation. - Acceleration: the developer knows the intended solution and wants help executing it without recalling every API detail. These are design heuristics, not a measured optimum for completion length.

Codeium reportedly inferred those modes from editing context. Working inside an if block suggests acceleration: the developer is carrying out a plan. An unsolicited else branch can pull attention into another decision before the current branch is finished. Multiline generation remains useful, but cognitive load helps determine where to stop.

Legal assistance illustrates how these requirements change by domain. In discussing Harvey.ai, participants emphasize both comprehensive coverage and grounded conclusions: citations do not help if relevant material is missing or the conclusion does not follow from the cited documents. Codeium’s stated strategy was to remain focused on code, because learning another domain’s users and requirements would demand substantial work.

Choose a model you can serve

Chinchilla, discussed through Training Compute-Optimal Large Language Models, asks how to balance model size and training tokens under a fixed compute budget. The questioner introduces a rough rule of 20 training tokens per parameter. A larger model can learn more from the same token count, but processing those tokens requires more computation.

A product adds deployment constraints. The conversation uses a 70-billion-parameter model on an 80 GB A100 to illustrate memory pressure, with lower-precision representations mentioned as a way to reduce storage requirements. Hardware fit and latency need consideration before training, because the model that best uses a training budget may be difficult to serve.

That can favor training a smaller model on more tokens. CodeGen’s approximately 2.7-billion-parameter model is offered as an example: it was recalled as having trained on hundreds of billions of tokens, including natural language and code, though the exact count was uncertain. Rounding its size to three billion parameters, the 20-to-one rule would suggest roughly 60 billion tokens. Spending more on the smaller model can make sense when a larger alternative would be harder to deploy.

The account of internal model development follows that logic: more parameters might improve quality while worsening latency or hardware fit. Training loss offers no obvious stopping point either. The speaker argues that improvements can continue at increasingly expensive margins, and that small changes in perplexity, a measure of next-token prediction performance, can accompany substantial changes in reasoning behavior.

A suggested capability transition around 20 billion parameters remains speculative, with no general threshold or explanation established. Better data offers another route to stronger small models: removing duplicates, near-duplicates and contamination is discussed as promising, without numerical results.

Give the model access to more information

Longer context—the information supplied to the model for a generation—creates another compute constraint. The transformer attention computation under discussion grows quadratically with context length. The speaker predicts architectural changes to reduce that cost, and expects cheaper context to encourage users to supply more of it. For largely stateless models, context is a principal way to provide information beyond fine-tuning.

DeepMind’s RETRO illustrates dynamic retrieval. In the explanation given, BERT embeddings support a search for nearby documents in a large database; the generating model then incorporates the retrieved material through attention. The retrieval encoder is frozen rather than trained jointly with the generator, limiting the sense in which the model chooses its own context. The database is described at terabyte scale; the trillion-token corpus sizes used to explain it are hypothetical.

Two possible directions follow: make long context cheaper to process, or let the model request particular information when needed. The latter suggests a loop in which the model requests database information, incorporates the result and tracks what remains missing through a state machine. Bing Chat is raised as a possible comparison, not a confirmed implementation of that design.

Retrieval brings the argument back to economics. A hypothetical ten-terabyte embedding database motivates concern about then-current per-gigabyte pricing. The expectation was to build internally first: application teams may encounter requirements before suppliers feel the same pressure. Outsourcing becomes attractive when a service meets the concrete workload and its scaling needs.

Evaluate the code, then the product

[HumanEval](https://github.com/openai/human-eval) evaluates a complete function generated from a docstring by running unit tests. This tests behavior beyond next-token prediction. Codeium reportedly built its own evaluation infrastructure to cover more languages and a wider variety of codebases.

Public benchmarks risk training contamination: a corpus collected from GitHub can include the benchmark itself, mixing training and validation material. Even a small benchmark can be memorized. The speaker suspects contamination could explain some striking online results, without identifying a confirmed case. Data cleaning and evaluation also require domain-specific judgments; high-quality code and high-quality legal documents have different criteria.

Evaluation is presented as a core competency for Codeium, but offline tests cannot capture the whole product, including work performed before prompt generation. The team therefore used A/B testing and an internal rollout system that gradually exposed changes to roughly 1%, 10% and 50% of users. Completion acceptance supplied feedback on the combined system’s usefulness.

Acceptance is only an immediate signal. Users can later change their minds, so Codeium also examined longer-term behavior, including whether they accepted suggestions or wrote roughly similar code. The speaker estimates that reliable signals can require hundreds of thousands or millions of completions. An anecdote about a Quora experiment that looked positive before deteriorating a week later illustrates the danger of early conclusions; those completion counts are not a formal sample-size calculation.

Make the cluster feel like one GPU

A strong base model trained on varied text remains valuable, while high-quality data specific to the workload is argued to improve both results and predictability. Training at scale also requires devices to communicate efficiently.

For machines containing eight H100 GPUs, the speaker cites inter-node bandwidth of 3,200 gigabits per second, or 400 gigabytes per second. Within a machine, NVLink is described as enabling lower-latency communication than PCIe. Fast communication supports synchronous distributed training, where devices coordinate updates. In the asynchronous alternative described, nodes update local model states and exchange updates periodically. The speaker is unsure why synchronous training became dominant.

Memory requirements also force distribution. A rough GPT-3 example places optimizer state—the state maintained during training—on the order of terabytes, requiring it to be split across communicating devices. The estimate is illustrative; its storage assumptions and device-versus-machine count are ambiguous. The desired programming abstraction is one enormous GPU, with communication overhead hidden and abundant compute and memory bandwidth.

Keep building with what already works

Midjourney earns praise in the closing discussion for improving image quality and producing predictably good results within a distinctive style. Eleuther supplies another source of optimism: GPT-J is described as comparable to GPT-3 Curie specifically in perplexity, and as having been trained by a university student. Its development through Discord collaboration and access to a research grant illustrates the potential of motivated people with practical computer knowledge.

Looking ahead from 2023, the prediction is that models will improve on specialized tasks while developers discover surprising uses for existing capabilities. ChatGPT is credited with stimulating that creativity, and the rapid appearance of legal applications is offered as an unexpected example. The encouragement is to build with models already capable of useful work.

The final product wish is a fitness assistant that recommends daily exercise and a regimen for longer-term goals. One participant reports a knee injury after cycling roughly 150 miles a week and wants guidance on training volume. The communities around Strava and Zwift suggest a possible audience, though willingness to pay remains a hypothesis.

The closing advice is to keep tinkering. A speaker recalls the frustration, six to eight months earlier, of watching others experiment while the company focused entirely on infrastructure—and anticipates substantial new applications over the following year or two.