Training a 70B Model Means Debugging the Whole Stack

Training data is a product decision

Enterprise customers repeatedly ask where a model’s training data came from. For Databricks’ text-to-image collaboration with Shutterstock, Jon Frankle could give a concrete answer: the Shutterstock collection, without additional web scraping or other datasets mixed in. The model took roughly a year to develop and was released through an API. Known data provenance addressed a customer concern, while the collection’s sensitivity and commercial value constrained distribution.

The June 2024 conversation then turns to DBRX, introduced by the host as a mixture-of-experts model: only part of its parameters are active for a given input. Its headline figures were 132 billion total parameters, 36 billion active per input, and pretraining on 12 trillion tokens of text and code. Frankle recalls betting against his team reaching an evaluation target two weeks before launch. They reached it, and he dyed his hair blue.

Josh Albrecht introduces the Imbue 70B work through the risks of training from scratch. The release comprised tools and evaluation resources, not model weights, addressing three connected problems: - Reliable infrastructure: diagnose hardware faults and low-level performance problems so training can run with fewer restarts. - Meaningful evaluation: clean existing benchmarks and add code-understanding tasks. - Affordable experimentation: use cost-aware hyperparameter tuning to learn at small scales before committing to expensive runs. Frankle emphasizes the separate hurdles before useful training: making GPUs communicate, running the job, obtaining good performance, and getting the model to converge.

A running GPU can still be faulty

Infrastructure trouble ranged from InfiniBand network cables stolen twice before delivery to recalled switches and new errors after driver updates. But the harder failures could leave execution running: - A slow GPU can delay the whole job. Correcting a memory problem can turn an otherwise operational GPU into a straggler. - Incorrect arithmetic can arrive without an error. One participant recalls GPUs returning wrong sums, making the fault harder to detect than an explicit failure. The participants treat these as consequences of operating complex, hot, power-intensive hardware at multi-megawatt scale, often in retrofitted data centers—not evidence that a particular vendor did poor work.

The cluster discussed in Imbue’s infrastructure account contains roughly 4,000 GPUs. Its three-tier network adds complexity beyond the two-tier arrangement described for clusters around 1,000 GPUs: maintaining consistent communication speeds becomes harder as the system grows. Work with Voltage Park extended into component selection, networking, construction, and commissioning. A wrong order could cost months as well as money.

Commissioning establishes confidence in stages: 1. Bring individual machines online. Provision one manually, then use Metal as a Service to install operating systems on the others. 2. Establish connectivity. Get the InfiniBand network working between machines. 3. Verify system health. Confirm that connected hardware behaves correctly. 4. Investigate training behavior. Address software, data, and performance problems on that foundation. Full-speed training does not end this process. Later failures and firmware changes require repeatable diagnostics to reproduce and explain regressions.

The servers provide an out-of-band management network, a separate connection for viewing boot and error screens even when normal operation is unavailable. Alongside InfiniBand and redundant 100-gigabit-per-second Ethernet, this makes remote diagnosis and automation possible. Centralized metrics collection, including Prometheus, exposes temperatures and other signals without individual logins. Factory testing is only a starting point: shipping can loosen components, and early hardware builds can expose firmware faults.

Turn failures into repeatable health checks

Direct control shortens the path to a fix. For the team describing Imbue’s setup, contacting Dell, NVIDIA, or the responsible data center directly was easier than routing faults through intermediaries; some problems required Dell firmware fixes. Cloud providers absorb physical logistics and other infrastructure responsibilities, but the Databricks/Mosaic experience still involved investigating firmware, power contracts, and suspect cable batches.

The released cluster health checks cover network interfaces, GPUs, disk space, Docker, and boot behavior. One check grew out of machines that developed new problems after rebooting. It examines whether boot-log lines are familiar, expected, and in the expected order. An anomaly enters triage to determine whether it is acceptable or needs investigation.

Automate detection after finding the root cause. Checks can catch recurrence and route a problem to its owner. Establishing hardware health also narrows the search when training slows: configuration and software become more plausible suspects. A figure discussed along the way—3% of machines breaking weekly—came secondhand from other operators, not Imbue’s measurements. Failures recurring on troublesome nodes also mean that rate does not imply every machine fails within a year.

The release offers reusable components; operators still need their own management infrastructure and log aggregation. Likewise, the internal team’s reported three-to-six-person size excludes substantial partner labor. The installation involved 12,000 cables and 24,000 ends to connect, while repairs could require unwiring a machine, removing GPUs, replacing a cable, and reassembling it. Dell, NVIDIA, H5, and others contributed substantially.

Local pauses become distributed delays

One training run gradually slowed as garbage collection across hundreds of machines drifted out of alignment. Increasingly staggered pauses disrupted training steps. The speaker described the problem as memory fragmentation; C-level execution traces isolated the expensive calls, and disabling garbage collection resolved the slowdown. Another participant reported the same behavior and used a callback to collect garbage at a shared interval. Monitoring CPU temperature and throttling similarly helped explain individual slow machines.

Software distribution created another bottleneck. Each Docker image change initially meant roughly 500 machines downloading many gigabytes from an external registry. Uber’s Kraken, described as a BitTorrent-based distributed registry, let machines exchange pieces locally after an initial download. The team reported faster distribution and no bugs encountered in its use. Limited internet bandwidth also motivated a simple local file mirror: a small web server for uploads and downloads.

Two infrastructure approaches serve different operating needs: - Simplify a specific workflow. Bash, Python, SSH, Docker, and direct launches keep faults accessible. A custom orchestration layer supports experiments that should run to completion despite disappearing machines, upgrades, and GPU failures. - Standardize across providers. Across six or seven clouds, small differences accumulate. Commodity training components, a standardized data loader, and Kubernetes provide common interfaces. Both approaches use abstractions. The shared advice is to choose the minimum complexity the work requires, including parallelism strategies, libraries, and architecture changes.

Model architecture changes the network problem

Using fewer parameters per token can make communication relatively more expensive. In the parameter-sharding setup discussed for mixture-of-experts training, parameters move between devices while each token uses only a fraction of them. This worsens the ratio of computation to communicated data. A participant involved in DBRX calls networking the project’s hardest part. Moving from the setup called FSDP to HSDP, which used smaller replica groups, helped alleviate network latency while retaining commodity components.

Hardware and training methods can evolve together. One participant describes Google’s training hardware as having a higher network-bandwidth-to-compute ratio that favors communication-intensive workloads, explicitly qualifying this as their understanding. The accompanying discussion concerns physical torus networking and ring all-reduces, not ring attention. No hardware generation or measured comparison is given.

NVIDIA’s then-announced hybrid networking offered another arrangement: high-bandwidth NVLink connections within GPU blocks and a separate network between blocks. Where communication happens—and where a GPU fails—now matters to system design. A hypothetical illustrates the pressure at larger scales: if a cluster experiences one GPU failure daily, a cluster 24 times larger could experience one hourly. Frequent restarts become less practical, motivating redundancy.

Faster or cheaper storage can support more frequent checkpoints, but vision and video bring multi-petabyte datasets. One participant recalls data centers without enough object-storage capacity for users’ datasets, forcing storage into another region from the GPUs.

Josh explains Imbue’s text focus through its applications. Although the company had worked on vision and self-supervised learning, coding and reasoning were its priorities. Visual subtasks could sometimes use textual descriptions or off-the-shelf multimodal models, including for browser interactions. A broad engineering stack can serve a narrow application goal: work on infrastructure, pretraining, reinforcement learning, fine-tuning, and products did not require building every modality into the base model.

CARBS: learn at small scales before committing to 70B

CARBS—Cost-Aware Pareto Region Bayesian Search—tunes hyperparameters, the configuration choices used to train a model. Its distinctive move is to model each experiment’s cost rather than assume comparable resource budgets. Candidates can differ in data fraction, model size, compute, or runtime, allowing the team to study how performance and useful settings change with expense.

The search includes layer count, learning rate, regularization, and dependencies introduced by Imbue’s custom tokenizer. The aim is to avoid repeatedly training expensive 70B models to discover those relationships. An intermediate run reportedly landed on the predicted scaling curve, giving the team confidence to proceed; the speaker tentatively recalls its size as around 14B.

Small experiments need a sensitive evaluation signal. Accuracy records whether an answer is right, but can fluctuate sharply at small scales. The team instead measures answer-token loss or perplexity on held-out multiple-choice tasks: a probability-sensitive signal targeted at the tasks it cares about. A small model might keep choosing A while becoming less confident in that choice. The metric can detect learning before the selected answer changes.

Hold the evaluation distribution fixed when changing the training mix. Measuring loss on a changing mixture of code and internet text changes the measurement itself. A fixed held-out task set gives CARBS a common basis for comparing mixtures, making evaluation design part of the scaling method.

Changing that mixture during training produced different experiences: - Small gains in the CARBS team’s settings. It hoped to teach basic features before emphasizing harder math or code. Gains were tiny, and many disappeared after tuning other parameters. - Substantial gains in some domain-specific settings. Another participant described the approach as early fine-tuning, with benefits dependent on the model’s intended use and training details. A complicated schedule therefore needs comparison against a well-tuned simpler baseline on the relevant metric. Different datasets, schedules, and configurations can produce compatible but different results.

This measurement choice also shapes the discussion of emergence. The CARBS explanation adopts the interpretation that some apparently abrupt capabilities reflect discrete metrics hiding gradual improvement. Another participant cautions that logarithmic plots can make trends look unusually clean or flat. Interpreting a scaling curve requires attention to both the metric and its plotted scale.

Inspect the questions before trusting the scores

Imbue’s evaluation work began by reading examples. Some candidate datasets seemed incoherent; even selected ones contained ambiguous questions and incorrect labels. Two reviewers could accept a question before a third exposed a different interpretation of one word. The cleaned evaluation resources covered 11 existing benchmarks, including BoolQ and ANLI, with approximately 450,000 human judgments about ambiguity and question quality.

To address training–test contamination—evaluating on examples already used for training—the team also created roughly 500–1,000 new examples per selected dataset from descriptions and examples. It reported that removing ambiguity often exposed benchmark saturation: models and reviewers agreed on clear questions, while remaining score differences could reflect guesses at confusing ones. Apparent leaderboard headroom was not necessarily meaningful room for improvement.

The new code-understanding benchmark asks concrete execution questions, such as what value a variable will have. Programmatic generation supplies fresh examples, which the team argues reduces contamination concerns. Its low-level focus also serves CARBS: small models need to make measurable progress on the tuning tasks. Other internal evaluations remained private because of redistribution constraints or their value as business assets.

Realistic coding tasks are harder to specify and judge. Architectural quality invites disagreement, and function-implementation prompts need clear requirements and type signatures. The team reviewed every example in an internal revision of the MBPP coding dataset, feasible because it contained only a few hundred examples. It did not perform equivalent exhaustive review for HellaSwag, and release of the MBPP revision remained uncertain.

A participant recalls published agent traces that passed benchmark tests but, in their assessment, failed to solve the intended problem. One used ASCII encoding while dropping edge cases that mattered in production. Passing tests is narrower than solving the software problem; a single reference patch also cannot represent every valid repair. More realistic evaluations make complete specifications and test coverage harder to achieve.

Measure usefulness without waiting for perfect evaluations

Define how to measure the customer’s goal before committing training resources. The participant giving this advice acknowledges difficulty following it internally. Imperfect benchmarks still provide useful optimization targets: neither abandoning them for loss alone nor becoming paralyzed by their flaws helps. One participant’s hiring criterion captures the needed temperament—tolerate unresolved problems while maintaining enough discipline to keep disorder from taking over.

Ambiguity itself becomes a capability to evaluate. A useful coding agent should recognize an underspecified request, assess whether it completed the task, and communicate uncertainty about its code. Reliable uncertainty signals are part of useful performance, rather than merely commentary accompanying an answer.

The discussion distinguishes three purposes for evaluation: - Base-model competence: direct answers, without step-by-step chain-of-thought reasoning, check whether basic capabilities work. - Agent reasoning: abstract tasks can help study problem-solving inside a larger agent loop. - Everyday usefulness: excellence at abstract puzzles or difficult coding competitions does not establish progress on routine programming. A participant offers IQ-style tests and SATs as a counterpoint: proxy tasks have long been used to predict broader capability. The question is whether a particular proxy serves the intended application.

Long context needs a task and a comparison

Jonathan places retrieval at the boundary between a system and an agent: when the model decides when to retrieve context or query a database, retrieval becomes agentic. Long context can supply many demonstrations as an alternative to fine-tuning and accommodate multimodal inputs. But human evaluation may require reading tens or hundreds of thousands of tokens, making annotation expensive and encouraging tasks whose answers are known by construction.

Needle-in-a-haystack tests fragment retrieval, not holistic context use. Jonathan proposes a more useful comparison: measure task performance with 10, 100, and 1,000 demonstrations in context, then compare with fine-tuning on the same data. Long context would compete against another way to solve the task, providing an automatically assessable baseline. This was a proposed evaluation, not a reported experiment.

Josh adds that putting an entire repository into context is a convenient starting point, but larger repositories still require selection. Filter and rank relevant code even when the context window is large. This can reduce confusion, inference time, and computation spent on material that does not help the task.

Code expands actions; applications set boundaries

Josh sees robust code generation as a broader action space than a fixed catalog of agent tools. Function calls become ordinary code invocations; the hard work is making code run correctly, debugging it, and assessing whether it is safe to execute. He expects stronger coding systems eventually to create interfaces for software that lacks APIs, presenting this as a motivation for Imbue’s focus.

Jonathan approaches tools through structured data: keep useful structure and let the model use its interfaces. Tables can remain tables, accessed through SQL or established APIs. He describes model-generated SQL as particularly useful for customers. He also questions flattening code into token streams when parsing and reference disambiguation already expose program structure, while acknowledging that he has no better solution for code itself.

Josh’s tool choices follow the task: - SQL for aggregation: query an order table to answer a sales question. - Knowledge graphs for relationships: use them when their definitions of entities and relationships map well to the problem. Ambiguous real-world boundaries limit their universality. - Code for combining approaches: select different libraries and abstractions as the task demands.

Choosing an abstraction is separate from granting access. Jonathan argues that some applications should permit SQL calls only. Web access and additional tools should depend on whether they improve solutions to the actual problem. Code’s expressive power does not determine which capabilities a product should expose.

Useful software—and work still worth doing

At the close, Josh describes Imbue’s goal as accelerating daily work through code generation, understanding, testing, and verification. He hopes internal prototypes will become useful enough to share over the coming months or quarters, without committing to a release date.

Jonathan describes customer work as the main priority, much of it outside public view. Exhausting model-development pushes leave scientific findings to explain afterward; the team was preparing more writeups, with open-source model releases emerging when there was something compelling to share and time and GPUs available. He reveals Abra as the internal name of an unreleased smaller DBRX model and Cadabra as DBRX’s name, then hints at a third through the Pokémon evolution sequence and Chatbot Arena without confirming a name, capability, or release date.

A closing hiring invitation seeks people interested in coding, reasoning, hardware, and model behavior, with everyday usefulness as the goal. The final encouragement is that familiar problems still reward fresh examination. Despite the field’s crowding and resource demands, cluster setup and evaluation datasets offer consequential work: inspect what actually fails, understand why, and share what you learn.