MPT-7B: Long Context and Training Without Babysitting

Proving a model could finish without a babysitter

Training a seven-billion-parameter language model on a trillion tokens meant keeping a long computation running without constant intervention. MPT-7B had three linked goals: train efficiently, finish without babysitting, and provide a base for further fine-tuning. Jonathan reported the result as 9.5 days and $200,000, with no human intervention. Completing the run unattended was part of the achievement.

In the May 2023 conversation, Jonathan explained why MosaicML had built MPT-7B: customers needed tools to create their own models, and Mosaic needed to demonstrate that those tools worked. Abhi traced the effort to language-model profiling that suggested training costs were lower than commonly assumed, followed by inspiration from LLaMA and other open releases. A separate $100,000 GPT-3 training figure remained a target; Jonathan explicitly declined to call it an achieved result.

Abhi characterized the base model as, for practical purposes, a recreation of LLaMA-7B. Chat, Instruct, and StoryWriter illustrated directions for customization. Users might build a long-context code model or specialize in a particular language; these were proposed extensions, not additional completed releases.

Train longer so the model can stay small

Chinchilla-style scaling laws guide the balance between model size and training data for an efficient training-compute budget. Mosaic also cared about inference—running the trained model—and wanted people to use it at home. That favored spending more training on a relatively small model.

Abhi’s approximate comparison was 140 billion tokens at the Chinchilla point versus one trillion tokens actually used, roughly seven times as much data exposure. The internal nickname llong boy referred to this extended training, not the context window; the doubled L nodded to LLaMA. Users’ attempts to run 7B models on home CPUs and port them to phones illustrated the adoption goal, though they were not deployment benchmarks.

A trillion tokens is not a data recipe

Choosing the training duration left a harder question: what should the model read? Consider the hypothetical raised in the discussion: ten billion excellent tokens repeated 100 times, or a trillion fresh, lower-quality tokens. Both provide a trillion token exposures, but make different bets on repetition, coverage, and quality. The team had neither a winning recipe nor a satisfactory definition of quality that settled the choice.

A participant gave Georgetown Law students the actual datasets, mixture information, and token counts and asked them to design a training mix. With little language-model knowledge, they still identified the central trade-offs: English coverage, multilingual material, and code. The discussion also challenged a common assumption about code: a strong code model does not by itself demonstrate that adding code improves general logical reasoning.

Experiments complicated intuition. The Pile was a solid baseline, but other mixtures scored better on the team’s evaluations. C4, described as a single 2019 web scrape, performed particularly well after reported deduplication against the evaluation set. English mC4 supposedly applied the same preprocessing to more crawls, yet performed substantially worse. The team could not explain the difference and gave no score differences or exact evaluation suite.

Strong scores did not vindicate every cleaning decision. A speaker criticized C4’s removal of material containing JavaScript, curly braces, or blocklisted words—including “gay.” Mosaic’s eventual rule was pragmatic: meet or exceed English mC4’s performance, then prioritize diversity, including code, scientific papers, and Wikipedia for different downstream uses.

The limiting factor was evaluation mismatch. Multiple-choice tests can score an answer letter or select whichever candidate the model considers most likely. Users often want open-ended generation instead. A participant similarly criticized the text-comparison scores BLEU and ROUGE because many different summaries can be good. Human judgment helped, but was too expensive to insert into the evaluation loop every ten minutes.

At 7B scale, some of the hardest tasks in the MMLU benchmark produced scores barely above chance. Development therefore leaned on a small set of commonsense-reasoning tasks and comparisons with other open models. Even apparent leaps in capability required care: a participant suggested that binary scoring or plot scaling could make gradual improvement appear abrupt, without claiming that this explained all emergence.

Long context needed speed, position handling, and stability

[FlashAttention](https://arxiv.org/abs/2205.14135) provided a faster, mathematically equivalent implementation of full attention. Mosaic made the implementation switchable: standard PyTorch attention for broad compatibility, or FlashAttention for GPU execution. A participant reported nearly twice the training speed and 50–100% faster inference, without specifying the hardware or workloads behind those figures.

[ALiBi](https://arxiv.org/abs/2108.12409) changed how the model represented token positions. Instead of adding a position-specific embedding to each token, it added a sloped bias to the attention map. Extending that bias across more positions allowed inference beyond the positional range used in training.

The techniques met at a third requirement: training stability. FlashAttention’s memory savings and performance made profiling contexts around 65,000 tokens feasible, but training at those lengths remained difficult. Mosaic reported that ALiBi dramatically improved stability, enabling [MPT-7B-StoryWriter-65k+](https://huggingface.co/mosaicml/mpt-7b-storywriter) to train on very long sequences.

StoryWriter’s context window—the sequence available to the model—had a deliberately loose upper bound. A speaker described it as technically unbounded given enough memory and said 84,000 tokens was an advertised length users were expected to reach easily. Extending the computation did not establish reliable use of distant information at arbitrary lengths.

The demonstration supplied The Great Gatsby and requested an epilogue. One internal continuation introduced Gatsby’s ghost; others impressed the team with Fitzgerald-like prose. A participant regarded these as qualitative signs that the model used the lengthy input. StoryWriter had been fine-tuned on books to encourage story writing, but the speaker did not know whether special creative fine-tuning techniques had been used beyond that choice.

The demonstration met a social dispute

StoryWriter’s release coincided with the Hollywood writers’ strike, where AI-generated creative work was contentious. Following controversy around Books3, Mosaic changed the model’s license for a couple of hours, then retained Apache 2.0. The change was a brief reconsideration, not a lasting withdrawal of commercial permission.

A participant reflected that concentrating on getting the model to train had obscured its societal context. In hindsight, a coding demonstration might have conveyed the long-sequence capability differently. Keeping the release open reflected a position about community access and licensing; it did not resolve copyright, fair use, or the boundary between memorization and learning. The advice for commercial users was to consult their own lawyers.

What unattended training required

The unattended run depended on both numerical stability and infrastructure recovery. Longer training made loss spikes harder to avoid, prompting changes to initialization, optimizers, and architecture. MPT-7B still had small intermittent spikes, but they reportedly recovered within a few hundred steps. A backup strategy could resume deterministically a few batches before a catastrophic event and apply interventions; it was prepared but never needed for this run.

Hardware failures needed separate machinery. A participant estimated roughly one infrastructure failure per thousand A100 GPU-days—about one every two days on a 512-GPU cluster. Networking failures and GPUs falling off the bus were examples. Mosaic’s watchdog automated the response: 1. Stop the failed job. 2. Test the nodes and isolate broken ones. 3. Relaunch on working hardware. 4. Resume quickly using deterministic software. The team described an overnight failure recovering within minutes while everyone slept. This followed months of manual incident response and incremental automation.

Why stop the whole computation? In synchronous data parallelism, every GPU works on each step, stays synchronized, and participates in averaging gradients. In this setup, a failed node interrupted the coupled computation and required recovery from a saved checkpoint. Dropping the failed GPU and its assigned data was an alternative, but Mosaic preferred reproducible runs with automated restarts. More extreme scale might change that trade-off.

Reuse the recipe and evaluate before scaling

Starting from MPT could mean using its trained checkpoint or reusing its training configuration. A speaker expected relatively few customers to choose the checkpoint itself. Replit illustrated the second path: while MPT was in development, it requested a three-billion-parameter model trained on its own data by the following week, and Mosaic shared its approach. Composer remained central to the stack, while [LLM Foundry](https://github.com/mosaicml/llm-foundry) supplied model configuration alongside the platform’s orchestration and fault tolerance.

Before a large run, the team recommended three inputs: - Evaluation metrics beyond training loss. - Pretraining or fine-tuning data, with a cleaning strategy. - A budget. Smaller training runs could help anticipate model quality before committing a few hundred thousand dollars. Mosaic cited its work with Replit and a Stanford group on biomedical language modeling as examples of this customer role.

The team also watched answers to a fixed set of prompts during training. One asked for games a three-year-old and a seven-year-old could enjoy together. For the participant describing it, changes in that answer were more informative than benchmark scores. These qualitative checks supplemented automated evaluation; HumanEval, despite its name, was clarified as an automated code evaluation.

LLM Foundry’s evaluation framework ran with Composer and supported multiple GPUs and large models. Mosaic reported tracking roughly 15–20 metrics during training in dashboards such as Weights & Biases, with overhead it described as negligible but did not quantify. The purpose was to assess progress before the final day of a run.

Cheaper experiments also made more science possible. One participant recalled starting lottery-ticket experiments on a laptop and eventually deleting more than a million PhD-era model checkpoints. Understanding behavior, reliability, and safety required repeated experiments. Mosaic argued that findings about data and architectures could benefit many customers, justifying shared research. PubMedGPT, a biomedical model trained from scratch, was one experiment the team considered successful, without presenting quantitative results here.

Custom models need continued training and serving

Mosaic’s business joined custom models, infrastructure, and integration—the idea a participant said inspired its name. Its roughly 15 researchers committed to sharing findings while acknowledging that they could not match historically larger labs’ research volume. A participant argued that exchanging ideas counteracted the intellectual monocultures that develop inside individual labs.

Open and closed systems could coexist. Some customers reportedly used both Mosaic and OpenAI, and a speaker explicitly declined to promise GPT-4-quality models, calling that beyond Mosaic’s means and intended scope. Mosaic offered specialization, training on proprietary data, customization, and control over where data went.

Customers also wanted to serve models within the same ecosystem. Mosaic had initially focused on efficient training, considering inference relatively well served; changing model requirements created an opportunity despite the risk of splitting its focus. In one customer anecdote, training finished in three or four days of a ten-day window, leaving time for fine-tuning. Making that workflow useful also required good inference.

The team rejected training as a one-time expense. MPT still had unfinished improvements, training data changed, and customers would want successive models with higher quality, smaller size, lower latency, and lower cost. Training belonged to an ongoing development cycle.

Efficiency claims had to survive implementation

Looking ahead from 2023, Mosaic expected H100 hardware to improve training efficiency over A100s and saw FP8, an eight-bit floating-point format, as another opportunity. Lower numerical precision could increase matrix-multiplication throughput. The team had begun profiling FP8 language-model training on H100s and reported promising improvements, without giving a measured speedup.

Mosaic was also exploring architectural changes, including gating and mixture-of-experts approaches. Together with hardware improvements, these supported its ambition to reduce a roughly $500,000 GPT-quality training run to $100,000 by the end of 2023. That remained a target. Newer architectures offered more unoptimized work: a participant described a roughly twelvefold Stable Diffusion cost reduction as easier than earlier ResNet-50 or BERT improvements. The twelvefold figure came from costs supplied in a host’s question, without matched training conditions.

For long context, Mosaic retained exact quadratic attention, accepting its growth in computation rather than introducing an approximation. FlashAttention accelerated that same attention calculation. Approximate, hierarchical, and other nonquadratic methods remained worth investigating, but a participant emphasized reimplementing promising research before relying on its benefits: practical results could weaken or reveal consequential caveats. The speaker later declined to name papers and explicitly rejected attributing those shortcomings to malice.

Nor was 64K context an obvious endpoint. More ambitious applications, including images and potentially video, could keep increasing demand. Architectural predictions were contested: one participant defended transformers’ longevity by pointing to the persistence of convolutional and recurrent networks; another asked what would still count as attention if its query–key dot product changed. A proposal that models might simplify toward multilayer perceptrons met the counterargument that they could become more complicated. The practical test was whether an idea lasted and enabled others to build on it.

Public research still had a commercial boundary. Open models demonstrated capabilities without implying that every configuration or technique would be published; some work remained customer-only. Free model releases could generate revenue through a per-token inference API, customization, and deployment on Mosaic’s platform.

Zeros save money only when hardware skips work

Sparsity means setting model weights to zero. A model can retain useful behavior after many weights are zeroed without becoming faster to execute. Unstructured zeros do not automatically reduce runtime or cost on the available hardware. That gap explained why sparsity was not a Mosaic priority.

A participant presented Cerebras as a promising exception, citing hardware designed for sparsity and research demonstrating sparse training, but giving no quantitative speedup or completed quality comparison. Standard hardware could accelerate some structured sparsity, yet another speaker argued that imposing structure harmed quality. Removing whole neurons illustrated the extreme: the layers simply became smaller. An accessible hardware-and-software combination for accelerating unstructured sparsity could change Mosaic’s priorities.

The broader argument was that available hardware shapes which architectures get developed. Sparse hardware might need to co-evolve with models that exploit it. Even transformers’ fit to existing accelerators prompted disagreement: one speaker praised their pairing with GPUs, while another favored TPUs.

How efficiently can strong capabilities be obtained?

Jonathan acknowledged repeatedly underestimating scaled next-token prediction. He had dismissed GPT-2 and initially seen GPT-3 as only a modest advance; OpenAI’s unusual bet had paid off. Another participant was surprised by the speed of chatbot adoption and by the emotional attachments described in communities discussing AI companions.

The remaining questions returned to efficiency. How far could numerical precision fall—to four-bit, two-bit, or binary weights, or even analog computation? Could larger models become easier to represent at lower precision or prune? These were possibilities to investigate. Jonathan’s question was broader: how efficiently can strong capabilities be obtained? His Lottery Ticket Hypothesis work had been one approach; at Mosaic, he hoped better modeling and algorithms could reveal paths beyond brute-force scale.

The closing advice was to stay balanced between hype and reflexive pessimism. One speaker urged recognition of extraordinarily useful tools while expressing the personal view that artificial general intelligence remained far away. Mosaic’s stated approach was to focus on practical usefulness and let experiments and data guide the work.

Another participant made the case for open scrutiny: more people can inspect models, identify problems, propose improvements, and examine their consequences. That community attention was presented as a way to improve safety as well as capability.