From Llama’s training budget to the compute an agent needs

An assistant that could edit the paper

Swapping two columns in a LaTeX table can take a scientist far longer than the request suggests. Galactica Instruct was meant to handle that work inside Overleaf, alongside finding papers and inserting bibliography entries. Thomas Scialom recalls successfully testing the integration. It never opened publicly: backlash against the original Galactica release stopped it.

That unfinished assistant introduces a problem running through this July 2024 conversation: turning a capable language model into a useful assistant. Scialom describes the original Galactica as a science model released without instruction tuning or reinforcement learning from human feedback (RLHF). Reactions were polarized, particularly around fabricated papers. ChatGPT’s arrival soon afterward brought strong leadership pressure to build a comparable assistant.

Scialom’s route into this work began with AlphaGo, which prompted his transition from trading to machine learning. A six-month executive course led to an industry-academic PhD with Recital and Sorbonne University, focused on natural language generation and reinforcement learning. He recalls starting roughly two weeks before BERT. His early language-model research tried to simulate human feedback with a discriminator—a model judging generated text—because an academic budget could not sustain ongoing paid annotation.

His contribution to BLOOM began before he joined Meta, and he was not a main contributor; Galactica was his first major Meta project. He offers an anecdotal explanation for Galactica Instruct’s citation behavior: pretraining on papers with references made that behavior emerge readily during instruction tuning. He also describes Galactica Instruct as Meta’s first RLHF annotation project.

For [Llama 2](https://arxiv.org/abs/2307.09288), Scialom and Hugo Touvron started from Llama 1 and focused on instruction following and chat. They used supervised fine-tuning—training on example prompts and answers—followed by RLHF, which uses human preferences between generated answers. The missing knowledge was operational: how many examples to collect, how much preference annotation to buy, which algorithm to use, and how often to retrain. Scialom’s range of 100,000 to ten million annotations describes the uncertainty they faced, not actual dataset sizes.

The training optimum is not the deployment optimum

The release discussed is Llama 3 405B: Scialom confirms 405 billion parameters for the text-only model, with a multimodal version still to come. Choosing that size required balancing a delivery date, available compute, training tokens, scaling laws, GPU memory, and inference cost. Community hardware mattered alongside Meta’s own infrastructure.

Chinchilla’s training-compute optimum, as explained in the conversation, scales model parameters and training tokens together: doubling parameters calls for doubling tokens. It addresses how to obtain the strongest model from a finite training budget. The Chinchilla trap is applying that objective without considering repeated inference. Model size continues to affect the cost of serving requests; the number of tokens used during training does not directly carry over into each request.

When inference efficiency matters, spending more training compute on a smaller model can be preferable. Llama 1 is presented as deliberately accepting lower benchmark performance than a larger model might have achieved with the same training budget, in exchange for a more useful community artifact. The largest Llama 3 model pursued frontier performance within the compute available when the project began.

A large model can also improve models that are cheaper to run. Distillation through annotations uses a larger model’s better answers as training material for smaller models; the discussion credits this process with improving the smaller Llama 3 releases. Lower precision offers another route to accessibility: the 405B model is described as fitting on one node in the lower-precision FP8 format, including a 128K-token context. The hardware configuration and throughput are unspecified, so this establishes a memory-fit claim rather than a serving-cost estimate.

Releasing large models gives the community an opportunity to improve inference efficiency, though making 405B practical on home hardware remains a hope. Low-precision training offers a related opportunity: achieve the same quality with less compute. FP8 and experiments with weights restricted to three values are treated as promising, with an unresolved question about which preliminary results will survive scaling.

More useful text per training budget

Llama 3’s reported changes concentrate on data quantity and quality, while retaining broadly the same architecture and training recipe as Llama 2. The training corpus grew from two trillion to fifteen trillion tokens, including for smaller models. More flexible computation remained a research direction: why should every token receive the same computational effort?

The 128K-token vocabulary is the set of units available to encode text. Expanding it helps represent more languages’ characters and words and can encode the same text in fewer tokens. The cost is a larger vocabulary-related matrix, which adds proportionally more overhead to a small model than to a large one.

More compact encoding can expose training to more distinct text for a given budget, provided it is not simply repeating the same data across training passes. It can also fit more text inside a context limit measured in tokens. Vocabulary size and context length therefore affect different parts of the same question: how much text the model can process.

Quality filtering and topic balancing make the data-efficiency argument concrete. The team used Llama to classify web text by quality and label topics such as law, politics, chemistry, mathematics, and reasoning. Quality filtering avoids spending compute on poor material; topic labels help adjust diversity. The data mixture was mostly shared across model sizes, while the pretraining team incorporated improvements discovered during the long training run.

Synthetic data can transform examples—or add information

Rephrasing or reformatting existing pretraining text is a form of data augmentation: exposing a model to transformed versions of the same material. Tool-assisted generation offers a different possibility. Search can retrieve information missing from the model’s weights; a calculator or code executor can supply an answer the model could not produce reliably. Those results can become new training examples.

The proposed expert-iteration loop targets weaknesses directly: 1. Have the model predict a result. 2. Check that prediction with an appropriate tool. 3. Train on corrected examples where the prediction failed. Arithmetic illustrates the mechanism, with harder problems as the intended opportunity. This is a future research direction, not the procedure used to train Llama 3.

A nearer bridge into post-training is continual pretraining: continue training on expert domains such as code, reasoning, or multilinguality before beginning RLHF. Stronger specialist capabilities can produce better material for preference annotation. Llama 2’s team had been occupied building the annotation arrangements, protocols, methods, and algorithms from scratch, so its scope concentrated on helpfulness and instruction following. Llama 3 expanded that effort to code, reasoning, and multilinguality.

The argument for large teachers does not necessarily imply a model devoted exclusively to generating synthetic data. The view advanced here is that better general models should also generate better training examples, even if their size makes routine inference impractical. The flagship’s dense architecture is likewise treated as a choice still open to exploration: the respondent frames it as the one-expert case of a mixture of experts, with expert count a hyperparameter the team had not yet optimized extensively.

Why judging answers can beat writing them

The annotation-budget question returns as a choice between writing answers and judging answers. In the supervised workflow, humans supply prompts and expected responses. In the RLHF workflow, the model generates two responses and a human chooses one. The respondent recalls initially favoring roughly 100,000 additional supervised examples and a smaller preference-annotation effort, despite having a multimillion-dollar budget.

A few weeks into Llama 2, human evaluations reportedly preferred model-generated answers over human-written annotation answers most of the time. Inspecting the annotations suggested that the model was already strong, rather than that the annotators were poor. Although no sample size or detailed protocol accompanies this account, it explains the change in approach: a person can judge an answer they could not create.

The example is a three-line haiku about language models. Writing one on demand may be difficult; choosing between two generated poems can be easier. In this explanation, behavioral cloning—learning to imitate human answers—reproduces their variation, including mistakes. Preference learning can shift probability toward the better answers already in the model’s output distribution. The annotator’s judgment can therefore support performance beyond their own writing ability, without establishing the same advantage across every task.

The respondent credits RLHF with the post-training improvements while retaining a necessary bootstrap role for supervised fine-tuning: turn a model that completes text into an initial chat model. Llama 2 reportedly used about 10,000 annotated examples for this step. Llama 3 instead used answers generated by the final available Llama 2 iteration. Its reliance on almost no human-written answers concerns answer authorship; human preference judgments remained a separate input.

Preference selection reaches a limit when both candidates are bad. Llama 3’s teacher critic method adds a correction step: a human edits the better candidate into a correct answer. That supplies a supervised target where repeatedly choosing among poor outputs might leave the model stuck. Code is the example, though which domains benefit most from each method remains open. The required annotation scale is also unresolved: no threshold is known, and no plateau had yet been observed.

Preference wins do not settle capability

The release discussion makes a strong qualitative claim: the new 405B model surpasses the originally released GPT-4, while remaining short of the leading model overall, with GPT-4o acknowledged as a possible stronger competitor. These are contemporary judgments without controlled benchmark conditions. The deeper evaluation problem is determining whether post-training gains extend beyond the tests being optimized.

The evaluation approach combines reward models that score outputs, models acting as judges, diverse capability benchmarks and prompts, human evaluation, and manual testing. Successive RLHF rounds provide a recurring comparison: generate an answer from the old model and the new model for each annotated prompt, collect a preference, and aggregate a win rate. Test diversity reduces opportunities to game a narrow evaluation, while stronger models make revealing manual failure cases harder to find.

The 70B preview illustrates the gap between community preference and difficult-task capability. Its ranking exceeded the respondent’s expectations and could place it alongside or above GPT-4, yet it still trailed on complex code and reasoning. Common prompts reveal helpfulness and tone more readily than demanding technical work. At recording time, the new release’s community leaderboard results were still unknown to the respondent.

Calibration, the ability to recognize uncertainty, needs attention alongside accuracy. A proposed test adds an abstention option to multiple-choice prompts that otherwise require A, B, C, or D. The conversation’s hypothetical comparison can be expressed over 100 questions: - Model A: 50 correct answers and 50 incorrect answers. - Model B: 50 correct answers, 10 incorrect answers, and 40 abstentions. Both answer half of all questions correctly, but B makes fewer incorrect assertions. The respondent prefers B and argues that post-training should improve this behavior rather than encourage an answer to every question.

Instruction following becomes agent infrastructure

The new release is described as fine-tuned for tool use, including zero-shot function calling under suitable system prompts. Search and code execution are supported, including code execution across multiple messages. The connection to [Toolformer](https://arxiv.org/abs/2302.04761) is practical: extending tool-augmented systems requires an underlying model that can follow instructions reliably.

[GAIA](https://arxiv.org/abs/2311.12983), a benchmark for general assistants with three difficulty levels, motivates the move from standalone models to agents. The respondent tentatively recalls standalone GPT-4 scoring roughly 0–5% on level one, versus agent systems reaching up to 60%, mentioning OS Copilot, Microsoft [AutoGen](https://github.com/microsoft/autogen), and a Hugging Face agent. The uncertain baseline and unspecified configurations make this a motivation for orchestration, rather than a controlled ranking of those systems.

The example orchestrator coordinates a short sequence: 1. Supply the task and available tools. 2. Ask the model to form a plan. 3. Execute the first step. 4. Check whether it succeeded and invite replanning when necessary. Each step depends on following instructions and constraints. The research ambition is to connect planning, backtracking, browsing, actions, and code execution into stronger systems. Orders-of-magnitude gains are an aspiration for that work, not a reported outcome.

Beyond spending more tokens to think

Text-based orchestration returns to the architectural limitation raised earlier. With fixed computation per token, generating additional tokens is the available way to spend more effort on a problem. Latent-space reasoning proposes doing intermediate reasoning internally, then producing an answer or action without expressing every step as text. Teaching that behavior is described as a medium- to long-term problem, far from solved.

Two ideas make adaptive computation concrete: - A universal transformer can apply a layer repeatedly, with the repetition count determined according to token complexity. - A mixture of experts with an identity expert can choose a path that leaves its input unchanged, allowing computation to be skipped. These remain preliminary directions, alongside incorporating the compute spent generating tokens into the training loss. They are possibilities for changing how effort is allocated, not established replacements for Llama 3’s architecture.

The research-versus-product question closes this technical thread. Meta’s priorities are described as balancing strategic leadership, progress toward artificial general intelligence (AGI), and immediate product needs. Large research models can transfer capabilities into smaller product models through distillation. Product use cases can return the favor by supplying evaluations that distinguish practical progress from gains confined to academic benchmarks.

Build a business that benefits from the next model

The closing startup discussion includes an endorsement of OpenDevin, described as an effort to reproduce Devin, before turning to the instability of technical advantages. Grammarly illustrates the competitive argument: a specialized grammatical-correction data advantage can be threatened when a general model offers similar functionality alongside many other capabilities. This is an argument about defensibility, rather than a measured product comparison.

The advice is to build something that becomes more valuable as models improve. A business devoted to compensating for a current model weakness risks losing its purpose when that weakness disappears. Another participant challenges how predictable improvement is, citing OpenAI’s real-time voice demonstration as a surprise that changed their view of voice-chat defensibility. The response is to plan for progress across dimensions, while acknowledging that assumption could prove wrong.

At the time, the respondent saw more visible startup success in models, compute clusters, and annotation infrastructure than in applications. Another participant offered a narrower observation: agent companies in their own orbit were gaining traction, while some prospective customers waited for better models because they lacked expertise. Both observations capture the difficulty of building applications while the underlying technology changes rapidly.

The conversation ends with a worldwide Llama hiring appeal and an invitation for generative-AI builders and founders to make contact on LinkedIn. The desired researcher profile emphasizes first-principles thinking, rigor, meticulousness, and structured work over extensive prior language-model expertise.