What Makes a Model Improvement Worth Using? — Part 2 of 3

What transfers when the stack changes?

Move between companies, and the infrastructure, training stack and data change. What transfers is the ability to solve the next unfamiliar problem. At the opening of Part 2 of this Latent Space conversation, the responding researcher compares a new leaderboard to a puzzle: confidence comes from knowing how to make progress before knowing the answer. The durable asset is the team’s ability to improve models. The team described here combines former Google colleagues with new hires, including recent PhD graduates.

Architecture puts that judgment to work. The researcher considers the transformer a formidable baseline because it has accumulated useful changes. The qualification matters: this is the evolved transformer; the original vanilla architecture would, in their assessment, be suboptimal.

Architecture changes must earn their implementation cost

Three choices illustrate what makes a modification attractive. These are qualitative assessments; the attention and activation mechanisms themselves are not developed here. - GQA improves the inference tradeoff. The researcher describes MQA as risking a model-quality penalty, while GQA retains faster inference without the same penalty in their experience. - SwiGLU offers useful gains with a simple implementation. They recall experimenting with it at Google when it attracted little outside attention. In a remembered study comparing transformer modifications, a SwiGLU-related activation variant and sparsity stood out, though the exact variant is not recalled. - RoPE makes context extrapolation attractive. In a recalled PaLM experiment against T5-style relative attention, it performed similarly to the alternatives, with extrapolation providing an additional reason to use it.

The informal label “Noam Architecture” refers to a lineage of changes: the researcher traces MQA to Noam, GQA to a follow-up by Google colleagues, and SwiGLU to a single-author paper by Noam. It does not designate an encoder–decoder configuration.

The researcher calls ease of adoption the implementation lottery. A simple change with broad benefits can spread; a complicated modification offering a hypothetical 0.1% improvement has a weaker case. Beating the established transformer requires either a substantial shift or a small change whose gains are general enough to justify integrating it.

Separate context processing from generation

The researcher declines to disclose the team’s exact setup, but distinguishes three designs by their attention masks and parameter sharing: - Causal decoder-only models use a causal attention mask. - Prefix language models differ primarily in the mask, allowing bidirectional attention over the prefix. - Encoder–decoder models process inputs and targets in separate transformer stacks with unshared parameters and an encoder bottleneck. UL2 explored these distinctions. The decoder in an encoder–decoder remains an autoregressive language model.

The first proposed benefit is intrinsic sparsity: separate encoder and decoder parameter sets change the relationship between total parameters and computation. The researcher offers UL2 20B as roughly matched in floating-point operations, or FLOPs, to a 10B decoder-only model. That approximation comes without sequence lengths or workload conditions; it is not a general half-cost rule.

A separate encoder also creates room to compress context without the decoder’s causal-mask constraint. It can process a long input using aggressive pooling or sparse attention, potentially becoming smaller or faster than the decoder while generation remains autoregressive. The researcher compares this separation to retrieval-augmented language modeling: preparing context and generating language need not use identical machinery.

Benchmarks wear out—and restricted access discourages adoption

The researcher regards Llama 3’s improvement over Llama 1 and Llama 2 as evidence that Meta has developed a strong training stack. The suggestion that Meta may have caught up with Google is tentative. Their skepticism about models premised on distillation is a general research preference: they acknowledge not following Phi closely or reading the relevant paper.

Establishing model quality requires evaluations that still distinguish useful behavior. The researcher finds the LMSYS evaluation platform more useful than conventional public benchmarks, though imperfect. They describe GSM8K and HumanEval as saturated and contaminated: a GSM8K score of 91 versus 92 has little practical significance, even while reporting such numbers remains customary.

The exchange develops three complementary approaches: - Withhold part of an internal evaluation. Keeping examples private reduces their exposure to community use and contamination. - Commission human evaluations and inspect outputs. The researcher reports using third-party data companies, while endorsing researchers’ direct examination of model behavior. - Generate fresh benchmark variants. The interviewer proposes evaluations generated from seeds that can be withheld or varied, averaging results across a chosen set. This is a proposed contamination defense, not a demonstrated solution.

Popularity can shorten an evaluation’s useful lifetime. The researcher includes VibeEval in this limitation: heavily used tests eventually need retiring. Yet protecting test data can undermine adoption. Evaluations that required researchers to package models and send them to organizers imposed enough work to discourage participation.

Benchmarking therefore has an incentive problem. Model developers want to demonstrate superiority; benchmark creators want visibility and citations. Accessible tests serve both goals. The interviewer describes SWE-bench as a breakout coding-agent competition at the time: it was considered difficult while allowing visible, rapid progress. Evaluation design gives academics a way to steer research even without frontier training budgets.

Multimodal boundaries can mirror team boundaries

The interviewer distinguishes late fusion, joining separately developed, frozen components, from early fusion, involving modalities from early pretraining. The researcher favors early fusion whenever possible. Compute constraints partly explain late fusion, but so do separate language and vision teams: one releases a language model, and another adds vision.

Organizational separation can become architectural separation, even inside large companies. The researcher predicts that teams and modalities will increasingly converge, much as previously separate tasks had unified since roughly 2019.

Choosing an Airbnb makes the overlap concrete. An agent must navigate the listing and interpret its photographs, including scenery and location cues. Screen understanding and natural-image understanding meet in the same task. The researcher expects both to become capabilities of a general model, while judging natural-image understanding stronger at the time and screen understanding more in need of improvement.

Training budgets and inference costs imply different choices

Chinchilla scaling optimizes training under a fixed compute budget. It does not define a ceiling beyond which training stops helping. For inference efficiency, the researcher advocates training a smaller model with substantially more data and compute until the capabilities that matter saturate.

That stopping condition is difficult to apply. Value per FLOP deteriorates, yet some metric may continue improving enough to justify another extension. Choosing which capabilities matter helps determine when to stop; otherwise, residual gains can repeatedly reopen the budget.

The researcher disputes the narrative that training beyond Chinchilla prescriptions began with Llama 2, citing T5 base, reportedly trained on one trillion tokens, as an earlier example. Conversely, they attribute the disappointing quality of OPT and BLOOM to training large models on too few tokens.

Long context must justify itself on the task

Accepting a long input differs from using it effectively. The researcher treats needle-in-a-haystack retrieval—finding a particular item within a large context—as a basic unit test. Richer evaluations, including coding tasks, are needed to establish what a model can accomplish with that context.

The participants disagree about deployment. The researcher sees long context as the future, while allowing combinations with retrieval-augmented generation (RAG), which retrieves material to supply to the model. The interviewer predicts that long context will serve prototyping and production systems will move toward RAG.

A complex story exposes the risk of retrieval: relevant information can be missed, and chunks can fragment material that needs to be understood together. The researcher imagines a user willing to pay $100 and wait a day for a correct answer. The interviewer counters that expensive calls would be used sparingly. The hypothetical illustrates differing application requirements, without promising that spending more ensures correctness.

Fact retrieval can favor RAG without settling the broader comparison. A task needing one isolated fact makes extra context look wasteful; that result says less about tasks whose answers depend on relationships across the input. The researcher argues for choosing or combining approaches according to the application.

Fine-tuning adds a third way to supply domain knowledge. The relevant conditions differ: - Long context: broader, interconnected material matters to the answer. - RAG: retrieval can supply relevant material without losing necessary context. - Fine-tuning: stable requirements may justify a one-time adaptation that avoids repeatedly supplying the same knowledge.

Changing domains weaken the case for fine-tuning. For specialized medical and legal models, however, the researcher considers it useful and a possible alternative to RAG. The interviewer proposes a further benefit: adjusting domain assumptions and word associations, beyond supplying individual facts.

Efficiency claims must survive scale

How should a practitioner assess a paper that removes model components successfully in a small experiment? The researcher agrees with the interviewer’s concern: a favorable task can conceal a weak general method. Preserving performance on one dataset is easier than establishing a useful tradeoff across capabilities and model sizes.

One warning sign is a zero-shot comparison at roughly one billion parameters where scores remain near chance. The researcher offers 50 versus 54 as a potentially uninformative improvement, without a statistical analysis of that difference. More broadly, matching a transformer on a few datasets at 1B, 3B or 7B does not establish behavior at 100B or 200B. Evidence sufficient for a paper may still be insufficient for adoption.

Sparse Upcycling enters as early Google work focused mainly on vision, with a smaller set of T5 experiments. The researcher corrects the interviewer’s description of their involvement to an advisory role. The broader attraction of mixture-of-experts (MoE) models is their computation-to-parameter ratio: they can have more total parameters while keeping active computation relatively low. Implemented well, the researcher argues, this improves performance per FLOP.

The researcher expects gains beyond eight experts but recommends no specific count. The open question is how MoE changes capabilities, particularly under large-scale instruction tuning. The interviewer initially sees upcycling as a modest boost to a strong base model; the researcher points out that training MoEs from scratch is also promising, which the interviewer acknowledges.

Parameter counts cannot substitute for throughput

The Efficiency Misnomer discussion turns to actual deployment cost. The researcher accepts active parameter count as an approximation, but calls for a broader assessment: - Active parameters and FLOPs help estimate computation. - Model fit and serving overhead matter, including whether the model fits on GPUs within a single node. - Measured throughput and speed expose implementation complexity, missing optimized kernels and poor hardware suitability.

Lower theoretical complexity can still produce lower throughput. The researcher illustrates this with a linear-time method whose required scan makes its implementation much slower in practice. A smaller model can likewise slow down when added operations lack optimized kernels or suit the hardware poorly. These are illustrative cases, without named methods or benchmark conditions. The interviewer regards missing optimization as fixable. The researcher qualifies that some problems are difficult to fix or require substantial additional optimization effort, challenging claims that parameter savings imply broader savings.

Even equal parameter counts can hide different costs. Moving parameters from a feed-forward network into a comparatively cheap embedding layer preserves model size while changing runtime behavior. Parameter-matched is not throughput-matched. A single favorable tradeoff plot can obscure other relationships among size, FLOPs and speed, even without intentional selective presentation.

Open weights and grassroots progress are different claims

The researcher distinguishes large-lab open-weight releases from grassroots model development. Having contributed to open source, they reject the characterization that they oppose it. Their criticism concerns the romanticized account of independent developers collectively challenging major labs, and incentives that reward renaming or lightly modifying models more readily than sustained improvement.

The participants identify two different leaderboard effects: - Encouraging variants: the interviewer argues that the Hugging Face leaderboard contributed to the proliferation of lightly modified models. - Exposing their limits: the researcher argues that climbing LMSYS required more substantial improvements, weakening the reward for lightweight modifications. The interviewer also credits leaderboard maintainers with providing a difficult public service.

Productivity, boundaries and the cost of keeping up

Asked about productivity, the researcher describes working almost constantly through their PhD and time at Google, with little boundary between work and life. Enjoyment supplies the motivation: running experiments can pull their attention away from television and back to code. The productivity came with an unhealthy lack of boundaries, they acknowledge. Having a baby has prompted an effort to make more room for life outside work.

The interviewer offers two deliberate practices for managing work and information: - Handwritten task lists: physically crossing off items helps them organize work more effectively than attempts to manage everything through a calendar. - A vetted information filter: an AI newsletter condenses Twitter, Reddit and Discord. Having checked its pipeline from beginning to end, they use it to keep up with news.

The researcher previously checked new papers each morning at 9:30 a.m. Singapore time, selecting one or two to skim. Startup work has reduced that reading. The interviewer challenges the old routine as novelty bias; the researcher defends it as a way to stay current, without claiming to have read everything. Part 2 ends with a changed filter: they now rely on Twitter to surface papers they consider important enough, reducing the need to monitor new releases directly.