NeurIPS 2023, Part 1 of 4: Training Choices That Change What Models Can Do

Word2Vec: learn word representations with almost no locking

About twenty CPU cores updated the same two-dimensional embedding array with almost no synchronization. In Jeff Dean’s account, Tomas’s single-machine C implementation still learned good word representations. Larger vectors and more training data improved results, creating the next problem: how to extend that training across machines.

The NeurIPS 2023 recap opens with Dean and Greg Corrado’s Test of Time retrospective, revisiting work published ten years earlier. The broader coverage is selective: Swyx describes roughly 15,000 attendees and 3,500 papers, far beyond what two people could cover.

Word2Vec learns word embeddings: dense numerical vectors representing words. The work began with an ICLR 2013 workshop paper comparing training objectives and optimization techniques. Skip-gram performed well enough to become the follow-up’s focus, alongside phrase embeddings. Dean credits releasing code and trained representations with encouraging downstream use. Corrado frames the retrospective’s lessons as ideas the work reinforced or foreshadowed, without claiming it originated or proved them all.

The learning recipe constructs supervision from the corpus itself. Sample a random window of sequence data, then generate inputs and targets algorithmically on the fly. Skip-gram predicts neighboring words from a word; other objectives fill a blank or predict a sequence’s ending. Corrado calls this semi-supervised learning and emphasizes on-the-fly construction as part of its power: a person need not label every example.

Distributed training extended the shared-update approach. Parameter servers stored parameters; independent workers fetched them, computed gradients, and sent updates back. Dean describes configurations with tens to hundreds of parameter-storage machines and hundreds to thousands of workers. Embedding training used 1,000–10,000 simultaneous updating threads, commonly with 300–1,000 dimensions and vocabularies of 100,000–1 million items. Those items could represent watched videos as well as words.

Corrado identifies avoiding tight synchronization as the biggest enabler. Other choices focused computation on useful learning: - Subsample frequent tokens. Probabilistically discard common tokens more often because they already offer many learning opportunities. Less common tokens can be more informative; subsampling can apply to inputs and targets. - Use faster objectives. The team moved through softmax, hierarchical softmax, noise-contrastive estimation, and negative sampling. Corrado reports that simpler, faster objectives also improved representation quality in their work. - Give multiword concepts their own tokens. Join pairs of individually not especially frequent words when they occur together much more often than expected.

The learned vectors captured syntactic and semantic relationships geometrically. Country–capital and disease–drug pairs appeared as similar vector displacements. Vector arithmetic recovered analogies such as sushi–Japan and bratwurst–Germany; adding Russian and river found nearby tokens such as Volga River. These were relationships encoded by the language model.

The retrospective closes with principles that endured and implementations that changed. Specialized hardware had enabled synchronized training at scale, while a return to asynchronous methods remained a prediction. More powerful attention models had shifted tokenization from phrases toward subword units. Simpler, more parallelizable methods and dense-vector representations remained central lessons.

Emergent abilities: evaluation can turn smooth progress into a jump

Rylan Schaeffer introduces joint work with Brando Miranda and Sanmi Koyejo through a tension: scaling data, compute, and parameters often yields predictable improvement, yet arithmetic and word unscrambling have appeared to switch abruptly from failure to success. Emergent abilities, as defined in the discussion, must be absent in smaller models, present in larger ones, and unpredictable by extrapolating smaller-model performance.

Schaeffer proposes that nonlinear or discontinuous scoring can create apparent emergence. More than 90% of the emergent abilities observed on the BIG-bench benchmark appeared under two metrics: - Exact string match: one point only for an entirely correct answer; every imperfect answer receives zero. - Multiple-choice grade: one point when the correct option receives the highest probability. Progress that leaves another option ahead earns no credit.

The toy example is addition of two five-digit numbers. Assume the model’s loss improves smoothly with parameter count, producing a smoothly increasing probability of a correct token. If that probability is approximately p, an answer requiring L correct tokens has approximate success probability p^L. Exact-answer scoring compounds token errors, producing an apparently abrupt gain from smooth underlying improvement. Counting token mistakes instead, using token edit distance, reveals a more predictable decline in errors.

The multiple-choice illustration exposes a different mechanism. A winner-takes-all score jumps when the correct option becomes most probable. A continuous Brier-score evaluation—using squared error between the target and assigned probability in the illustration—changes smoothly. The underlying model is unchanged; the apparent transition comes from the score.

Tests on four GPT-3-family models reportedly matched these predictions: exact-match accuracy declined geometrically with target length, while rescoring the same outputs with token edit distance yielded quasi-linear scaling. Increasing evaluation data and plotting accuracy logarithmically also revealed predictable differences between models whose success rates had been too low to estimate adequately. The BIG-bench meta-analysis found apparent emergence under only four examined metrics.

A small vision model supplied another demonstration. A shallow nonlinear autoencoder trained to reconstruct CIFAR-100 images showed smoothly improving squared reconstruction error as parameter count increased. Awarding success only below an error threshold produced apparently abrupt behavior.

The conclusion is conditional: analysis choices can produce apparent emergence in some cases. Forecasting capabilities requires attention to metric choice, evaluation-data quantity and quality, and coverage of smaller model sizes. The host praises the execution but questions whether this disproves emergence or warrants best-paper recognition. The practical check is narrower: rescore fixed outputs to expose partial progress, and ensure the evaluation can resolve small success probabilities.

DPO: fit preferences without a separate reinforcement-learning stage

Eric introduces Direct Preference Optimization (DPO) as a simplification of reinforcement learning from human feedback, or RLHF, which trains model behavior toward human preferences. The described pipeline begins with a capable, supervised fine-tuned model. For each prompt, it generates two completions and asks a human to choose the better one. These preference triples train a reward model; a separate reinforcement-learning stage then optimizes the language model against that reward.

The Bradley–Terry preference model turns differences between response rewards into preference probabilities, allowing reward fitting through a binary classification loss. The policy—the model’s distribution over completions—then seeks high reward while staying close to the reference model. This restriction matters because the human-labeled examples came from that reference: large changes can move the policy beyond the reward model’s reliable regime.

The reinforcement-learning optimizer discussed here, PPO, brings many interacting components. DPO instead exploits a closed-form optimum of the stated RLHF objective: reweight the reference distribution by exponentiated reward, giving higher-reward completions more probability. Rearranging this identity expresses reward as a scaled log ratio between policy and reference probabilities, plus a prompt-dependent normalization term.

That normalization term would require summing over all possible completions. But Bradley–Terry fitting needs only the reward difference between two completions of the same prompt. The shared normalization term cancels. Substituting the remaining policy-based reward difference into the classification loss gives DPO: fitting preferences also obtains the corresponding policy, without a separate reinforcement-learning stage.

The gradient increases the preferred completion’s log probability and decreases the rejected completion’s. Updates diminish as the induced reward model ranks the pair correctly, reducing repeated pressure on learned examples. The policy still implies a reward; eliminating a separate reward network does not eliminate reward modeling conceptually.

The experiments examine both optimization and task performance: - Synthetic movie-review preferences. A pretrained classifier ranks samples from a small GPT-2 model using IMDb data. DPO reportedly achieves the best balance of reward and divergence from the reference policy among tested methods. PPO does not match it even when given the scoring model that generated the preferences; other approaches without reinforcement learning, including preferred-answer fine-tuning, improve less or prove unstable. - Summarization and dialogue. On Reddit-post summarization and a public helpful-and-harmless dialogue dataset, DPO reportedly matches or exceeds tested baselines. The summarization PPO model is described as almost twice as large. - Generalization to news. Across sampling temperatures, DPO and PPO perform comparably in distribution. Both deteriorate substantially on CNN and Daily Mail news summarization, but DPO generalizes as well as, or perhaps better than, PPO despite PPO’s additional sampled training data.

At the DPO poster: implicit rewards, offline data, and open trade-offs

Before the poster interview, host commentary introduces a competing assessment: DPO is easier and cheaper to train, but critics reportedly find weaker alignment or benchmark results than PPO. At the poster, the researcher emphasizes the implementation burdens DPO removes: repeated model sampling, value-function training, and interacting optimization components.

The questions separate three consequential distinctions: - Initial preference collection remains. DPO uses an initially collected dataset but avoids PPO’s additional online completion sampling. - Optimality belongs to the stated objective. The mathematical solution assumes a given reward function; it does not guarantee optimal real-world behavior. - A well-tuned PPO pipeline can perform similarly. The researcher identifies high-variance estimation as an optimization difficulty, without claiming that PPO must perform worse.

For experimental comparability, the researcher describes starting with the same base model and preference dataset. PPO first fits a reward model, then performs reinforcement learning. The team also sought existing RLHF models trained on the same datasets to reduce the risk of a weak baseline caused by its own tuning. Zephyr and Mixtral Instruct are cited as DPO adoption examples. The efficiency advantage centers on one preference-training stage instead of reward fitting followed by reinforcement learning, without a measured speedup in this exchange.

The researcher declines the interviewer’s suggestion that DPO has no trade-offs: more research is needed, and PPO may do better in some settings. The origin story is more definite. Rafael’s algebra suggested removing the reinforcement-learning stage; Chelsea Finn provided lab infrastructure and urged the team to pursue the idea promptly. According to the researcher, the paper came together in the three weeks before the NeurIPS deadline.

The closing exchange raises supervision beyond pairwise preferences as a possible complication. The researcher recounts that John Schulman tried DPO and said it worked, while asking about richer reward-model supervision. Another possibility is a reward model conditioned on information unavailable to the policy. These are proposed boundary conditions, not confirmed details of another team’s undisclosed system.

Data-constrained scaling: spend more compute on repeated data

Scaling Data-Constrained Language Models, also introduced as the Datablations paper, asks how to pretrain when unique data is fixed. Niklas motivates it with a prior forecast that high-quality language data, including papers and books, could be exhausted the following year—a forecast made in the 2023 context. Low-resource languages already face the constraint.

The first intervention is repetition, contrasted with cited GPT-3 and PaLM practices that avoided it. In a validation-loss comparison, two epochs—two passes over the dataset—with half as much unique data perform approximately like one pass over fresh data. One configuration repeats 28 billion unique tokens twice. A few repeats remain competitive; extensive repetition eventually degrades performance.

To determine how to allocate compute, the researchers fix unique data at 100 million tokens and vary model size and epoch count. Their fitted equations favor increasing epochs faster than parameter count. This departs from extrapolating Chinchilla’s equal scaling of model size and training data into the repeated-data setting. Gains eventually saturate as the fixed corpus yields less additional value.

A larger comparison holds both computational work and the 25-billion-token unique-data budget constant. The allocation derived for data constraints uses fewer parameters and more epochs: 9.7 passes, or approximately 242 billion training tokens. It reportedly achieves lower loss and better downstream performance than the Chinchilla-based allocation. The result concerns allocation under those constraints, rather than a universally best number of passes.

Complementary strategies are evaluated through average performance across 19 natural-language tasks: - Repeat available text. Roughly four epochs provide a useful trade-off in many tested setups, including a 25% data budget. Heavier repetition eventually causes substantial decline. - Replace missing text with Python code. Substantial substitution preserves natural-language performance, with some tasks improving. A 50% code mixture is recommended for most tested setups. - Filter, then repeat. Perplexity-based quality filtering followed by two epochs outperforms the original dataset in the presented comparison. The proposed explanation is that quality gains outweigh the repetition penalty. - Preserve enough data when deduplicating. Aggressive fuzzy deduplication can worsen performance by shrinking an already constrained corpus.

The presentation closes with FinGPT, a Finnish-language model trained with repeated data: 38 billion unique tokens, eight epochs, and 13 billion parameters. The host contrasts the findings with the paper titled One Epoch Is All You Need, emphasizing uncertainty about the right number of passes, and praises Hugging Face’s well-documented experimental and filtering code.

QLoRA: compress frozen weights and the constants that describe them

Tim Dettmers introduces QLoRA as a way to fine-tune large models with less memory. It compresses the pretrained network to four bits and trains small added adapters, aiming to preserve performance comparable to sixteen-bit models. The underlying operation is quantization: replace many possible numerical values with a small set of representable values.

Dettmers’s four-bit integer example divides the empirical minimum-to-maximum range into sixteen equal-width bins and replaces each value with its bin’s midpoint. He generalizes this through a codebook: stored indices mapped to representable values, normalized between −1 and 1. Quantization and reconstruction follow this procedure: 1. Divide the tensor by its absolute maximum to normalize its values. 2. Find each element’s nearest codebook value using binary search over the sorted codebook, and store its index. 3. Reconstruct by looking up each index and multiplying the recovered value by the original scale.

An intentionally awkward two-bit codebook shows why placement matters. With values [-1, 0.3, 0.5, 1], the input [10, -3, 5, 4] reconstructs as [10, 3, 5, 5] in the example’s mapping. One value changes sign. Compression must keep quantization error small enough to preserve network performance.

Fine-tuning makes this memory saving consequential. Dettmers accounts for sixteen bits per parameter for weights, sixteen for gradients, and sixty-four for Adam optimizer state: twelve bytes per parameter, or 840 GB for a 70-billion-parameter model. Adapter fine-tuning freezes the pretrained weights and updates only small added components, while passing gradients through the frozen layers. This reduces trainable state, but the described base weights still require sixteen bits each.

QLoRA quantizes those frozen weights to four bits. In the presented configuration, the average footprint falls from 17.6 to 5.2 bits per parameter, with memory totals falling from 150 GB across eight consumer GPUs to 46 GB across two. The full sizing assumptions are not specified, so these are configuration-specific totals.

Dettmers briefly mentions an optimizer mechanism for preventing memory spikes when processing large documents. The precision mechanism receives more explanation: four-bit NormalFloat uses equal-probability bins, rather than equal-width bins, under a normal distribution. Equal expected numbers of values fall into each bin. He argues that this makes the representation information-theoretically optimal under that distributional assumption.

Part 1 ends with double quantization: compress the scale constants too. Blockwise quantization stores compressed weights plus an absolute-maximum constant for each block. Those constants consume about 0.5 bits per parameter on average in the presentation’s accounting. Quantizing them again saves about 0.4 bits per parameter—multiple gigabytes at large model sizes, potentially enough to determine whether a model fits on consumer GPUs.