ICLR 2024, Part 2 of 4: Adversarial Examples, Transformer Workspace, and Long Context

Tiny image changes—and attacks that transfer

Take a correctly classified image and find the smallest additive change that makes the classifier give a different answer. This optimization problem opens Part 2 of 4 of the ICLR 2024 recap. In Christian’s account, adversarial examples are constructed to induce errors. Intent distinguishes an attack from accidental corruption.

The original implementation repeatedly optimized a relaxed objective with the L-BFGS optimizer until the classification changed. Christian recalls finding smaller perturbations than the faster gradient sign method introduced a year later. MATLAB’s available optimizer helped determine the method: it supported box constraints, which keep pixel intensities within the allowed range. Omitting those constraints had produced smaller but invalid perturbations.

Christian built the initial implementation from Andrew Ng’s deep-learning tutorial code, with most experiments on MNIST. Several comparisons widened the finding: - Autoencoder pretraining did not remove the vulnerability. The pretrained models still admitted adversarial examples. - Linear classifiers were vulnerable too. Logistic regression misclassified subtly perturbed images, and Christian reports the same susceptibility in support vector machines. - The deeper MNIST models tested required smaller perturbations than the linear classifiers. This was a result for those models, rather than a universal relationship between depth and robustness. A collaborator named Wojciech later ran ImageNet experiments on AlexNet, extending the investigation beyond MNIST.

Christian recalls dismissive early reactions at Google, followed by Geoff Hinton expressing concern if the findings were correct. The presentation considered possible attacks on Google systems, including circumventing CAPTCHAs; these were proposed uses, not established successful attacks. Replication on AlexNet helped persuade the researchers to publish. Early unpublished adversarial-training experiments helped little, he says, while the larger datasets used by later models alleviated vulnerability without eliminating it.

The more consequential result was transfer across architectures and training-data partitions. To test whether the errors were peculiar to one fitted model, Christian split MNIST into two partitions and trained different architectures on different partitions. Adversarial examples still transferred to some extent. That enables black-box attacks: an attacker can construct perturbations using accessible models and try them against a target without possessing its internals.

Christian regards transfer as a durable contribution, even though he believes others were close to discovering adversarial examples themselves. He disavows the original paper’s spectral-analysis explanation and recommends a follow-up analysis credited primarily to Ian, whose experimental arguments undermined it. His closing research lesson is to resist pressure to include an explanation one does not trust. The host then offers a separate interpretation: networks overfit to clean images and compress different features from humans. That interpretation should remain distinct from what the transfer experiments established.

Registers give vision transformers working space

Timothée Darcet’s Vision Transformers Need Registers begins with a puzzle: why does a vision transformer concentrate attention on seemingly arbitrary background patches? Its CLS token gathers information about the whole image from patch tokens through self-attention. Reshaping its attention scores over those patches produces a spatial heat map. DINO produced smooth maps, but DINOv2 showed artifacts, so self-supervised training alone could not explain their absence.

The investigation connects three observations: - Attention spikes coincide with unusually large token vectors. In DINOv2, some token norms were around 500; a norm above 200 selected outliers for further tests. - Outliers arise in visually redundant patches. Comparing each patch’s raw pixels with its four neighbors gave cosine similarities concentrated near one, indicating very similar content. - Outliers reveal less local information and more global information. Linear probes—simple predictors trained on frozen token representations—recovered less pixel and position information from outliers, while image-classification probes performed better on them than on normal patch tokens.

Darcet’s hypothesis is that sufficiently trained large models repurpose redundant patches as global workspace. A token whose visual content repeats its neighbors can surrender local information and become a place to store, process, or retrieve information about the whole image. The probes support this interpretation by showing what information they can recover; they do not prove exactly how the model uses each token.

The intervention follows directly: registers are extra tokens that carry no image information at input. They participate in self-attention, but their outputs are discarded rather than used directly for the loss. This gives the model working space without requiring it to appropriate an image patch. In the presented comparison, training with one register removed the visible attention artifacts. Darcet reports DINOv2 gains on classification, segmentation, and depth estimation, along with cleaner attention maps and token-norm distributions in tests on DINOv2, CLIP, and DeiT-3.

Different registers also sometimes attended to different objects without an explicit training requirement to separate them. Darcet treats this as a possible research direction: the behavior was intermittent and came from a one-off experiment. He closes by recommending Massive Activations in Large Language Models, which he describes as extending many of these observations to language models while proposing different interpretations and a simpler fix.

Pause tokens add computation before an answer

The next presentation asks whether a language model can spend more computation before predicting a difficult answer. Pause tokens add input positions before answer generation—ten in the initial example—providing additional attention computations before the response begins. Simply appending them at inference did not work in the reported experiments. The presenter argues that models must learn to use pauses during pretraining.

Pretraining inserts pauses at random positions. A pause between “is” and “playing” in a sentence adds a computation step before predicting “playing.” The presenter reports a 10% insertion frequency, with sequences clipped back to their original length to match the baseline’s floating-point operation count. This setup reportedly improves reasoning, reading comprehension, and natural-language understanding benchmarks; the presentation gives neither benchmark names nor gain sizes.

The theoretical motivation concerns computation at a fixed depth. In a two-layer sketch, each attention position performs an operation on a pair of input tokens, while the desired function may require as many as N(N−1)/2 such operations. Extra positions provide more places to perform them, expanding the representable function class under that setup. Reading comprehension supplies an analogy: additional positions could support more comparisons between sentences and answer choices. This illustrates the argument rather than demonstrating the learned attention behavior.

The discussion distinguishes three roles for intermediate computation: - Pause tokens increase input width. Extra positions provide computation before ordinary decoding begins. - Chain-of-thought increases effective depth. Intermediate reasoning tokens are generated autoregressively, so later computation can depend on earlier generated steps. - Backspace tokens support scratchpad correction. The presenter characterizes them as a way to repair intermediate reasoning. Whether pauses improve chain-of-thought, and how much the specific decoded reasoning tokens matter, remain open questions.

The learned pause embedding did not have an obvious interpretation among nearby word embeddings. Nor did punctuation or instructions to think more reproduce the effect. The intervention depends on learning to use the additional positions, rather than on an ordinary-language meaning for the pause.

Adaptive pausing remains a proposed extension. The experiments append a fixed number of pauses to the prompt. Alternatives include pausing periodically during generation, inserting pauses when prediction confidence falls, or letting the model generate them until confidence recovers. The last option requires changing pretraining: the current objective ignores the output immediately before an inserted pause and therefore never teaches the model to predict one.

The closing exchange connects this search for useful computation and supervision to multi-token prediction. Predicting the next four tokens lets a target contribute to losses from four preceding positions, distributing responsibility for predicting it across more positions. The connection is offered as a research direction, without comparative results.

The best training subset may favor easy examples

An autonomous vehicle collects many repetitive images, but those images need not contribute equally to learning car detection. Pulkit introduces Towards a Statistical Theory of Data Selection under Weak Supervision through this problem: training, labeling, storage, and quality-control costs grow with the collection. In the opening logistic-regression example, curated subsampling matches full-dataset test misclassification performance after discarding 65% of the data, and improves on full-sample error while retaining 60%. These are results for that example, not general retention targets.

The procedure selects examples before acquiring their labels: 1. Score unlabeled examples with a surrogate model that supplies label information better than random guessing. 2. Select a subset using those scores, potentially through probabilistic sampling rather than retaining only the highest scores. 3. Acquire labels for the selected examples and train the downstream model. This is weak supervision because selection has imperfect label information from the surrogate. That surrogate could be a foundation model, a model trained on another dataset for the same task, or one trained on a small labeled portion of the target dataset.

The formal setup assigns each example a selection probability π_i and a training weight w_i, then minimizes the weighted training loss. Unbiased subsampling weights examples inversely to their selection probabilities, preserving an unbiased loss after selection. The framework examines biased and unbiased schemes, low- and high-dimensional settings, and imperfect or perfect surrogates. Here, a perfect surrogate knows the true data distribution.

For binary logistic regression, the surrogate’s class probability p gives a hardness score `p(1−p)`. Confident predictions near zero or one count as easy; predictions near one-half count as hard. A hardness exponent α controls selection: positive values favor hard examples, negative values favor easy ones, and zero gives random sampling. Synthetic experiments use isotropic Gaussian features and test cases where the model matches or mismatches the label-generating process. The presentation reports close agreement between simulations and theoretical predictions of high-dimensional test error.

The results challenge several plausible defaults: - An unbiased loss need not give the best test error. Biased selection wins in presented examples, and specific theoretical constructions make unbiased selection arbitrarily worse. - Harder examples need not be more useful. Favoring easy examples gives the lowest errors in the presented real-data settings, which vary regularization and signal-to-noise ratio. - A stronger surrogate need not select a better subset. One comparison gives better downstream performance with a weaker surrogate. - Full-sample training need not win under model mismatch. In one selected synthetic logistic-regression example, retaining 40% of the data improves misclassification error.

The proposed explanation for the last result is that mismatched models and losses make examples unequally useful to the training objective. The recommendation remains conditional: uncertainty-based sampling can help, but selecting the hardest examples can be catastrophic. Whether to favor easy or hard examples depends on the setting.

DoRA learns image representations from a walking tour

Shashank’s Is ImageNet Worth One Video? changes the source of supervision. Self-supervised learning obtains its learning signal from data without manual labels, avoiding repeated relabeling as collections evolve. The presentation identifies three limitations of image-based approaches: controlled training on unlabeled ImageNet, dependence on carefully designed augmentations, and difficulty using richer real-world context. Video offers temporal, visual, audio, and motion cues.

Walking Tours contains ten egocentric videos—recorded from the walker’s viewpoint—from cities in Asia and Europe. They are described as 4K, typically one to three hours long, rich in semantic categories, and licensed under CC BY.

DoRA discovers and tracks objects using the vision transformer’s own attention, without external object detectors or optical-flow networks. The teacher model’s penultimate-layer output supports generation of multi-object masks. Those masks feed the student model, and a distillation loss trains its image-level CLS embedding against the teacher’s embedding.

Inside the tracker, a subset of attention heads pools patch queries into object prototypes. Refinement improves correspondence between those prototypes and patch embeddings while encouraging diverse tracked objects. Cross-attention then tracks objects across frames. The presentation does not clearly establish the refinement procedure’s name.

With one walking-tour video containing 200,000 frames, DoRA reportedly outperforms DINO pretrained on ImageNet-1K on object discovery and semantic segmentation, while matching it on object detection, video object segmentation, and multi-object tracking. Numerical scores are not given. In the separate discussion of fine-tuning on image datasets to address distribution shift, one-video pretraining is comparable to DINO, while pretraining on all ten videos outperforms it.

LongLoRA makes context extension cheaper

LongLoRA addresses the expense of fine-tuning pretrained language models for long inputs. It combines shifted sparse attention with an enhanced form of LoRA adaptation. Shifted sparse attention limits computation to groups while connecting neighboring groups: half the attention heads shift token positions by half a group, allowing information to cross group boundaries.

The implementation uses three operations: 1. Split the features into two chunks. 2. Shift tokens in one chunk by half the group size. 3. Divide tokens into groups and reshape the groups into the batch dimension. Attention masks prevent potential information leakage. The implementation is compatible with FlashAttention, and evaluation can use full attention even though fine-tuning uses sparse attention. The attention pattern is designed to extend models already pretrained with full attention.

The second change makes embedding and normalization layers trainable alongside the LoRA parameters. The presentation assumes familiarity with LoRA itself; its contribution here is this additional trainable parameter set. The addition reportedly matters for context extension and can close the performance gap with full fine-tuning. Perplexity evaluations, including PG-19, are described as comparable to full-attention and full-fine-tuning baselines with better efficiency.

The reported extensions cover different model sizes and training conditions: - Llama 2 7B: 4K to 100K context on eight A100 GPUs. The presentation reports the 7B extension; the host’s recap supplies the model identity and starting window. - Llama 2 70B: extension to 32K on the same hardware. The recap clarifies the context length. - Llama 3: progressive extension to one million tokens. The presentation reports strong qualitative needle-in-a-haystack retrieval performance, without specifying model size or an exact score.

On LongBench and LongEval, the presenter reports comparable or better performance than the other evaluated 7B long-context models after roughly four hours and 300 million tokens of supervised fine-tuning. Large batches and long training schedules are additional contributors. LongAlpaca, the accompanying instruction dataset, contains about 12,000 question-answer pairs covering book questions, short questions, summarization, and paper review. Preparing high-quality long examples required substantial manual work.

Demonstrations include reviewing a paper against guidelines and comparing two papers’ structure, technical depth, theory, experiments, and writing style. A book example asks about Ye Wenjie’s motivations in The Three-Body Problem. Supplying the book’s content produces a more concrete, detailed answer than supplying its title alone, according to the presenter—even though the model may have encountered the book during pretraining. These are illustrative outputs, not measured review or literary-analysis results. The presentation closes by announcing the release of code, models, and data on GitHub and Hugging Face.

YaRN preserves local position behavior while extending context

Bowen Peng’s discussion of YaRN turns to positional representation. RoPE represents position through dimensions that rotate at different speeds. Positional interpolation stretches that representation to support a longer window. Peng credits interpolation with enabling extension using a relatively small amount of fine-tuning, estimated at roughly 1% of the pretraining dataset. He explicitly corrects the interviewer’s suggestion that no fine-tuning is needed.

His objection to uniform interpolation is that fast-rotating, short-wavelength dimensions support local attention. Stretching them changes behavior that already works. The proposed alternative is to preserve short-wavelength behavior while interpolating long-wavelength dimensions. Peng reports lower loss and post-training perplexity, plus better passkey retrieval than uniform interpolation, while acknowledging degradation near the end of the evaluation.

Peng says the comparisons held context length and training amount constant, with effort spent optimizing each method. Temperature adjustment was especially important at 128K context. He reports approximately 100% passkey retrieval in the presented results and context extension with about 400 training steps. These figures describe that retrieval evaluation; the full setup and most exact comparison values are not provided.

The subsequent exchange describes LongRoPE as replacing hand-tuned parameters with machine-learning-based optimization, with reported improvement over YaRN. It also establishes a practical condition: extension training needs sufficiently long examples. For a 32K target window, suggested example lengths are around 15K–20K. Efficient methods reduce the cost of extension, but the respondent argues that additional compute and data can still produce better models.

The questions distinguish three responsibilities: - Positional methods govern behavior beyond the original window. A proposed comparison between RoPE and the alternative positional method ALiBi would hold architecture and pretraining data fixed. The respondent says the team cannot afford that Llama-7B-scale experiment and regards existing comparisons as inadequate. - Ring Attention addresses compute efficiency. It is separate from positional extension: memory and training resources remain constraints as windows grow. - Training data must teach long-context tasks. Multi-hop reasoning and multi-turn conversation do not follow automatically from a longer window or successful passkey retrieval.

The respondent predicts that positional context extension will become a solved problem, leaving suitable training data as the harder capability bottleneck. The closing exchange does not settle that view: one speaker proposes high-quality, varied long-context data as the remedy, while another argues that attention limitations also require architectural changes. Their final distinction challenges the label infinite context: encoding information for later retrieval differs from maintaining an indefinitely large context. Part 2 ends with that distinction still under discussion.