Self Pre-Training and Fair Long-Sequence Benchmarks — Part 4 of 4

Training from scratch leaves out learned priors

Transformers performed poorly on Long Range Arena, a benchmark testing dependencies between distant sequence elements. Yet pre-trained transformers showed strong capabilities in text summarization, code completion, and protein folding. In this closing portion of the ICLR 2024 recap, Ido presents work with Jonathan Berant and Ankit Gupta asking whether that discrepancy reflects architecture or training practice.

Weak transformer results on Long Range Arena (LRA) helped motivate architectures designed to capture long-range dependencies. But evaluating those designs requires accounting for inductive bias: the preferences a model brings to learning a task. Architecture supplies some of those preferences; Ido argues that the pre-training objective supplies others. Two training settings therefore measure different things: - Random initialization followed by task supervision tests how well a model learns the benchmark from scratch. - Pre-training followed by fine-tuning includes preferences learned before supervised task training, reflecting the practice Ido describes as common in real applications. A comparison confined to the first setting leaves out the contribution of pre-training.

Self pre-training learns an initialization from task data

Self pre-training (SPT) first pre-trains a model on the downstream task’s own training data, then uses the resulting parameters to initialize supervised task training. This incorporates the bias introduced by the pre-training objective while keeping the data source fixed: no external pre-training corpus is required.

The experiments use denoising objectives for this preliminary stage. Ido motivates the choice with prior work reporting that denoising pre-training solely on downstream training data often approaches the performance of pre-training on large datasets. That finding motivates a practical comparison using only task data; it does not establish that the two data regimes are universally interchangeable.

SPT changes the LRA comparison without changing architecture

With denoising SPT, Ido reports that transformers match the LRA state of the art without architectural changes. The gains also extend to S4, a model he describes as a reparameterized linear recurrent neural network (RNN): - Transformers improve by more than 30% on average compared with training from scratch. - S4 improves on five of six tasks after adding SPT. - S4 with SPT solves PathX-256, with a reported 20% improvement over training from scratch.

The presentation does not specify whether those improvements are relative percentages or percentage-point gains, and gives no absolute scores or uncertainty estimates. The results nevertheless illustrate the central evaluation issue: training procedure materially changes the architectural comparison.

Learned priors let a simpler model approach S4

The final experiment asks whether pre-training can also reduce the advantage of architectural complexity. Ido compares two recurrent designs: - S4 reparameterizes the linear RNN and uses theoretically derived initialization, rather than learning its parameters directly. - Diagonal Linear RNN (DLR), attributed to Gupta and colleagues, simplifies S4 by replacing its reparameterization steps with a single diagonal matrix. The comparison treats these design differences as handcrafted biases for learning long-range dependencies.

When both models train from scratch, DLR underperforms S4. Adding SPT shrinks the gap to roughly 1%, again without specifying relative percent versus percentage points. Ido argues that learned priors can therefore let a simpler model approach a more elaborate architecture. The practical implication is to include the intended pre-training regime before attributing a performance advantage to architectural complexity.

Next in the ICLR coverage: reasoning and agents

The recording closes with an announcement that the next installment of ICLR coverage will focus exclusively on reasoning and agents—the host’s subjective choice for the conference’s biggest topic for AI engineering.