Beyond Attention, Toward Everyday Inference — Part 4 of 4

Finding common ingredients in sequence models

Sequence-model papers can look very different to researchers outside their immediate subcommunity, even when they share a few central ideas. The closing architecture discussion in the NeurIPS 2023 recap takes inspiration from the T5 paper’s approach: make the common components clear so more researchers can participate.

Three ingredients organize that account: - Long convolutions: convolutions whose extent grows with the input, though not necessarily across its entire length. - Multiplicative gating: component-wise multiplication within the sequence. - Data dependence: making computation depend on the input data. Mamba combines data dependence with subquadratic runtime—runtime that grows more slowly than the square of the input length.

Attention-free results do not settle scaling

The new convolutional architecture reportedly shows dimension scaling somewhat like attention on MQAR. This is a qualitative comparison: the task is not defined here, and no experimental setup or numerical results accompany it.

In the week leading up to NeurIPS 2023, Mamba and an architecture transcribed as “Base” reportedly achieved lower perplexity without attention. The speaker interprets this as evidence against a fundamental barrier to attention-free models, not a demonstrated advantage at 100 billion scale. The reported smaller scale is only “350”; its units, the comparison baseline, and the exact name rendered as “Base” remain unresolved.

Efficient sequence processing also leaves another problem: MLPs can become a bottleneck at larger model dimensions. These model components are not explained in this part, but slimming them down is identified as a separate efficiency task, with a pointer to Dan Fu’s talk. The architecture discussion closes by connecting model efficiency to classical ideas from signal processing and databases, which may also help rethink the systems built around foundation models.

Daily inference creates two different systems problems

By this late-2023 account, models had moved from demonstrations into daily use. Training once and running repeatedly had become an operational reality, giving inference—the repeated use of a trained model—new urgency. The speaker describes rapid improvements in commercial inference speed over the preceding three or four months, without benchmark measurements. Named examples of optimization work include MQA, GQA, and speculative decoding; their mechanisms are not explained in this part.

The workload determines what “fast” needs to mean: - Low latency: return an individual response quickly, potentially within milliseconds. - High throughput: complete a large aggregate workload—perhaps 100 documents or a million—when individual responses need not arrive immediately. Internal data cleaning is the concrete example of a throughput-oriented application. For that workload, the amount of work completed can matter more than the delay before any single result.

These workloads open questions across the surrounding infrastructure: - Programming: how should foundation-model workloads be expressed? - Hardware: which accelerators and other hardware suit them? - Underlying data systems: what systems of record should support them?

Closing the experimental recap

The host closes the NeurIPS 2023 coverage with an invitation to respond to its experimental format: live interviews, stage-talk audio, commentary, and backing music assembled into one recap. Listeners are asked to share feedback, share the episode with a friend, and participate in the listener survey. Planned follow-ups will revisit startups encountered at the conference, including returning guests.