Where AI Progress Could Come From: January–February 2024 Recap, Part 1 of 3

Long inference: what should a model do with ten minutes?

A report that takes ten minutes to generate poses a different product question from an immediate chatbot response: is the result worth the wait? Alessio describes trying such a product in this January–February 2024 recap. The attraction is that users can choose how much computation to spend on a task. A developer can retain a shorter training cycle while customers pay for additional work where the outcome matters.

That possibility leads five research directions, ranked in order: long inference, synthetic data, alternative architectures, Mixture of Experts, and online large language models (LLMs). The ranking favors ideas likely to endure beyond daily news cycles, before citation counts establish their influence. It is a judgment shaped by January's developments, not a comprehensive research agenda.

Long inference gives a system more computation when performing a task. It includes Code Interpreter, step-by-step prompting, and flow engineering: chaining operations across multiple turns and incorporating feedback. Instead of putting the whole task into one prompt, an engineer organizes a workflow that can revise its work. SGLang from LMSYS earns a recommendation for expressing these flows concisely; DSPy is also considered useful, though one speaker finds its abstractions less transparent.

The scaling argument concerns remaining room to grow. GPT-3's reported 175 billion parameters and 300 billion training tokens give way in the discussion to DeepMind's roughly 20-token-per-parameter compute-optimal ratio, then Llama-related regimes using 200 or sometimes 2,000 tokens per parameter. As attention shifts from model size toward data, another dimension becomes attractive: time spent solving a task. An illustrative three-month training run might become three years, but thirty years becomes impractical. Inference could instead extend to hours, days, or months. That headroom is a research opportunity, not a demonstrated quality scaling law.

Extra time must buy useful search, planning, or feedback; merely extending autoregressive, token-by-token generation is argued to have limits. AlphaGeometry provides a consequential example: the discussion describes geometry solutions exceeding 100 steps and performance at a human Math Olympiad gold-medalist level, without specifying the evaluation protocol or runtime. Alessio's product criterion is more immediate: improve performance for the customer's application, rather than pursue an absolute state-of-the-art label.

Preserve successful plans instead of rediscovering them

Expensive inference creates a reuse problem. Stochastic models produce different plans; once someone finds a successful procedure, others could benefit from storing and sharing it. Alessio proposes an “NPM for skills”—a package library for reusable agent capabilities and plans. Voyager's procedure for obtaining a diamond pickaxe in Minecraft illustrates the idea: a successful plan becomes an artifact another agent can reuse.

The exchange separates three levels of reuse: - Skills share integrations and capabilities. Building the integration tooling can still be difficult or expensive. - Plans depend on the user's environment. Different needs and preferences make personalization a central obstacle. - Meta-plans share a way to construct plans. A template for planning and testing may transfer more readily than a finished procedure.

A suggested meta-plan specifies a role, intention, requested actions, and prohibited actions. Multiple templates could then be tested with task-specific details filled in. LangChain and LlamaIndex hubs are cited as attempts to make prompts reusable, although one speaker doubts their adoption because people prefer writing their own.

Synthetic data: verify candidates and preserve variation

Synthetic data is generated training material. The opportunity emphasized here is self-improvement without a teacher model: an LLM generates data that helps improve itself. That differs from the practice associated with 2023 of collecting GPT-4 outputs to train another model. It also differs from using a constitutional or more aligned judge model to provide feedback.

Verification makes math and code favorable starting points. Generate candidate solutions, check correctness, then train on accepted candidates. The case becomes less clear where correctness is fuzzy. Another route is rewriting existing material: the Apple work discussed as WRAP, or rephrasing the web, reportedly used Mistral to rewrite the C4 dataset and trained on the rewritten corpus. The reported benefit is faster, cheaper training than raw C4, without a quantitative comparison or evaluation conditions.

Rewriting may also remove useful variation. If generated text consistently cleans up misspellings, a model trained on it might lose opportunities to learn how incorrect spellings relate to correct ones. The speakers connect repeated loss of variation with mode collapse. The typo example is a hypothesized failure mode, not a measured result from the Apple work. It makes the central uncertainty concrete: can synthetic data sustain improvement, or will its benefits reach a nearby ceiling?

Improving a pretrained model and pretraining entirely on synthetic data are also different ambitions. The latter remains an unrealized step at substantial scale in this discussion. A simpler response to data scarcity is to train again on existing tokens. OLMo reportedly used 1.5 epochs—passes over its data—and the work referred to as Datablations is said to show continued loss improvement through roughly four epochs. The methodological point is stronger than any universal epoch recommendation: exhausting a compute budget does not establish that another pass would stop helping.

Alternative architectures: sequence length is not retention

Mamba, RWKV, and Monarch Mixer anchor the alternative-architecture discussion. Together and RWKV are seen as potential sources of leading models, amid concern about how few independent groups pursue these approaches. February complicates the January thesis: Gemini's long-context capabilities challenge architectures whose principal attraction is efficient processing of long sequences. Sora also brings diffusion transformers into focus; alongside Stability's Hourglass Diffusion, they represent attempts to mix architectural approaches whose eventual scaling success remains open.

Two distinctions govern the long-context argument: - Attention complexity versus total cost. Attention's quadratic dependence on sequence length need not predict overall inference cost: it may account for only a small share of computation, and implementation techniques affect practical scaling. GPT-4's expansion from 8K to 32K context without a sixteenfold price increase is offered as an illustration, not a controlled cost measurement. - Sequence capacity versus retention. Processing a long input does not establish that a model can recover its details. The discussion relays criticism that Mamba and RWKV's limited hidden memory can cause forgetting.

Two further research bets broaden the category. Text diffusion is proposed as a different approach to text generation, reasoning, and planning, motivated partly by an anecdotal report of work at Midjourney rather than a demonstrated result. Contextual AI is described as incorporating retrieval inside the language model, with RETRO cited as related work. Retrieval-augmented generation (RAG) supplies retrieved information to support a model's answers; moving retrieval inside the model changes where that capability sits.

Retrieval creates an ongoing maintenance obligation: the information must stay current. A fixed model can change behavior when its retrieval data changes, so maintaining the corpus becomes part of maintaining the system. The speakers also reject an investor's promise that Contextual AI's approach would solve hallucination.

Experts, model routing, and weight merging

Mixture of Experts (MoE) routes work among expert components. January's DeepSeekMoE is described as using 64 smaller experts plus one or two always-on experts for common knowledge. The distinctive idea is to share common knowledge while other experts specialize, rather than route all knowledge through specialized components. The discussion highlights possible benefits for inference and memory retention during training.

DeepSeekMoE's reported improvement over compared open-source models comes with material limits: Mixtral was excluded, benchmark scores are not given, and the parameter comparison does not distinguish total from active parameters. One speaker rates the direction as medium potential, expecting a useful gain from converting a base model into an MoE but limited scope to repeat that gain indefinitely.

SambaNova's Samba-1, described as a trillion-parameter combination of existing models, raises the possibility that independently developed specialists could contribute to one system. Its architecture remains unresolved in the exchange, which separates several mechanisms: - Routing among models selects existing specialists, potentially with less joint training than a conventional MoE. - Connecting models through an adapter links components such as a vision model and a language model. The reported inclusion of CLIP prompts this question, without establishing how Samba-1 uses it. - Merging model weights combines model artifacts, with stacking and grafting discussed as variants aimed at combining complementary strengths.

Weight merging attracts attention because the merging operation can be cheap: adding, dividing, and otherwise performing simple arithmetic on weights can run without GPUs. One proposed explanation for apparent gains is regularization—combining weights might reduce overfitting and improve generalization. That remains an intuition, with limited gains expected.

Online LLMs: freshness can correct or bias an answer

Online LLMs add search access. One speaker attributes Gemini Pro's reported January rise above GPT-4 Turbo on LMSYS to Google Search integration and better answers about recent events, though no controlled comparison isolates search's contribution. The broader proposal is an emerging baseline for model offerings: - Chat or instruction tuning for responding to user requests. - Structured output through JSON and function calling. - Online access for searching beyond the model's existing knowledge.

The suggested system pairs RAG for internal knowledge with online search for external information, citing Perplexity and Exa as search providers. Search alone does not guarantee strong overall preference rankings: Perplexity's roughly thirtieth-place LMSYS ranking is offered as a counterpoint. The useful question is task-specific—news and social-media work especially benefit from recent information, while many other tasks may not need it.

Freshness can correct stale knowledge or overemphasize fashionable topics. Choosing an offline model can silently exclude relevant developments when the user does not realize recent events matter. Online access can challenge outdated knowledge, but it can also steer a research answer toward whichever topic dominates today's headlines.

Sora: convincing motion versus dependable physics

February exposes an omission in the five-direction framework. Its author says the piece appeared on February 14, just before Sora on February 15, and acknowledges repeatedly underrating multimodality. A focus on reliable reasoning and agents had made image and video generation seem less central. Sora prompts a reassessment of their cultural and human impact.

The optimistic argument is that depicting motion requires capturing physical regularities beyond those needed for a static image. Water splashing around a ship provides the example. A more demanding next step would be to interpret a video and explain why the ship rocks excessively or sinks—potentially useful for industrial work such as oil-rig deployment. Generating motion and diagnosing its causes are different capabilities; the proposed diagnosis has not been demonstrated in the discussion.

The skeptical response distinguishes visual continuity from physical consistency. The speakers describe fingers, limbs, and chairs appearing or disappearing in generated video. Such failures limit the case for a dependable world model, a representation of how the physical world behaves. Jim Fan's description of Sora as a data-driven world model is recalled as a vision, with data requirements, generalization, and tolerable error still unresolved. Producing convincing video may not produce a useful physical model.

Synthetic video could connect generation to understanding by providing training examples for video-understanding models. A thought experiment asks whether a video model available before Newton could reveal useful regularities in falling apples. Models might observe enough examples to expose patterns individuals miss, although turning those patterns into science would still require substantial formalization. Game engines offer another potential benefit for synthetic datasets: generating labels alongside examples.

Sora is judged a substantial advance over the earlier short clips and Runway Gen-1/Gen-2 results discussed, with practical applications still to be demonstrated. For builders, control over generation is a separate product concern. One speaker predicts that a future Sora API will offer limited control and argues for an open alternative enabling ComfyUI-style pipelines. Stability's intention to build a competitor remains an ambition, accompanied by uncertainty about resources for simultaneous work on code, language, and video models.

Groq math: fast inference needs sustained utilization

Groq's inference hardware shifts the discussion from model capabilities to economics. Its viral demonstration reportedly runs a model called Mistral at roughly 500 tokens per second, without specifying model size, concurrency, or benchmark methodology. The investment question is whether fast, high-volume inference can repay a larger hardware commitment.

A speaker's spreadsheet comparison for hosting Llama 2 assumes a five-year hardware lifespan: - Eight H100 cards: $300,000 upfront and $60,000 in annual depreciation. - 576 Groq cards: $2.3 million upfront and $460,000 in annual depreciation. These figures imply about a 7.7-fold capital-cost difference. An earlier assertion of a thirtyfold difference uses an unreconciled comparison basis.

The architectural explanation trades memory placement against hardware count. Keeping model data in on-chip memory, described as SRAM rather than HBM, reduces memory-transfer time but requires many cards. Compiler-based scheduling or routing is offered as a second source of higher utilization, although its precise mechanism is not established. The economic argument depends on enough concurrent work on the same model to keep the hardware busy.

To make reported pricing of $0.27 per million tokens plausible, the spreadsheet assumes 500 simultaneous requests and 80% model bandwidth utilization. Under those favorable conditions, the estimated base cost falls into the twenty-something cents per million tokens. The 80% figure is an input, not a measurement.

Another speaker argues that continuous full utilization would favor Groq while 50% utilization would favor Nvidia; these are judgments, not measured crossover points. Dylan Patel's more pessimistic view of the economics is relayed secondhand. The favorable case remains conditional: low token prices could work if the system sustains the assumed demand and utilization.

Gemini's million-token context meets per-token billing

Part 1 closes with Gemini 1.5 Pro, described as accepting video, audio, PDFs, and text within a million-token context window. That capacity is attractive for prototyping because more material can go directly into the model. It also raises the question of whether long context could replace retrieval-augmented generation.

Context capacity still carries a per-token cost. One speaker argues that RAG has better economics than filling a large window, while making the comparison dependent on how prices change. The longer-term choice balances two pressures: - Retrieval complexity can grow as more data sources are added. - Token costs for a fixed model are expected to fall, making larger inputs more affordable; that decline is a prediction.