Worthwhile Research for building SOTA LLMs (Jan 2024 Recap)
5 directions we think will stand the Test of Time: Search, Synthetic Data, Subquadratic LLMs, MoEs and Merges... Also: our usual highest-signal recap of top items for the AI Engineer from Jan 2024!
This is the sixth in our new monthly recaps for supporters. We’re leaving Dec 2023 open since it got shared a lot. See the rest of the month-by-month must-reads here!
Please join us on Feb 23 for Latent Space: Final Frontiers, and June 25-27 for the biggest AI Engineer conference of the year!
The most challenging problem for people keeping up on AI is sifting through signal vs noise1. Given your total useful knowledge is, by definition, the cumulative sum of still-relevant knowledge2, your greatest (possibly only) proxy for useful signal should be the “test of time” estimator: how likely are you to still be caring about3 {a given model, paper, technique, library, or news item} in 1/3/12/60/120 months from now?
Everything changed, yet nothing changed
With that lens, January 2024 was a month where it felt like a lot happened…
The Rabbit R1 “won” CES by earning a ton of press and selling out of multiple production runs, despite criticism of both the presentation style and skepticism on practical viability of the tech and business model. Tab made a smaller splash.
Perplexity “crossed the chasm” with a critical mass of hype building around their Series B funding announcement, most notably backed by Jeff Bezos. Lepton made a well received open source clone.
OpenAI released a wealth of updates across GPT3/4 Turbo, embeddings, its API key platform, the GPT Store, and made nontechnical moves around the NYT lawsuit, 2024 elections, and collaborations with the military.
Open source code models took several leaps forward in each LLM weight class, with Meta releasing CodeLlama 70B, and the Chinese labs releasing WizardCoder 33B, and Deepseek Coder 7B (AlphaCodium notwithstanding)
ElevenLabs became the newest AI unicorn with a $80m Series B, the latest major player in the Multimodality war (not that it surprises anyone watching!)
LangChain crossed a major landmark with its first v0.1 semver release, pushing through a major rearchitecture of core vs community packages (LlamaIndex did the same with v0.10)
>30,000 tech workers got laid off including at Google, Microsoft, PayPal, SAP, Unity, Twitch, Citrix, Salesforce, Snap, Docusign, Amazon, iRobot, Brex, Discord…
…and yet it’s not clear that the fundamental assumptions of the AI landscape have budged at all in 2024 vs 2023:
Keep reading with a 7-day free trial
Subscribe to Latent Space to keep reading this post and get 7 days of free access to the full post archives.