Latent Space’s 2024 Year in Review — Part 2 of 3: From Model Demos to Daily Work

An image edit should take a request

Give a model a picture of a car and ask for a small modification. In a Gemini 2 image-output demo discussed in Latent Space’s 2024 review, the model returns the edited image directly. One speaker sees the appeal immediately: a natural-language request could replace manual masking and inpainting—selecting regions and regenerating their contents in workflows such as Stable Diffusion and ComfyUI. The demo shows a simpler interaction; it does not establish editing reliability across tasks.

That example gives the “multimodality war” a concrete shape: specialists in individual modalities compete with one model that handles many. Text-to-video has become a major battleground, with Veo 2 and Sora joining the field. ElevenLabs and Pika suggest that focused businesses still have room to succeed. One speaker judges specialists to be ahead for now, while expecting general models to catch up.

Midjourney’s advantage is the iteration workflow, according to a speaker who uses it for all their thumbnails. Its editor and variation controls make it easy to keep working. In their experience, the Black Forest workflow produced one image and required regeneration to continue. Calling the API four times, as the other speaker suggests, does not address the time spent managing those iterations. They agree that the distinction is product usability, rather than a demonstrated limitation of the underlying model.

Rankings answer a different question. Recraft V3 reportedly overtook Flux 1.1 in Artificial Analysis’s image arena, though the discussion gives no scores or evaluation details. For video, one speaker expects DeepMind’s work on Genie, Genie 2 and VideoPoet to benefit Veo 2, while cautioning that selected demo clips cannot establish its advantage over Sora. Generating audio from video and synchronizing the two are proposed as the next frontier.

Broad promises concentrate attention—and complicate deployment

The infrastructure discussion begins with a measurement problem: GitHub stars can outgrow adoption. One speaker prefers PyPI downloads as a signal and reports contrasting trends over the preceding six months: - LangChain and LlamaIndex: downloads continued growing, with commercial products suggested as a reason users stay. - CrewAI: stars rose while downloads remained roughly flat. A package split might explain that divergence, but neither speaker reports finding one. These are directional observations, without numerical download counts.

AutoGPT’s promise of generality illustrates the proposed mechanism. A goal as broad as increasing someone’s net worth attracts people with many unrelated objectives, concentrating their interest in one project. Satisfying those objectives makes deployment harder, while team execution remains another constraint. AutoGPT’s cloud offering reportedly still had a waitlist—a tangible gap between attention and availability.

Devin presents a similar availability gap: its March announcement preceded general availability by roughly nine months. One speaker expects a Devin-like product to succeed without assuming Devin will win. Their recommendation is to promise a recognizable task, such as code completion or pull-request review. They regard ChatGPT, and perhaps NotebookLM, as unusual successes for a broad product promise.

Agents need execution environments they can revisit

As products move beyond chat, code execution becomes an immediate infrastructure need. Together AI’s reported acquisition of CodeSandbox, E2B’s fundraising and Bolt’s web-container work all feature in this shift. Perplexity’s finance charts supply a concrete use case: the speaker describes Perplexity as an E2B customer using execution to produce richer output than a conversational answer.

The speakers identify a recurring agent toolset: - Web browsing to access information. - Code execution to perform work through programs. - Memory or planning, whose requirements remain less clearly defined.

Rewind and fork make branching execution possible. Creating a virtual machine, running code and destroying it serves a linear job. An agent exploring a tree of alternatives may need to return to an earlier state and fork it into several paths. The argument is that the execution environment should preserve those states, allowing the agent to revisit work rather than treating every run as disposable.

ChatGPT Canvas combines editing and execution. Following the 12 Days of Shipmas updates, one speaker considers it to have absorbed much of Code Interpreter’s role, though both still have separate toggles in custom GPT creation. They predict that diff-based editing will become standard in 2025. Anthropic, OpenAI and Fireworks are cited as having shipped diff modes, with Aider and Cursor as earlier examples.

Memory should learn from repeated refusals

An assistant repeatedly proposes a hike at 6 AM; the user repeatedly declines. Useful memory should infer a preference from that behavior, one speaker argues, without requiring an explicit declaration. Their criticism of contemporary memory products is that they mostly summarize or extract statements. Both speakers see the implementation as immature, while agreeing that the need is real.

The architectural distinction concerns the information’s role: - Knowledge concerns the world: company documents, an existing retrieval corpus or information found through external search. - Memory concerns interactions over time: what a system has experienced with a particular user. Either can use vector databases or knowledge graphs. Storage technology does not define memory.

Time introduces a need for decay and review. A sleep-like processing phase would revisit the day’s interactions, derive insights, persist them and bring relevant results into future context. LangMem reportedly had been incorporated into LangGraph as a memory service with the same API. The broader case is for reusable infrastructure: even a million-token context window only postpones the need to compress long-lived interactions and retrieve selectively.

The other speaker prioritizes user-owned, portable memory: people should choose which memories to bring into a new product instead of making every application relearn them. A protocol is proposed as a possible mechanism. Anthropic’s Model Context Protocol (MCP) reportedly includes a simple memory implementation of roughly 300 lines, but that example is not presented as an established standard for portable memory.

Standardization could also emerge from a community, with llama.cpp and ComfyUI offered as precedents. Roleplay is suggested as fertile ground because interactions run for so long. In one participant’s inspection of Silly Tavern and Kobold, users shared character cards in several formats but lacked exportable memory. That bounded observation points to an opportunity for continuity across products.

New capabilities need harder evaluations

The year-end benchmark discussion tracks a change in emphasis. MMLU, GSM8K and the first Hugging Face open-model leaderboard give way to SWE-bench, LiveBench, MMLU Pro and AIME, with o1 particularly associated with AIME. FrontierMath reflects growing attention to difficult mathematics alongside coding. Within software evaluation, the speakers emphasize SWE-bench Verified, rather than treating every SWE-bench version as interchangeable.

One speaker relays Noam Brown’s assessment that GPQA was effectively exhausted, while acknowledging continued attention to GPQA Diamond. The discussion reports SWE-bench progress from roughly 13% at the start of 2024 to about 50%, with OpenHands near that level. The benchmark variant and evaluation conditions are unspecified. An 80% result by the end of 2025 is a hope, not an achieved score.

MLE Bench evaluates machine-learning research automation, a potentially consequential capability for frontier labs. One speaker argues that researchers automating their own work could change the pace of AI progress. The accompanying advice is to watch which benchmarks labs emphasize: familiar tests may stop exposing remaining gaps, while new capabilities create a need for new evaluations.

Production readiness depends on the task

A proposed capability taxonomy starts with mature: broadly available and dependable enough for production. One speaker places general knowledge, long context, much of document retrieval for generation, and batch transcription in or near that category. These late-2024 judgments immediately invite finer distinctions.

The emerging category contains different engineering situations: - Multi-file code generation remains emerging even if simpler forms of generation are relatively mature. - Tool use is approaching maturity, with that year’s structured-output releases cited as a reason for optimism. - Vision-language models, including their use for PDF parsing, may already qualify as mature. The speaker acknowledges that the categorization is about two months old.

A hardware detour introduces another constraint. One speaker interprets reported enthusiasm for Blackwell as a positive sign for NVIDIA’s transition from H-series to B-series hardware. The discussion also identifies execution risk in a faster hardware cycle: moving from a two-year to a one-year cadence means a single misstep can delay delivery.

In a December 31 recording insertion, swyx continues with real-time transcription. He points to Whisper WebGPU, but says speaker diarization remains difficult, citing the work required on Latent Space transcripts. He relays Dwarkesh’s use of Gemini 2.0 Flash for diarization and suggests it might help with cross-talk, while retaining Pyannote as a conventional option. He offers no comparative accuracy result.

Swyx’s remaining categories separate demonstrable promise from established use: - Frontier capabilities lack settled everyday applications. Examples include long inference, interruptible real-time voice APIs, on-device models and additional modalities. - Niche capabilities retain interest without broad use. He places base models, state-space models and RWKV here, questioning how much their practical position changed during the year. There are concrete uses within the niche category: he says Latent Space uses Cartesia for projects including NotebookLM-style podcast clones and regards it as a challenger to ElevenLabs.

The cost of a fixed performance level keeps falling

The pricing retrospective asks how much a fixed level of measured performance costs. Beginning around Claude 3 Haiku’s March 2024 launch, the speaker tracks prices against LMSys Elo ratings. Their comparison places 2023’s GPT-4 and Claude 3 Haiku near 1175 Elo, at quoted prices of roughly $40 and $0.50 per million tokens. The input/output token mix is unspecified: similar Elo does not establish equivalent cost or quality for every workload.

The reported price–performance frontier moves repeatedly through the year: - July: Gemini 1.5 Pro surpasses Claude 3 Opus on both price and Elo in the comparison; GPT-4o mini and DeepSeek V2 appear as price–performance outliers. - August: an updated Gemini 1.5 Flash offers roughly the same Elo as its June version at a substantially lower price. - September: Gemini 1.5 Pro joins the frontier after halving its prices. The speaker interprets its alignment with competing models as evidence that the price–Elo relationship has predictive value.

The trend has exceptions. Claude 3.5 Haiku is described as four times the price of Claude 3 Haiku; a larger underlying model is only an unconfirmed explanation. Amazon Nova Pro, Lite and Micro occupy the efficient frontier around 1200–1300 Elo in the latest comparison discussed, but it excludes Gemini 2 Flash, Flash Thinking and several reasoning models. The frontier is a dated comparison, not a permanent purchasing guide.

The speaker’s central assessment is that cost reductions at fixed Elo accelerated beyond their earlier tenfold-per-year rule of thumb. H200 availability and Blackwell could contribute further, but the trajectory is difficult to predict: inference costs depend on more than GPUs.

January to March: larger contexts and credible alternatives

The review then rewinds through 2024. January brings Jeff Bezos’s backing of Perplexity, which reportedly reached a $9 billion valuation by recording time. February brings a more uncertain product story: confidence in custom GPTs had weakened, in the speaker’s view, though they still favor continued marketplace iteration. ChatGPT Memory and Gemini’s million-token context window stand out as technical developments from the period.

Larger windows qualify the earlier maturity claim: retrieving from a long input and reasoning across it are different capabilities. RULER is described as a multiple-needle retrieval evaluation, while reasoning over long inputs requires further evaluation. Magic.dev’s advertised 100-million-token model attracts attention, but the speaker says it remained unreleased and could not yet be assessed.

Claude 3’s March launch gives production users a credible frontier-model alternative, in the speaker’s assessment, with Sonnet becoming Anthropic’s workhorse. Devin’s announcement supplies a contrasting lesson about promises and delivery. By the review, it was reportedly available for $500 per month, with mixed reactions. Beyond delivering the feature, an application must withstand imitation: branding, product quality and engineering provide differentiation when successful features are copied.

April to June: new modalities meet product expectations

April’s music-generation discussion contains a small product lesson. The podcast experimented with Suno and Udio intro songs; the speaker also describes enjoying a later workflow using o1 to help craft lyrics. Enjoyment in creating a feature did not guarantee listeners wanted it. The intros were removed because some listeners skipped them, though the actual skipping rate was unknown.

April also brought Llama 3’s 8B and 70B models, highlighted as a significant contribution to openly available frontier models. May’s GPT-4o combined an efficiency improvement with a compelling demonstration of the omni-model direction, particularly native voice. The distinction matters: earlier GPT-4 models already supported vision.

June’s Apple Intelligence announcement represents an on-device deployment direction. The speaker describes a 3-billion-parameter foundation model running on the phone with hot-swappable LoRA adapters, and praises Apple’s technical disclosure. Their product verdict is restrained: decent, but not the transformative reason to upgrade an iPhone anticipated around the announcement.

A direct research mission or products along the way

Returning to the conversation, the speakers consider Ilya’s reported billion-dollar raise through two strategies: - Pursue one final product: a direct path to superintelligence provides focus and may avoid the cost of supporting intermediate products. - Ship along the way: the approach associated here with Tesla and OpenAI makes intermediate products part of the journey. The suggestion that a billion dollars might fund the direct path is speculation. A consequential question remains unresolved: where would its data come from?

The speakers connect Ilya’s departure to a broader exodus from OpenAI’s safety wing, citing Jan Leike and describing essentially the entire superalignment team as having left. The retrospective then returns to products: Artifacts is compared with Canvas, with the qualification that Artifacts is more code-oriented. June’s AI Engineer World’s Fair reportedly drew more than 2,000 people, excluding the speakers themselves.

The assistant moves into everyday work

An always-on assistant that sees the user’s screen is one speaker’s desired software experience: something that can converse while seeing what they see. They predict running it continuously on their own machine within a year, conditional on comfort with the privacy settings. Comparable assistance outside the screen, they expect, will take several more years.

The chronology continues through Llama 3.1 in July, structured outputs in August and o1 in September. The speakers rank o1 among the year’s most significant releases for opening new possibilities in inference-time compute. One praises the rollout’s speed, while remaining uncertain about the internal-demo date and exact sequence of ChatGPT and API availability. Their continued use of Sonnet for some work coexists with that assessment of o1’s significance.

Voice mode supplies a simple adoption anecdote: one speaker says their wife talks to it frequently. October’s Canvas returns the discussion to drafting. Another speaker uses it extensively as a document-editing environment with an assistant alongside the document, judging its AI integration better than Google Docs with Gemini for some editing tasks.

That experience grounds Part 2’s closing argument. The speaker sees Canvas and OpenAI’s search extension as a gradual challenge to Google through additions to existing workflows. Success remains uncertain, but the appeal is concrete: assistance alongside the document or search task already at hand. They welcome a challenger after what they regard as a long period of insufficient competition.