LLMs in 2024: Cheaper Capability, Harder Boundaries

A phone call with Santa

By the end of 2024, Simon Willison could talk to a model impersonating Santa Claus through his phone and show it what he was seeing. That playful example captured a consequential change: large language models had become multimodal, handling images and audio alongside text. They were also cheaper and faster, with larger context windows for the material they could work with. Even without a dramatic leap beyond GPT-4, those improvements expanded their uses.

The January 2025 Ride Home conversation takes up themes from Willison’s year-end post, Things we learned about LLMs in 2024. Looking ahead, he expects inference-time compute—spending more computation while answering a request—to help models such as o1 and o3 tackle harder problems, at greater cost and latency. A GPT-5-scale jump remains possible, but continued gains in price, speed and capability would be exciting enough.

The GPT-4 barrier gives way to efficiency

Willison counts 18 organizations with models that beat the older GPT-4 baseline: the version available roughly a year before the discussion, not necessarily GPT-4o. Several models he considers comparable to that baseline now run on his laptop. His assumption that such capability required a rack of expensive GPUs had broken down as smaller models became more capable.

Microsoft Phi-4 makes the change concrete: Willison ran a 14 GB download on a MacBook Pro. Its benchmark results looked competitive with GPT-4o, although his experience suggested weaker practical performance. DeepSeek V3 illustrated progress at a different scale. Too large for his laptop, it led the available open-weight models on the benchmarks he referenced. He recalls its initial Hugging Face release as a giant binary upload without a README.

The economics require separating training cost from prompt cost. Willison cites a reported $5.5 million training cost for DeepSeek V3, with no accounting scope specified. Separately, he describes OpenAI usage prices as roughly 100 times lower than GPT-3 prices about two and a half years earlier. Brian McCullough checks that this comparison concerns charges for running prompts, rather than training models; the contemporary model and pricing basis remain unspecified.

Gemini 1.5 Flash 8B combines the two trends. Willison describes it as 27 times cheaper than GPT-3.5 Turbo had been a year earlier, while offering image recognition and a million-token context window. That historical ratio comes without the billing assumptions needed to reproduce it. He also relays reports that Gemini and Amazon Nova can cover serving costs at low prices, while explicitly excluding training and research staff from that calculation. Covering prompt execution does not mean recovering development costs.

McCullough asks whether inexpensive competitive training undermines the expectation of escalating budgets and fewer organizations able to participate. The longer-term trend remains open. One hypothesis is that researchers had shifted from discovering capabilities to making known GPT-4-level capabilities cheaper, including removing weights that contributed little. If 2024 captured many easy efficiency gains at once, subsequent improvements could slow.

A pelican, a bicycle and visible reasoning

The reasoning-model discussion includes an important correction: a participant withdraws an initial claim about running DeepSeek R1 locally after mixing up models. In this January exchange, R1 is described as available through an API. The laptop examples concern Alibaba’s Qwen models, QWQ and QVQ; QVQ accepts images so that visual puzzles can accompany a prompt.

The Qwen models emit dozens of paragraphs of intermediate reasoning text, whereas o1 hides that process. In one reported experiment, QWQ produced extensive Chinese text before delivering an SVG drawing of a pelican riding a bicycle. The participant liked the drawing; translating the preceding text revealed a discussion of SVG files. This is a distinctive output behavior, though the visible explanation need not faithfully describe the model’s internal computation.

Agents need a definition and a trust boundary

Agent can mean an LLM using tools in a loop, perhaps on a schedule, or a service trusted to book travel on someone’s behalf. Those definitions create different expectations. Here, the provisional definition is a system that accepts a job and goes away to perform it. Under that definition, reliability depends partly on whether the system can resist misleading information encountered along the way.

Prompt injection exploits that weakness by placing malicious instructions where the system will encounter them. A participant recounts an adversarial demonstration against Claude Computer Use, whose Docker-based environment could open a browser and click through tasks. A webpage reportedly persuaded it to download and execute malware that joined a botnet. Material encountered during the assigned job had become authority to perform another action.

Narrower agent patterns receive more confidence. A Gemini Deep Research beta earns qualified praise for gathering webpages, placing their contents into context and producing a report. One participant describes useful answers from this process, with 56 websites offered as an example of its reach. Search and long context make the workflow practical, but deliberately deceptive pages could still mislead it.

ChatGPT Code Interpreter supplies a concrete feedback loop: 1. Write Python for the task. 2. Execute it. 3. Inspect errors. 4. Rewrite the code and try again. The participant endorses this as an already useful pattern. Their skepticism concerns trusting a system to make autonomous decisions, particularly purchases. A Stripe agent toolkit prompts a suggestion to cap a virtual spending card at $50: a way to bound financial loss, without making the decisions themselves reliable.

Travel booking also tests the value of delegation. One participant estimates that automating their flight-search input would save about 15 seconds, while they would still want to compare options and reject certain airlines regardless of price. The product question is therefore which work the user wants to hand over, including which decisions they want to retain.

Long context meets eyes and ears

NotebookLM combines a tool for searching supplied material with a podcast generator. A participant relays an account of two internal agents behind podcast generation, though calling that an agent loop depends on the definition. The synthesized voices draw praise; the formulaic introductions and conclusions draw criticism. Convincing delivery and strong composition are separate achievements.

The podcast feature also illustrates a distribution advantage: users can share an output that demonstrates the product. The discussion compares this with Midjourney, whose shareable images can attract attention to the tool that made them. NotebookLM’s voices brought visibility to an existing research product.

Video understanding has a useful decomposition too. A participant describes a common mechanism as sampling one frame per second and placing those images into a long context. Vision plus long context lets a model work across a sequence of frames. But frames omit sound: early Gemini 1.5 Pro reportedly ignored the audio track. Processing images and audio together would preserve information that either stream alone misses.

For a user, the change can be as simple as turning on a phone camera during a conversation and asking about a tree. A participant reports doing this in the ChatGPT iPhone app, while leaving open whether the implementation samples still images. Cheap image processing also makes larger jobs worth considering: one calculation estimates $1.68 to caption 68,000 personal photographs with Gemini 1.5 Flash 8B. This was a cost estimate, with caption length and other billing assumptions unspecified, rather than a completed run.

Generated media earns its place in a workflow

Generating video remains comparatively expensive, even as consuming it becomes cheaper. Veo 2 and Sora also expose a comparison problem. One participant prefers Veo 2 after experimenting in its beta, including with pelicans on bicycles. Another objects to comparing selected Veo marketing examples with poor outputs from the public Sora version, which they describe as distilled. They allow that Veo 2 may still be better; the disagreement concerns both quality and the conditions under which it is judged.

The creative opportunity is easier to explain through individual production tasks. One participant invokes Everything Everywhere All at Once, describing a five-person visual-effects team whose members sometimes learned techniques from YouTube. Another reports that the team used Runway ML, but does not know which features or how extensively. The forward-looking question is what skilled teams could accomplish with generated effects shots and other short production elements.

An asset’s role determines its demands. Background footage, music and sound effects may be useful even when a model struggles with foreground action. A participant cites a flawed Sora gymnast clip to explain why consistency failures become conspicuous when viewers are watching the main subject closely. Short inserts and background elements offer other places for creative teams to select and use generated material.

HeyGen supplies a narrower, existing workflow. A participant records an MP3 in their own voice each day, feeds it to a trained avatar and gets a lip-synced talking-head video. They judge the result convincing for casual viewing, though imperfect. The value is removing camera setup, lighting and some editing from a human creator’s process. Predictions of autonomous AI influencers meet more skepticism: novelty can attract an audience without sustaining it.

Credibility belongs to the publisher

As generated content becomes abundant, the discussion places more value on editorial responsibility. A model’s answer depends on prompting; a publisher can stake a reputation on deciding that an output is accurate, useful and worth sharing. Generating alternatives, selecting one and publishing it under a human name still requires judgment. Choosing among five variations changes the creative task from starting with a blank page.

One proposed definition of AI slop is content that is both unrequested and unreviewed. Meaningful review requires someone to decide that the result deserves other people’s time and to attach their credibility to it. Under this standard, responsibility for the published result matters even when AI helped produce it.

Make the interface part of the conversation

The blank chat prompt offers little guidance about what a model can do. A questioner compares it to dropping a new computer user into a Linux terminal. ChatGPT Canvas offers a more concrete interaction by letting a person and model work on the same document. TL Draw’s make it real takes a drawn interface as input and generates an implementation.

Claude Artifacts demonstrates another approach: answer a request with a small HTML and JavaScript application. The proposed next step is to close the interaction loop. A model could generate a map to ask for a location, or sliders to capture preferences, then receive the user’s selections as input. In the January 2025 account, that connection remains unfinished: Artifacts can generate knobs and dials, but Claude cannot observe the user manipulating them.

Bolt receives praise for generating attractive app designs from prompts for Spotify or Airbnb clones. A proposed data-exploration workflow combines generated charts with a form that inserts rows into a database table. One participant predicts that custom interface generation will become an expected web-application feature within six months—a forecast made at the start of 2025.

This produces a useful disagreement: individual tasks can get easier while the whole system gets harder to understand. One participant now asks Gemini to write Google Sheets formulas. Another points out that every added feature introduces boundaries that may be undocumented, and that models explain their own limits poorly. They describe users unsuccessfully trying to build Artifacts that call external APIs. Whatever the underlying restriction, a plausible-looking generated interface had concealed an important capability limit.

Local models must share the computer

One participant had nearly lost interest in local models because hosted Claude 3.5 Sonnet was so much better. Improvements in the preceding months made local models useful again, even while they still lagged Sonnet. Memory remained a constraint: running a large Llama model on a 64 GB machine required closing browser tabs and VS Code windows. Fitting the model is only part of the requirement; the computer must still support the surrounding work.

Apple Intelligence raises a related distinction between capability and discovery. Participants criticize model quality and the lack of guidance about when to use its features, while notification summaries receive mixed reactions. Small-model progress inspires hope that Apple could improve within six months. A separate prediction imagines Gemini Nano shipping through Chrome to give web applications access to a bundled local model. Both are expectations expressed in early 2025.

For trying local models, the recommendations offer several entry points: - MLC Chat on iPhone, demonstrated with a Llama 3B model producing Christmas movie plots and pun titles, though its recommender does not find it capable enough for their practical work. - Ollama, which packages models and exposes an API, including access through a plugin for the LLM command-line tool. - LM Studio, recommended for a graphical interface.

Disk space is another practical cost. One participant’s worthwhile small-model downloads start around 2 GB, while the models they find especially impressive tend to occupy 20–30 GB; these are personal observations, not universal requirements. Local experimentation also coexists comfortably with hosted API use.

Useful AI removes a specific piece of work

A Super Whisper user describes holding Caps Lock to dictate and releasing it when finished. Speech recognition captures the words, then GPT-4 rewriting removes fillers and restatements. That second step turns speech into usable long-form input. Rosebud is also recommended for journaling, with affordability central to its appeal; this is a personal recommendation, without clinical outcome evidence.

MacWhisper supports a compact publishing workflow: transcribe a YouTube video or MP3, paste the transcript into Claude and ask for show-note suggestions. Claude returns a list of mentioned items that may deserve links. The combination turns an existing recording into an editorial starting point.

Automated podcast editing supplies a concrete personal saving. Instead of manually cutting three recorded tracks, a participant uses a smart-edit feature to select the active speaker or show the group during overlapping speech. They report saving three or four hours per posted recording and say Riverside offers the feature. Another participant continues to use a human editor: the choice depends on the workflow and the owner’s standards.

Better criticism requires understanding usefulness

Asked about OpenAI’s prospects, Willison expresses concern about apparent talent losses and a shrinking competitive lead. He credits o3 with restoring some of its position, praises Gemini’s progress during 2024 and names Claude 3.5 Sonnet as his personal favorite. His assessment reflects a year in which credible alternatives had spread across providers.

His call for better criticism holds usefulness and harm in view together. He regards environmental impact and unlicensed training data as legitimate concerns, while rejecting blanket claims that LLMs are useless. He reports using them dozens of times daily, with value dependent on learning their strengths, weaknesses and tendency to fabricate information.

Willison also separates legality from fairness. In his view, unlicensed training is probably protected by fair use, yet taking someone’s work without permission to build a competing model can still be unfair. That legal assessment is his opinion. More precise discussion matters because people—including artists and lawyers—need to make career decisions as these tools affect their professions.

The respondent proposes regulating how LLMs are used, with an insurance claim denied by a model whose decision nobody can explain as the concrete concern. They suggest reinforcing and updating existing laws for such cases, alongside understandable privacy rules that would prohibit training on users’ inputs under specified circumstances. The aim is to let people understand those protections without navigating complex terms and conditions, addressing fears that private inputs could become training data and later be exposed to others. These are proposals for stronger protections. The respondent also cautions that poorly designed regulation can have unintended consequences, citing EU-style cookie banners as an outcome they would want to avoid.

Screens, wearables and the next work to build

Willison’s near-term interests require no new breakthrough. Inexpensive inference, streaming video and screen sharing already offer enough to explore for the coming year. He has experimented less with screen sharing but expects it to be useful. Its privacy requirement is concrete: he does not want a model trained on everything visible on his display.

Swyx predicts a stronger year for AI wearables, despite negative sentiment around Rabbit R1 and Humane. His proposed use case is workplace memory assistance: extend tools such as Granola and Fireflies from online meetings into in-person conversations. He describes Limitless as recording only an opted-in wearer’s voice, though he supplies no mechanism or reliability results for that filtering. The acceptable boundaries of recording remain unresolved.

Another participant imagines earbuds and smart glasses connected to an LLM through a phone. Willison connects that hardware possibility to 2024’s software gains: affordable audio, image and video processing makes the software behind wearable assistants substantially more feasible than a year earlier.

The closing announcements return to concrete building work. Swyx introduces an agents-focused AI Engineer Summit in New York for February 20–21, 2025, with a leadership day followed by a day for hands-on engineers. The announced program includes lab presentations and Instacart’s work on recommendations, background jobs and internal jobs. Willison describes plans for his open-source data-journalism tools: plugins that use LLMs to craft queries and build dashboards.

The exchange closes by pointing listeners to Willison’s blog and Techmeme Ride Home, a 15-minute technology-news podcast released each weekday.