The Four Wars of the AI Stack

A new model still has to fit the workflow

A new code model appears, but a developer working comfortably in VS Code, Cursor, or GitHub has no easy way to try it inside the existing workflow. Alessio offers this explanation for why experimentation with code models seems less fluid than experimentation with text models. Model availability does not automatically produce adoption. Integration work and satisfaction with the current tool can outweigh curiosity about the next release.

That friction sets the tone for the December 2023 audio recap, recorded in mid-January 2024. Drawing on the written December recap, the hosts organize the stack into four competitive fronts: data quality and quantity, GPU-rich versus GPU-poor participants, multimodality, and retrieval and operations tooling. Their framing follows money, people, power, and impact.

Alessio sees a shift during 2023 from LangChain, LlamaIndex, and vector databases toward Llama 2 fine-tuning and Mixtral. swyx leaves agents outside the four main fronts because he judges their momentum lower than at the start of the year. He treats competition to serve open models as a compute business battle rather than making open source a separate category. These choices describe that moment’s competitive attention; omitted subjects could still matter.

The production implication is less glamorous than the demos: AI engineering can become data engineering and traditional machine-learning engineering. swyx expects work hidden by a compelling demonstration to become part of the engineer’s actual job.

The data war: quality, permission, and synthetic supply

Alessio credits Jeff Huber at Chroma with the analogy of low-background tokens: older internet text as a comparatively clean source of human writing, before widespread model-generated material complicated new web crawls. A Common Crawl update may now mix human and generated text. Filtering obvious model self-identification phrases is a basic response, but the larger change is conceptual: a familiar dataset source does not make every item good training material.

Quality intersects with ownership. The New York Times lawsuit against OpenAI and emerging data partnerships turn attention toward consent, attribution, and compensation. The hosts describe a spectrum of positions: - Unrestricted permission: creators may welcome training on their work; the hosts put their podcast transcripts in this category. - Recognition or payment: creators may seek disclosure, attribution, compensation, or a combination. - Opposition and removal: some object to training itself or seek destruction of affected artifacts.

One host predicts that the Times case could reach the Supreme Court and shape fair use for AI, with harm to the original content’s market as the central contested issue. That host supports compensating creators while worrying that a ruling against OpenAI could substantially slow AI progress. The difficulty is both legal and empirical: suspected market harm would still need to be proved.

A host who moderated Rap Genius recalls music-label objections to its reuse of lyrics, followed by licensing deals. The hosts also remember distinctive alterations in its lyrics corpus appearing in Google’s results, exposing copying. They disagree about the exact alterations and are unsure whether Google ultimately paid. Detecting reuse and resolving commercial rights are separate problems.

One host argues that a dispute confined to a particular publisher could prompt developers to substitute competing publications. The same host criticizes restrictions around user-generated content on Reddit and Stack Overflow: attempts to capture value for individual platforms could reduce the collective value of training on mixed sources. The tension is between a platform’s incentive to control its corpus and the benefits of combining it with others.

Two approaches could create additional training material: - First-party human data: Alessio proposes asking employees to write five question-and-answer pairs each, accumulating a distinctive dataset. He supplies no measured training result. - Synthetic data: model-generated examples could supplement scarce human material. Another host reports Luis Castrejón’s ambition to create fully synthetic datasets reaching a trillion tokens, but does not understand the proposed workaround for dependence on the generating model.

Verifiable answers offer a selection criterion for synthetic data. One host recalls a DeepMind paper discussed at NeurIPS in which generated math and code answers could be checked for correctness. The unresolved question is how to extend that approach to tasks without checkable answers. Correctness also leaves a second problem: examples can be individually right while repeating the same phrasing and structure.

The hosts describe this loss of variety as a concern about mode collapse. Resampling a flawed model distribution introduces no new human information. Yet one host proposes a different objective: synthetic data might reshape the distribution toward useful assistant behavior, rather than more faithful imitation of human speech. Correctness, diversity, and usefulness remain distinct goals.

The inference war: service conditions determine the comparison

Mixtral’s release sparks the inference discussion: a rapid price war, followed by a dispute between Together and Anyscale over whose service was faster and whether the benchmark represented real performance. The actionable criticism is that demo access may not represent production infrastructure. One host alleges that Anyscale’s comparison missed this distinction.

For an enterprise-scale load test, that host recommends: 1. Notify the provider about the planned load. 2. Obtain the appropriate API key and endpoint for the production service being evaluated. 3. Test that infrastructure, rather than assuming public trial access represents it.

Reproducing comparisons adds friction: a customer may need accounts across several clouds, whereas a Postgres benchmark can be easier to reproduce locally. The hosts argue that this gives published AI benchmarks unusual influence. They welcome Artificial Analysis, an independent service probing production API endpoints, while recognizing the work required to keep comparisons current.

Provider selection requires minimum acceptable service levels across several dimensions: - Latency: responses must arrive quickly enough for the application. - Uptime and reliability: dependable operation must be demonstrated over time. - Throughput: the service must sustain the workload. - Price: cheap tokens are useful only when the service meets critical requirements.

Using Dylan Patel’s Mixtral write-up as an input, one host estimates a minimum serving price of $0.50–$0.75 per million tokens under favorable assumptions. Those assumptions are aggressive amortization of hardware expenditure, 50% utilization, batch size 16, and generation at 100 tokens per second. The host explicitly rejects the suggestion that this assumes maximum utilization and considers the upper end more realistic.

The quoted prices at the time are Perplexity at $0.56 per million output tokens, Anyscale and OctoAI at $0.50, Abacus AI at $0.30, and DeepInfra at $0.27. The host interprets prices below the estimated break-even point as loss-making. The loss-making conclusion depends on the cost assumptions: provider-specific costs are unknown, and input-token treatment is not consistently specified across the comparison.

If low prices are deliberate subsidies, where does the profitable upsell happen? A retail loss leader gets customers through the door, but the equivalent in inference is unclear. One hypothesis is integration retention: a customer adopts Together for cheap Mixtral inference, then stays when a newer model offers better unit economics. The hosts leave that strategy speculative.

Sparse models save computation, but still need memory

Mixtral illustrates a serving challenge for mixture-of-experts models, or MoEs: only some weights are used at a time, while all weights remain loaded in the setup described. Less active computation does not remove the memory requirement. The hosts expect this mismatch to change hardware needs, workload handling, and batching optimization, further separating training requirements from inference requirements.

Fire Attention provides a concrete example of specialized optimization. Fireworks reportedly built a custom CUDA kernel for Mixtral on H100 and announced quantization—reducing numerical precision—from 16 bits to 8 bits without performance loss. That reported result comes without a metric or evaluation conditions. The broader engineering burden is clear: inference providers must improve developer experience while writing low-level kernels and confronting memory-bandwidth limits.

One host predicts that specialized batching, caching, and MoE optimization could change pricing beyond separate input- and output-token rates. No replacement scheme is proposed, but exposing these efficiencies through a service could give providers a way to compete beyond advertised price.

The GPU-poor side of the competition consists of approaches that reduce dependence on large GPU fleets. Alongside efficiency and distillation methods, the hosts highlight Apple’s MLX for on-device inference: running models locally can avoid cloud inference altogether.

Mamba makes efficiency the argument for an alternative architecture. One host had found extreme context length less compelling because 100K context, retrieval, and recursive summarization already served their needs. Mamba draws interest through a different comparison: less compute at the same useful context length, or more performance within a fixed compute and model-size budget. This is a qualitative assessment, without comparative measurements in the discussion.

The architectural debate prompts a philosophical distinction. The bitter lesson, as described here, favors scalable general-purpose systems over domain-specific rules. One host’s proposed sour lesson says human cognition need not be the blueprint for machine intelligence: an airplane can take inspiration from a bird without copying its mechanism. The host explicitly concedes that this caution about recurrent and state-space approaches is philosophical, not a rebuttal of their mathematics or empirical results.

Multimodality: easy experimentation can create new markets

The discussion of models working beyond text begins with Stable Diffusion, remembered for bringing image generation into people’s homes. The hosts regard SDXL Turbo and Video Diffusion as incremental relative to that earlier impact. Their attention then moves to Midjourney’s commercial uses through freelancers, including product-placement imagery made with an apparently consumer-oriented product.

DALL-E 3 exposes a disagreement about usability. One host remembers an attractive Colosseum-with-llamas image from a simple request, then finds later results less appealing in the combined GPT-4 experience. The other suggests prompting skill may explain the difference. The dissatisfied host counters that automatic prompt improvement was part of the product’s appeal, while allowing that unlucky generations could explain the experience. This remains a personal quality judgment, not a measured regression.

The broader thesis is citizen art: people who would not normally commission art may pay modest amounts for generated images they find good enough. One host argues that conventional estimates of the existing art market miss this new demand. The related multiverse-versus-metaverse thesis contrasts digital scarcity with generating another personally satisfying image by changing a seed. Abundance could change where value accrues, though the host does not claim to have solved its monetization.

Voice has a different constraint. One host listens almost daily to an ElevenLabs-generated Hacker News recap but questions speech’s low information rate. Another reports that a spouse talks to ChatGPT while driving. A possible question about supplies for replacing a sink, asked on the way to Home Depot, illustrates access when hands and eyes are occupied. Adjustable playback speed could also make listening more useful.

Useful technology does not necessarily provide a repeatable sales process. A voice API can attract many unrelated uses and earn recurring revenue while leaving its company unsure which customer segment to pursue. The counterargument invokes Twilio: sometimes a general capability lets customers discover applications that early investors could not anticipate.

One host locates the decisive improvement upstream: better generation of what to say makes speech synthesis more useful. A support system reading a fixed script has a narrower role than one whose language model can converse with the customer. The speech channel gains value from the intelligence supplying its content.

The modality competition has two product approaches: - Broad multimodal models: OpenAI and Google combine capabilities; transfer of learning across domains is a possible advantage, not an established outcome here. - Specialist products: focused companies serve users who need more than a general model’s baseline capability.

The hosts also see a path from broad models to specialists. A developer reluctant to build a computer-vision pipeline might upload an image to GPT-4, discover a useful application, and then implement it with Roboflow. Lowering the cost of trying a capability can expand the specialist’s market.

RAG and operations: owning storage invites competition for the workflow

Retrieval-augmented generation, or RAG, supplies selected context to a model by retrieving material relevant to the query. Its surrounding databases, frameworks, and operations tools can work together—until vendors expand into adjacent layers.

The hosts identify competition moving in several directions: - Frameworks into operations: LangChain’s LangSmith competes with operations vendors. - Operations into frameworks: vendors build software development kits that compete with frameworks. - Existing databases into vector storage: dedicated vector databases face established systems adding vector features.

The database decision starts with a familiar discovery: a team attracted by LangChain or LlamaIndex searches for a vector database, then realizes it can use `pgvector` in its existing Postgres deployment. MongoDB, Cassandra, Redis, and Elasticsearch also appear as alternatives for storing or indexing vectors. For dedicated vendors, the proposed differentiation shifts toward making the retrieval pipeline useful, beyond storing the vectors.

The counterargument is that data represented in vectors may be materially different from existing application records. One host wonders whether context or memory could support a new database category, analogous to the rise of document and unstructured data. That category remains undefined, but the strategic incentive is clearer: a vendor owning the data may also want to own the RAG interface, competing with frameworks such as LlamaIndex.

Framework ownership offers another route to adoption. One host uses Vercel and Next.js to argue that a framework can drive cloud and platform usage. The other questions generalizing from a single competitive outcome; Gatsby then serves as a counterexample to guaranteed business success. Control of the developer interface may help distribution without settling monetization.

The expectation that 2024 production deployments will clarify these questions meets an objection: a “year of AI in production” needs a catalyst or threshold, or the prediction can recur indefinitely. The response is that enterprise procurement and implementation take time. Interested companies may still need more than several months to buy and roll out a system; wider deployment should expose both successes and failures.

Underneath the market debate is a technical choice: - Retrieve selected context: inject material chosen for the query. - Supply a much larger context: rely on the model to find what matters within it. One host expects users to prefer selected context but allows that this could change; the other strongly rejects that possibility. They also distinguish continued demand for retrieval from the revenue prospects of any particular open-source framework.

Coding agents: the sixteenth button tests maintainability

After the four main fronts, the hosts return to code. Natural-language programming extends a familiar abstraction: a sorting method hides an algorithm, and a model might translate a request for a signup form—with name and email fields sent to an endpoint—into implementation. The optimistic case is that people who understand business requirements can make production changes without passing every small request through an engineer.

The bear case has two parts: - Coordination: more people authoring changes can compound existing integration problems. - Maintainability: a model may satisfy each request with another local patch instead of improving the codebase.

The memorable test is the sixteenth button-style change: does the agent recognize that it should introduce a shared class? Completing the visible task and leaving the codebase better organized are different capabilities. One host treats reliable supervised use as a prerequisite for greater autonomy: users must first be able to request a change, inspect it, and judge it ready.

Another host proposes a deliberately simplified distinction: - Inner loop: work between Git commits, usually inside the integrated development environment, or IDE. - Outer loop: work after a commit is pushed for a pull request, usually in GitHub. That host favors inner-loop tools at this point, using Code Interpreter as bounded, supervised autonomy, and discloses an investment in CodeGen while remaining cautious about outer-loop agents.

The disagreement concerns where supervision can happen. One host imagines moving from autocomplete, to asking Cursor how to make a change, to requesting changes directly in the product interface. Comments in a preview environment could become code changes through a supervised interface. The other calls this outer-loop work; the response is that models may change where the loops begin and end. Code creation outside the IDE can still retain human review.

Prompt sensitivity and the gap between launches and use

A recalled Claude long-context demonstration returns the discussion to reliability. One host remembers answer quality varying with the location of relevant information. An Anthropic suggestion to add assistant-prompt wording introducing the most relevant sentence reportedly improved the demonstration dramatically. Without a model version, numerical results, or exact protocol, the recollection supports no general claim about long-context reliability.

Both hosts object to reliability depending on small wording tricks. They discuss step-by-step instructions, requests for complete scripts, hypothetical tips, and emotional appeals, but offer no controlled comparisons that would make these a validated prompting recipe.

Launch enthusiasm does not predict adoption timing. One host reports hearing little uptake of JSON mode; the other recalls their excitement about ChatGPT plugins. For GPTs, the question becomes what motivates developers to build substantial products inside the platform if the uses they observe mostly involve better system prompts or workarounds for base-model limitations.

The final model highlight is Gemini. One host criticizes its marketing while welcoming it as a credible OpenAI competitor, with Llama 3 still anticipated. This is a competitive assessment from that moment, without an accompanying benchmark comparison.

Personal context—and a physical off button

The closing hardware discussion takes place after CES and the Rabbit R1 launch. The hosts appreciate Rabbit’s toy-like nostalgia while questioning its lasting utility. One argues that an assistant like the one in Her requires both device engineering and AI engineering: hardware can capture context that software alone does not receive.

That host discloses an investment in Tab, whose proposed advantage is unique personal context. A store of what someone says and does could support applications unavailable to competitors without that information. The host reports that much of Tab’s $600 price goes toward GPT processing costs, without giving a breakdown. A developer API is a desired future capability, after the hardware and its social acceptance are established.

The social argument remains unsettled. One host relays the view that people often prioritize convenience over privacy, drawing an analogy with social-media adoption. The Tab investor sees socially uncomfortable products as potential consumer opportunities, but recognizes a hardware failure mode: intense initial interest followed by a device sitting unused in a drawer.

The last concrete design request is a physical off button. The Tab investor wants to show another person that a private conversation is not being captured, and reports tension with Avi’s preference for an always-on device. Continuous collection promises richer context; a visible control makes interruption demonstrable. The hosts leave that tradeoff open and close by inviting listeners to say what worked, what did not, and what should change in this first audio recap.