Fireworks’ bet on open models and compound AI

From research code to production demands

A framework that researchers enjoy using still has to survive production. At Meta, that meant ranking ads, detecting harmful content, translating text, and running models on phones and VR devices. Lin Qiao recalls spending five years adapting PyTorch to those workloads’ stability, reliability, and latency requirements while preserving its usefulness for research.

Qiao came from distributed systems and database management. She connects Meta’s transition from mobile-first to AI-first to the data generated by greater mobile engagement. PyTorch provided another reinforcing cycle: researchers published innovations in open source, downstream teams put them into production, and external adoption reinforced Meta’s internal use. Research was the top of the production funnel.

Companies she encountered through PyTorch often lacked the infrastructure and teams to adopt AI at all. Their needs differed from Meta’s pursuit of extreme scale and optimization. That gap motivated Fireworks, whose own first two years brought operational lessons: the Silicon Valley Bank run, an accidental data deletion, and growth that forced a scramble for capacity.

The initial blueprint was a PyTorch cloud. Qiao’s assessment of the 2022 market was that cloud providers offered some support, but lacked the first-class, PyTorch-first hosted experience she wanted. After starting the company in September 2022, the team chose a narrower customer problem before attempting a broad platform. Requests following ChatGPT’s announcement directed it toward generative AI, despite the weakness of early open models.

The original conversation includes an advisory disclosure: Swyx advised Fireworks on several launches after speaking with Qiao for the NeurIPS 2023 podcast.

A familiar API, a workload-specific deployment

Fireworks began with a distributed inference engine—running models to answer requests—behind an OpenAI-compatible API. It initially served large language models, targeting application developers and product engineers. Existing foundation models let these users build without first assembling model-training and GPU operations teams; the familiar API hid the complexity of serving them.

API compatibility also raises a standards question. Llama Stack represents Meta’s effort to standardize the software above Llama models and build a community around it. An interviewer questions whether model adoption will automatically carry the stack along. The response acknowledges that the stack is early: its success depends on community adoption and improvements informed by partner feedback.

Below the API, Fireworks describes custom CUDA and ROCm kernels, caching, distributed execution, and a specialized PyTorch build. A general-purpose endpoint still leaves room for customization. Customer requests differ from a model’s training distribution, and Fireworks argues that adapting to the actual workload can improve quality, latency, and cost.

Fire Optimizer makes those priorities explicit. A customer supplies an inference workload and an objective function specifying the desired tradeoff; the system returns a deployment configuration and model setup. Different uses within one company can require different operating points. The discussion describes this interface and its goals, but provides no measured optimizer gains.

Why one model is not enough

Customers building text assistants soon wanted audio input and output. Fireworks expanded into audio transcription, translation, speech synthesis, and text–audio alignment, with an associated announcement expected by publication.

Documents provide a concrete reason to combine models. Information may arrive in a PDF, screenshot, or image. A vision model first extracts its content; a language model then processes that content and produces a response. Fireworks previews a multimedia extraction endpoint for this workflow, without quantifying its promised accuracy.

The catalog also includes embedding models for semantic search, text-to-image generation, and image-to-image models, with text-to-video being added. These capabilities run above the optimizer and inference engine.

The case for [compound AI](https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/) extends beyond handling different media: - Models can generate misinformation. - Models have different specialties, making it useful to divide a problem into narrow tasks. - Training cannot supply all the real-time or proprietary information an application needs. A compound system combines models with public and internal APIs, storage, databases, and knowledge systems to produce an answer. Model selection, task decomposition, and access to information become parts of the application architecture.

What the serving platform contributes

Fireworks describes distribution at three levels: - Model execution: split a model into pieces and scale each according to its bottleneck. - Geography: use regional affinity to reduce latency and global load balancing to accommodate demand across time zones. - Hardware: match hardware types to long-context, short-context, or long-generation workloads, potentially spreading one workload across different hardware. At the time of the conversation, the company reported operating in North America, Europe, the Middle East, Africa, and Asia.

The company reports custom kernels for all its language models and almost all other models it offers. Fire Attention primarily serves language models, with customized kernels for other modalities. The introduction claims 15× higher throughput than vLLM, but gives no hardware, workload, precision, or latency conditions; that figure is not a general deployment comparison.

An interviewer presses the competitive question: other providers can serve the same open models. Fireworks’ answer starts with what application developers need. A presentation tool might let authors develop a story without first fitting it into slides. Such experiences require fast interaction and sustainable serving costs. Product-market fit can accelerate failure if every additional user increases losses.

Fireworks’ proposed advantage is accumulated serving expertise: an automated feedback loop that uses customer workloads to improve quality, latency, and cost while developers focus on the product. Open-model developers supply innovations across audio, video, and other modalities; Fireworks aims to turn those innovations into usable services. The feedback loop’s implementation and measured gains are not disclosed.

A compound system behind a model interface

Frequent model releases create repeated decisions: whether to upgrade, how to divide an application into subproblems, and which model should handle each part. Fireworks distinguishes imperative systems, where developers specify execution steps, from declarative systems, where developers specify a desired result and the system determines how to produce it.

SQL provides the analogy. A query expresses the result wanted; the database chooses indexes, disk access, and execution nodes. An explicit data pipeline instead specifies triggers, actions, and failure recovery. Fireworks favors declarative integration for application developers, while maintaining that both approaches have value and neither will subsume the other.

Fire Function was an initial step: a function-calling model trained to dispatch requests to APIs, with additional APIs configurable by users. After three versions, the team previews a declarative system presented as a model, inspired by OpenAI o1, that would remove the need to use a separate function-calling model. The episode introduction identifies [f1](https://fireworks.ai/blog/fireworks-compound-ai-system-f1) as Fireworks’ long-inference reasoning offering built on open models; during the interview, the release is still forthcoming.

Fireworks says it trained the offering using available models rather than starting from scratch. It shares the goal of improving quality through work performed at inference time, while describing its approach as different from o1’s. Claims of approaching OpenAI’s quality are unquantified: the team reports a submission to LMSYS and promises benchmarks, but supplies no scores, evaluation conditions, or specific comparison model.

Specialists, scaling, and reasoning budgets

Fireworks bets that the capability gap between open and closed models will shrink, increasing the value of inference engineering. It also argues that specialists can compete in particular dimensions, citing coding and Fire Function’s API dispatch. The prediction is that narrow expert models can outperform general models within their specialties; no comparative measurements accompany these examples.

An interviewer challenges this with the bitter lesson: specialist systems may deliver temporary gains, then be overtaken by a general model scaled with more compute, data, or inference. Fireworks responds that training is approaching a data limit, making model-generated synthetic data and inference-time scaling more important. These are competing expectations about how capabilities will improve; the interviewer remains undecided.

Building on open models does not mean releasing the resulting system as open source. The team confirms that the forthcoming offering will be available through an endpoint. Whether to expose reasoning traces remains undecided, and pricing is unavailable. Early testing anecdotes include apparently correct answers to difficult mathematical questions and an internal prompt about artificial general intelligence that produced a complicated graph-shaped thinking process. Those anecdotes come without prompts, outputs, or correctness checks.

A concrete interface request is control over the reasoning budget. An interviewer argues that users sometimes know a problem’s difficulty and should be able to allocate time or money—for example, asking the system to spend two minutes on it. Fireworks says it wants to support that control eventually, but not at the initial release.

Cursor turns optimization into a product

The discussion returns to execution. An interviewer says Fireworks’ shipping pace changed their skepticism about another provider entering a crowded market. The company reports a team of 40 and describes its priorities as results and customer business value, including deploying models and reallocating capacity overnight or on weekends.

Cursor provides the example. Fireworks praises its decision to partner on infrastructure despite having the technical ability to build more of the stack. Cursor’s demands for low latency and high quality reportedly drove new features and rapid expansion across regions, with engineers coordinating through an always-on Slack channel. The precise hosting arrangement remains undisclosed; the interviewer’s suggestion that Fireworks serves Cursor’s in-house models is not confirmed.

Workload-specific optimization deployed for Cursor helped motivate a separate optimizer product. The team describes speculative decoding as a family of latency-reduction approaches: pairing a small and large model from the same family, or using components called EAGLE heads or Medusa heads. Their internal mechanics are not explained; the operational point is that the choice and alignment of components depend on customer traffic. An interviewer mentions 1,000 tokens per second, but supplies no model, hardware, or workload conditions, and the response does not explicitly validate it.

The recommended development sequence separates discovery from optimization: 1. Establish product fit. Choose a good model and test whether generative AI improves the experience. 2. Choose an operating point. Once the product works, optimize quality, latency, and cost for its users. 3. Validate the deployed experience. Test inference configurations and model versions, including A/B tests, rather than stopping at offline evaluations. A model with lower evaluated quality can be the better product choice if it improves speed and cost without affecting the experience users value.

The same evaluation principle governs quantization choices. Fireworks describes different schemes applied to different model components, with customer-specific configurations selected through evaluation and a default chosen for self-service. Asked about a competitor’s public criticism, the company disputes the fairness of vendor-run comparisons and advocates third-party evaluation using agreed methodology. The disputed results are not established in the exchange.

Sharing a base model changes customization economics

Fireworks intends to earn margins from hosting open models. Its pricing argument is that undifferentiated services face downward pressure, while workload optimization adds value. Serving externally developed models also avoids their original training costs, whereas closed-model providers must recover training investment through inference revenue. That distinction concerns the underlying models: Fireworks also reports training its own specialized offerings.

The closing technical example is multi-LoRA serving. A LoRA adapter supplies a fine-tuned customization of a base model. Fireworks says developers can upload an adapter and receive an endpoint that behaves like a new model, without an on-demand deployment and at the base model’s token rate.

Many adapters share one base model, reducing memory duplication and pooling traffic around the component that dominates serving cost. Fireworks says a base model can support 100–1,000 adapters, although it specifies no hardware, model size, or workload conditions for that range. Sharing the base is what makes the claimed pricing possible.

Feature discovery remains a problem. Multi-LoRA had reportedly been available since the previous year but was poorly known. An advisor had not realized speculative decoding was released, and Fireworks says prompt caching had also been available since the previous year.

The team closes by asking developers for successful use cases, failures, and feature requests. Small-group product launches typically begin on Discord, which also serves as a feedback channel. At the time of the conversation, hiring spanned frontend engineering, cloud infrastructure, backend optimization, applied research, and researchers experienced in post-training and fine-tuning.