NeurIPS 2023 Startups, Part 1 of 4: From Model Building to Useful Products

MosaicML: differentiation after model building becomes commonplace

MosaicML wanted to remain independent, but Databricks offered a compelling fit: data expertise and large language model expertise, both focused on enterprises. Jon Frankel describes that complementarity as the reason to combine. Introducing the conversation, Swyx reports the $1.3 billion acquisition and notes that discussions were already underway during their May MPT-7B interview.

By late 2023, Frankel sees building large language models (LLMs) becoming a broadly available capability. His question for 2024 is what teams will do with it: which products they can build and how they will differentiate. MosaicML’s quieter public presence reflects acquisition onboarding and larger projects, he says, but he leaves those projects unannounced for people to judge when they arrive.

Multimodality—working across text, images, audio and video—makes the product question pressing. Frankel points to LLaVA, Bard, Gemini and GPT-4V as existing capabilities. Which production applications will justify the investment in building them? He also wonders whether larger open-source multimodal models will approach GPT-4V and trigger another competitive race.

Frankel sketches two possible markets: - Interchangeable providers: models converge, leaving customers to choose through interface preferences, familiarity and organizational habit. He considers this outcome uninteresting and unlikely. - Specialization through data and expertise: organizations build around distinctive resources. Adobe brings a creative perspective to Firefly; OpenAI has invested heavily in text data. YouTube is a potential resource for Google, although Frankel explicitly does not know whether it is using it.

His preferred future extends beyond the largest providers. Replit’s code model illustrates specialization, with finance and healthcare models as further possibilities. Helping customers use their own data is central to his description of MosaicML and Databricks’ business. Yet training a model is not a required endpoint: the discussion describes a possible progression from using an API to retrieval, fine-tuning and eventually training, without requiring every organization to complete it.

Evaluation: use synthetic data to make human judgment go further

The conversation turns to an unexpectedly ordinary NeurIPS. One participant contrasts the anxiety around scale at ICML 2022 with a late-2023 conference where models containing tens of billions of parameters feel mundane. Academics are finding research directions, industry researchers are continuing their work, and the community has endured substantial changes in roles. There is more secrecy, but plenty of open discussion remains.

Asked for underrated papers, the guest resists ranking research and objects to best-paper awards; both speakers favor test-of-time recognition. The question becomes what a PhD student should investigate. The proposed priority is measurement as a prerequisite for scientific understanding: researchers need reliable ways to evaluate model properties before they can explain them.

Synthetic evaluation data means model-generated examples aimed at testing particular behaviors. The discussion offers two complementary reasons to use it: - Target selected properties. The host recalls Kanjun from Imbue changing his view: generated data can deliberately emphasize useful properties instead of trying to reproduce human preferences. - Make human curation more productive. A model can draft examples for people to select and edit, reducing the work of building a useful dataset from scratch.

The proposed result is a collection of small evaluation sets that act like unit tests for niche LLM behaviors. Human selection and editing keep judgment in the feedback loop. The research opportunity lies in making many such tests practical to construct through creativity and ingenuity, without a huge compute budget; no measured improvement is presented.

Workshop posters: encounter ideas before consensus forms

Timing complicates judgments about promising ideas. In the guest’s view, being early in deep learning can mean being only six months early: a missing piece may arrive and change an idea’s prospects. The conference advice follows that uncertainty: wander through workshop poster sessions to meet students exploring early, creative work.

The venue distinction matters: - Main-conference posters may present papers that have already circulated for months. - Workshop posters can feature work created that fall, before it has attracted much attention. Smaller crowds also make conversations with researchers easier.

The attraction is assessing an uncertain idea before online consensus settles its reputation. Something confused or incomplete today may become important later. Recording poster presenters also gives exposure to researchers whose work has little online reach.

The interview closes with a more ordinary picture of NeurIPS: coffee, academic conversations, poster promotion, hiring and commercial interests. After the pandemic and rapid technical change, a conference that feels normal offers room to catch one’s breath.

Between interviews, the narrator returns to synthetic data through DeepMind’s Beyond Human Data: Scaling Self-Training for Problem Solving with Language Models. In his account, computer-generated math and coding data improved PaLM 2 on benchmarks in those domains; he gives no benchmark names or effect sizes. The harder question is extending self-training beyond verifiably correct answers. He suggests reinforcement learning from human or AI feedback as a possible direction beyond math and code.

Fireworks: specialize inference around production workloads

Fireworks is an inference-serving platform, providing infrastructure for running models in applications. Lin Qiao says its founding team started the company in 2022 after five years building Meta’s AI platform and PyTorch, pursuing one framework for research and production. By their departure, she reports, PyTorch handled more than five trillion inferences per day across 50 Meta data centers—the team’s prior operating environment, not Fireworks’ serving volume.

Qiao describes PyTorch’s compile effort as preserving a simple programming API while automating optimization. Fireworks follows that philosophy with a narrower workload scope that permits more specialized optimization. It began with ranking and recommendation, then expanded into language models, image generation and other modalities at customers’ request. She sees this focus as complementary to PyTorch’s broader responsibilities.

She identifies three approaches: - Address resource bottlenecks independently. CPU–GPU communication, compute and memory bandwidth can each constrain inference. Qiao says Fireworks’ scaling algorithm tackles them separately. - Build specialized kernels. The team implements operations including attention, matrix multiplication and AllReduce. Qiao claims superior performance, without hardware details or comparative benchmarks. - Adapt to recurring workloads. Qiao says performance improves as the same workload runs repeatedly, without specifying what adapts or quantifying the gains.

Reusable model components also shorten release turnaround. Qiao reports launching Mistral 7B the afternoon its weights arrived and supporting Mixtral within a day of release. Mixtral initially arrived without code, she says. Fireworks’ componentized libraries let the team reuse architectural building blocks instead of implementing each model from scratch.

She describes the CTO adapting a Llama implementation to released weights, then addressing a separate requirement: the working base model still needed chat tuning. The team tuned a chat model overnight. Qiao says some users later preferred it even after the service switched to an official instruction-tuned version; that was user feedback, without a controlled comparison.

Production demand extends from chat to changing recommendations

Qiao’s customer examples show several kinds of inference demand: - Creative and developer applications: Tome uses Fireworks for presentation generation, while Sourcegraph’s Cody provides coding assistance. - Varied chatbot experiences: Qiao says Fireworks provides an inference backend for Poe as it expands its open-source model offerings, seeking better performance and lower costs alongside more varied experiences. Those benefits are qualitative. - Ranking and recommendations: the DoorDash stack supports advertising and restaurant search recommendations.

Why not precompute restaurant recommendations? The response emphasizes changing preferences, participating restaurants and menus: the matching problem keeps moving. But adaptive recommendations do not necessarily require online training. For the enterprises discussed, models were trained offline, refreshed with new information and redeployed periodically; the interviewee had not seen a need for online training.

Fireworks also describes a LLaVA variant using its own training data in place of GPT-4-derived data, with the aim of making the model commercially usable. That is a stated goal, without a licensing analysis or quality comparison. Conversations with startups at NeurIPS point toward further product development and partnerships to reach broader audiences.

Its late-2023 hiring priorities reflect that scope: - Systems engineers with cloud infrastructure experience to scale inference serving. - Researchers focused on data and quality for model training and task-specific fine-tuning. - Commercial teams spanning marketing, solutions architecture and sales.

For the host, the unresolved business question is whether proprietary serving software creates a durable competitive advantage. Operators see substantial value in that software; outsiders may see commodity infrastructure. Fireworks’ move into training and releasing models adds another dimension to the question.

Cursor: shorten the path from idea to experiment

The host introduces Cursor, an AI coding editor, through his own switch to its workflow. Aman Sanger reports an $8 million raise from OpenAI and growing adoption inside the organization. His ambition is turning an idea into experiment code immediately: he sees implementation as a research bottleneck, while acknowledging that Cursor is only a small part of the way toward that goal.

Model expertise does not necessarily produce more product feedback. Sanger suspects OpenAI users give less feedback than users unfamiliar with language models because they already understand the models’ deficiencies. Feedback also travels the other way: Cursor shares observations with OpenAI about early alpha releases.

KV caching: reuse the unchanged conversation prefix

Successive chat requests share earlier messages, creating an opportunity to avoid repeated computation. A key-value (KV) cache retains computed keys and values for tokens. The additional technique discussed here is keeping that cache across requests, so an unchanged conversation prefix does not require recomputing its keys and values.

The described serving sequence is: 1. Move computed keys and values from GPU memory into CPU memory. 2. Retain them until the cache evicts them. 3. When a request arrives with a matching prefix, load the retained state back into GPU memory. Avoiding recomputation can free compute for more requests. The speaker claims higher throughput but gives no numerical latency or throughput result.

A system prompt containing today’s date illustrates reuse: it changes once a day, while more personalized prompts may complicate matching. A proposed retention policy is least recently used eviction—keep entries within a memory budget and discard the least recently used when space runs out. This is a suggested design, not a disclosed provider policy.

For a participant using dedicated GPT-4 instances, token volume and cache hits affect capacity planning, even though the service exposes little operational detail. Tokens contribute to memory pressure; cache hits can reduce compute demand. Understanding both helps estimate when more requests can be admitted without latency spikes or failures.

A speculative extension would retain computed state for an entire corpus in blob storage such as S3. Its usefulness depends on loading retained state faster than recomputing it. The participant suggests attention variants with smaller keys and values could help, but gives no model configuration or benchmark for the proposed reductions. The idea is described as difficult, with few compelling applications and no implementation known to the speaker; codebases are one possible use case.

The caching exchange closes with the Assistants API as discussed in late 2023. One participant criticizes charging for the full context on each new message and sees a missed opportunity to offer caching. A possible operational explanation is that machine-local caches are easier to exploit on dedicated instances than behind serverless endpoints. The participants do not know OpenAI’s implementation or rationale.

Synthetic retrieval data: judge relevance instead of generating answers

Synthetic data returns through a concrete retrieval workflow: ask GPT-4 to judge which documents are relevant to a query. This depends on a narrower assumption than training a weaker model to imitate GPT-4’s answers. The model needs to make good relevance judgments among candidate documents.

The procedure builds a larger ordering from small comparisons: 1. Give GPT-4 a query and four candidate documents to rank by relevance. 2. Repeat the ranking task across candidate groups. 3. Combine the judgments using TrueSkill, a ranking method described in the conversation as an Elo variant, to order a larger set—for example, 100 documents. The resulting ordering supplies synthetic retrieval-ranking data. The participant reports high-quality rankings without a quantitative evaluation.

Confidence in models as judges leads to a distinction between better harmlessness and greater helpfulness. Recalling an unnamed paper, one participant says reinforcement learning with AI feedback improved harmlessness but showed no statistically significant helpfulness advantage over reinforcement learning from human feedback. The other emphasizes the value of improving one dimension without degrading another. They agree that a better tradeoff matters, while differing over how much additional capability it demonstrates; this remains a recollection of an unspecified study.

NeurIPS: access to expertise beyond online visibility

Part 1 closes back at the conference. NeurIPS can be overwhelming, but gathering paper authors and other people of interest in one place makes it valuable for meeting and catching up. One participant finds that repeated attendance improves the experience.

The NeurIPS startup recordings aim to give nonattendees a sense of those encounters. A final observation captures their appeal: expertise need not come with online visibility. Some of the people worth meeting have little or no Twitter presence; their knowledge comes from having done the work.