Together AI: Making Open Models Practical Across the Stack

From a working model to an accessible platform

At Apple, Vipul Ved Prakash worked on question answering across a broad range of subjects. A complicated information-retrieval system was struggling; an alternative based on a 2016 Facebook neural-network paper took about a week to implement and, he recalls, performed strikingly well. That experience helped convince him of deep learning’s potential, though it was a personal assessment rather than a measured comparison.

In the February 2024 Latent Space conversation, Prakash describes Together AI as a platform for open models and company-owned models. Founded in June 2022, it aimed to make independent AI systems accessible beyond frontier labs’ proprietary developer platforms. Its cloud combined data centers around the world, outside the hyperscalers.

Prakash brought experience in both openness and product design. He had created the collaborative spam filter Vipul’s Razor and founded CloudMark around open-source and commercial offerings. Apple reinforced the value of hiding complex technology behind fluid products. His deep-learning work there, followed by research on how model capabilities improve with scale, helped shape Together’s ambition.

Ce identifies a systems problem connecting that ambition to a decade of research: data movement. Distributed computing optimizes communication between machines; FlashAttention optimizes movement through the memory hierarchy. Together’s early research asked whether compressing the traffic needed for gradient averaging during training could make disparate GPUs connected across the internet useful. Prakash’s premise was that capital requirements posed a harder barrier than access to web-derived data—and more efficient systems could lower it.

RedPajama makes data selection reusable

RedPajama addressed a concrete reproducibility gap: Llama 1 described its training-data recipe without supplying the data. Together attempted a best-effort reproduction that the community could inspect and improve. Ce credits earlier open datasets, including Eleuther’s Pile and Google’s C4, as inspirations. SlimPajama’s subsequent deduplication of RedPajama illustrated the intended result: an open dataset could support new datasets as well as models.

RedPajama V2 extended that idea with roughly 40 precomputed quality signals over a corpus described as containing 30 trillion tokens. Depending on the application, filtering and deduplication might reduce it to one or two trillion. The headline size described a pool for selection; the signals let developers make their own choices about what belonged in training.

The team said those signals enabled a best-effort reproduction of C4 filtering in about 20 lines of code. Developers could combine filters or learn their combinations. AI2’s Dolma offered another flexible format for quality signals, and RedPajama was expected to work with its tooling. The larger design principle was to preserve intermediate selection choices: different applications value different subsets, so the final training corpus is only one useful product of a data pipeline.

Choose data for the model you need

The data-selection method DSIR starts with a small reference dataset and finds similar documents in a larger corpus. A validation set can guide selection toward an intended task; Wikipedia can serve as a broader reference for finding more Wikipedia-like material. This can amount to weighting different slices of data, filtering a corpus or expanding the reference collection.

Selecting for a known purpose raises a question about generality. The response treats it as a spectrum: selection can target a distribution of tasks as well as a single application. For a customer with a specific use, a targeted dataset may yield a better, smaller model that is easier to serve. The tradeoff spans generality, accuracy, robustness, model size and deployment cost.

Together also links data recipes to its architecture research. A goal of 5,000 tokens per second, and the expense of learning longer sequences with transformers, motivate exploring alternatives. Open research provides recipes that customers can use to build custom models; in response to a question about StripedHyena, the team confirms that it was available for inference on the serverless platform at the time.

Data scarcity introduces a different constraint: useful material may exist without being accessible to open-model developers. Organizations hold private collections, while books, transcripts, video and audio remain incompletely integrated into open datasets. The proposed direction is better access for model building, while recognizing that some underlying material should remain private.

Quality also depends on the application. When an interviewer worries that YouTube transcripts might teach models to sound like livestreamers, the response is that conversational speech suits some uses and formal language suits others. A further concern is fragmentation: data owners building separate models can leave useful material isolated. A proposed creator-data marketplace would encourage participation through fairer value sharing, but its design remains unresolved.

Training is a continuing workload

Together’s usage challenged the expectation that customers train once and then move permanently to inference. At the interview, its top five inference models were said to be probably all customer-fine-tuned versions of open models, adapted on and off the platform. More than twenty large customer model builds were underway. Training remained the larger workload because customers kept training successive versions, often at increasing scale.

The fleet then comprised roughly 7,000–8,000 GPUs, mostly A100s and H100s, and was growing monthly. Research access was improving through institutions such as Hessian AI in Germany, which shared compute with the community. Commercial purchases remained tight, with minimum waits of roughly two to three months—even with NVIDIA serving as both investor and supplier.

One speaker’s rough market estimate put annual GPU sales at four to five million units and GPU spending at $80–100 billion, rising toward $250 billion after servers, facilities and electricity. Supplier coverage and the accounting basis were unclear. A comparison with a stated $88 billion in prior-year AWS revenue conveyed the scale of the anticipated build-out, but compared infrastructure spending with a different financial measure.

Inference economics depend on the configuration

An interviewer cites a SemiAnalysis argument favoring two 80 GB A100s over an H100-based system for Together, along with speculation about its decoding method. The response challenges the apparent assumption that input tokens were unpriced and emphasizes tradeoffs between batch size and generation speed. Together describes its inference stack as combining roughly fifty techniques; the exchange does not establish which decoding method it used.

Hardware selection was empirical and model-specific: measure price-performance on H100s, A100s and L40s, then choose a configuration for the model and sometimes the customer’s traffic. Memory bandwidth alone cannot explain serving economics. Batch choices, model behavior and the rest of the system also matter.

This serving advantage defined a boundary around Together’s openness. Broadly useful work such as FlashAttention and FlashDecoding was published, while some implementation details stayed closed to preserve competitive price-performance. Research publication also helped recruit people who wanted their work to advance the field. Software embedded deeply in Together’s cloud might be released later, but opening the complete stack remained a possibility.

Disaggregated cloud describes resources that differ in geography, reliability and underlying infrastructure. Together’s intended service layer would absorb those differences so developers would see a reliable product. The team preferred that term because “federated learning” had accumulated meanings that made it less precise for their purposes.

The optimization thesis is to improve algorithms, architectures, systems and hardware jointly. A further tenfold inference improvement was a goal of that approach. But optimizations do not automatically compound: the team reported that some combinations gave results similar to individual methods, while others produced substantial gains, without quantifying them. It considered deep tuning for a particular model, GPU generation and bus configuration worthwhile, and maintained multiple systems for different combinations.

Benchmarks shape what vendors optimize

The discussion of Anyscale’s inference benchmarks focuses on incentives. Together credits the effort as a community service, while asking what happens when vendors optimize aggressively for a ranking. A useful standard should reward engineering that benefits users. The proposed direction is an industry-agreed comparison protocol, drawing on database benchmarking as a precedent, preferably operated independently of competing providers.

Together reported that Anyscale’s results differed from its internal cross-service tests, without measurements that settle the disagreement. The engineering requirements for a useful comparison were more concrete: - Test location and time: client location affects time to first token; service load varies through the day and week. - Latency and throughput: response delay, generation performance and latency percentiles reveal different behavior. - Reliability and capacity: aggregate usage can expose limits invisible in a single successful query. - Output quality: acceleration techniques such as sparsification and quantization can change quality along with speed.

A transparent protocol needs quality checks before services belong in the same price-and-latency table. A shared model name does not make differently optimized deployments interchangeable. Artificialanalysis.ai appears as an emerging independent effort, with the speaker explicitly withholding judgment on its methodology.

Embeddings improve retrieval and training data

Together’s customer engagements ranged from pretraining on supplied datasets to helping smaller-model teams scale, or starting with a task and choosing data and models together. The goal was to turn more of that consulting into a repeatable product. Meanwhile, small open-source engineering collectives were bringing experiments with data recipes, synthetic data and model merging into commercial applications.

Embeddings, vectors that represent semantic information, connect application retrieval with model development. They help retrieve knowledge for retrieval-augmented generation, or RAG, and help build better training datasets. The proposed feedback loop is better embeddings → better datasets → better models → better embeddings.

The research agenda includes richer semantics, better handling of negation and faster execution at different vector sizes. Together had only just released its first embedding model. Asked to explain the apparent strength of Chinese embedding research, the respondent declined to speculate while the team was still learning how to build its own.

State-space models and hybrid layers

The motivation for state-space models extends beyond long context. The central idea here is to keep the model’s state small and avoid dependencies that grow quadratically with sequence length. In the team’s account, this can make execution more efficient, permit larger batches and reduce the cost of long context. Those benefits could also matter to applications processing shorter sequences; no numerical comparison is given.

StripedHyena and Mamba-3B explored different questions: - StripedHyena combined components described as state-space and transformer layers. Its quality target was to match Mistral and potentially surpass it through fine-tuning. - Mamba tested how far a pure architecture could scale, progressing from small models to three billion parameters against a strong baseline at that size. These were research objectives, not demonstrated victories over the baselines.

A hybrid architecture makes the choice at the layer level. Some layers might obtain the required benefit from state-space components; others could use transformers for a more global view of the sequence. Mamba, Hyena and transformer components become composable building blocks. The hypothesis is that different components handle information propagation differently enough to improve the combined model. The best mixture remains unknown.

Turning those experiments into practical models requires systematic study of scaling, data quality and dataset size. Whether state-space models should use the same training data as transformers is an empirical question. Hardware and kernel efficiency belong in that study too: faster execution permits more training data, which can affect final quality. The proposed approach is patient progression from small models through one-, three- and seven-billion-parameter scales.

Why aim for 5,000 tokens per second?

5,000 tokens per second remained a goal, motivated partly by applications that consume model output directly. Software does not need to wait for a person to read or hear every token. The team described demand for generation speeds that providers could not adequately satisfy.

Higher throughput per GPU could support more applications, while nearly immediate answers could enable different interfaces. Together’s observation was that users rarely requested less performance, with text-to-speech consumers offered as a possible exception. Its broader view was that developers had adapted to slow model responses; faster models could change what they expected an application to do.

Serverless development and the work ahead

The closing product direction is serverless AI development: letting developers start with open models without committing thousands of dollars to compute. Inference and fine-tuning were already described as serverless; training was still in progress. Together then offered $25 in credits, which it said could cover fine-tuning, running a model and building an application, depending on the model.

A hackathon framed as fine-tuning versus RAG yielded a practical observation: combining them worked well, according to the team, though no quantitative evaluation was given. The same ambition to combine improvements shaped the company’s work across applications, model development and machine-learning systems.

At the interview, Together reported 38 people and more than twenty openings across GPU kernels, research, cloud infrastructure, frontend development and applications. Professional AI experience was unnecessary for some roles: people with Kubernetes, CUDA or DevOps expertise could apply their systems skills and learn machine learning on the job.

Asked what they would do outside Together, one guest imagines returning to a professorship, valuing the freedom to explore without justifying immediate usefulness. Previous work included quantum machine learning; a renewed interest was connected edge devices, with satellite links such as Starlink and new batteries opening questions about how to compress communication further.

Another calls for an expert-informed, realistic vision of life with advanced intelligence. Dystopian science fiction, especially Terminator, had too much influence on how people imagined that future, in this speaker’s view. A better framework could guide the industry’s direction. For their own future, the answer was simpler: building Together was a dream job they hoped to keep doing for a long time.