Replicate’s path from runnable research to an API for AI builders

Software should respond like a physical tool

Press a button, receive no feedback, then wait half a second for something to happen. Ben Firshman uses that frustration to explain what software should avoid. His ideal interface feels like a physical machine: an action produces an immediate, tangible response. Electronics, woodworking and bicycle repair inform that sensibility alongside programming.

Firshman helped write the Command Line Interface Guidelines, a command-line counterpart to the Mac Human Interface Guidelines. Yet Fig and Compose initially missed his responsiveness goals. He attributes some of the delay to Python’s runtime startup and prefers Go’s startup behavior. His suggested response times—roughly 50 to 100 milliseconds—are design targets, not benchmark results.

Feedback also means helping a user finish. Swyx describes an ambition for the Netlify command-line interface (CLI): when a command encounters a missing precondition, it should gather information or wait, then resume the original operation. Firshman connects this to conversational CLIs. Silence on success may suit a script, but a novice copying a file may interpret it as failure. A tool that guides people through a task offers a natural setting for language-model interaction.

From readable papers to runnable models

After leaving Docker, Firshman turned toward science infrastructure. As a tool builder, he hoped to make many scientists a little more effective. Publishing offered a starting point: even openly available research arrived as PDFs, underusing the web’s capacity for linked, structured documents.

In 2017, a vacation in Greece turned that frustration into [arXiv Vanity](https://www.arxiv-vanity.com/). Andreas was reading a machine-learning paper on his phone, zooming into the PDF and scrolling line by line. The collaborators began building LaTeX-to-HTML converters beside the pool. They later spent a few months helping integrate the work into arXiv as volunteers before shifting their attention to Replicate. A contributor to an underlying library completed the integration; by the time of the conversation, Vanity had recently shut down.

Making papers readable left a deeper problem unsolved: a description of software is not a runnable artifact. While applying research at Spotify, Andreas encountered missing code, repositories that would not run and model weights that had disappeared from Google Drive. Docker suggested a bridge: package models in containers so researchers could reproduce prior baselines and engineers could apply the software to practical problems.

Replicate’s development began in 2019, following the collaboration on Vanity. The eventual division was Cog for packaging models and Replicate for publishing them, but roughly two years of intervening work separated the container idea from that product.

A useful research tool still needed a business

The initial proposal applied the idea of continuous integration (CI)—automated testing—to model comparisons. Researchers would submit packaged models to a service that ran standard benchmarks. Someone evaluating a new model could inspect the results, download its predecessors, verify their results locally and run the same evaluation. This shared benchmarking service was intended to reduce repeated environment setup while creating a supply of reusable containers; those benefits remained a proposal.

The team entered YC with the concept and began a prototype. Researchers were interested, but revenue remained uncertain. Attempts to sell adjacent, unbuilt products—internal research sharing, model testing, deployment and versioning—went poorly. The team was more comfortable building than selling a promise. Well into the batch, it still had no usable product, users or clear business.

The pandemic disrupted the batch in March 2020, shortly before demo day. The team remembers the interruption as a reprieve from fundraising it was unprepared for. YC’s lasting value came afterward: Jared Friedman and other staff helped with subsequent rounds, while the alumni network brought users and customers. As an interviewer observes, shared membership can get an email opened without securing a purchase.

After unsuccessful projects involving contact tracing, food delivery and volunteer coordination, the team returned to machine learning with an open-source experiment tracker. Command-line tools suited the founders, and an open-source project promised fewer operational and shutdown obligations. But researchers wanted visual graphs and charts and were comfortable with a cloud service. The tracker, originally named Replicate and later renamed Keepsake, attracted too little interest to remain the focus. The builders’ preferred interface did not match their users’ work.

The community that kept coming back

Returning to containers and research sharing, the team raised seed funding with a small prototype and built [Replicate](https://replicate.com) around interactive model pages and downloadable Docker containers. It cut integrated benchmarking as too complicated. Researchers could publish demos alongside papers, and others could download the models for their own evaluations. There was no public API.

That workflow created a retention problem: a researcher might upload a model and disappear until the next paper, roughly six months later. In early 2021, image-generation experimenters offered a different rhythm. They combined [CLIP](https://openai.com/research/clip) with image-generating models, producing projects such as Big Sleep and VQGAN-CLIP. Through Discord and Colab notebooks, they continually copied, modified and forked one another’s work.

A VQGAN-CLIP bot in the Eleuther Discord made generation social: roughly a hundred people could see one another’s outputs and build on the ideas. That experience helped motivate bringing these models to Replicate. PixRay, recalled as one of the first models published primarily there around mid-2021, produced pixel art and other images that were appealing even when they lacked crispness. Users kept returning to the web forms, although the business model remained unsettled.

The first customer copied the request

Replicate’s API business began when a user inspected its web form, copied an internal API request as cURL and started generating images from a script. The endpoint was unsecured, and the resulting traffic caught the team’s attention.

The team recognized demand for programmatic access. It documented the API in Notion, created a Stripe payment form and offered access for approximately $1,000 per month. The user accepted. That first customer was PixRay’s creator, who generated art to sell as NFTs and referred other creators. An official API and billing beyond the initial fixed fee followed.

Portability had not been enough. A Docker container made a model distributable, but getting it running on a GPU still required substantial setup. Hosted execution let software developers invoke it with a small amount of code. Replicate considered building models itself but stayed focused on developer tools. The user’s workaround had exposed the missing product capability.

Open models brought both supply and demand

Stable Diffusion amplified the community Replicate already served. Its combination of sufficient image quality and modifiability encouraged both experimentation and product development. The team recalls animation adaptations and game-texture models using circular convolutions to produce repeatable textures within the first week. Within weeks, fine-tuning enabled personalized outputs incorporating a user’s face. Model builders supplied new variants while developers arrived wanting to use them.

Llama 2 reportedly brought Replicate its biggest growth week up to that point, although no growth metric is specified. At the time of the conversation, the reported audience was two million users, many of them developers. Latent Space itself used hosted Whisper diarization for podcast transcription.

Customers worked at three levels: - Off-the-shelf models: Unsplash used BLIP to generate searchable text descriptions of catalog images. - Fine-tuned models: Pieter Levels built products around adapted image models. - Custom inference: companies with machine-learning teams wrote their own Python and used Replicate to run custom models.

Company size did not cleanly separate those needs. Indie developers and larger organizations both wanted model capabilities without managing internals and servers. Replicate also challenged the suggestion that indie customers were inherently erratic: some solo developers ran substantial businesses and ranked among its largest customers. For them, the platform effectively supplied a machine-learning team.

Cog defines the callable model

[Cog](https://github.com/replicate/cog) preserves the original reproducibility idea through an open packaging format. It defines how to invoke a prediction and describe its inputs and outputs. The package is a Docker container that can attach to a CUDA device for GPU execution, with an OpenAPI specification stored as a Docker image label describing the model’s interface.

Cog presents the model as a function with human-readable inputs and outputs, while allowing arbitrary software inside. The package therefore need not prescribe a model’s internal representation. Its interface is intended to survive changes in model formats and inference servers—the software that runs models to answer requests. Docker supplies the container; Cog adds the prediction contract and execution conventions.

The authoring interface follows researchers’ existing knowledge. A notebook user may know the required Python version, Python packages and system packages without knowing how to write a production Dockerfile. Cog’s YAML configuration lets them declare those requirements. Its broader design aims to absorb recurring work around CUDA versions, GPU integration, serving, input and output schemas, and batching.

The open format also provides an integration boundary outside Replicate. Organizations unable to use a hosted service could run Cog models internally; developers could incorporate them into custom inference pipelines. A proposed testing workflow makes the value concrete: a CI system pulls the package, tests the model through its defined interface and passes it on for deployment to Replicate.

Connecting libraries, laptops and cloud execution

Transformers and Diffusers complement Cog: these Python libraries still need dependency installation and an inference server. Many Replicate models ran them inside Cog. A more direct integration for deploying Hugging Face models through Cog into Replicate was still in progress.

Ollama and llamafile focus on local execution. Cog was designed chiefly for servers with CUDA and Nvidia GPUs, and laptop execution worked less well. Replicate wanted models to move in both directions between local machines and its cloud, but the shared layer remained unresolved. The goal was interoperability and conversion, with fewer competing standards.

The local/cloud split echoed earlier Docker work. The people behind Ollama had worked on Kitematic, a desktop application that evolved into Docker Desktop; the other startup’s successful product became Compose. Collaboration continued in AI. Replicate likewise drew on its team’s experience at Docker, Heroku, GitHub and in the iOS ecosystem, while recognizing that older abstractions could not simply be copied into the new domain.

Short requests, long GPU commitments

Replicate reported running production primarily on GCP and CoreWeave, with some other providers; its own GPUs were for experimentation. Its capacity strategy was demand aggregation: combine customer workloads, make large commitments to cloud providers and sell access in smaller pieces—even a 100-millisecond API request on an A100.

The illustrative procurement contrast is asking for one A100 versus committing to a hundred for two years. Providers value sustained use, while startups may need small, intermittent allocations. An interviewer compares the business to a bank connecting short-term deposits with long-term loans. Replicate accepts the financial implication: capacity commitments are bets on future demand and GPU value.

Planning depends on model requirements as well as customer growth. Larger models may require more GPU memory or multiple GPUs; smaller models could favor smaller devices. Replicate reported multi-node A100 and H100 systems capable of running the mixture-of-experts models raised in the discussion, while acknowledging uncertainty about future requirements. It had sufficient supply for its own demand at the time—a company-specific assessment, not a forecast for the GPU market.

Optimization is part of the user experience

GPU availability does not ensure efficient inference. Replicate’s community published almost all its models, with usage concentrated in a popular subset and a long tail of less optimized implementations. The company worked with Meta and Stability on Llama 2 and Stable Diffusion, but acknowledged substantial inefficiency elsewhere. Slow execution wastes capacity and makes users wait.

Its optimization work included vLLM and TRT-LLM for language-model serving, model compilation and ordinary Python efficiency improvements. Replicate had helped customers improve code and rewritten popular implementations. Its Stable Diffusion implementation was compiled for faster inference and available as open source, though no measured speedup was given.

Lower-precision models offered an appealing tradeoff: reduced memory use, faster execution and lower cost in exchange for some quality loss. The discussion provides no model or benchmark conditions for a numerical comparison. Broader assistance—guides, access to optimized servers, compilation and possible help with quantization—remained partly exploratory. Whether Cog would automate compilation or make it easier to configure was undecided.

An API should leave room to change the model

Replicate’s pricing aim was developer affordability with sustainable economics. More customers should not mean larger losses. It claimed comparable price and performance to other services for Llama 2 and Mistral; the respondent could not recall whether Mixtral belonged in that comparison.

The stronger distinction was customization beyond the API. Developers could fine-tune a model, retrieve its source, edit it and deploy their own version. Hosted execution needed to preserve the experimentation and model combinations that had brought Replicate its early community.

Licensing introduces a tension between that freedom and funding model development. Replicate supported creators charging for commercial use, including arrangements that released code for experimentation while sharing commercial proceeds with the original trainer. Fine-tuning made relatively inexpensive downstream adaptation possible despite costly base-model training. The unresolved task was to compensate original trainers while preserving derivative work. Replicate also hoped to help smaller closed-source providers reach developers and earn revenue.

Learn the capabilities by building with them

The closing career advice follows the same abstraction argument. Building AI products need not require working directly in PyTorch, just as building a website need not require understanding network-stack internals. Developers do need practical knowledge of model capabilities and limitations. Experimenting with language models, diffusion models, fine-tuning and prompting builds that knowledge; dataset construction and prompt writing may become ordinary parts of the job.

The invitation extends to tool builders. Replicate began with a software engineer wanting to use capabilities a machine-learning collaborator could already build. Its closing argument is that usability constrains what product developers can do with AI: capable models exist, but the tools and abstractions that make them broadly useful remain incomplete.

Teaching was part of that work. Replicate’s developers produced articles, videos and example applications, and the conversation highlighted a hacker-in-residence opening available at the time. The team had originally been called developer relations, or DevRel; its account credits Zeke’s dislike of conventional titles with the move toward “hacker.” An interviewer closes by describing the San Francisco office and meetups as a hacker space—a fitting setting for a business whose first paying customer arrived through an unsupported API request.