Replit’s code-model sprint: data pipelines, contextual completions, and the limits of benchmarks

A second-by-second market question

What happens to a company’s stock in the seconds after someone mentions it on television? In 2009, Reza Shabani investigated by recording CNBC for ten hours a day, extracting closed captions, and joining the resulting second-by-second transcript with high-frequency trading quotes. He used natural-language processing to identify company names, then examined subsequent price and trading-volume changes. Building that dissertation pipeline forced him to learn Python and programming beyond specialized data-analysis languages.

His route into engineering was circuitous. The dissertation attracted more interest from computer scientists than economists, and he received no academic offers. He declined a BlackRock offer to start a company, then returned two and a half years later after the startup failed. Later, an unsuccessful conversation about a product role at Replit opened another route through its data and analytics needs. His economics background still prompted questions about whether he could code.

Swyx recognizes the path: he also learned Python in quantitative trading, building pipelines with unversioned notebooks while his code supported management of a billion dollars. Both accounts connect an analytical question to the engineering needed to make it answerable.

When a technology mention becomes a signal

At BlackRock, Shabani gravitated toward analyst reports, earnings reports, and conference-call transcripts. An experimental strategy called emerging technologies asked whether companies adopting new technologies ahead of competitors would outperform. Turning that hypothesis into a candidate trading signal required three steps: 1. Assemble startup descriptions and the timing of their funding. 2. Use non-negative matrix factorization to group the descriptions into technology clusters, such as mobile, social media, and digital advertising. 3. Identify publicly traded companies discussing those technologies and showing exposure to them.

The consequential distinction was volunteering a trend versus answering an investor’s question about it. Shabani recalls Walmart proactively discussing mobile while Sears did not; investor prompting might indicate that a company was already late. Domino’s also surfaced, with its mobile pizza tracker as a concrete example. These comparisons illustrated the hypothesis, but Shabani does not think the team ultimately traded it and gives no backtest results.

By the time of the conversation, Shabani thought widespread AI mentions would weaken this signal: mentioning a technology distinguishes little when nearly every company discusses it. The broader trading problem was equally difficult. Historical tests could fail under changed market conditions, and an overlooked factor could explain returns previously credited to a strategy’s unique edge. Limited usable history and competitors actively trying to outmaneuver the system made a convincing pattern a fragile basis for action.

Ghostwriter began with data infrastructure

Replit’s initial problem was more basic than training a language model. It had production databases but lacked infrastructure for querying and processing data at scale. Even asking which repl—a Replit project—had been forked most could time out. Modernizing the pipelines made analytics and extracting projects for search practical. The team described work becoming possible in minutes instead of days or longer, though it did not specify a benchmark workload.

Those pipelines became the foundation for training models and further training them on Replit data. The AI and data team grew to cover [Ghostwriter](https://replit.com/learn/intro-to-ghostwriter) code completion, explanation, generation, transformation, and chat inside the integrated development environment, or IDE. It also supported search, large-scale processing, and abuse detection. Language models shifted attention toward text processing, but the discussion emphasized the continuing value of other specialties: analyzing variables over time requires distinct skills that Replit used relatively little.

The original Ghostwriter shipped in August of the preceding year, predicting the code that should follow existing text. It used a smaller Salesforce CodeGen model trained on public code, with distillation and other optimizations to reduce its footprint and serve completions quickly enough. Replit wanted behavior better suited to its users’ JSX, HTML, and inline CSS workflows. Fine-tuning CodeGen did not deliver that behavior, prompting investment in training from scratch.

HumanEval passed; the editor experience did not

HumanEval offered a clear evaluation procedure: give the model a Python function definition and a description of its intended behavior, let it complete the function, then run unit tests. Some fine-tuned CodeGen models performed well on this benchmark and still disappointed in use. One seemed overtrained despite strong scores, although that was an impression rather than an established diagnosis.

The complementary test was AmjadEval: Amjad probing a model with carefully chosen prompts to assess its behavior quickly. Contextual completions deep inside functions were an important target. Several comparisons discussed during the demonstration show what the team valued: - In process-cloning code, Replit’s completion used readable constants where CodeGen initialized an integer-related value to zero. - In a Python expression, Replit generated an inline lambda while the competing completion used the nearest available variable. Replit’s model used only preceding text, not context after the insertion point. - In a tokenization assertion, the presenter described CodeGen as inventing tokens. The demonstration also included a CodeGen win: Ghostwriter imposed a list requirement on a function argument that the prompt had never specified.

A JavaScript loop containing setTimeout produced a memorable reversal. Bard’s explanation initially looked right to both discussants. They then recognized the behavior identified by Replit’s completion model: the callbacks read the iteration variable after it had reached its final value, reported as five, so that value repeated. The exact snippet is unavailable here, so its loop bounds and timing cannot be reconstructed. The example remained an informal comparison, and the presenter agreed with a discussant’s correction: it resembled an interview question more than a test of a model’s feel.

The broader vibe test combined low latency, useful suggestions, a felt improvement in productivity, and avoidance of plausible-looking mistakes. A screenshot could show a completion but could not convey its responsiveness. Replit needed to evaluate both the generated code and the experience of working with it; isolated function tests did not settle both questions.

Unexpected reasoning, uncertain causes

Another comparison posed a nonsensical question about the square root of a banana, with instructions to return unknown for nonsense. The original 175-billion-parameter GPT-3 reportedly answered nine; Replit’s model returned unknown. Neither model in that comparison had been instruction tuned—given additional training aimed at following instructions. Replit planned that step, but its unexpectedly strong natural-language behavior had already prompted evaluation on HellaSwag. No HellaSwag scores were given in this exchange.

The code data contained substantial natural language in comments, Markdown, reStructuredText documentation, and HTML. The presenter doubted that comments alone explained the behavior and proposed a broader hypothesis: code training might develop logic useful for subsequent natural-language training. The causal explanation remained open.

That uncertainty constrained deliberate improvements. The team could not explain why its model solved the programming example when Bard failed, or reliably isolate the qualities behind its feel. Its practical lever was the data mixture: choose material representative of the generations users should receive, with instruction tuning as a later option.

From predicting text to predicting IDE actions

The proposed direction for Ghostwriter extended beyond completing a single file. It would select models for particular tasks and help with project scaffolding and cloud configuration. One envisioned workflow began with a website receiving more traffic: inspect its code and traffic, generate a backend database schema, and let the developer deploy it in Postgres. Another would handle configuration work that otherwise required reading documentation, inspecting the codebase, asking colleagues, and preparing a pull request.

[Ghostwriter Chat](https://blog.replit.com/ghostwriter-building) already used IDE context to inspect an error traceback and suggest a fix. An autonomous agent capable of driving the IDE—including executing and repairing code—was still a goal.

The proposed mechanism was next-action prediction over development sequences. A developer might clone a repository, create a file, change references elsewhere, update configuration, run code, respond to errors, and deploy. Replit reported terabytes of operational transform data recording editing sequences such as file creation, naming, typing, and coordinated changes across files. The plan was to train a subsequent model on development history that also included execution, errors, and developers’ responses to them.

A small model with a large token budget

[Replit Code V1 3B](https://huggingface.co/replit/replit-code-v1-3b) was described as a 2.7-billion-parameter model trained on 525 billion tokens of permissively licensed code across three epochs—three passes over the training set. The presenter used LLaMA-style to mean a relatively small model exposed to a large token budget. The announced plan was to release the completion model and tokenizer that week; releasing the additionally Replit-tuned variant remained under consideration.

Replit trained its own tokenizer and vocabulary on code instead of reusing CodeGen’s. The vocabulary contained roughly 32,000 entries, compared with CodeGen’s vocabulary in the 50,000s. The presenter credited this code-specific choice with faster training and inference and more relevant output. The Replit-tuned variant retained the same model size and vocabulary, adding training on public Replit data described as permissively licensed.

Beginner material survived quality filtering, yet the resulting model reportedly reasoned better and felt more instruction tuned. The team suspected that educational assignments, explanations, and answers exposed programming logic more directly than complex legacy codebases. That explanation had not been isolated experimentally. A reported commonsense-benchmark improvement from 20% to 30% meant ten percentage points, or 50% relative improvement; the benchmarks and evaluation protocol were not identified.

The one-week sprint rested on months of preparation

The rapid training push followed months of infrastructure work on a platform said to contain roughly 230 million repls, each potentially holding many files. An internal demo reportedly extracted every Python function signature across Replit’s data in under two minutes, though hardware and workload details were unspecified. Before the larger run, the team had established a training-to-production loop with 1.3-billion-parameter models trained separately on JavaScript or Python using roughly 20–30 billion tokens.

Open questions included how to prefix chunks for multilingual training. Amjad pushed the team to use all available data before completing its planned series of smaller experiments. The hesitation was practical: additional cleaning might help, training was expensive, and a disappointing result after a long run would be difficult to diagnose. The run took about two days on 256 A100 GPUs, with evaluation requiring more time afterward.

Michele proposed increasing token exposure through multiple epochs over the same data. The team considered overfitting a risk but proceeded and judged the base model successful. With the GPUs available, it continued into additional runs. Existing pipelines let it prepare Replit data in about two days, including selection, deduplication, and filtering for personal information and harmful content.

The team also completed a run identified as UL2, intended to support fill-in-the-middle completion: using context after the insertion point as well as before it. Deployment and testing were still ahead. This would extend the earlier demonstrated model, which used only preceding context.

The account credits Amjad’s push with compressing perhaps two or three months of planned work into the week before Developer Day: one week of concentrated experiments after months of preparation. The team interpreted the results as evidence that code models had been undertrained, while emphasizing how few expensive runs had been tested thoroughly with real product users. More compute and token exposure appeared helpful in these runs; the finite supply of code constrained how far that approach could go.

Keeping training productive

Replit had operated its own training infrastructure before using MosaicML. Mosaic let it obtain GPUs from multiple providers without separately signing up for each cloud, separating training compute from its predominantly Google Cloud infrastructure. The team also valued handling of GPU and node failures, preconfigured training setups, and recovery from loss spikes. It reported spending little time troubleshooting underutilization, crashes, or out-of-memory errors, though it supplied no measured utilization or failure-rate comparison.

The choice to stay with GPUs rather than Google’s alternative training hardware rested on ecosystem adoption: the team wanted systems other developers were already building for. It was an argument about available tooling and experience, not a measured hardware-performance comparison.

Replit’s hiring needs at the time showed how much engineering surrounded the model: - Applied AI and machine learning: data pipelines, training variations, output testing, human evaluations, and troubleshooting. Tokenizer training alone required sampling across languages and distributed pipelines before feeding data to SentencePiece. - Inference: optimizing transformer layers to reduce latency and scaling the serving stack. - Full-stack product work: integrating completions and chat into the workspace, building planned agents, and supporting questions about documentation and projects.

Beyond chat, and beyond prompting workarounds

The closing discussion returned to the speed of change. A Black Mirror story about recreating a deceased partner’s conversational style from social-media data illustrated how a once-surprising chatbot premise had become familiar. Perceived threats to white-collar work were also making questions about helping workers and sharing economic benefits more urgent. Looking roughly a year ahead, the forecast emphasized applications in specific products and industries beyond general chat.

A hypothetical travel assistant made that direction concrete. Before an Airbnb booking, it could generate queries for nearby noise complaints and construction permits, checking whether construction overlapped the dates of the stay. The proposed value was connecting a traveler’s concern to specific records and date conditions.

For builders, the requested product was an LLM IDE—a development environment for working with large language models. It would let users create prompts and compare them across providers without repeating manual work. Nat.dev supplied the reference point: a playground that sent one prompt to several models, reportedly commissioned by Nat Friedman through a Replit bounty and later put behind a paywall because usage was expensive.

The proposed extension would retain responses alongside their generation parameters and track differences across models, prompting strategies, and time. That tooling mattered even to people training models: code-completion inference and instruction tuning still involved constructing prompts.

The final technical disagreement separated two meanings of prompt engineering. One participant expected model improvements to eliminate workarounds, comparing them to middleware that rewrites requests into keywords for a weak search engine. Another argued that communicating intent, supplying context, and giving instructions would remain necessary even for general intelligence. The positions converged: workarounds may disappear; communicating intent remains. Management supplied the analogy—clear instructions, communication frameworks, and revision when the recipient misunderstands.

The closing advice was to learn to use AI, understand its limitations, and identify where it improves your work. The speaker acknowledged that some societal fears might be warranted, while comparing AI with the internet: a powerful change that creates opportunities for misuse and broad benefit.