Teaching Genie the work behind the patch

An app generator that barely worked

Cosine’s early prototype took a mobile-app idea, chose a stack, broke the project into increasingly detailed pieces and asked Codex to write every component from scratch. With roughly 4,000-token context windows, it relied on chains of prompts carrying information forward. Occasional outputs had functioning backends and databases, but the system barely worked. Alistair “Ali” Pullen attributed much of its fragility to regenerating everything without templates.

Pullen and his co-founder Sam came from iOS and Android development, respectively. Their student consultancy grew into larger projects, and they later became essentially the entire engineering team at grocery-delivery startup Fancy, building customer apps, backends, stock management and driver routing. In retrospect, Pullen would have bought more components off the shelf. After Fancy’s acquisition by GoPuff and an integration period, they left around 2022 to pursue their own startup. GPT-3’s ability to produce tables and JSON drew them into experiments; discovering Codex directed those experiments toward software generation.

By the August 2024 Latent Space conversation, that ambition had become [Genie](https://cosine.sh/blog/genie-technical-report), an agent trained to perform software-engineering work. Pullen’s thesis was to teach the process behind a change: information lineage, meaning where the information behind a decision came from, alongside incremental discovery and step-by-step decisions. The episode’s introduction described work with OpenAI on GPT-4o fine-tuning using billions of tokens, much of them synthetic. Swyx disclosed that he was a small angel investor in Cosine.

Useful engineering needed both capability and context

The pivot toward existing repositories happened during the team’s YC interview. Accepted with two weeks to build a minimum viable product, they produced a command-line tool they considered poor. They then concentrated on retrieval infrastructure while waiting for models capable of useful engineering work. Running semantic code search locally would minimize sending customer code to the cloud, but its search index consumed excessive RAM inside VS Code on repositories with millions of lines. Building that infrastructure occupied roughly the first six to eight months.

Context capacity and model capability were separate constraints. A 16K-token window could still be smaller than a Python file. Experiments with the model described as “3.5 16K” suggested that the proposed loop and data-generation approach had signal, but neither its context capacity nor its capabilities supported useful engineering performance. Fine-tuning GPT-4 with an 8K window helped despite the smaller context. A 128K window finally looked practical, and experimental access to GPT-4 Turbo fine-tuning let the team combine a stronger model with its already-built data pipeline.

The grounding goal was to make generated changes traceable to information in context, reducing the need for the model to invent missing code. The first trained Genie nevertheless performed badly: arbitrary dataset choices became behavioral problems, and generating Git diffs that reliably applied required iteration. SWE-bench gave development a concrete evaluation target—whether the agent could solve software issues—where earlier checks had relied on subjective judgments of usefulness.

A pull request loses the investigation

A pull request preserves the final changes but little of the investigation that produced them. Cosine’s training pipeline attempts to reconstruct how developers reached a solution, including why they made particular decisions. The team reports that, after addressing output structure and reliable patch generation, extracting more of this historical problem-solving signal became its largest driver of SWE-bench performance. That is a qualitative finding, without a quantified comparison isolating its contribution.

An early, loosely cleaned open-source dataset taught the wrong mix of behaviors. The model became good at README edits and comment rewriting, but also absorbed argumentative review exchanges and pushed back on feedback. Documentation changes dominated the scraped material enough to require substantial filtering. Developer activity carries both a task distribution and social habits; Cosine had to filter for the work and responses it wanted in customer workflows.

Access to private workflows imposed another constraint. Cosine reported that willingness to share code and historical data varied mainly by sector. Large prospective customers often wanted an initial sandbox and imposed other conditions, while remaining interested in potential productivity gains.

Retrieval becomes a learned investigation

Finding the right repository files is more fundamental than accessing external information. Cosine’s criticism of agents built by wrapping foundation models in tools did not mean those tools were unimportant. Genie had a URL-reading tool, used Perplexity’s API to ask questions and had code-interpreter access. The priority was getting the repository context right so that the agent could act on the task.

The difficult retrieval problem was searching by functionality—for example, finding authentication logic from an English request. Cosine’s approaches evolved through three forms: - Match an English query to code: split the repository into chunks, embed the chunks and query, and return the top k matches by cosine similarity. The team reported poor results, attributing them to the semantic mismatch between English descriptions and code. - Search through hypothetical code: generate a snippet resembling the answer, embed it and search for similar code. This reportedly improved retrieval and became part of an engine combining semantic search, keywords and language-service navigation, with a model choosing methods for each request. - Teach the model to navigate: Genie moved away from that engine. Training examples generated through self-play taught it how to investigate a repository using keyword search, basic semantic search and code-graph navigation.

Genie starts with directories and filenames, opens plausible files, then follows imports, definitions and references. A running language service supplies operations such as go-to-definition; the agent does not operate inside VS Code. Each file can reveal where to look next, allowing retrieval to proceed as an investigation rather than a single search.

Cosine measured retrieval using lines edited in historical pull requests as targets. The speaker recalled finding roughly 65–66% of target lines, and reported that adding language-service navigation and encouraging developer-like traversal moved coverage from about 54% into the 60s. These were internal comparisons of line coverage, not task-success rates, without controlled experimental details.

That measurement exposed a narrowing funnel: retrieve the necessary information, reproduce the issue, then solve it. Early Genie development had concentrated on solving issues once the right information was available. Improving retrieval offered a way to get more tasks far enough through the funnel for that work to matter.

Planning and coding have different context demands

Cosine’s answer to planning was to train on the desired reasoning behavior. The team argued that prompting a foundation model to think step by step was insufficient for its target performance, while acknowledging that reasoning remained constrained by the underlying model. Its longer-term bet was to adapt the training data to stronger models as fine-tuning became available. Better reasoning models might help twice: generate better training examples, then learn from those examples. This was an anticipated bootstrapping strategy, not a measured result.

The planned interface would let users intervene throughout execution: - Edit the plan while it runs, redirecting the agent’s approach. - Supply Markdown playbooks describing preferred workflows. - Correct a generated patch and resume, so a small error would not invalidate an otherwise useful run.

To examine where the model was more or less certain, the team averaged token log probabilities at each position across a 128K context window. It reported greater certainty during code generation than during retrieval or planning—an observation about token probabilities, not proof of patch correctness. Genie was trained to emit diffs instead of whole rewritten snippets, conserving tokens as the loop continued.

A 128K window did not mean equally reliable performance throughout it. In Cosine’s reported SWE-bench analysis, the probability of solving an issue declined approximately linearly with context length, reaching about 50% at 60K tokens. Above that length, failure was more likely than success. The speaker attributed the decline mainly to the underlying model, despite training examples showing successful solutions around 110K tokens.

The analysis did not establish the cause of that relationship or whether a larger window would move the threshold proportionally. It did guide small optimizations intended to keep Genie within its more reliable context range, without structural changes to the system.

Language coverage, tests and supervision have boundaries

Language coverage reflected internal needs and generation cost. An interviewer cited a training mix of 21% JavaScript, 21% Python, 14% TypeScript and 14% TSX—49% across the JavaScript-related categories. The response described reasonable generalization without per-language evaluation results. The team wanted a future mix roughly following GitHub language popularity, but broader coverage required more resources.

Existing continuous integration (CI) was the prerequisite for the planned testing workflow. Genie would trigger the repository’s GitHub CI pipeline, read its outputs and feed those results into context. Autonomously setting up a repository was outside the intended scope; execution depended on infrastructure the customer already operated.

The supervision modes made a separate tradeoff: - Headless execution: assign a ticket and let Genie attempt it without asking questions. - Website interaction: allow clarifying questions, especially when a broad request could send the agent in the wrong direction. Once work was underway, issue comments and pull-request review comments could steer it.

Scaling training capacity—and teaching recovery

Fine-tuning at Cosine’s data volume required changes beyond the dataset itself. The team reported access to larger fine-tuning adapters, called LoRA adapters in the discussion, than the public offering, along with changes to the learning-rate schedule. Adapter capacity became an unresolved scaling question: would additional training hit the adapter’s limit or the underlying model’s limit? Cosine described exploring larger adapters and pre-training extension with OpenAI. Its dataset size was constrained by generation cost, rather than a demonstrated point of diminishing returns.

A separate weakness appeared when training contained only successful problem–answer pairs: the model was poorly prepared to recover after a wrong first attempt. Recovery had to appear in the training examples. One technique deliberately altered abstract syntax trees to introduce faults such as out-of-bounds accesses or references to nonexistent variables. These errors were injected probabilistically to represent mistakes models sometimes made.

A second technique used the model’s own failures to build repair examples: 1. Train an initial model on clean problem–solution pairs. 2. Let it attempt another problem, producing an intermediate codebase that may be incorrect. 3. Compare a failed state with the known correct state to derive a corrective diff. 4. Train on the repair and repeat, teaching the model to continue from mistakes instead of depending on a correct first patch.

Benchmark outcomes and the work behind them

The episode introduced launch scores of 50% on SWE-bench Lite, 30% on the full benchmark and roughly 44% on [SWE-bench Verified](https://openai.com/index/introducing-swe-bench-verified/). The later discussion supplied the conditions for the Verified figure: 219 solved instances out of 500, or 43.8%, calculated by matching Verified instance IDs against an existing full-benchmark run. It was not a fresh Verified-only evaluation. Cosine described the score as state of the art to its knowledge at the time, while acknowledging that few leaderboard entries limited the comparison.

The leaderboard dispute concerned trajectories: the intermediate context and steps leading to a patch. Cosine’s representative said the team expected to submit final patches and the evaluation harness’s report.json, but reviewers required trajectories. Cosine withheld them because it believed they exposed the structure of its proprietary training data and could help competitors reproduce its behavior. The speaker said the patches remained public for others to run. Checking a patch’s outcome and inspecting the process that generated it therefore remained distinct.

Evaluation speed constrained model iteration. Full runs were expensive and slow enough that Cosine considered a roughly 300-instance subset preserving the benchmark’s difficulty distribution; the team regarded Lite as too concentrated on easier tasks for that purpose. It welcomed Verified’s smaller, human-reviewed set of solvable problems as a way to shorten iteration. Cosine did not disclose its actual evaluation cost.

Even a faster evaluation leaves a diagnostic gap: binary success hides where a run failed. An agent can make substantial correct progress and lose the task through a late mistake. Cosine planned to inspect failed solutions for the point where they diverged from a correct state, then triage those failure modes. The intermediate work mattered for evaluation as well as training.

Repository specialization and a different developer workflow

An internal variant called “the John” specialized Genie for Cosine’s own repository. The team ran its main data-generation pipeline on that codebase, then used the resulting examples to fine-tune base Genie. The speaker described strong repository-specific usefulness without a quantitative evaluation. Alongside expanding the dataset and trying more underlying models, this offered another direction: teach general engineering behavior, then specialize it to a team’s code.

The closing discussion praised Cursor for putting assistance where it helps and getting out of the way when it does not, while retaining a familiar VS Code experience. The speaker had initially doubted that users would download a separate editor fork, but now regarded that choice as successful. Building Genie had reinforced a related lesson: small teams should attempt difficult work because it can offer the greatest payoff.

Genie’s own product thesis asked developers to adopt a different workflow: the model writes code while the developer guides the work, retaining the ability to inspect and edit code when needed. Cosine was seeking customers willing to try that arrangement.

At the time, its customer and hiring priorities followed that thesis: - Start with well-represented languages: TypeScript, JavaScript, Python and Java were preferred starting points. The team claimed broader language and repository-size support, while qualifying that performance would vary. - Prioritize willingness to experiment: company size mattered less than openness to the workflow and manageable information-security constraints. - Combine experimental work with shipping: hiring plans spanned product engineering, AI, machine learning and datasets, seeking people with strong interests who wanted to turn experimental technology into usable products.