Distillation, borrowed answers, and the limits of SWE-bench

The same API loop, two different purposes

Send mathematics questions to an LLM and save its answers: the same loop can produce benchmark results or training examples for another model. One participant in SAIL Live #6 describes using DeepSeek V3.2 through OpenRouter for distillation and almost the same script for evaluation, including MATH-500 and a mathematics dataset with 12,000 examples. The provider sees answer generation; training happens later, elsewhere.

That visibility gap shapes the February 2026 debate over Anthropic’s allegations of distributed distillation activity associated with Chinese labs. One participant questions the framing of this activity as an attack and argues that GPU shortages make API-generated training data attractive. Another challenges the suggestion that enforcement primarily targets Chinese companies, recalling that Anthropic had also blocked OpenAI and xAI.

Distillation uses a larger teacher model’s outputs to help train a smaller student. It predates LLMs, but its meaning has broadened: - Logit-based distillation trains from the teacher’s output-layer signals, called logits, rather than only its selected answer. - Generated-text training uses the teacher’s answers as synthetic training data for the student. DeepSeek R1, a 671-billion-parameter flagship, illustrates the second approach: its outputs train smaller models suitable for local use. The disputed boundary is generating that data from another company’s model. Nathan Lambert’s linked discussion of distillation and Chinese LLMs also introduces this broader contemporary meaning.

Restrictions on using API outputs to train competing models are older than this dispute. A participant recalls similar concerns around Alpaca and other research models after ChatGPT’s release, followed by limited enforcement and a quieter debate. They interpret the controversy’s return as a sign of stronger competitive pressure.

Traffic signals do not establish downstream use

Several proposed detection signals have legitimate-use explanations: - Volume: large collections may suggest training-data generation, but large benchmark suites and popular customer chatbots also produce many requests. - Repetition: repeated questions may look suspicious, yet internal evaluations can run the same questions three to five times. - Topic distribution: broad coverage might suggest general-purpose distillation, while a mathematics benchmark or support chatbot could be narrower. This is a proposed heuristic, not a confirmed provider practice. Patterns across related accounts might add information, but the participants do not establish how accounts were linked in the reported cases.

Recognizing benchmark questions could help classify traffic, but content-based detection requires examining customer content. One participant finds that uncomfortable even while cautioning against assuming remotely generated data is private. Inspection still cannot directly reveal whether an answer later enters a training dataset outside the provider’s platform.

Collection scale also changes the economics. Substituting an API into an existing generation pipeline is a small experiment compared with collecting millions of exchanges across rate-limited accounts. OpenRouter provides hosted access and provider comparisons for open-weight models too large to run locally, exposing prices and token-generation rates through an OpenAI-style API. One participant reports occasional failures around apparent provider switches, while allowing that their own script could be responsible.

Comparisons between MiniMax’s and DeepSeek’s reported traffic expose another missing variable: request totals need a collection window. Four weeks and six months imply very different activity. A participant recounts an allegation that MiniMax redirected nearly half its traffic when Opus 4.6 launched, interpreting the rapid switch as evidence of a coordinated operation. Whether training-cycle timing or better data selection explains differences between labs remains contested. Without a common observation window or average exchange length, totals cannot reliably establish relative intensity or token volume.

The strongest model may be the wrong teacher

Collecting answers is only useful if the student learns from them. In supervised fine-tuning (SFT), the student trains on teacher-generated question-and-answer pairs using an autoregressive, or next-token prediction, loss. A stronger teacher does not necessarily produce a stronger student. The proposed explanation is teacher–student compatibility: a teacher’s answer style or token distribution may be too different from what the student’s base model can readily learn.

A participant describes dense Qwen models as effective teachers for many open-weight students, including OLMo, and GPT-OSS as another useful teacher. Their hypothesis is that many students are Qwen-based or already resemble Qwen’s distribution. Replacing a teacher with GLM-4.7, a larger DeepSeek model, or a newer large Qwen model does not reliably improve SFT results, even with the same prompts and suitable sampling settings. Teacher quality must be measured through the student’s improvement, rather than inferred from the teacher’s own performance.

The training signal matters too. An explicit correction establishes that the OLMo training discussed used generated tokens, not teacher logits. Participants identify Gemma as an example of logit-based distillation. Open-weight teachers offer control over the full output distribution; proprietary APIs, in their account, expose at most limited top-token information rather than full logits.

Demonstrate learning signal before scaling collection. Substitute teachers in the generation pipeline, train students on their outputs, and compare performance. This kind of controlled comparison—an ablation—should also consider throughput: a medium-sized teacher may generate more useful data in the available time than a larger model. The imagined rush to collect 100 billion Claude tokens represents a decision after finding useful signal, not a reported training run.

Product exclusivity versus API demand

Teacher selection leads to speculation about models the public never sees. One participant imagines Gemini Ultra serving as an internal teacher, without establishing that Google uses it this way. The participants also disagree about whether labs routinely distill between particular architectures, while broadly agreeing that internal teachers may differ from public releases. Serving speed cannot settle model-size questions: concurrent demand, hardware, and allocated capacity all affect observed performance.

Reserving models for proprietary products could reduce exposure through APIs, but the business arguments pull in different directions: - Product exclusivity: a lab could reserve differentiated capabilities for consumer or coding products rather than compete in a crowded API market. Its advocate presents this as a provocative possibility, not a confident forecast. - Profitable API access: developers build applications the lab cannot anticipate, and heavy usage can exceed subscription allowances. API demand can support a business when token prices cover costs. OpenClaw supplies the demand example: a participant describes users needing API access instead of subscription-plan access in that context, with substantial token consumption.

Temporary exclusivity does not establish a permanent strategy. One participant recalls the last three GPT-5 releases having Codex variants available inside Codex two to four weeks before API access. Another suggests the latest window could have been intended to promote a newly released app.

SWE-bench: tests can reward the wrong success

SWE-bench evaluates coding work using open-source issues and the pull requests that resolved them. A model must make failing tests pass while preserving tests that already pass. Unlike HumanEval-style completion, it specifies a problem and desired outcome while leaving the route to the solution open—an agentic task. Its original sampling concentrates many issues in relatively few popular projects, limiting coverage of software work.

SWE-bench Verified, introduced in 2024, is OpenAI’s curated subset of 500 original tasks. A participant describes three human reviewers assessing each task’s quality. The issue identifiers, problem statements, and test information are public, making the benchmark inspectable while also exposing material that training corpora can absorb.

Tiny leaderboard gains need careful interpretation. One participant estimates run-to-run variation of roughly 0.5–1 percentage point and warns that choosing the highest run can exaggerate progress; no repeated-run measurements accompany that estimate. Another supplies an arithmetic check: on 500 binary-scored tasks, one additional success is 0.2 percentage points. Averaging runs can yield finer increments. A separate suggestion about partial credit does not establish that Verified uses it.

A more consequential problem emerges in the account of OpenAI’s re-audit of difficult tasks near an apparent 80% plateau. A participant reports six reviewers per task, another team for flagged cases, and 59% of the examined tasks judged unsolvable. The 59% concerns the examined sample, not all 500 Verified tasks; the exact sample size is unspecified.

One example asks for a string in the output, but its test allegedly requires a particular identifier that the stated requirements never establish. An overly specific test can reject a reasonable implementation and reward recall of the existing one. The participants propose repurposing that failure mode as a safeguard: a canary task whose suspicious success flags possible contamination. This is a proposed diagnostic, not a validated detection method.

Contamination through ordinary training data

Benchmark contamination occurs when training exposure supplies information that compromises the evaluation’s intended test. A participant reports that OpenAI found GPT-5 reasoning using knowledge of Django versions released after the task’s historical setting. Another recognizes a related failure in everyday programming assistance: models suggest newer APIs for scripts targeting older library versions.

Contamination does not require deliberate benchmark training. Relevant information can arrive through repository clones, copied tests in other libraries, or blog posts and conference talks explaining later features. Removing the benchmark’s own page leaves these other routes of exposure.

A participant recounts a more direct probe: OpenAI supplied SWE-bench task IDs to Gemini Flash and Opus and asked them to repeat the tasks. The models reportedly reproduced problem statements and solutions from identifiers alone. Without exact model versions, task counts, or success rates, this describes a failure mode rather than its prevalence.

How little exposure can produce exact recall remains an open question here. One participant wonders whether one or two corpus appearances could suffice, but supplies no exposure-count experiment. Another describes replaying older data during continued pretraining to reduce forgetting. Retaining useful knowledge and retaining evaluation answers are intertwined concerns, without a reliable replay percentage or general rule for single-pass memorization established in the discussion.

Replacing the benchmark requires new tasks and continued scrutiny

Scores can converge while practical usefulness remains different. One participant describes MiniMax M2.5 and GPT-5.2 performing similarly on Verified while preferring GPT-5.2 in personal use. Their hypothesis is that defective tasks prevent stronger models from earning additional credit. The comparison reflects personal experience, rather than a controlled measurement of relative utility.

SWE-bench Pro introduces a new problem set, whereas Verified selected tasks from the original benchmark. The improvements discussed address distinct weaknesses: - Private and public splits address exposure of evaluation material. - Newer task dates refresh the historical problems being tested. - Broader repositories and languages reduce concentration in a few projects. - Improved tests aim to align acceptance criteria with stated requirements. A participant reports that OpenAI is moving its reporting from Verified to Pro.

Fresh review does not guarantee valid tasks. Verified also had reviewers and professional incentives behind it; defects emerged after extended use and further scrutiny. A participant cautions that Pro could follow the same pattern, without claiming defects have already been found. Private evaluation’s access protocol also remains unresolved: agreements and server-side scoring are discussed as possibilities, not established operating details.

Beyond coding, evaluation becomes a system problem

Other domains need better evaluations too. Coding and mathematics offer comparatively objective checks; outside them, judgments often return to subjective preferences. Computer-use evaluation resembles the move from unit tests to end-to-end tests. An LLM controlling a computer must complete broader workflows whose outcomes are harder to specify and measure than those of isolated coding tasks.

One participant forecasts frontier evaluation costs reaching tens or hundreds of millions of dollars, widening the divide between frontier development and other research. These are predictions, not disclosed budgets, but they express a consequential concern: measuring progress may become a major resource barrier of its own.

The session closes with SAIL and the informal “eval pals” collaboration, described as a way for people with aligned interests to amplify each other’s work. Another participant values the simpler benefit of direct conversations with human experts amid an apparent rise in AI-generated emails and social responses. The closing emphasis is on keeping that human exchange alive.