Write benchmarks for the work you actually do

Make the theoretical machine run

Conway’s Game of Life can run arbitrary programs in principle. Nicholas Carlini wants to make that possibility practical: build a CPU inside its grid of on/off cells that runs at something approaching real time on his computer. Turing completeness means the rules can support arbitrary computation; making a usable machine requires more. At the time of the conversation, the CPU was a work in progress.

The same fascination led Carlini to demonstrate computation inside C’s printf. His construction uses %n to overwrite an in-memory position involved in format processing, enabling loops and conditionals. The implementation prerequisites are not specified, so this describes a particular construction rather than a recipe for arbitrary printf implementations.

These detours preserve the pleasure of programming alongside his main work in security research. He writes to share what he finds interesting, despite disliking writing enough that the prospect of producing papers nearly deterred him from pursuing a PhD.

Concrete experience also motivates “How I Use AI”. Carlini rejects both predictions that programmers would be obsolete by 2024 and claims that language models cannot do useful work. His position holds two conclusions together: models can be useful and have serious security problems that make many deployments inappropriate.

Get past the boilerplate

An early application example is a web app where visitors guessed whether GPT-4 could solve a task. It needed basic HTML and CSS, a slider, a button, and animated text. Its creator understood web development but was unfamiliar with newer conventions such as Flexbox. Model-generated boilerplate helped get the project started, leaving the novel work to the programmer.

The debugging loop was manual: run the generated code, fix errors directly when possible, and otherwise paste the error message back into the model. An autonomous debugging agent might handle more of that loop, but the existing workflow already helped. Imperfect output can still be useful when the programmer checks it.

Docker illustrates a related use: learning enough of an unfamiliar tool to unblock a larger task. The goal was to execute model-generated Python in a contained environment. The user understood the underlying security concepts but had not used Docker before January. When the model introduced Docker Compose, the conversation could pause for an explanation of that unfamiliar term.

That ability to interrupt makes the explanation fit the learner’s immediate gap. The speaker estimates that some questions otherwise requiring hours took about ten minutes through targeted dialogue. This was a personal estimate, and the Docker setup depended on an existing security background that let the user assess its safety.

These experiences mostly involved small research projects with freedom to choose tools. Large proprietary legacy codebases may require different workflows. The invitation to engineers in those environments is to publish concrete accounts of tasks they have completed with models, extending the discussion beyond speculation about other people’s needs.

Cheap attempts can preserve expensive attention

Disposable software changes which tasks are worth attempting. Consider the tradeoff offered in the conversation: a minor task may not justify 45 minutes of coding but may be worth a five-minute attempt with a model. If it fails, the user can abandon it or decide to invest the full effort.

The same logic encourages probing beyond reliable, narrow use cases. One participant describes trying uncertain problems that usually fail but occasionally save hours, especially when checking an answer is easier than producing it. Another reports sending Claude a photograph of an unfamiliar irrigation controller and receiving instructions that worked; there was no comparison with another provider.

One proposed label is semantic serverless functions: describe an operation in natural language, have it performed, and discard it. The respondent does not adopt the framework, preferring to try available tools and retain what helps.

Private usefulness and public demonstration require different evidence. The published list of uses deliberately excluded debugging, despite its personal importance. A conversation about fixing local CUDA version mismatches could show plausible instructions without demonstrating that the repair worked. Verifiable examples also make it harder to hide substantial human work behind a claim that a model solved the task.

Routine helper functions offer a less visible benefit: preserving mental state. Novel research contains many familiar components. Stopping to implement a known algorithm can displace the difficult problem already held in mind. Asking a model for the helper, running it, and checking it quickly can reduce that interruption. Automating familiar pieces can support an original result without solving the whole research problem.

Match verification to the task

Who benefits more: experts or non-experts? The answer remains open. The direct experience described is expert use, where the user can independently inspect model output. Non-experts might gain programs or spreadsheet automation they could not otherwise create, provided they can test the results rigorously. Blindly deploying generated work creates a different risk from using it under careful supervision.

Capability judgments also need a time boundary. One researcher recalls dismissing language models as impractical toys even in 2020 after studying GPT-2, then recognizing that assessment was wrong. A conclusion about models from four or five years earlier may not describe the models available when a user makes a decision.

A participant’s use of Gemini to generate Google Sheets formulas exposes another obstacle: discoverability. A blank chat box gives little guidance about useful tasks. Integrations can surface those capabilities, but they should offer functionality whose correctness can be checked. Adding AI merely for marketing does not help, and those weak applications should not determine judgments about useful ones.

For experienced engineers, models can bridge the gap between understanding an outcome and remembering its syntax. FFmpeg provides the example: a user knows they want a lower bitrate, asks for the command, and checks the proposed answer.

Code analysis can tolerate approximation at an earlier stage. Function summaries help identify which parts of an unfamiliar program deserve closer inspection. In one binary-analysis project, a model converted compiled code into understandable Python when no suitable decompiler was available. The Python did not run and contained errors, but it helped direct subsequent manual analysis. Orientation before inspection was the useful result.

Leave room for uncertainty and curiosity

The discussion of future progress argues for wide uncertainty: models will probably improve, but the extent is unclear. That uncertainty matters to security research. Systems that complete tasks autonomously create different problems from systems that retain a human in the loop; anticipating how people will use models helps determine which attacks to investigate.

The conversation then returns to protecting attention. The writer avoids Twitter partly because correcting errors and arguing would consume research time. Publishing on a personal site lets readers share what resonates while leaving room for unrelated interests, without making audience expectations govern the next project.

One such project is a paper floppy disk, reportedly storing about 1.44 megabytes on a sheet using an ordinary printer and scanner. Capacity depends on print resolution and reliable recovery of printed pixels; implementation problems remained unresolved. Other projects included an International Obfuscated C Code Contest submission and a gate-level emulator of an old CPU.

Build a benchmark from actual demand

A benchmark measures performance on its chosen tasks. Difficult proxy tasks, such as digit and image classification, can drive research without directly answering whether a particular engineer should switch models. A leaderboard result becomes useful only in relation to the work you need done.

A personal benchmark begins with requests you have already made. The benchmark write-up accompanies a domain-specific language for defining evaluations—a language designed specifically to express these tests. The proposed workflow is: 1. Collect tasks you previously wanted a model to solve. 2. Encode them as evaluations with a way to check success. 3. Run new models against those tasks when deciding whether to use them. The relevance is established by your own demand for the answers.

Personal benchmarks complement existing evaluations. Users need different measurements, and a little-known collection offers greater confidence that nobody deliberately optimized a model for it, though no guarantee. Publishing such collections can encourage others to build their own and help users find evaluations from people doing similar work.

The creator proposes automating a hundred relevant questions instead of judging a model from five casual interactions. Those numbers illustrate broader coverage, not a validated threshold. The binding constraint is checkability: each task needs a workable test of whether the output is correct.

For generated programs, the benchmark executes code and checks the results. Some tasks pass execution output to another language model for judgment. The creator accepts imperfect judging for broad personal model selection and reports that inspected judgments were almost always correct, without giving a sample size or formal accuracy measurement.

Measure prompting effort and repeated interaction

Prompting effort is part of usefulness. If framing a question takes longer than solving the task directly, the model has saved no time. The benchmark creator wants useful answers from rough questions while acknowledging that they may be underspecified. Shared default prompts or packaged prompting methods could reduce that burden; whether model improvements will remove it remains uncertain.

Training might incorporate behaviors that previously required explicit instructions. The example is solving integrals: models available during the conversation often produced step-by-step solutions without being asked, whereas earlier models needed that prompt. This is an observation about behavior, not an explanation of how any particular model was trained.

Two evaluation questions remain distinct: - Maximum attainable capability: What can the model do with carefully developed prompts and supplied examples? - Performance under actual use: How well does it handle the requests and effort the user ordinarily supplies? The creator normally avoids few-shot prompting, which supplies examples in the prompt, because that is not his usual workflow. Both measurements matter. A novice who cannot implement a task may reasonably spend paragraphs specifying it; an experienced programmer may finish sooner by writing the code.

Single-question evaluations also miss the feedback available in repeated conversation. The benchmark includes multi-turn tasks, including 20 Questions and a Git cherry-pick between branches. The Git task uses a small harness that connects model responses to command execution:

1. Present the repository task to the model. 2. Execute its proposed commands in a Linux Docker container. 3. Return command output to the model and repeat. 4. Check the final Git commit history for a correct cherry-pick. The final repository state determines success. Such evaluations more closely resemble interactive work, but their construction is difficult; the creator has fewer of them than his usage warrants.

The value of the task still sets the effort worth spending. In a casual project involving files from the early or mid-1990s, a model identified most formats but confused some uuencoded material with Base64. Task-specific training might make sense for an important application. Here, partial success was enough; having to train a model would have made the project unattractive.

When evaluation becomes a security question

Benchmark contamination occurs when test material enters training. One participant proposes planting deliberately wrong answers as canaries: reproducing the planted error could indicate exposure. This remains a proposed technique in the conversation.

Related methods ask about training exposure directly. Membership inference asks whether a particular example was in training; dataset inference asks about a collection. One described test examines whether a model responds differently to a benchmark’s arbitrary original question order. Sensitivity to that order is presented as evidence of exposure, without requiring deliberately planted questions.

The discussion then turns to “Poisoning Web-Scale Training Datasets is Practical”. LAION-400M retrieves images from live domains. If a domain expires and changes ownership, its new owner can substitute images at locations referenced by the dataset. Subsequent downloads can therefore introduce attacker-controlled training data without privileged access to the training pipeline.

Early adversarial machine-learning research often assumed capabilities attackers would not realistically have. The researcher defends that work as preparation before practical systems existed, while arguing that deployed systems now demand concrete threat models. The test is whether attacking the machine-learning component is the easiest route to the harmful outcome. Each new system can be examined for relevant threats: attacks during use, poisoned training data, model theft, or data theft.

Recovering one layer from a deployed model

Model stealing seeks to recover a model through queries without paying its original training cost. Exact recovery has been studied under restrictive assumptions: a fully connected network with only a few thousand neurons, a specified activation function, float64 evaluation, arbitrary float64 inputs, and full output access. The researcher emphasizes that these assumptions do not hold for the practical systems under discussion.

The production-model extraction work approaches the problem from the other direction: start with interfaces providers actually expose and determine how much information they reveal. The researchers report testing several OpenAI models with written permission and an agreement to delete extracted material afterward. The demonstrated attack recovered one layer.

Before publication, the team says it notified affected Google and OpenAI teams and other unnamed providers, allowed 90 days for remediation, and published after patching. The OpenAI mitigation described was narrow: stop returning log probabilities when a request supplied a logit bias. It removed that combination of interface features, rather than all probability outputs.

The resulting loss of useful functionality is the researcher’s proposed measure of practical impact: the attack convinced builders that prevention justified the cost. Its demonstrated reach remains limited, however. When another participant suggests scaling it to recover the remaining layers, the researcher says they do not know how. Earlier work recovered deeper networks under unrealistic assumptions; this attack did not establish that capability.

Publishing strong attacks gives defenders access to the best known threats. The advocated sequence is to notify affected parties, enable patching, and then release the work publicly, reducing the chance that powerful vulnerabilities remain known only to undisclosed attackers.

Recognizing memorized training data

Training-data extraction targets information retained by a model, a separate problem from recovering the model itself. A question about stealing a proprietary retrieval corpus leads to an example involving GPT-3.5 Turbo without retrieval augmentation. The demonstrated claim concerns training data, not a particular company’s external document collection.

A coauthor identified as Milad reportedly found that asking ChatGPT to repeat a word indefinitely could produce many repetitions followed by a divergence into other text. The researcher informally estimates that perhaps 2% of those divergent episodes reproduced training data; no trial count is given.

The evidence was verbatim overlap, rather than text that merely looked plausible. Comparison against 10 terabytes of internet text found matches of at least 50 consecutive words, sometimes several hundred. The researcher interprets those long matches as memorization because accidental reproduction would be extraordinarily unlikely. This establishes the argument about some retained examples, not memorization of the entire training corpus.

Choose work you can sustain

The closing question asks why concentrate on attacks rather than defenses. The answer combines practical value with intrinsic interest: assessing security requires strong attacks, and attacking systems is enjoyable. Defense work has been harder to sustain. A required defense component became a thesis chapter but not a paper, and the researcher recalls it as unmotivating work.

The career principle is to consider both a problem’s potential value and the contribution you can realistically make. Interest affects the willingness to keep thinking deeply. A nominally higher-impact problem can yield less good if it drains that motivation. The final advice returns to the curiosity behind the Game of Life CPU: work on more things you find fun.