Code Interpreter, Part 2 of 3: The Missing Manual and the Self-Debugging Loop

A manual for capabilities the model may deny

Code Interpreter sometimes says it cannot perform a task that its execution environment can handle. That problem opens Part 2 of the Code Interpreter conversation: participants propose feeding it website documentation and working examples to help it use its capabilities. A shared introductory prompt and a library of successful snippets would record what works and how to elicit it, including when extra prompting is needed.

That manual would need continual revision as OpenAI changed the product. Oren speculates that multiple models might refine ideas together and that tooling could eventually absorb users’ successful approaches. The immediate constraint is more concrete: sessions start fresh, and their files eventually disappear. One proposed workaround is to export work into a downloadable file, perhaps a SQLite database, and upload it into a later session.

A small media task makes the interface’s value tangible. One participant uploads an MP4, asks for its audio, and gets an MP3, attributing the result to ffmpeg. The conversational request makes that tool useful without requiring the user to remember its command-line parameters.

Reading old grammar books inside an isolated environment

Daniel’s problem is recovering text from old grammar books for languages with little available data or machine-translation support. His experiments with languages in Nigeria and Indonesia use those books to help an agent generate well-formed sentences. This requires optical character recognition (OCR): recovering text from page images while retaining useful structure such as table rows and columns.

In his informal comparison, Code Interpreter performed better on these materials than the GPT-4 visual model and image-to-text plugins he had tried. On one Nigerian grammar page, he estimated four or five transcription mistakes and said it recognized the table structure; he supplied neither a page length nor a formal accuracy measure. He first requested a reproduction, then explained what the document represented and continued toward a cleaner format. The OCR engine was not identified.

Reported resources did not establish usable capacity. A participant recalled Python inspection showing roughly 16 CPUs and 60 GB of RAM, yet workloads exhausted memory far below that apparent capacity. They did not know the current specifications or why the reported memory was unavailable in practice.

Surya proposes connecting the interpreter to external services through installed plugins—for example, querying a personal database or combining Wolfram Alpha with local computation. Two requirements pull against each other: - Scope external access: approved plugins could proxy API calls and constrain which services the interpreter reaches. - Prevent data exfiltration: another participant argues that malicious instructions entering through plugins could trick a model with Python execution, private data, and outbound HTTP access into sending that data elsewhere. The proposed integration would expand capabilities while weakening an isolation boundary with a concrete security purpose.

Store memory in files, retrieve only what is needed

A workaround credited to Nick Dobos uses chatgpt-memory.txt to hold summaries that the interpreter can read and update. Downloading and re-uploading the file carries information between conversations. A proposed extension separates file storage from the model’s context—the text it receives for a response. Search with grep or Python, then return a match and three lines on either side. A participant suggests this could support a 100 MB memory file provided the model never tries to load the whole file into context; that capacity is hypothetical.

Participants also propose uploading libraries to generate embeddings locally and use a vector database for memory retrieval. This remains an implementation idea: one person reports that transformers is absent, while others suggest uploading Python files or an offline model. Instructor embeddings later appears as a candidate, without a demonstration inside Code Interpreter.

An anecdote about the Noteable plugin extends file-based memory into sustained execution. An unnamed developer reportedly used three Jupyter notebooks for separate responsibilities: - Long-term memory: retain information across work. - To-do list: track remaining tasks. - Output: collect results. Repeated model calls reportedly kept the system running; claims of indefinite operation and token efficiency were unquantified. Participants also propose open-source alternatives built on StarCoder or Replit’s code-model work, with specialized fine-tuning suspected—but not established—as a replication challenge.

Watching failure becomes part of learning

Gabriel Cohen’s sentiment-analysis experiment shows the execution loop adapting to a missing dependency. Code Interpreter tried the Natural Language Toolkit (NLTK), failed to download a required lexicon because it lacked internet access, and improvised a naive method using roughly 30 positive-sentiment words. The next day, the same download failure led it to TextBlob instead. Fallbacks changed between runs, without an accuracy comparison establishing which result was better.

Participants value seeing code execute, fail, and change approach: the visible process teaches troubleshooting alongside the answer. One describes releasing a Python source-search tool despite unfamiliarity with the ast module. The interpreter parsed source into an abstract syntax tree, a structured representation it could search for functions, declarations, and type annotations, then executed and tested the resulting code.

Two requests push beyond those demonstrations: - Interrupt a run that goes off course: automate the stop-and-recalculate intervention a participant already performs manually. - Recover data from a chart image: Manny wants numerical values when the underlying dataset is unavailable. A response identifies mapping chart content to X–Y coordinates as a likely difficulty and suggests a dedicated tool might work better. No successful extraction is demonstrated.

Shyamal describes his brother-in-law, who had no programming background, analyzing financial CSVs and learning about Python and data analysis within an hour. Another participant emphasizes removing development-environment setup and learning from commented code, including after 25 years of programming. Shyamal anticipates students learning through the interface and invites feature requests and security reports.

Requests include Node or Deno support and runtime extensibility; participants report that previously working package-upload and binary-execution approaches had stopped working. Asked about business use, Shyamal says ChatGPT for business is still being specified. Possible inclusions are plugins and Code Interpreter, with stronger data security, controls, and team or company licensing as initial distinctions. These are plans at the time of the recording.

A successful prototype can still be an awkward workspace

One participant successfully split a video into frames, selected frames, and cropped them to requested dimensions. Resource constraints forced them to use a much smaller clip, producing a hybrid workflow: validate code on a sample, then process larger files locally. Switching between the hosted session and their machine introduced a second problem: the hosted state could disappear while they worked elsewhere.

They recalled a timeout after perhaps half an hour, though the interval was uncertain, and said replaying earlier code did not reliably recreate the previous state. These constraints sharply reduced their use despite strong initial results. Participants express willingness to pay for dedicated hardware and fuller control. Upload size and execution memory remain distinct constraints; the discussion establishes no reliable limit for either.

Alex introduces AgentEval, a project for visualizing why autonomous agents fail, and proposes tracking recurring failures and incorrect outputs in Code Interpreter. His example is a hallucinated dependency during ffmpeg-related work, although he is unsure which library was involved.

Other reports show why capability discovery remains unsettled. One participant successfully splits a 90 MB, 1280 × 720 video into three pieces; it feels faster than their M1 machine, but no timings support that comparison. Another overcomes a stated inability to execute code by asking the model to try so they can inspect the error—and receives working output.

A speaker introduced as Shroominik describes independently recreating the experience in Discord after seeing a demonstration, before gaining access to OpenAI’s version. The architecture gives each user a sandbox session with an interactive Jupyter kernel. He is building an API intended to create Python environments, run code, return outputs, and transfer files, which he proposes combining with a conversational LangChain agent.

Better answers do not establish a better model

One participant uses Code Interpreter for questions and essays because its text feels better than default GPT-4. They say the same prompt worked well in April, performs worse in the default interface at recording time, and produces something resembling the April result in Code Interpreter. Simon challenges the inference: isolated comparisons cannot separate model changes from output variability. An earlier answer may have been a lucky sample; detailed published comparisons would help.

The interface also lacks exposed temperature controls for controlling sampling. Simon offers a hypothesis conditional on degradation being real: plugin and interpreter variants might retain behavior from older GPT-4 checkpoints used for fine-tuning. Both the degradation claim and that model history remain unconfirmed.

Max AI returns to a concrete workflow: learning piano with compositions generated in ABC music notation, a compact format for written music. He requests changes in complexity, dynamics, rhythm, and voices. His desired extension is MP3 conversion with playback inside the interface; the conversation does not demonstrate that conversion pipeline.

Why experienced programmers delegate

Aravind, who has not yet tried Code Interpreter, recalls early Perplexity work using Codex to plot follower distributions and likes over time. He expects GPT-4 to work well but questions the added value for experienced programmers. Simon’s answer is that lower effort changes which tasks he undertakes: the assistant makes him more ambitious.

For a tutorial with H1, H2, and H3 headings, Simon wanted a nested HTML table of contents. Given a paragraph of instructions and example code, the interpreter reportedly produced about 15 lines of Python in 30 seconds; he estimates writing it himself would have taken five to ten minutes. He watched it make familiar off-by-one mistakes, inspect its output, and correct them. The time saving mattered because he might otherwise have skipped the feature.

The more demanding example is Symbex, an installable open-source tool for finding symbols in Python source through the ast module. Simon estimates learning the necessary tree manipulation himself would have taken a day. Code Interpreter generated, executed, debugged, and refined code he incorporated into the larger tool, taking on unfamiliar details and an initial execution-and-repair cycle.

The debugging can proceed without another prompt. Simon supplies a problem, desired result, and example data; the interpreter writes code, runs it, inspects failures, and retries. He has watched five or six rounds without intervening. His practical guidance depends on what those rounds are doing: - Let productive iteration continue: the interpreter can find and repair mistakes itself. - Redirect repeated fixes: stop a loop and suggest another strategy, such as line-by-line processing instead of a regular expression. - Restart a persistently stuck conversation: a fresh session beginning with a smaller part of the task can sometimes escape repeated failures or capability denials.

Participants extend delegation to trivial tasks they already know how to implement and to unfamiliar expressions of familiar ideas. A construct from Wolfram Language, for example, can frame a request for its Python equivalent. They distinguish enjoying programming from enjoying the mechanics of typing code: describing an idea and requesting revisions can remove an irritating part of the work.

Execution feedback, visual inspection, and developer control

Alex Graveley argues that execution adds outcome signals to generated code. A generated program alone does not reveal whether it works; running it, critiquing failure, and regenerating it could produce valuable training material. This is an argument about the potential value of execution traces, not confirmation of OpenAI’s training practices.

A participant then reports an input-length test using text measured with OpenAI’s tokenizer. Pasted inputs just above roughly 8K tokens were rejected for Code Interpreter, the plugins model, and ChatGPT 3.5, while smaller inputs worked; they report roughly 4K for default ChatGPT 4. Pasted-input acceptance is narrower than total context capacity: this test does not establish how history, internal instructions, or output consume the available context.

Karl, who has not used Code Interpreter directly, argues that an API implementation must also reproduce the interface’s presentation of graphs and statistical results. His enterprise media system indexes footage across platforms, splits it into scenes, describes their content, and supports natural-language search. He wants supported image-captioning and parsing APIs, while questioning whether generating small applications on demand suits that task as well as a dedicated image-processing model.

Simon doubts that installed libraries and small uploaded models would match Karl’s dedicated system, and sees plugins as a more promising extension. Two further requests distinguish control over execution from the ability to inspect its results: - Supply the execution environment: Simon wants paid API access to the underlying model, if it is specially fine-tuned, with his own code-evaluation function and infrastructure such as a Kubernetes container with network access. - See the generated chart: Kyle reports invented descriptions of plots. He wants vision combined with execution so the model can inspect the actual chart and use it to propose further analyses.

A developer reports that a newly released function model handles interpreter-style tasks better than their previous model, without establishing that it shares Code Interpreter’s fine-tuning. Proposed next steps include exporting results to train models such as Llama or Salesforce’s XGen, and releasing a LangChain wrapper around a Discord interpreter’s infrastructure API. The training work is a suggestion; the wrapper is a planned release.

The final contribution in this part begins with an experiment making a video from one photo and identifies self-debugging toward the user’s intent as the central advance. That enthusiasm comes from simple tasks; success on more complicated work remains an expectation. The closing request turns from model capabilities to people: let users make themselves available to collaborate with others pursuing similar prompting goals, connecting work now happening in separate conversations.