Code Interpreter: Run, Fail, Revise — Part 1 of 3

An uploaded database becomes executable work

Upload a SQLite database, ask questions about it, and download a new database containing the results. That was one of Simon’s early uses for Code Interpreter. Python’s standard library supplied SQLite support, allowing the assistant to write SQL, execute queries against uploaded data, and produce a reusable file.

At the time of this Latent Space discussion, Code Interpreter was rolling out as a beta to paying ChatGPT users, enabled in settings and selected under GPT-4. Its basic workflow connected three capabilities: upload files, execute code, and download results. Simon had used it almost daily for at least a couple of months.

The consequential addition was execution feedback. The model could write code, run it, inspect an error, revise the implementation, and retry. Simon had seen tasks take four or five attempts before reaching a working solution. Execution did not prevent invented APIs or coding mistakes; it gave the assistant feedback it could use to repair them.

The environment included pandas for data processing and matplotlib for graphs. Simon had previously added libraries by uploading Python wheel packages, though that no longer seemed to work. He also explored asking the model to read binary bytes and interpret a file format from its existing knowledge when no dedicated library was available. That remained an exploratory technique, without a validated format-specific result.

Test capabilities with small executable probes

A 150 MB CSV exceeded the stated 100 MB upload limit. Simon compressed it into a ZIP archive; Code Interpreter unpacked it and began processing the CSV. Another participant uploaded an entire project repository the same way. These experiments exposed a recurring problem: the assistant could deny capabilities its environment supported. Reminding it that it could execute code sometimes got an initially rejected task working.

Simon treated it as a fast coding intern with uneven judgment and patience for repeated redirection. An experienced programmer could understand the runtime better than the assistant did and coach it with a small executable probe: read the first 20 bytes of a file, or import Python’s SQLite module and report the installed version.

Tedious programming tasks with awkward edge cases were especially attractive. Simon could watch the assistant execute, test, and revise code he would otherwise debug himself. He estimated that some work taking him roughly an hour could take a couple of minutes this way—a personal comparison, not a productivity benchmark.

Inspecting installed packages revealed less obvious capabilities. Simon used os.listdir on the package directory and found TensorFlow and Tesseract-related support. Participants reported using Tesseract for optical character recognition (OCR)—extracting text from uploaded images. Package inventories became a way to discover possible tasks, though their completeness was uncertain.

The conversation can outlive the runtime

Participants reported no network access and a 100 MB upload limit. Support for additional executables was less clear. One had previously uploaded Deno, a JavaScript runtime distributed as a single binary, and a Lua interpreter, then run them through subprocesses. They believed that route had since been restricted and wanted broader execution within explicit CPU, disk, network, and time limits.

A more immediate failure was conversation history surviving after files and variables disappeared. Disconnections or resets could invalidate download links while leaving the chat intact. The assistant might then keep changing its code as though missing files still existed. Participants recommended stopping that loop and rebuilding the work in a new session, using the retained conversation as a record.

Time limits had a more mixed effect. During exploratory analysis of a large table, one participant saw the assistant abandon a slow operation and switch to shorter code using a sample. That improved responsiveness, but the participant wanted a choice between quick exploration and waiting for the full computation.

Two techniques reduced repeated text generation within a live session: - Refactor for reuse. Smaller functions let later turns call existing code. One participant described follow-up changes shrinking from roughly 50 generated lines to five. - Keep large text in files. Uploading text, or writing it to a file once, let code access it without repeatedly reproducing its contents in chat. Both targeted token consumption and interaction speed. The actual context-window size—the amount of text available to the model at once—remained untested; 8,000 tokens was only a guess.

Repository editing required similar coaching. When the assistant said it could not write a change, a participant asked it to generate code replacing the whole file; it accepted that formulation. Git commits and repository round-tripping remained proposals, while separate Git-binary and GCC experiments had failed or been abandoned. A narrower approach already worked: generate diffs with Python’s `difflib` and download them.

Separate model knowledge, execution, and display

Network isolation complicated data enrichment. A participant wanted to fill a table of superhero names with information the model already knew. Code Interpreter preferred writing code, but that code could not call OpenAI over the network to supply the missing information. The suggested division of work was to obtain structured names and biographies from regular GPT-4, then upload the resulting JSON for processing.

Visualization exposed more misleading refusals. A participant obtained drawings after the assistant initially denied it could draw, and found Folium, a mapping library, installed despite another denial. Maps, network graphs, and matplotlib plots were useful possibilities. One participant argued that mature libraries benefited from having many examples available before the model’s training cutoff.

The chat renderer did not define the output formats code could produce. Participants distinguished: - Inline images: they believed chat displayed image outputs such as PNGs and GIFs, rather than interactive JavaScript or SVG. - Interactive downloads: generated HTML, CSS, and JavaScript could embed data and run outside chat. Examples credited to Ethan included flight maps and 3D music visualizations. - Documents: downloadable Excel files and PDFs extended the workflow beyond charts.

Ethan’s experiments also challenged the idea that only programmers could use the tool effectively. A participant described him as a nonprogrammer whose results showed how far someone could get without programming expertise.

Media and machine-learning capabilities had different levels of testing: - Torch ran successfully. Participants initially suspected hardware acceleration, then concluded it was running on fast CPUs; they supplied no hardware measurements. - Audio transcription remained a proposal. Torch Audio and another speech library were reportedly installed but untested, and uploading Whisper was a suggested next experiment. - Video processing remained partly unresolved. A participant reported FFmpeg availability and requested a three-way video split, without confirming the output files. MoviePy and PyFFmpeg were also reported present, but whether processing used bindings or subprocesses was unclear.

A supermarket question becomes an analysis pipeline

The participant running Datasette, an open-source SQLite-based web application for exploring data, described an unsettling product moment. Originally aimed at helping data journalists find stories, the project now faced a tool that appeared to perform everything on its next two years of roadmap. The response was to redirect work toward making Datasette combined with language models more useful than Code Interpreter.

One investigation made that change concrete. Reporting described 1,000 police calls over a year and a half at a San Francisco Whole Foods—but was that unusual for a supermarket? The participant obtained a CSV described as containing roughly 250,000 San Francisco police-call records from 2018 through the experiment, including locations, categories, and dates. A zipped upload succeeded after the original file proved too large.

The participant supplied coordinates for Whole Foods and a similarly sized Safeway roughly a mile and a half away, then asked for calls within 500 meters of each point plotted over time. Code Interpreter reportedly built the procedure itself: 1. Calculate distances from latitude and longitude with a Python Haversine-distance function. 2. Filter the records to the two 500-meter areas. 3. Aggregate calls by month for each location. 4. Plot both series in a comparative line chart, without instructions specifying the chart type.

The participant interpreted the chart as showing substantially more calls near Whole Foods. The measurement covered surrounding areas, not calls proven to originate inside the stores. One comparison location also could not establish a general supermarket baseline. The result supported a local contrast more directly than the broader conclusion drawn from it.

A follow-up request converted the results into a downloadable SQLite database. The participant estimated roughly two minutes for the workflow versus half an hour to an hour using their own data-journalism tools. That personal estimate accompanied a consequential sequence: question, executable analysis, chart, reusable database.

Could a developer reproduce the execution loop?

External information could enter through a person acting as a proxy: let the model request an operation, perform it elsewhere, and return the result. A participant had used this pattern before plugin access existed. A proposed Chrome extension would automate it by recognizing a prescribed response format, fetching a URL, and pasting back the content.

A broader proposal used the GPT-4 API and function calling to expose code execution in a developer-controlled sandbox, allowing choices about containers and networking. Participants wondered whether matching the interface would match the behavior: they suspected task-specific fine-tuning because the tool worked so well. That remained a hypothesis, and reportedly extracted prompt fragments had not been authenticated as complete or genuine.

The underlying infrastructure was also uncertain. Kubernetes began as a guess; later, a participant said runtime output confirmed its use, though the output was not shown. Discussion of a dependency vulnerability produced no demonstrated sandbox escape. Confidence in network isolation was a participant’s judgment, not a security assessment.

Live experiments reveal unstable boundaries

Audience experiments complicated earlier reports. ZIP uploads failed for one commenter but worked for participants who retried. A participant offered a writeup of getting Deno working; a subsequent report distinguished preinstalled programs that ran from uploaded executables that were killed, even after permission changes with chmod. Binary execution remained an unsettled, apparently changing boundary.

Cheap setup made small benchmarks attractive. One participant estimated that comparing Python idioms took seconds through the assistant versus ten minutes constructing a benchmark manually. Yet participants also reported variable speeds and timeouts, while one Torch job completed after about five minutes. Resource allocation and execution-time limits remained unknown; shared CPUs were a hypothesis.

Hardware inspection again put execution at odds with self-description. A participant reported obtaining a 54 GB memory reading after repeated prompting, only for the assistant to call the result a simulation and ask them to disregard it. That report did not establish memory available to an individual session, but it illustrated the confusion caused by the assistant’s contradictory account of its capabilities.

Image-processing experiments included reported digit classification using what a participant described as OpenCV pretrained models, face detection with Haar cascades, and Canny edge detection. No accuracy measurements or model identifiers accompanied those reports. Another participant said Hugging Face Transformers and Diffusers were absent: an extensive installed toolset still had gaps.

Autonomy includes knowing when to ask

A proposed debugging workflow would supply a code snippet and error context, then ask the assistant to reproduce the issue and iteratively suggest repairs. Its proposer had not tried it. A separate observation concerned when the assistant paused: it sometimes offered choices and asked how to proceed, resembling a data analyst clarifying a vague question.

Ask for alternatives when exploration is the goal. A broad request for different visualizations could produce bar, pie, and time-series charts. The participant’s rationale was that several options could cover plausible interpretations without a separate clarification turn for each.

Documentation search suggested two further experiments: - Search words, then read passages. Upload documentation as a ZIP, use Python for keyword matching, and inspect surrounding text to answer questions. - Upload externally computed embeddings. Supply the numerical representations in a Python pickle file and compare them using cosine similarity inside the interpreter. Both were untested proposals. The second assumed embedding computation elsewhere; it did not establish a complete retrieval workflow.

A successful 100 MB download also prompted speculation about exporting model outputs to train another model. Another participant challenged the missing connection: running code did not explain how those outputs would be obtained. No working mechanism followed. Separately, observations of different browser-request routing and parameters renewed speculation about a different underlying model without confirming it.

Available files are not automatically model context

The closing discussion distinguished two coding workflows: - For a targeted change, supply a minimal example. One participant pasted reduced code into chat, asked it to run against supplied data, then requested a feature refactor. - For a larger project, execute uploaded files. Python could evaluate code from a ZIP and return errors without the model first reading every source file into its context.

Nisten reported running Tinygrad, which he described as George Hotz’s simpler alternative to PyTorch, after uploading its GitHub ZIP archive. The report supplied a concrete project for the upload-and-execute workflow, though no workload or performance result.

Surya wanted Code Interpreter integrated with plugins. A plugin under development would store chat summaries in a user’s vector database and retrieve relevant material during later turns. Documentation retrieval combined with code execution was another proposed extension. The intended benefit was continuity beyond the immediate conversation, not a demonstrated increase in context capacity.

Part 1 closes on the distinction underlying these ideas: uploading a repository makes its files available; it does not put the whole repository in context. The interpreter can execute files, and the model can inspect relevant contents when needed. Guidance still matters: when information is missing from context, the assistant may need to be directed to read its files before declaring that information unavailable.