Summer 2023: From Model Releases to Working AI Systems

Code Interpreter turns a request into an executable method

Someone without programming experience asks an engineer for help with a small task. The implementation may be easy, but the requester cannot do it independently. Code Interpreter brings that interaction inside ChatGPT: the user describes the task, and the assistant can write and execute code and work with uploaded files. Code becomes a means of completing work, even when the goal has little to do with software development.

This capability opens the Latent Space × AI Breakdown summer roundup, covering July 2023’s developments for builders. Swyx explains that Code Interpreter appeared in the March plugins announcement but was selected separately in ChatGPT’s model dropdown. Its sandbox let the assistant run, test and iterate on code. Users were already uploading binaries and probing the container, including attempts to escape it; no successful escape is reported.

Calculation makes the mechanism clear. Instead of producing a numerical answer directly, the model can write a program that computes the answer. One participant attributes some earlier models’ mathematical weakness to how digits are represented as tokens. Code execution changes the task: the model supplies an executable method, and a tool performs the calculation.

That distinction underlies two arguments for Swyx’s “GPT 4.5” nickname: - Possible model improvements: Swyx suspects a different underlying model, partly because of reports of better performance on non-code tasks. - Greater system capability: another participant argues that the nickname fits because users can accomplish more, whether or not the underlying model changed. Neither establishes an announced OpenAI model identity.

Swyx connects this to adaptive inference—varying the computation spent answering a question with its difficulty. His comparison of simple addition with proving the Black–Scholes theorem illustrates the principle; it is not a timing measurement. In the proposed search pattern, a decision with many branches receives more exploration, while a simple decision takes a shortcut. Tool use and additional effort at answer time offer routes to greater capability beyond larger datasets and more training.

Model updates need evaluation and a stable contract

The question of whether GPT-4 had become worse exposes a measurement problem: correctness, presentation and workflow compatibility are different judgments. Participants consider some reported examples clear regressions, while others may reflect disliked formatting changes. A useful evaluation must identify which behavior changed.

Several examples show why that is difficult: - Accepted code may not survive use. Copilot is cited as tracking whether suggestions remain one, two and five minutes after acceptance, adding a persistence signal. - Open-ended quality is harder to score. Podcast show notes lack the relatively concrete feedback available for code suggestions. - Preference training can reward unintended features. A model may learn to give longer answers when the intended preference was for better answers. - Safety behavior can block ordinary work. In a reported Llama 2 interaction, the model refused to explain how to kill a Linux process because it interpreted the word as harmful. The anecdote comes without a reproducible model configuration.

The proposed response is to lock model versions and evaluate them against one another. Comparing a service at widely separated dates can hide intervening updates. The organizational challenge is equally consequential: a research lab may treat model changes as routine, while infrastructure customers need predictable behavior. Small disruptions can have large effects once users have redesigned workflows around a model.

Owning the model lets an application builder control when that dependency changes. Measuring improvement remains difficult, however. The discussion contrasts advertising’s immediate signals, such as click-through rates, with sparse thumbs-up and thumbs-down feedback on generated answers. Control over updates and evidence of quality solve different parts of the reliability problem.

Personalization makes application advantages easier to copy

Custom instructions give ChatGPT persistent user information and behavioral guidance. Participants see the feature as useful for nontechnical users but familiar: the OpenAI Playground already allowed system prompts, while Perplexity and character-based chat products offered precedents for personalization.

The host uses Perplexity’s sourcing interface to pose a business question: could the underlying model provider copy an application’s best features? The response turns on how quickly comparable models become widely available. Office, Notion and Superhuman can add AI where users already keep their work, combining existing distribution with data that helps tailor responses. A startup’s opportunity depends partly on how much time it has to establish an advantage.

Llama 2 separates deployment control from full openness

Llama 2 makes model ownership commercially relevant. One participant describes it as a commercially usable GPT-3.5 equivalent—a qualitative comparison, without benchmark conditions. Its central benefit is control: organizations can run the model on their own infrastructure or cloud and fine-tune it with access to its weights and internals. Government, healthcare and finance are cited as settings where that flexibility could matter.

Commercial usability and full openness remain distinct. Participants acknowledge that Llama 2 does not meet the conventional open-source definition while emphasizing the community work it enables, including GGML quantized versions and efforts to expand how much context the model can handle. The host also defends precise terminology: practical benefits do not erase the importance of an established licensing definition.

Three release patterns make the distinctions easier to inspect: - RedPajama shares data, training code and weights. These represent separate dimensions of access. - StableLM shares weights without the training data. Users can download the model but lack the data needed to reproduce its training. - Llama 2 shares models with use restrictions. Its training data is also withheld.

Shared weights can coexist with guarded data. One participant interprets that pattern as a shift in competitive value toward data, even amid a GPU shortage. Another urges open-source contributors to build excellent specialized datasets instead of repeatedly funding another training run. Model releases attract attention, but dataset work may offer the more useful contribution.

Disclosure can also attract disputes. A participant recounts copyright objections to MPT’s reported use of Books3, described as roughly 190,000 books, and unresolved questions about fair use. A position attributed secondhand to Shawn Presser favors trying available data in public research and addressing objections as they arise. That is a research stance, not a legal determination. The crucial distinction is that availability on the internet does not establish a license to train.

Open and local models raise the bar for buying software

As model access expands, buyers need stronger reasons to choose an application vendor over an internal implementation. One participant argues that a polished interface around an OpenAI model could have been enough six to nine months earlier; by July 2023, customers expected more. Competition shifts toward making AI applications work in production. A prediction that OpenAI would open-source GPT-3 within six months reflects that perceived pressure, not an announced plan.

The MLC team’s work running Llama 2 on MacBook GPUs offers another tradeoff: local automation without repeated payments to OpenAI. One participant would accept execution five times slower for menial tasks under those conditions. That is a personal preference, not a measured speed comparison.

Llama 2 is only weeks old at this point. The group expects more derivatives tuned to follow instructions, analogous to Vicuna and Alpaca, now that commercial availability broadens their potential use. That expectation runs ahead of the observed results: a participant reports few striking downstream developments so far.

AI companions connect personalization with privacy

The discussion moves from enterprise deployment to personal relationships. One participant describes romantic AI companions as an underdiscussed market with millions of users, without supplying a measurement basis. The host anticipates a social flashpoint and a possible generational divide: children might find an artificial relationship meaningful despite their parents’ objections. Another participant imagines a Nobel Peace Prize over a 30–50-year horizon if companions could substantially reduce loneliness and violence. These are forecasts and hypothetical benefits, not demonstrated effects.

Proposed uses extend beyond romance: - Practice for human interaction: rehearse jokes or conversation topics and receive feedback before speaking with people. - Personal reflection: work through concerns with a conversational partner without romantic attachment. - Advice and accountability: consult a panel of experts or check in with an AI boss; the personal boss is a planned project.

Private journaling makes deployment control part of the product’s appeal. Users may want help with their most personal thoughts without sending them to OpenAI. Open models offer a reason to consider alternative deployments, though their use alone does not ensure that an application keeps data local. Circlechat.co supplies a different interface example: a conversation in which a group of AI friends chimes in.

Claude’s long context reduces chunking, but retrieval still needs testing

Claude 2 gives Anthropic a concrete capability advantage: public availability after a waitlist, multiple-file uploads and a 100K-token context window—the amount of material available to the model in context. One participant reports submitting whole podcast transcripts for transcript and timestamp work, avoiding the chunking workflow they had considered for GPT-4. Another cites businesses including Sourcegraph choosing Claude for perceived latency, reliability and code-synthesis advantages, without a controlled comparison.

Developer enthusiasm is less settled. One participant sees Anthropic’s marketing and community support lagging its technical quality; another describes a recent Claude hackathon with roughly 300 participants and substantial activity at 2 a.m. In informal side-by-side use, one participant preferred Claude on about 30% of their own tasks, without a defined task set or scoring rubric. The practical recommendation is to compare models on your work before committing to one toolkit.

A larger input allowance creates a separate evaluation question. A participant describes an unnamed paper finding a U-shaped retrieval pattern: information near the beginning or end of a long context was easier to retrieve than information in the middle. A plausible summary does not demonstrate reliable factual retrieval. Accepting an entire transcript simplifies input preparation without establishing that the model can recover every needed detail.

That leaves two architectures to test: - Large-context input: provide the material together and evaluate whether the model retrieves the needed facts, including those in the middle. - Smaller context with retrieval: use semantic search to select relevant material for a 4,000–8,000-token working context. This is the approach discussed as retrieval-augmented generation: retrieve material to support the generated answer. The choice depends on factual retrieval in the application, not simply on context size or a summary’s fluency.

Bard’s new capabilities renew the reliability question

Bard’s July updates added image recognition and more languages. A participant highlights image uploads for descriptions, saying Bard made this capability generally available before GPT-4. Another argues that incremental model announcements are losing their appeal: a concrete improvement to an existing product, such as Google Maps search, would make the benefit easier to see.

The reliability concern returns through a reported regression. A participant says questions used to promote Bard’s implicit code execution in June no longer worked. They describe the feature as a limited form of code-interpreter functionality. The allegation reinforces the earlier infrastructure concern: users need a shipped capability to remain dependable across updates.

Useful agents need scope, domain knowledge and evaluation

Interest in AutoGPT and autonomous agents remains strong, but one participant sees a central product problem: narrowing general-purpose ambition into useful scope. Evaluation tools are emerging to monitor prompt and agent success, version them and potentially support sharing, though their product boundaries remain unsettled. LangSmith, newly launched as a LangChain commercial product, is described as focused on observability—seeing what the system does—with evaluation capabilities expected to grow.

A participant who reports investing in two industry-specific agent companies argues that domain expertise can improve the general-purpose approach. Financial-document retrieval supplies the example: similarity alone does not capture importance. This quarter’s earnings may matter more than a similar document from ten quarters earlier. Founders are exploring document hierarchies and additional scoring to encode that priority alongside similarity-based retrieval.

The closing survey returns to two other areas of development: - SDXL and text-to-image generation: a participant highlights continued progress despite less attention than the previous year. - AI characters as social interfaces: a participant claims strong Character AI retention without figures or cohorts, then speculates about representing personality and intelligence in reusable characters. The further analogy to mind uploading is explicitly science fiction.

The host adds an unconfirmed Financial Times report that Meta might introduce personas the following month, including Abraham Lincoln and a surfer-style travel adviser. The report is described as relying on three people familiar with the project, without confirmation from Meta.

Agent clouds and the emerging AI engineering profession

Agent clouds are proposed as infrastructure built for programmatic action: environments with dependencies preinstalled, controls agents can operate and feedback they can consume. A participant reports AutoGPT and E2B developing a protocol to let agents run cloud infrastructure, expanding execution from individual machines to clusters. They also anticipate safety concerns about that increase in reach.

Looking ahead from summer 2023, one participant expects companies to spend August preparing open models for production and to publicize their use from September through year-end. The argument is that limited disclosure understates adoption, sustaining the perception that open models are toys rather than production choices.

Another participant sees AI engineering becoming a profession, moving from informal groups, Slack channels and meetups toward courses, certifications, job titles and dedicated teams. Messages from people reorienting their careers provide anecdotal support. A reported AI engineer title within Meta is a narrower signal: the speaker qualifies it by noting employees’ apparent flexibility to choose internal titles.

The host closes by forecasting a quieter August 2023 followed by renewed activity in September, with Facebook Connect and the return to business after summer. Bay Area hackathons would keep going in the meantime, and one unexpected release could overturn the forecast.