Long Context, Working Agents, and Learning in Public — Part 2 of 3

When the whole document fits

Upload all of Harry Potter, change one fact in one sentence, and ask the model to find it. A host describes Gemini beta testers succeeding at that experiment. It makes the opening engineering question concrete: when does maintaining a retrieval pipeline cost more than sending the material directly? If an application’s model already works well enough, falling inference costs could change that decision.

In this January–February 2024 recording, one host expects RAG to retain an economic advantage at the cutting edge. Long context could become cheaper for established workloads once pipeline construction and maintenance are included. The nearer-term prediction is narrower: long context reduces the need for retrieval during prototyping.

Needle-in-a-haystack retrieval tests whether a model can recover a specific fact buried in a large input. Moving the fact between the beginning, middle, and end checks whether recall depends on position. The hosts discuss reported 100% recall across Gemini’s million-token window—on this retrieval test, not arbitrary reasoning tasks.

Ring Attention enters as a community hypothesis for handling long inputs, not a confirmed account of Gemini’s implementation. It distributes attention and feedforward computation in blocks; a host describes a target of roughly 10–100 million tokens. But distributing computation does not, by itself, explain unusually strong recall. Context capacity and retrieval quality are separate questions.

Video as an agent’s input

The discussion moves from context length to training data and video understanding. The hosts report Google’s $60 million Reddit deal, then describe a Simon Willison demonstration: scan a bookshelf with a short video, pass it to Gemini, and receive JSON describing the books.

The bookshelf example suggests video as an agent’s primary input. An agent could interpret an interface visually instead of relying on a textual representation of the page’s structure. Long multimodal context could let it take in extended visual input as its working environment changes. The hosts credit Joseph from Roboflow with the broader proposition that these models could make anything visible programmable.

The enthusiasm prompts a joke about professional incentives: vision practitioners favor vision, retrieval practitioners defend RAG, and fine-tuning practitioners favor fine-tuning. The hosts then raise two distinct objections to Gemma, Google’s 2-billion- and 7-billion-parameter releases: - Quality: One host relays disappointing reports, without benchmarks establishing an advantage or disadvantage against alternatives. - Licensing: Another disputes the open-source label and objects to reported language requiring reasonable efforts to update. This is a licensing criticism, not a precise reading of terms reproduced in the conversation.

Prompt rewriting and model alignment

Gemini’s controversial image generation shifts the discussion toward control. A host describes alignment problems as extending beyond Google, citing reported incidents involving Meta models and Microsoft’s Copilot. A Grimes post read aloud treats Gemini’s unintended offense and contagiousness as a performance-art effect. Another host interprets homogenized images as mode collapse from human-feedback training: outputs converging toward a narrow preferred pattern.

Two possible causes of unwanted behavior need separate treatment: - Application-level prompt rewriting: A host attributes the image controversy to hidden changes intended to make prompts more diverse. The implementation and review process remain unestablished in this account. - Model-level alignment: The hosts separately question whether Gemini’s text behavior reflects excessive constraints. Explaining image prompt rewriting would not resolve that issue.

Their practical demand is visibility into how an application changes a request. Invoking a Hamel Hussein post, a host argues that inspecting prompts is necessary for user control. The host also recalls demographic descriptors being appended to DALL-E prompts as a precedent: users should be able to see such interventions.

One host extends the argument to models as national soft power. A widely adopted assistant could subtly influence reasoning and decisions through embedded biases, creating incentives for countries such as China and Russia to develop their own models. This is a claim about incentives, not a specific influence operation.

A reported Gemini response hesitating to compare the harm of Elon Musk’s tweets with Hitler’s actions exposes a diagnostic disagreement. One host suspects a data-pipeline problem; another compares it to failing the equal-weight feathers-and-bricks question. Neither explanation is established by the anecdote. The product question remains: what comparative judgments should a useful assistant be expected to make?

Lindy: the answer succeeds, the handoff fails

The recap ends with Compute Provider month: difficult to coordinate, but successful by the hosts’ reading of post statistics. They invite suggestions for further thematic series. The recording then moves into Final Frontiers, Latent Space’s February 2024 first-anniversary demo day. Its introduction reports more than 750,000 downloads and more than one million unique Substack readers.

Florent Crivello presents Lindy.ai, recently opened to the public, with a claimed 3,000-plus integrations selectable through the interface. Users need not supply integration specifications themselves. Its agent-composition mechanism lets a support agent pass ticket details to a separate reporter that logs them in a spreadsheet—an arrangement the presenter compares to object-oriented programming.

The live refund test separates those responsibilities. Lindy receives an email, subscribes to the thread, and sends a refusal. But the reply succeeds while delegation fails: it does not pass the ticket details to the support reporter as instructed. Completing the customer-facing response leaves the workflow unfinished.

RWKV: efficiency for devices, languages, and access

Eugene Chia presents RWKV/Eagle as an attention-free architecture with linear scaling, alongside Mamba in the search for scalable transformer alternatives. He reports that its 7-billion-parameter model beat other models of that size after one trillion training tokens, without identifying the comparison benchmarks. The size matters to him because it can run on many users’ laptops and devices.

Chia reports single-GPU concurrency of 256 or 1,000 users at near-ChatGPT speed, contrasting it with what he describes as eight or sixteen concurrent transformer requests. The hardware-to-model mapping, context length, and numerical generation speed are unspecified. His subsequent estimate of serving OpenAI-scale output with far fewer GPUs is a capacity projection, without demonstrated equivalence in quality or workload.

His objective is affordable access across countries and languages. Two contributions matter: - Local execution: Users valued running the model on laptops. Some reportedly ran it on a Raspberry Pi, very slowly. - Multilingual data and tokenization: Chia attributes the claimed language advantage to refined training data and a custom tokenizer, rather than architectural superiority. The international team wanted the model to work for their own families.

That goal creates an evaluation tension. Chia argues that multilingual training can hurt English evaluation scores while improving usefulness elsewhere. He reports training on 100 languages but having evaluations for only 23. Benchmark coverage falls short of the intended user population, making English leaderboard performance an incomplete guide to the project’s objectives.

Chia says the model and code were placed with the Linux Foundation to preserve their availability independently of the founding team. A new commercial entity had attracted support for another trillion-token training run using sixteen H100 nodes. At the event, he anticipated a two-trillion-token model the following month and a hosting, training, and fine-tuning platform by March 15, 2024—plans at the time of the presentation.

Pixee: repair useful findings, triage the noise

RWKV and Pixee AI receive the event’s two judge prizes. Ryan, a Pixee software engineer, presents security automation as a way for a limited security workforce to keep up with increasing code generation. He describes a PDF web application built using ChatGPT that received multiple suggested fixes within three weeks of installing Pixee Bot, including an issue he considered a real vulnerability.

Pixee’s recently shipped Sonar integration ingests code-quality and security findings and automatically remediates them. In the deliberately vulnerable demo application, Sonar reports roughly 2,600 issues and an estimated 33 days of effort. That is a scanner estimate, not measured Pixee savings. The economic argument is that one-minute fixes become substantial work across large backlogs.

Some findings are wrong or not worth fixing. Pixee therefore separates two actions: - Remediate useful findings: Apply security and code-quality improvements, including hardening changes that do not correspond to a fully exploitable vulnerability. - Triage irrelevant findings: Its Triage command-line interface pulls issues from other tools, uses codebase context and a large language model to categorize them, and prompts the user to update their state with an explanation.

A GitHub app integrates fixes into existing development workflows. A command-line tool also lets developers try automated fixes on smaller projects before committing them.

Julius: execute, inspect, revise

Rahul introduces Julius as an AI data scientist, reporting 300,000 users about six months after launch and several thousand daily users. His demo loads a 20,000-row dataset covering a century of human height across countries. Julius initializes a notebook, imports pandas, and displays sample rows; the next request asks it to plot male height in France over time.

The mechanism is an execute–inspect–revise loop in a cloud virtual machine assigned to each user. The AI creates a Jupyter Notebook, writes and runs analysis code, examines the output, and decides whether more code is needed. Rahul says it can recover from initial failures. The team chose Python after substantial work with SQL, seeking deeper analysis such as regression and access to libraries including Matplotlib, Plotly, pandas, NumPy, and scikit-learn.

Asked how Julius differs from ChatGPT Code Interpreter, Rahul emphasizes specialization for frequent data work: - Prevent recurring repair loops: Collect failures from daily usage and add rules or prompting changes to prevent familiar errors at runtime. - Let users finish the analysis: Expose generated code for direct editing and accept natural-language refinements, such as changing a line graph into a bar chart. The claimed advantage is qualitative; he supplies no comparative error rates.

Paper clubs and the meaning of “world model”

The anniversary conversations turn to listeners. Balázs Némethi recalls a Discord paper club where the paper’s author joined partway through. The extra explanation beyond the written paper made the session valuable, and he asks for more author-attended discussions despite the organizing effort.

A host announces an upcoming discussion of Matryoshka Representation Learning, which organizes embedding vectors so they can be truncated to fewer dimensions. The host says OpenAI’s new embedding models expose this reduction through the API. Shorter vectors motivate the claim of major vector-database savings, but no retrieval-quality measurements or workload-specific costs accompany it.

Sylvia Tong’s conversation returns to Sora and the ambiguity of “world model.” A host recounts initially taking Jim Fan’s description to imply an underlying physics model, then understanding Fan to mean a learned representation of the world. This is the host’s account of Fan’s position. The consequential distinction is between learning such a representation and reliably reproducing physical behavior.

The host cites chairs and spoons appearing and disappearing as inconsistencies that undermine confidence, while predicting—without a supporting measurement—that these problems will be resolved within two years. Tong recommends longer conversations to unpack compressed Twitter claims. The editorial challenge is to celebrate progress while explaining what the system actually does.

Publish the evaluation idea before it is finished

RJ Honicky brings telecommunications experience to model evaluation. At the time, he led a data team using telemetry to infer network health and detect improvements or regressions in product and hardware builds. He proposes exploring whether methods for evaluating wireless systems could inform language-model evaluation: both involve nondeterministic behavior. It is a starting question, not a completed method.

The advice is to publish without seeking permission, while accepting that nobody owes the work attention. The host calls the starting mode explorer: share notes as learning progresses, primarily for yourself. Documenting work and reasoning can help your team and future successors even when few people read it.

Make experiments inexpensive and follow the overlap between interest and demand. Publish in several directions, notice what you enjoy, and watch for requests for more. Accumulated experience can become polished teaching grounded in work you have done. Honicky’s aim is to share the learning process, rather than become an influencer.

Platform choice combines publishing features, audience relationships, and familiarity: - Developer-oriented publishing: The host recommends Hashnode as a starting point for developers. - Recurring contact: For Latent Space, the host values Substack’s email distribution and the ability to take the email list elsewhere. - An audience you understand: Spending time on a platform helps reveal what people respond to. That familiarity does not remove production costs; the host describes video as especially expensive to make.

Bringing remote engineers into the conversation

Honicky’s feedback exposes a constraint of technical audio: paper-focused material was hard to follow when he had to locate the papers separately. The host clarifies that the NeurIPS recap had no video. Startup interviews offered another kind of access, introducing people Honicky would otherwise neither know about nor encounter. The host values bringing remote listeners into the community, despite the production burden; the narration expresses an intention to make more recaps when suitable occasions arise.

Jan Zheng describes his route into that community: GitHub answers to Svelte questions led him to the eventual Latent Space creator, then the podcast helped him learn about large language models. After moving to Sydney to work on a clinical trial, he says listening partly motivated his return to San Francisco for its AI activity. He also identifies himself as a member of Build Club, the Sydney AI community described in the conversation as expanding into additional cities.

This part ends with a product engineer explaining what he listens for: orientation on AI and questions he would not know to ask himself. He does not follow machine-learning papers. The value is a technically informed way into the field for someone whose daily work is building products.