How Claude Plays Pokémon: Tools, Memory, and the Long Game

Fifty-two hours in Mount Moon

At the start of the Latent Space conversation, published in March 2025, the host checked the ongoing Pokémon run: Claude had been stuck in Mount Moon for 52 hours. Creator David Hershey confirmed the fifty-something-hour stay. It was somewhat worse than his usual observations, but characteristic of the model’s navigation difficulties. Claude could catch Pokémon and reach new areas while still struggling to interpret the screen.

Claude Plays Pokémon began around June 2024 as Hershey’s environment for experimenting with long-running agents. He built on an initial connection attempted by someone else at Anthropic. Nostalgia drove the choice of game: Pokémon would keep him interested through months of debugging. It also accommodated model latency. Doing nothing for five seconds typically has no consequence, allowing the agent to act through successive snapshots.

Successive releases brought visible changes: - June Claude 3.5 Sonnet showed limited signs of life. - October Claude 3.5 Sonnet could leave the starting house somewhat reliably, obtain a starter Pokémon, and sometimes name it. - Early Claude 3.7 Sonnet runs caught Pokémon and got out of Viridian—milestones Hershey had not previously observed. These were qualitative observations, without quantified success rates.

The project accompanied the release of Claude 3.7 Sonnet and Claude Code, with play broadcast on the Claude Plays Pokémon stream. For Hershey, watching the game provided a qualitative measuring stick for model behavior and a way to explain agent progress to audiences unfamiliar with software-engineering benchmarks.

A conversation that takes actions

Hershey kept the agent harness—the code that supplies context and executes the model’s actions—deliberately modest. He believed a stronger Pokémon-solving program with Claude in the loop would be straightforward to build. The project’s purpose was to understand what Claude could do with a simple harness.

The core loop is an expanding conversation: 1. Build a prompt from instructions, stored knowledge, and conversation history. 2. Call the model and execute its returned tool calls. 3. Append the tool results and continue. 4. Periodically summarize the history to keep play within the context window—the information available to the model in a request. There are essentially no user interjections during play. The instructions define three tools, explain their use, supply roughly six Pokémon facts, and address a few severe mistakes.

The tools let Claude press emulator buttons, manage a knowledge base, and invoke Navigator, a local movement helper. Each button action executes a sequence of presses and returns two screenshots: the game image and a copy with coordinate overlays. Those coordinates let Claude specify destinations for Navigator.

The implementation exposes nearly all current game state programmatically, although it uses only a fraction of that information. Claude Code helped map memory addresses into Python. Tool responses include a short state description read directly from the game; explicit location information is particularly useful because Claude otherwise sometimes invents successful transitions between zones.

Helping Claude read the screen

One overnight failure explains the need for intervention: Claude mistook a red doormat for a dialogue box and spent twelve hours pressing A to dismiss it. The harness added reminders to interrupt stuck behavior, including detection of prolonged lack of movement—roughly thirty time steps was an example threshold—and prompts to use the knowledge base.

Prior Pokémon knowledge could also distort perception. Early in the Twitch run, Claude identified an unrelated character in southern Pallet Town as Professor Oak. Feedback sometimes helped: after trying Thundershock on a Geodude and seeing it fail, Claude acknowledged that the move did not work. The developer remained unsure whether existing game knowledge helped more than it harmed.

Even identifying the player character was unreliable. Instructions specifying exact coordinates, the center of the screen, and a red hat did not consistently resolve the confusion. Claude struggled with relative positions and sometimes lost track of which character it controlled.

Navigator automates movement to a visible target. Claude selects a coordinate on the current screen, and the tool presses the buttons needed to reach it. It cannot request arbitrary destinations across the whole map. Without this assistance, Claude repeatedly tried to walk through walls toward destinations on the other side. Extensive prompting had not resolved that failure.

How much history helps?

The system prompt was estimated at about 1,000 tokens. The knowledge base, which stores longer-term concepts and memories, was capped at about 8,000 tokens to encourage selection of useful information. Conversation history consumed most input tokens as a run progressed, especially through enlarged screenshots. The harness retained some previous images and removed others.

Periodically, a summary replaces the accumulated history as the first user message in the next stretch of play. Informal tuning favored an interval of thirty over twenty or forty. The discussion uses messages, turns, and tool-call cycles interchangeably, so thirty is a reported tuning choice rather than an exact message-count specification.

Individual API requests were estimated at 5,000–100,000 tokens, depending on summarization and knowledge-base state. That upper figure describes one request, not the total needed to finish the game. Extensive experimentation was estimated to consume at least thousands of dollars in tokens; no exact total or measured per-run price was given.

More visual history did not clearly improve Claude’s understanding of change over time. Giving it twenty or thirty screenshots at once produced no obvious benefit, and whether different locations needed different history lengths remained uncertain. Navigation prompts also helped little unless they specified exactly what to do. Claude handled easy areas reasonably well, but its progress through complex mazes could resemble a random walk.

Apparent route knowledge could make matters worse: in one episode, Claude spent twelve hours pursuing its belief that an exit lay on an eastern wall. Discovering what to do without following a memorized walkthrough was part of the challenge. The developer valued the strengths and weaknesses revealed through play, despite disagreements about how much assistance should count as acceptable.

Better models, fewer prompt workarounds

The transition from 3.5 to 3.7 Sonnet brought no serious degradation in the developer’s experiments. Both ordinary operation and reasoning mode, in which the model spends time reasoning, proved useful. Across model upgrades, the recurring harness change was deleting prompt workarounds: instructions addressing older failures became unnecessary, and prescribed strategies could constrain a newer model’s approach.

A smaller prompt change had a surprising effect. After an instruction encouraged Pokémon nicknames for entertainment, the developer observed Claude immediately healing newly caught, injured Pokémon it had named—a behavior reportedly absent before. This was an anecdotal association, not a controlled comparison.

Successful long runs sometimes accumulated useful strategies in the knowledge base. Claude 3.7 Sonnet occasionally recorded observations about its own limitations, including visual mistakes it should avoid repeating. One potentially transferable lesson concerned how many actions to take before checking the result: pressing too many buttons at once made Claude lose track of the game state.

The knowledge base was roughly a Python dictionary appended to the prompt. Reusing it after a restart or in another Pokémon game remained a proposal, without a demonstrated transfer result. Keeping memory continuously available had a practical advantage: when Claude had to choose to read, load, and save external memories, it could underuse them or forget information. Better memory management remained a promising improvement.

Recognizing a mistake without escaping it

New models had made a larger difference than small harness adjustments. One failure illustrated the remaining gap: Claude left Oak’s Lab, recognized that it needed to travel north toward Route One, and pressed up—walking straight back into the lab. It exited and repeated the move, alternating up and down roughly twelve times. Recognizing the immediate mistake did not prevent the action that recreated it.

Seeing, navigating, and remembering visual context remained major capability gaps. The developer considered Pokémon a useful proxy for tasks requiring sustained progress over many actions, while leaving open whether better image or video interfaces were also needed. At recording time, the developer expected the current stream to miss the discussed sixteen-day completion target, particularly doubting that it would clear Victory Road in the remaining thirteen days. This was a forecast, not a final result.

The first victory over Brock was nevertheless a highlight after roughly eight months of work. A Slack progress update alerted the developer to the impending battle that morning. The research blog’s progress graph represented a single best run, which reached at least Surge’s gym and continued a little farther. Its reporting endpoint reflected the time available before publication, rather than a reported terminal failure.

Evaluate the whole run

The closing discussion returned to other games. Before joining Anthropic, the developer had experimented with training an open-source model on Magic draft data to improve card selection from packs; no quantitative result was reported. Open-sourcing the Pokémon project had also been discussed, but no plan was settled. It remained outside the developer’s day job.

The broader promise was improved course correction, plan updates, and persistence. Pokémon made those changes visible alongside persistent failures. The developer expected those abilities to support practical agents, though successful application outside the game remained an expectation.

The final question returned to measurement. Small scenario tests helped inspect particular behaviors, but the preferred evaluation was to run a configuration ten times and measure how quickly it reached game milestones, such as gym badges. This is an expensive integration test of the combined system: whether the model, tools, memory, and navigation support repeatedly turn decisions into progress.