Play Pokémon Blue with an agent
The Gemini Plays Pokémon agent reads game RAM and screenshots, turns actions into button presses, and uses pathfinder and boulder-puzzle sub-agents. Task: Suggested implementation: An independent developer, Joel Zhang, built a harness that gives Gemini 2.5 Pro a subset of Game Boy RAM as text, overlaid on a screenshot. Gemini picks actions that become button presses. The harness summarizes history every 100 turns and runs a critic instance every 25 turns. It also provides pathfinder and boulder-puzzle sub-agents. Human role: Source evidence is unavailable regarding specific human roles or manual interventions in the agent's gameplay.; tools: Gemini. Human role: Source evidence is unavailable regarding specific human roles or manual interventions in the agent's gameplay.; limitations: Prerequisites include access to a Game Boy emulator with RAM reading capabilities and Gemini 2.5 Pro API access. The model struggled to read raw screen pixels, necessitating RAM-to-text translation. Behavior degraded once context grew well past 100k tokens. Source reliability is secondary, and the original URL was blocked or too short to verify details. Human approvals are not documented..
Tools
Published on Agent Work Atlas:
Overview
The Gemini Plays Pokémon agent reads game RAM and screenshots, turns actions into button presses, and uses pathfinder and boulder-puzzle sub-agents.
Workflow
Suggested implementation: An independent developer, Joel Zhang, built a harness that gives Gemini 2.5 Pro a subset of Game Boy RAM as text, overlaid on a screenshot. Gemini picks actions that become button presses. The harness summarizes history every 100 turns and runs a critic instance every 25 turns. It also provides pathfinder and boulder-puzzle sub-agents. Human role: Source evidence is unavailable regarding specific human roles or manual interventions in the agent's gameplay.
Reported result
The agent beat Pokémon Blue in 813 hours on run 1, then in 406.5 hours on a fully autonomous second run. It looped on long contexts, hallucinated items from other games, and struggled with raw screen pixels. These outcomes are attributed to the author's reported experiments. No independent verification of these metrics is available from the provided source.
Human role
Source evidence is unavailable regarding specific human roles or manual interventions in the agent's gameplay.
Limitations
Prerequisites include access to a Game Boy emulator with RAM reading capabilities and Gemini 2.5 Pro API access. The model struggled to read raw screen pixels, necessitating RAM-to-text translation. Behavior degraded once context grew well past 100k tokens. Source reliability is secondary, and the original URL was blocked or too short to verify details. Human approvals are not documented.
Try it
Act as an AI agent developer. I want to replicate the 'Gemini Plays Pokémon' experiment. 1. **Context**: Use Gemini 2.5 Pro to play Pokémon Blue via a Game Boy emulator. 2. **Inputs**: [Replaceable: Path to Pokémon Blue ROM], [Replaceable: Emulator configuration]. 3. **Steps**: - Build a harness that extracts a subset of Game Boy RAM and converts it to text. - Overlay this text on a screenshot of the game. - Send the combined input to Gemini 2.5 Pro to select actions (button presses). - Implement history summarization every 100 turns and a critic instance every 25 turns. - Integrate pathfinder and boulder-puzzle sub-agents. 4. **Output**: A log of actions taken and game state changes. 5. **Quality Check**: Verify RAM-to-text translation accuracy against known game states. 6. **Safeguards**: Run a small sample dry run (first 10 turns) and require human confirmation before proceeding with longer sessions. 7. **Missing Inputs**: If emulator RAM access is not available, ask for clarification on alternative input methods. Do not claim this prompt was tested or copied from the source.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.