Play Game Boy games through an emulator

Tools

Claude

Overview

An AI-powered game playing agent using Claude and PyBoy

Workflow

An AI-powered game playing agent using Claude and PyBoy Overview Claude Player is an AI agent that lets Claude play Game Boy games through the PyBoy emulator. The agent observes game frames, makes strategic decisions, and controls the emulator through button inputs. I have been working on this project for a while, and have been meaning to clean it up and release it, and with the release of Claude 3.7 (especially given their semi official https://www.twitch.tv/claudeplayspokemon stream of a similar project), I thought it was a good time to do so. I've taken some imspiration from their official implementation by adding additional memory tools and summarisation, however mine differs in that I don't have any coordinate based movement helpers: it is purely button based. Additionally, the emulator only ticks when the AI sends inputs, so it is not running at real time speed. Features AI-Powered Gameplay: Uses Claude 3.7 to analyze game frames and determine optimal actions Dual Emulation Modes: Turn-based: Emulator only advances when AI sends inputs (default) Continuous: Real-time gameplay with periodic AI analysis Turn-based: Emulator only advances when AI sends inputs (default) Continuous: Real-time gameplay with periodic AI analysis Memory System: Short-term and long-term memory to maintain game context Automatic Summarization: Periodically generates game progress summaries Tool-Based Control: Structured tools for game interaction and state management Screenshot Capture: Automatically saves frames for analysis and debugging Requirements Python 3.10+ PyBoy emulator Anthropic API key Game Boy ROM files Optional: Saved state files Installation Clone the repository: git clone https://github.com/jmurth1234/claude-player.git cd claude-player Clone the repository: Install dependencies using Pipenv (recommended): pipenv install Install dependencies using Pipenv (recommended): Create a .env file with your Anthropic API key: ANTHROPIC_API_KEY=your_api_key_here Create a .env file with your Anthropic API key: Place your Game Boy ROM file in the project directory Place your Game Boy ROM file in the project directory Configuration Configuration is loaded from config.json (created automatically on first run if not found). The settings are structured to avoid duplication between different modes. You can customize Claude's behavior by adding custom instructions that will be injected into the system prompt. You can customize these settings by: Editing the generated config.json file directly Creating your own configuration file and specifying it with: python play.py --config my_config.json Usage Activate the Pipenv environment: pipenv shell Activate the Pipenv environment: Run the agent: python play.py Or specify a custom configuration file: python play.py --config my_config.json Run the agent: Or specify a custom configuration file: For setting up a saved state, you can use the included utility script: python emu_setup.py This script runs the emulator to help you create a saved state that you can reference in your configuration. For setting up a saved state, you can use the included utility script: This script runs the emulator to help you create a saved state that you can reference in your configuration. Game Controls The agent uses a structured notation for game inputs: Single Press: A (press A once) Hold: A2 (hold A for 2 ticks) Simultaneous: AB (press A and B together) Wait: W or W2 (wait for 1 or 2 ticks) Sequence: R2 A U3 (right for 2 ticks, A once, up for 3 ticks) Available buttons: U (Up), D (Down), L (Left), R (Right), A, B, S (Start), X (Select), W (Wait) Tool System The AI uses several tools to interact with the game: send_inputs: Send button sequences set_game: Identify the current game set_current_goal: Set the current gameplay objective add_to_memory: Store important information remove_from_memory: Remove outdated information update_memory_item: Update existing memory items toggle_thinking: Dynamically enable or disable thinking capabilities (when both THINKING and DYNAMIC_THINKING are enabled) Debugging Game frames are saved to ./frames/{timestamp}/ Detailed logs are written to game_agent.log Contributing Contributions welcome! Please feel free to submit a Pull Request. License MIT License Acknowledgments Anthropic for the Claude AI model PyBoy developers for the Game Boy emulator

Reported result

No measured results or success metrics are reported in the source. The project is described as a released tool with features like automatic summarization and screenshot capture, but actual gameplay performance or strategic effectiveness is not quantified. A measurable acceptance check would be verifying that the agent successfully completes a specific in-game objective without human intervention.

Human role

The original post does not say how a person stayed involved.

Limitations

Prerequisites include Python 3.10+, PyBoy, an Anthropic API key, and Game Boy ROMs. The emulator only advances when inputs are sent in turn-based mode, so it does not run at real-time speed. Source reliability is secondary as it is a GitHub repository description without execution logs. Human approval is required for initial setup and configuration.

Try it

I want to set up the Claude Player agent to play a Game Boy game. First, clone the repository from [replaceable URL: https://github.com/jmurth1234/claude-player.git] and install dependencies using Pipenv. Create a .env file with my Anthropic API key [replaceable input: API_KEY] and place a ROM file [replaceable input: ROM_NAME.gb] in the directory. Configure the agent to use turn-based mode. Run a dry test with a simple game sequence to verify the agent can send inputs and receive frame analysis. Show me the logs and screenshots from this test before proceeding to a longer session. If the agent fails to interpret the screen, adjust the custom instructions in config.json. Do not start a full game session until I confirm the dry run was successful.

More cases · Content & creative

• Play a button-clicking game via a browser agent
• Generate ads from a website URL
• Design a personal site with Codex CLI and a Markdown spec
• Turn an outline into a long-form post with SEO bundle
• OpenAI dots: Proposing and Building a Product Manager Toolbox