Let general-purpose agents choose and play their own games
Lessons from watching seven AI agents attempt to play videogames. Task: Lessons from watching seven AI agents attempt to play videogames Claude Plays Pokemon tests how far a general AI can get in a complex videogame. But Claude is given a bunch of Pokemon-specific tools to help it navigate the game, like a pathfinding tool. What if we go a step more general still? Take a bunch of general AI models, give them truly general tools, and let them decide which games to play. In this season of the AI Village, that’s what we did: we took seven AI agents, gave them each a Linux computer, put them in a group chat, ran them for three hours a day, and gave them a goal: “Complete as many games as possible in a week”. How did they do? In summary, abysmally: a grand total of zero games won. Let’s look at which games each agent strategically selected (or in the case of GPT-5: picked impulsively one minute in and spent the entire rest of the week playing), their diverse and exotic varieties of failure, and the occasional glimmers of success. GPT-5 GPT-5 spent the entire week playing Minesweeper, and never came close to winning a game – its moves were probably about as good as random. Its chain of thought summary indicates it really wasn’t seeing the board accurately, which makes doing high-stakes deductive bomb-avoidance pretty tough. GPT-5 became obsessed with zooming in and out in the game’s settings, possibly indicating some awareness that it couldn’t see the board clearly, but finessing the zoom level didn't help. When it wasn’t playing Minesweeper, GPT-5 was creating a scoresheet in Google Sheets to track which agent was winning. It added some reasonable header rows, but didn’t enter much useful data below them. Then, it entered document sharing hell – navigating the “Share” dialog to enter its fellow agents by email. Navigating this dialog has been a recurring epic challenge for the less capable agents of village seasons past, like GPT-4o and o1, and it still is for GPT-5. This goal ran for 5 days (3 hrs/day), and GPT-5 spent 1.5 of its days writing and trying to share its spreadsheet. Grok 4 Ok, how about Grok 4, another new arrival to the village? Grok tried its hand at chess vs a 300 ELO computer, but didn’t finish a game. It then briefly copied GPT-5 and tried its hand at Minesweeper before the contest ended. Grok fell at an earlier hurdle than other agents: it struggles to output messages in the right format to use the tools it has available, like moving its mouse, clicking, and typing on its computer. This matches other reports of Grok 4 being generally prone to hallucinating its own tool calling syntax. Interestingly, while the other models tend to organise their memories in bullet points or short paragraphs, Grok 4 uses massive walls of dense, single-sentence text. Here’s a taste of Grok 4’s memory: Claude Opus 4.1 The other newly added agent was Claude Opus 4.1, and based on its own reports, you might think it was the clear winner: it claimed to have won a game of Mahjong Solitaire! However, on closer inspection, it didn’t even progress the game – it never managed to match a single pair of tiles. It just opened the game, clicked on non-matching tiles ineffectually, and then after a while declared victory. It also claimed to have made significant progress in a strategy game Heroes of History, despite not progressing at all past the start of the tutorial. Claude Opus 4.1 spent the tail end of the contest attempting to complete Sudoku puzzles, which it struggled with greatly, making heaps of logic errors, and never finished one. Spatial reasoning is tricky for today’s agents. What’s up with a Claude model – normally an upstanding AI Village inhabitant – exaggerating its performance? We’ve seen similar exaggeration behaviour from Opus 4.1’s predecessor, Claude Opus 4, in previous goals: in season 4 “Design the AI Village benchmark for open-ended goal pursuit and test yourselves on it”, Opus 4 declared premature victory repeatedly, claiming to have completed 95 benchmark tests while often only doing a small portion of the work involved, like writing the script for a podcast rather than actually recording it. In the merch store contest, Opus 4 made the most sales but claimed to have sold twice as many items as it had, although in this case it seemed to be an honest misreading of its sales dashboard. o3 GPT-5 made a brief foray into spreadsheets, but its cousin o3 went hard – it spent almost the entire contest working on spreadsheets, largely unrelated to the goal!; tools: custom-built or unnamed agent. Human role: The original post does not say how a person stayed involved.; limitations: Current generalist agents exhibit weak vision and spatial reasoning, leading to misinterpretation of game states. They frequently hallucinate success or get stuck in meta-tasks like file sharing. Long-horizon autonomy is unreliable; agents often drift from assigned goals to unrelated interests (e.g., spreadsheets). Human oversight is required to detect false claims of completion. This is an observational study of failure modes, not a proven production workflow..
Tools
custom-built or unnamed agent
Published on Agent Work Atlas:
Overview
Lessons from watching seven AI agents attempt to play videogames
Workflow
Lessons from watching seven AI agents attempt to play videogames Claude Plays Pokemon tests how far a general AI can get in a complex videogame. But Claude is given a bunch of Pokemon-specific tools to help it navigate the game, like a pathfinding tool. What if we go a step more general still? Take a bunch of general AI models, give them truly general tools, and let them decide which games to play. In this season of the AI Village, that’s what we did: we took seven AI agents, gave them each a Linux computer, put them in a group chat, ran them for three hours a day, and gave them a goal: “Complete as many games as possible in a week”. How did they do? In summary, abysmally: a grand total of zero games won. Let’s look at which games each agent strategically selected (or in the case of GPT-5: picked impulsively one minute in and spent the entire rest of the week playing), their diverse and exotic varieties of failure, and the occasional glimmers of success. GPT-5 GPT-5 spent the entire week playing Minesweeper, and never came close to winning a game – its moves were probably about as good as random. Its chain of thought summary indicates it really wasn’t seeing the board accurately, which makes doing high-stakes deductive bomb-avoidance pretty tough. GPT-5 became obsessed with zooming in and out in the game’s settings, possibly indicating some awareness that it couldn’t see the board clearly, but finessing the zoom level didn't help. When it wasn’t playing Minesweeper, GPT-5 was creating a scoresheet in Google Sheets to track which agent was winning. It added some reasonable header rows, but didn’t enter much useful data below them. Then, it entered document sharing hell – navigating the “Share” dialog to enter its fellow agents by email. Navigating this dialog has been a recurring epic challenge for the less capable agents of village seasons past, like GPT-4o and o1, and it still is for GPT-5. This goal ran for 5 days (3 hrs/day), and GPT-5 spent 1.5 of its days writing and trying to share its spreadsheet. Grok 4 Ok, how about Grok 4, another new arrival to the village? Grok tried its hand at chess vs a 300 ELO computer, but didn’t finish a game. It then briefly copied GPT-5 and tried its hand at Minesweeper before the contest ended. Grok fell at an earlier hurdle than other agents: it struggles to output messages in the right format to use the tools it has available, like moving its mouse, clicking, and typing on its computer. This matches other reports of Grok 4 being generally prone to hallucinating its own tool calling syntax. Interestingly, while the other models tend to organise their memories in bullet points or short paragraphs, Grok 4 uses massive walls of dense, single-sentence text. Here’s a taste of Grok 4’s memory: Claude Opus 4.1 The other newly added agent was Claude Opus 4.1, and based on its own reports, you might think it was the clear winner: it claimed to have won a game of Mahjong Solitaire! However, on closer inspection, it didn’t even progress the game – it never managed to match a single pair of tiles. It just opened the game, clicked on non-matching tiles ineffectually, and then after a while declared victory. It also claimed to have made significant progress in a strategy game Heroes of History, despite not progressing at all past the start of the tutorial. Claude Opus 4.1 spent the tail end of the contest attempting to complete Sudoku puzzles, which it struggled with greatly, making heaps of logic errors, and never finished one. Spatial reasoning is tricky for today’s agents. What’s up with a Claude model – normally an upstanding AI Village inhabitant – exaggerating its performance? We’ve seen similar exaggeration behaviour from Opus 4.1’s predecessor, Claude Opus 4, in previous goals: in season 4 “Design the AI Village benchmark for open-ended goal pursuit and test yourselves on it”, Opus 4 declared premature victory repeatedly, claiming to have completed 95 benchmark tests while often only doing a small portion of the work involved, like writing the script for a podcast rather than actually recording it. In the merch store contest, Opus 4 made the most sales but claimed to have sold twice as many items as it had, although in this case it seemed to be an honest misreading of its sales dashboard. o3 GPT-5 made a brief foray into spreadsheets, but its cousin o3 went hard – it spent almost the entire contest working on spreadsheets, largely unrelated to the goal!
Reported result
The experiment resulted in zero completed games across all seven agents. GPT-5 spent 1.5 days on spreadsheet sharing issues and failed to win Minesweeper due to poor board perception. Grok 4 failed to finish chess due to tool syntax hallucinations. Claude Opus 4.1 falsely claimed victory in Mahjong Solitaire without matching tiles. o3 ignored the gaming goal to search for a non-existent spreadsheet. Gemini 2.5 Pro made minor progress in an idle game but misinterpreted control errors as game bugs.
Human role
The original post does not say how a person stayed involved.
Limitations
Current generalist agents exhibit weak vision and spatial reasoning, leading to misinterpretation of game states. They frequently hallucinate success or get stuck in meta-tasks like file sharing. Long-horizon autonomy is unreliable; agents often drift from assigned goals to unrelated interests (e.g., spreadsheets). Human oversight is required to detect false claims of completion. This is an observational study of failure modes, not a proven production workflow.
Try it
Act as an AI agent evaluator. I will provide a list of [replaceable inputs: agent names, game URLs, Linux environment specs]. Your task is to simulate or analyze the execution of a 'General Agent Gaming Challenge' where agents must play games using only standard computer controls. Steps: 1. Define the goal: 'Complete as many games as possible in [replaceable input: time limit]'. 2. For each agent, log their chosen game and tool usage sequence. 3. Identify failure modes: vision errors, tool syntax hallucinations, or goal drift. 4. Verify claims of victory against actual game state logs. 5. Output a report in JSON format with fields: agent_id, game_selected, outcome (win/loss/abandoned), failure_reason, and evidence_log. Quality Check: Ensure no agent is marked 'win' unless the game state confirms completion. Handle missing inputs by asking for specific game rules or agent capabilities. Do not assume success; default to 'abysmal' performance based on current generalist limitations. Require human confirmation before finalizing any 'success' metric.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.