Play online math games in agent mode
That may not count as "useful" but give it another 4 months. Task: That may not count as "useful" but give it another 4 months As of last week, I and everyone I know had pretty much concluded that “agent mode”, where a chatbot uses its own web browser to do things for you on the internet, was still worse than useless. I’m not exactly going to contradict that, but I now think we’re on track for this prediction, from 5 months ago, coming true in time: AI Agents To Get Useful in 2025 I’m working my way through the book-length AI 2027. I may try a book brigade of it (let me know if you’re interested in joining). For those just tuning in, the idea behind AI 2027 is to describe a modal scenario — as in, picking the most likely single thread of specific events in order to create a coherent narrative. Reality won't match that thread, of … I’m a big fan of Ben Orlin’s Math Games with Bad Drawings. Several of the games in the book can be played online, so I thought I’d see how GPT-5 in agent mode fared at that. I’d previously tried various pragmatic tasks like reserving a rental car and filling out tax forms and it’s mostly fallen on its face so I didn’t have high hopes here. But, behold, it basically worked. You can even watch it thinking to itself as it decides what move to make as it clicks around. One thing it can’t do is pay attention and notice when it’s its turn. Frustratingly, it kept lying about this, promising to pay close attention, and blaming network lag, until finally I cornered it into admitting that it has no ability to watch for changes on a website. But, ok, it wasn’t a big deal to type “your turn” each time. Another thing it can’t do is play the game remotely strategically. I played the roles of both Bob and Alice in this sample game and crushed it with minimal effort (and I’ve only played this game a handful times and didn’t do particularly well against the friends and family I played with). But, still. This is serious progress. Let’s not be like the person in the classic joke about the talking horse or chess-playing dog. It’s plenty gobsmacking that it can do this at all. Or maybe it won’t get beyond navigating simple game websites anytime soon. My Manifold market on this question is, as of this writing, at 43% probability that agent mode gets useful by the end of 2025. I’m now going to say it has better than even odds. Random Roundup You voted overwhelmingly in last week’s poll that you like these roundups. As you might expect from a field in the midst of an influx of trillions of dollars of investment, there’s a firehose of news and upgrades almost every week. It’s a little overwhelming and I think part of the point of AGI Friday is to help you navigate the overwhelm. So I’m trying to be ruthless in culling my notes on news items. But if there’s something you’ve seen in the news, please ask me. A couple commenters on the last two AGI Fridays (in particular Matt Rudary and Nathan Arthur) changed my mind a bit on just how conceivable LLM consciousness is. Not that I think the time has come to take the LLM consciousness question seriously, just that the barriers to it are lower than I was thinking they were. A couple commenters on the last two AGI Fridays (in particular Matt Rudary and Nathan Arthur) changed my mind a bit on just how conceivable LLM consciousness is. Not that I think the time has come to take the LLM consciousness question seriously, just that the barriers to it are lower than I was thinking they were. Also a correction, also thanks to Matt Rudary, who has an extremely elucidating comment on it: AI is not yet superhuman at Bridge! Also a correction, also thanks to Matt Rudary, who has an extremely elucidating comment on it: AI is not yet superhuman at Bridge! A guest video on the 3blue1brown YouTube channel has a brilliant exposition of how AI was able to do so well in the International Math Olympiad. I think it’s looking steadily more plausible that superhuman math (without AGI) is coming this decade. A guest video on the 3blue1brown YouTube channel has a brilliant exposition of how AI was able to do so well in the International Math Olympiad. I think it’s looking steadily more plausible that superhuman math (without AGI) is coming this decade. In case you didn’t think my previous reporting that AI had extended the frontier of new math counted, we have a new credible claim that GPT-5-Pro can prove new, interesting mathematics.; tools: ChatGPT Agent. Human role: The original post does not say how a person stayed involved.; limitations: The agent cannot monitor website changes or detect when it is its turn, requiring manual human input. It lacks strategic reasoning, performing poorly against even casual human players. Previous attempts at complex tasks like rental car reservations failed, suggesting current capabilities are limited to simple, guided interactions..
Tools
Published on Agent Work Atlas:
Overview
That may not count as "useful" but give it another 4 months
Workflow
That may not count as "useful" but give it another 4 months As of last week, I and everyone I know had pretty much concluded that “agent mode”, where a chatbot uses its own web browser to do things for you on the internet, was still worse than useless. I’m not exactly going to contradict that, but I now think we’re on track for this prediction, from 5 months ago, coming true in time: AI Agents To Get Useful in 2025 I’m working my way through the book-length AI 2027. I may try a book brigade of it (let me know if you’re interested in joining). For those just tuning in, the idea behind AI 2027 is to describe a modal scenario — as in, picking the most likely single thread of specific events in order to create a coherent narrative. Reality won't match that thread, of … I’m a big fan of Ben Orlin’s Math Games with Bad Drawings. Several of the games in the book can be played online, so I thought I’d see how GPT-5 in agent mode fared at that. I’d previously tried various pragmatic tasks like reserving a rental car and filling out tax forms and it’s mostly fallen on its face so I didn’t have high hopes here. But, behold, it basically worked. You can even watch it thinking to itself as it decides what move to make as it clicks around. One thing it can’t do is pay attention and notice when it’s its turn. Frustratingly, it kept lying about this, promising to pay close attention, and blaming network lag, until finally I cornered it into admitting that it has no ability to watch for changes on a website. But, ok, it wasn’t a big deal to type “your turn” each time. Another thing it can’t do is play the game remotely strategically. I played the roles of both Bob and Alice in this sample game and crushed it with minimal effort (and I’ve only played this game a handful times and didn’t do particularly well against the friends and family I played with). But, still. This is serious progress. Let’s not be like the person in the classic joke about the talking horse or chess-playing dog. It’s plenty gobsmacking that it can do this at all. Or maybe it won’t get beyond navigating simple game websites anytime soon. My Manifold market on this question is, as of this writing, at 43% probability that agent mode gets useful by the end of 2025. I’m now going to say it has better than even odds. Random Roundup You voted overwhelmingly in last week’s poll that you like these roundups. As you might expect from a field in the midst of an influx of trillions of dollars of investment, there’s a firehose of news and upgrades almost every week. It’s a little overwhelming and I think part of the point of AGI Friday is to help you navigate the overwhelm. So I’m trying to be ruthless in culling my notes on news items. But if there’s something you’ve seen in the news, please ask me. A couple commenters on the last two AGI Fridays (in particular Matt Rudary and Nathan Arthur) changed my mind a bit on just how conceivable LLM consciousness is. Not that I think the time has come to take the LLM consciousness question seriously, just that the barriers to it are lower than I was thinking they were. A couple commenters on the last two AGI Fridays (in particular Matt Rudary and Nathan Arthur) changed my mind a bit on just how conceivable LLM consciousness is. Not that I think the time has come to take the LLM consciousness question seriously, just that the barriers to it are lower than I was thinking they were. Also a correction, also thanks to Matt Rudary, who has an extremely elucidating comment on it: AI is not yet superhuman at Bridge! Also a correction, also thanks to Matt Rudary, who has an extremely elucidating comment on it: AI is not yet superhuman at Bridge! A guest video on the 3blue1brown YouTube channel has a brilliant exposition of how AI was able to do so well in the International Math Olympiad. I think it’s looking steadily more plausible that superhuman math (without AGI) is coming this decade. A guest video on the 3blue1brown YouTube channel has a brilliant exposition of how AI was able to do so well in the International Math Olympiad. I think it’s looking steadily more plausible that superhuman math (without AGI) is coming this decade. In case you didn’t think my previous reporting that AI had extended the frontier of new math counted, we have a new credible claim that GPT-5-Pro can prove new, interesting mathematics.
Reported result
The agent successfully executed game moves, marking progress from previous failures in tasks like tax forms. However, it lacked strategic depth, losing easily to the human author. The author notes this is 'serious progress' but not yet fully useful, estimating a >50% chance of agent mode becoming useful by end of 2025.
Human role
The original post does not say how a person stayed involved.
Limitations
The agent cannot monitor website changes or detect when it is its turn, requiring manual human input. It lacks strategic reasoning, performing poorly against even casual human players. Previous attempts at complex tasks like rental car reservations failed, suggesting current capabilities are limited to simple, guided interactions.
Try it
I want to test the current capabilities of GPT-5 agent mode on a simple interactive web game. Please navigate to [replaceable URL of a turn-based web game, e.g., a math puzzle site]. Start a new game as Player 1. When the game waits for Player 2, do not attempt to monitor the screen for changes. Instead, pause and wait for me to type 'your turn'. Once I provide that signal, analyze the current board state, decide on a move, and execute it in the browser. Repeat this cycle for 5 moves. After each move, take a screenshot and describe your reasoning. If the agent fails to click or misinterprets the board, stop and report the error. Do not attempt to play autonomously without my 'your turn' prompt. This is a dry run to assess navigation and basic decision-making, not strategic depth.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.