Play a button-clicking game via a browser agent
A developer wired Gemini's Computer Use model to Playwright in Java and had it play a simple button-clicking game to verify click-coordinate scaling. Task: Guillaume Laforge implemented a Java agent loop using Gemini 2.5 Pro Computer Use and Playwright to automate browser interactions. He created a test game where a red button moves randomly upon clicking, requiring the agent to locate and click it ten times. The system captured screenshots after each action, scaled coordinates from a normalized 1000x1000 grid to the viewport, and executed clicks via Playwright.; tools: Gemini. Human role: Not stated in the source; limitations: The model requires a fresh screenshot after every action to maintain context, and a one-second delay is necessary to ensure the page has fully rendered before capture. Without this, the model may receive blank or incomplete screens. The implementation relies on a specific normalized coordinate system (1000x1000) that must be scaled to the actual browser viewport dimensions..
Tools
Published on Agent Work Atlas:
Overview
A developer wired Gemini's Computer Use model to Playwright in Java and had it play a simple button-clicking game to verify click-coordinate scaling.
Workflow
Guillaume Laforge implemented a Java agent loop using Gemini 2.5 Pro Computer Use and Playwright to automate browser interactions. He created a test game where a red button moves randomly upon clicking, requiring the agent to locate and click it ten times. The system captured screenshots after each action, scaled coordinates from a normalized 1000x1000 grid to the viewport, and executed clicks via Playwright.
Reported result
The agent successfully clicked the moving button ten times and reported a final score of 10, confirming that the coordinate scaling and click handling logic worked as intended. The source does not mention any failures in this specific test, though it notes general challenges like page rendering delays and pop-ups in broader experiments.
Human role
Not stated in the source
Limitations
The model requires a fresh screenshot after every action to maintain context, and a one-second delay is necessary to ensure the page has fully rendered before capture. Without this, the model may receive blank or incomplete screens. The implementation relies on a specific normalized coordinate system (1000x1000) that must be scaled to the actual browser viewport dimensions.
Try it
Act as a QA engineer verifying browser automation logic. I need you to set up a Java environment with Playwright and the Gemini API. First, create a simple HTML page with a red button that moves to a random position when clicked and increments a visible score counter. Next, write a Java script that launches a headless Chromium browser with a 1000x1000 viewport. Implement an agent loop that sends the current screenshot and a prompt to Gemini 2.5 Pro Computer Use, asking it to click the button 10 times. Ensure your code scales the model's returned coordinates from the 1000x1000 grid to the actual viewport before executing the click. Add a 1-second sleep after each action to allow for page rendering. Finally, capture the model's text response confirming the score. Do not publish or spend real money; this is a local test. If the API key is missing, ask me for it before proceeding.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.