Test screenshot-to-click scaling in a simple web game

A developer used Gemini with Playwright to confirm that click coordinates lined up with the browser view.

Done with Gemini· Intermediate· Nov 2025

What they did
Guillaume Laforge connected Gemini’s computer-use preview model to Playwright in Java and ran it in headless Chromium with a 1000x1000 viewport. After each action, the agent received a fresh screenshot, then returned click targets on a normalized 1000x1000 grid; his code converted those points to the actual viewport and clicked. He also built a tiny game where a red button shifts after every click and asked the agent to hit it 10 times and read back the score.
How it went
The agent followed the moving button and ended with a score of 10. That confirmed the click scaling and input handling were working in this test, and no failures were reported.
Tips
Send a new screenshot after every action, and wait about a second before capturing so the page has fully drawn. If you skip that pause, the model may act on a blank or partially rendered screen.

Try it yourself with Gemini

Open [URL of a simple browser game] and play it by inspecting each screenshot and clicking the target element. After every click, wait for the page to update, then take a fresh screenshot before choosing the next move. Use [number] clicks total, and after each one, say whether the click matched the intended spot and whether the target moved as expected. Pause before any irreversible action and ask for approval first. When you finish, report the final score and any mismatch between the screenshot position and the actual click position.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on glaforge.dev ↗

1 more report of this

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.