Build an agent to play Pokémon Blue from RAM and screenshots

Joel Zhang wired Gemini 2.5 Pro into a Game Boy emulator so it could read game state and press buttons in Pokémon Blue.

Done with Gemini· Advanced

What they did
An independent developer, Joel Zhang, built a setup that fed Gemini 2.5 Pro a text version of part of the Game Boy’s RAM alongside a screenshot. Gemini chose moves, and the harness converted those choices into controller inputs. It also kept a running summary every 100 turns, checked the agent with a critic every 25 turns, and added Gemini-powered helpers for pathfinding and boulder puzzles.
How it went
The system finished Pokémon Blue in 813 hours on its first run, then in 406.5 hours on a second fully autonomous run. It still had trouble with raw pixels, drifted when the context window grew past 100k tokens, and sometimes invented items from other games.
Tips
A RAM-to-text layer helped because the model could not reliably interpret screen pixels on its own. Long-running games also need context trimming or summarization, since performance dropped sharply as the conversation grew.

Try it yourself with Gemini

Help me set up an agent to play [game title] in [emulator/setup]. Give it [RAM/state data] as text plus [screenshots or another visual feed], have it choose the next move, and convert that choice into button presses. Add a small helper for [specific subtask], keep a short log of decisions, and after every [N] turns pause and ask me to approve any irreversible action before continuing.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on arxiv.org ↗

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.