Have Gemini write robot control plans from vision and speech

A developer used Gemini to turn camera input and robot tools into Python plans, then drove robot actions with voice commands.

Done with Gemini· Advanced

What they did
They gave Gemini 2.5 a small robot control interface with calls for spotting objects, finding grasp points, and opening or closing the gripper. With no examples, it generated Python for tasks like placing a banana in a bowl. For box packing and dress folding on an ALOHA robot, they added 10 examples of thoughts and actions to the context. They also exposed the same robot calls through Live API tools so spoken instructions could be turned into plans while the camera stream was active.
How it went
Gemini came up with two workable banana plans, and one of them adjusted when the right arm could not reach far enough. The ALOHA robot completed box packing and dress folding using the examples. No success rate was reported.
Tips
Start with a tight set of robot actions and test in simulation first. If you want the model to handle failures, include examples that show it revising a plan when a reach or grip problem appears.

Try it yourself with Gemini

You are controlling a robot in [simulator or real setup]. Use the available tools to inspect the scene, identify [target objects], find grasp points, and move the gripper. Write the plan in [language] as code that calls my API for [available actions]. First, outline the plan and ask me to approve it before you run anything that would send, delete, submit, pay, or move the real robot. If a step fails or an object is out of reach, revise the plan and show the updated code before continuing. Goal: [task].

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on developers.googleblog.com ↗

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.