Run Spot step by step with Gemini

A developer paired Gemini Robotics ER 2 with Boston Dynamics Spot so it could plan, check, and continue multi-step chores.

Done with Gemini· Intermediate

What they did
They turned Spot’s movement and hand controls into callable tools, then fed the model live streams of video, speech, or text through the Gemini Live API. Gemini Robotics ER 2 decided what to do next, inspected the camera feed to see whether a step had actually finished, and then advanced to the following step. In the demo, Spot also responded to a spoken command by retrieving items like a popcorn snack.
How it went
For progress classification, which places each frame into five stages, the model reached 57.4% accuracy. For moment finding, it reached 91.3% accuracy with a 0.96s mean absolute distance. The post does not report a success rate for the trash-bag task.
Tips
For real-world robot work, keep a person ready to intervene before anything irreversible happens. The demo code is on GitHub, and the model can be tried as a preview in the Gemini API and Google AI Studio.

Try it yourself with Gemini

Act as a controller for [robot name]. You can use [navigation tool], [gripper tool], and [other tool names] to complete [task]. Take [video/audio/text input], outline the work in order, choose one safe action at a time, and verify with the live camera that each step is done before you proceed. Stop and ask me to approve any irreversible action, including [payments], [sending], [deleting], or [submitting]. If you cannot confirm a step, describe what you observe and wait for my next instruction. Start with a brief plan, then carry out the first step.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on blog.google ↗

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.