Control robots with generated code and voice commands
Gemini detects objects and writes robot API code, and with 10 in-context demos an ALOHA robot packed boxes and folded a dress; voice commands became robot plans. Task: Researchers provided Gemini 2.5 with a robot control API (detect, grasp, move, gripper) to generate Python plans zero-shot, such as placing a banana in a bowl. For complex tasks like ALOHA box packing and dress folding, they added 10 interleaved reasoning-action demonstrations to the context. The Live API was configured to convert voice commands into executable robot plans via function calling.; tools: Gemini. Human role: Not stated in the source; limitations: The ALOHA setup relies on an open-source Colab with specific example demonstrations. Access to Gemini Robotics-ER requires joining a trusted tester waitlist. The Live API voice console is available on GitHub, but real-world deployment requires strict safety validation and human oversight for physical robot actions..
Tools
Published on Agent Work Atlas:
Overview
Gemini detects objects and writes robot API code, and with 10 in-context demos an ALOHA robot packed boxes and folded a dress; voice commands became robot plans.
Workflow
Researchers provided Gemini 2.5 with a robot control API (detect, grasp, move, gripper) to generate Python plans zero-shot, such as placing a banana in a bowl. For complex tasks like ALOHA box packing and dress folding, they added 10 interleaved reasoning-action demonstrations to the context. The Live API was configured to convert voice commands into executable robot plans via function calling.
Reported result
Gemini produced two feasible banana plans, with one re-planning around the right arm's reach limit. The ALOHA robot successfully packed boxes and folded a dress using the in-context demonstrations. Voice commands were translated into robot actions through the Live API integration.
Human role
Not stated in the source
Limitations
The ALOHA setup relies on an open-source Colab with specific example demonstrations. Access to Gemini Robotics-ER requires joining a trusted tester waitlist. The Live API voice console is available on GitHub, but real-world deployment requires strict safety validation and human oversight for physical robot actions.
Try it
Act as a robotics developer. I need to test Gemini 2.5's code generation for a simulated robot. 1. Define a Python class `SimulatedRobotAPI` with methods: `detect_object(obj)`, `get_grasp_pose(obj)`, `move_gripper(pos, ori)`, `open_gripper()`, `close_gripper()`. 2. Provide the API schema and a task description: "Pick up the red block and place it in the blue bin." 3. Ask Gemini to generate the Python code to execute this task. 4. Review the generated code for syntax errors and logical safety (e.g., does it check for object detection success?). 5. Run the code in a local simulation environment. [Replaceable Inputs]: - Task: [Insert specific pick-and-place task] - API Schema: [Insert your robot's actual API definitions] - Simulation Env: [Insert path to simulator] Deliverables: - Generated Python script. - Log of simulation execution. - Success/Failure status. Safeguards: - Do not execute on real hardware. - Verify all coordinates are within safe bounds before running. - If the code fails, analyze the error and ask Gemini to re-plan with the error message as context.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.