Abstract
Passive human videos of everyday activity offer robots access to a massive amount of human experience without requiring a demonstration for each task. However, using this experience requires recovering knowledge from unstructured video recordings and adapting it to a different body (i.e., the robot). To overcome these challenges, we present mobile code-as-policy (MoCaP), an agentic framework that turns passive human video recordings into guidance for an end-to-end mobile manipulation policy. MoCaP extracts spatial memory and cues of how people prepare to interact with objects, which inform navigation and the search for a manipulation-ready configuration. Guided by passive human experience and current observations, MoCaP uses a closed-loop code-as-policy to coordinate planning and tool use primitives to solve motions under the robot’s geometry and kinematic constraints. On a real mobile manipulator, MoCaP achieves 80% success on each of two mobile manipulation tasks in an indoor environment whereas existing approaches cannot succeed at the task. Video-derived spatial memory improves object-navigation success from 70% to 100%, and human approach prior raises the rate of finding a reachable base pose from 0% to 100% at an unfavorable start location.
Video
How MoCaP Uses Human Video
Spatial memory guides where to search, and the human approach prior guides base placement for manipulation. The code agent integrates this guidance with live observations and execution feedback, while robot-specific checks and planning determine the motions to execute.
Human video as a prior for navigation and manipulation
Spatial memory for object search
An offline builder organizes the recording into text and timestamped reference frames. At task startup, an advisor relates this memory to the instruction and initial robot view.
Human approach prior for base placement
Recorded interactions retain the view preceding the reach. MoCaP estimates where the person stood relative to the target and uses that direction to guide the robot’s approach. The robot chooses its own distance from the object and checks whether its arm can reach a grasp.
Closed-loop Code-as-Policy
A coding agent generates short Python programs that call the robot’s navigation and manipulation tools. Execution feedback and a fresh observation inform the next code-generation step.
# Approach from the side suggested by the human video
dock = dock_to_visible_object("can")
arm = dock.get("suggested_arm", "right")
# Check nearby base poses before grasping
prepare_for_manipulation("can", arm=arm)
position, quat = sample_grasp_pose("can", arm=arm)
open_gripper(arm=arm)
goto_grasp_pose("can", position, quat, arm=arm)
close_gripper(arm=arm)
lift_grasped_object(arm=arm)
finish("can grasped and held")MoCaP in Action
All experiments run on the real YOR robot: an omnidirectional base, a vertical lift, two 7-DoF AgileX Nero arms, and a ZED 2i RGB-D camera. The human recording was made with Aria Gen 2 glasses, using RGB only. A single memory built from it is shared by every task.
01Complete mobile manipulation
The box task starts with the target in view. The can is outside the robot’s initial view. Success requires grasping the requested object and keeping it securely held. We compare against CaP-X through its mobile-manipulator API, ported to our robot with the same task instructions, policy model, time limit, and program budget.
Grasp the cardboard box on the white cabinet, 4×.
02Spatial memory for object navigation
The robot navigates to a blue trash bin from the kitchen. An episode succeeds if the base stops within 1 m of the bin and the bin is visibly identified in the terminal robot view. CoW and ApexNav cannot consume an offline prior. They are compared against the no-prior variant. The comparison with full MoCaP measures the effect of spatial memory.
Navigate to the blue trash bin from the kitchen, 4×.
03Human approach prior for base placement
We compare approaches with and without the human prior. Both conditions use the same perception, checks of nearby robot positions, and grasp planner. Only the approach direction changes.
We test starts A, B, and C to examine how the approach direction affects arm reach. Without the prior, the robot keeps its arrival direction. The human video shows the person standing on the C side before reaching for the can. From A, this prior guides the robot toward that side, helping it find a position from which its arm can reach a grasp.
4×.
BibTeX
TBD