We want humanoid robots to be able to perform complex, long-horizon tasks in the real world — putting away the groceries or cleaning a room, for example. This requires diverse loco-manipulation skills, which can be easily parameterized to handle object affordances and interact with the world around it safely.
In HANDOFF, Lizhi Yang proposes a 10-D learned whole body controller, which can be converted to whole body actions, and which can be used by a VLM-driven agentic planner to perform complex, multi-step manipulation actions in the real world.
We then discuss how it’s possible to deploy such controllers in the real world, how to make them safe around people via controlled barrier functions and safety functions. This enables humanoids which can move safely through dynamic, crowded environments in the real world.
Learn more in Episode 90 of RoboPapers, with Michael Cho and Chris Paxton.
Abstract
For a humanoid robot to be deployed in the real world, the choice of command space (i.e., the interface between task planning and whole-body control) is crucial. Existing whole-body controllers typically demand dense kinematic or spatial references that planners struggle to synthesize from task semantics. We instead propose a compact, explicit interface that is intuitive, general, modular, and expressive enough for diverse loco-manipulation skills. To this end, we introduce HANDOFF, a single humanoid whole-body controller that follows this interface and is distilled via multi-teacher KL distillation under a context-conditioned gating scheme into a mixture-of-experts student from three complementary specialists: whole-body motion tracking with safety-filtered data, locomotion, and fall-recovery. On the Unitree G1, HANDOFF matches state-of-the-art velocity tracking and offers one of the largest robust manipulation workspaces. We further demonstrate hardware feasibility through multiple natural-language-driven task roll-outs, powered by a VLM-driven agentic planner with no task-specific data or controller fine-tuning.
Learn More
Project page for HANDOFF: https://lzyang2000.github.io/HANDOFF/
Code: https://github.com/lzyang2000/HANDOFF
ArXiV: https://arxiv.org/abs/2606.06493
