Contact-rich manipulation is still very challenging for robotics. Problems like opening a jar, or in-hand reorientation of an object, require making repeated contact with different parts of a robot’s hand, and this is hard to do with pure vision. Instead, research is moving towards using tactile sensors in combination with visual policies. But what’s the best way to learn how to handle multi-point contact?
Zhengtong Xu and Yeping Wang tell us about their new work Contact-Grounded Policy (CGP). CGP predicts future robot state and tactile feedback, and predicts this into actions for a compliant robot controller so that a four- or five-finger robot hand can perform complex tasks involving precise manipulation, delicate grasping, and tool use.
To learn more, watch Episode #89 of RoboPapers, with Chris Paxton and Jiafei Duan.
Abstract
Contact-rich dexterous manipulation with multi-finger hands remains an open challenge in robotics because task success depends on multi-point contacts that continuously evolve and are highly sensitive to object geometry, frictional transitions, and slip. Recently, tactile-informed manipulation policies have shown promise. However, most use tactile signals as additional observations rather than modeling contact state or how their action outputs interact with low-level controller dynamics. We present Contact-Grounded Policy (CGP), a visuotactile policy that grounds multi-point contacts by predicting coupled trajectories of actual robot state and tactile feedback, and using a learned contact-consistency mapping to convert these predictions into executable target robot states for a compliance controller. CGP consists of two components: (i) a conditional diffusion model that forecasts future robot state and tactile feedback in a compressed latent space, and (ii) a learned contact-consistency mapping that converts the predicted robot state-tactile pair into executable targets for a compliance controller, enabling it to realize the intended contacts. We evaluate CGP using a physical four-finger Allegro V5 hand with Digit360 fingertip tactile sensors, and a simulated five-finger Tesollo DG-5F hand with dense whole-hand tactile arrays. Across a range of dexterous tasks including in-hand manipulation, delicate grasping, and tool use, CGP outperforms visuomotor and visuotactile diffusion-policy baselines.
