人間の物理的インタラクションから視触覚ワールドモデルを学習するACTS
ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
手持ちのGelSightセンサで人間が物体を操作した視触覚データを収集し、触覚センサ位置の動作と力変化を共通行動空間として、シーン・手首・触覚の未来を予測する視触覚ワールドモデルを学習した論文。
著者: Yuxiang Ma, Weihan Xu, Pengfei Ye, Anna Mei, Yuxin Ray Song, Tingle Li, Haonan Chen, Edward H. Adelson
分類: cs.RO
原文アブストラクト
Abstract Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React. Without fine-tuning, in the robot teleoperation rollouts we inspected, predicted tactile imprints still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions exceed frame copying on PushT, are mixed on Rope, and fall below it on Motherboard, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot.