見て学ぶ：人間の動画からの操作軌道のスケーラブルな抽出
Learning from Watching: Scalable Extraction of Manipulation Trajectories from Human Videos
人間の操作動画から、大規模基盤モデルと点追跡を組み合わせてタスク関連キーポイントの密な軌道を抽出する手法を提案し、ロボット学習のデータ収集を効率化する。
分類: cs.RO, cs.CV
原文アブストラクト
Collecting high-quality data for training large-scale robotic models typically relies on real robot platforms, which is labor-intensive and costly, whether via teleoperation or scripted demonstrations. To scale data collection, many researchers have turned to leveraging human manipulation videos available online. However, current methods predominantly focus on hand detection or object pose estimation, failing to fully exploit the rich interaction cues embedded in these videos. In this work, we propose a novel approach that combines large foundation models for video understanding with point tracking techniques to extract dense trajectories of all task-relevant keypoints during manipulation. This enables more comprehensive utilization of Internet-scale human demonstration videos. Experimental results demonstrate that our method can accurately track keypoints throughout the entire manipulation process, paving the way for more scalable and data-efficient robot learning.