日常の人間ビデオを用いたロボット操作ポリシーの共学習で重要な要素は何か？
What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?
ロボット操作ポリシーの共学習に日常の人間ビデオを用いる際、手のポーズの品質とネットワークの専門化が重要であることを示し、低ロボットデータ環境で成功率を29.7%向上させるレシピを提案した。
著者: Richard Li, Aditya Prakash, Andrew Wen, Saurabh Gupta, Yilun Du, Pulkit Agrawal
分類: cs.RO, cs.AI, cs.CV, cs.LG
原文アブストラクト
Human video datasets used for cotraining robot manipulation policies largely consist of curated demonstrations where motions are orchestrated to resemble robot behavior and 3D hand poses are captured with specialized hardware. A more plentiful source of data is everyday Internet video, but it is an open question what factors enable transfer from such videos to robots. We investigate this using a new dataset of 532 human videos with 28 hours of high-quality triangulated hand labels and natural motions. We find that hand pose quality affects transfer, but even with accurate hands, the inherent motion gap hinders transfer unless the vision and policy networks specialize to each embodiment. Our cotraining recipe yields consistent improvements, with an absolute success rate gain of $29.7\%$ in the low-robot-data regime across six manipulation tasks.