EgoGeneralist：動画ネイティブモデリングで一人称行動動画タスクを統合
EgoGeneralist: Unifying Egocentric Action-Video Tasks through Video-Native Modeling
事前学習済み動画生成モデルを適応させ、一人称操作における行動条件付き動画生成・姿勢推定・行動動画同時生成を統合する汎用フレームワークを提案した。
著者: Yanlin Jin, Bowen Yang, Cheng Zhang, Yifan Yang, Zhe Li, Jianfei Cai, Kai Zhu
分類: cs.CV
原文アブストラクト
Modern video models possess strong visual priors, yet do not natively represent human action in metric 3D space. We present EgoGeneralist, which adapts pretrained video generators to unify action-conditioned video generation, pose estimation, and joint action-video generation for egocentric manipulation. Our role-adaptive action renderer provides video-native representations for multidirectional action-video learning, making metric motion usable as both a spatial control signal and a recoverable generation target. Cross-stream action-video coupling establishes close correspondence between metric motion and visual dynamics through lightweight adaptation while preserving the pretrained visual prior. Together, these designs establish a generalist framework spanning six action-video tasks. With only narrow-domain data, EgoGeneralist generalizes to diverse unseen scenes and interactions with effective action control and visual quality, bridging pretrained visual knowledge and metric 3D human motion.