StreamEgo: 三人称から一人称へのストリーミング動画生成
StreamEgo: Streaming Exocentric-to-Egocentric Video Generation
三人称視点の動画を一人称視点へリアルタイム変換するフレームワーク。二重分岐事前分布と領域認識型学習で操作領域の品質と汎化性を高め、単一GPUで24.3FPSのストリーミング生成を実現した。
詳しい要約
1. どんなもの?
2. 先行研究と比べてどこがすごい?
3. 技術・手法の肝は?
4. どうやって有効だと検証した?
5. 議論はある?
6. 次に読むべき論文は?
※ AIが要旨から生成した要約です。正確性は原文をご確認ください。
著者: Tianyi Xu, Boyang Lyu, Yunkai Tang, Kairan Ding, Jiahang Li, Weiyu Guo, Yandong Guo, Shuchen Weng, Boxin Shi
分類: cs.CV
原文アブストラクト
Exocentric-to-egocentric (Exo2Ego) video generation transforms third-person recordings into first-person viewpoints, unlocking immersive AR/VR experiences and scalable demonstration synthesis for embodied robotics. Extending it to a streaming setting is expected to further enable live applications. However, existing methods suffer from degraded manipulation regions, overfitting and confinement to fixed-length clips with prohibitive latency. To address these limitations, we present StreamEgo, a real-time framework for unbounded, generalizable Exo2Ego video generation. To resolve intricate manipulation dynamics while supporting the streaming pipeline, a streaming dual-branch prior provides pixel-aligned guidance for translating appearance and skeletal kinematics. We also introduce a region-aware loss curriculum to improve generalization and stabilize multi-objective diffusion training. Furthermore, we distill the model into a chunk-wise autoregressive model, augmented with a region-aware corruption mechanism and a first-chunk curriculum, enabling high-quality, generalizable real-time streaming Exo2Ego generation at 24.3 FPS on a single H100 GPU.
