Genie 4D: 意味的先行知識で導く4D動的シーン再構成
Genie 4D: Semantic-Prior-Guided 4D Dynamic Scene Reconstruction
手持ちスマホの映像から、意味的に接地され行動制御可能な4D世界モデルを構築するフレームワークを提案。リアルタイムの慣性ガウススプラッティングと意味的特徴を先行知識として用いた4Dバックボーンを組み合わせ、動的トラッキング精度と再構成品質を向上させた。
著者: Yiru Yang, Zhuojie Wu, Nishant Kumar Singh, Max Schulthess
分類: cs.CV, cs.AI, cs.RO
原文アブストラクト
At the intersection of computer vision and robotic perception, 4D reconstruction of dynamic scenes connects low-level geometric sensing with high-level semantic understanding. We present Genie 4D, a framework that turns hand-held phone capture into a semantically grounded, action-controllable 4D world model. Genie 4D couples a real-time visual-inertial Gaussian splatting front-end for metric geometry with a feed-forward 4D backbone regularized by frozen DINOv3 features acting as structural priors. The semantic priors suppress identity drift during dynamic tracking, while a short conditional diffusion refiner recovers high-frequency surface detail that regression backbones smooth away. Finally, a lightweight latent-action head exposes the reconstructed 4D state to a Genie-style world model trained with a JEPA-style next-embedding objective, so that the scene can be rolled forward under user actions. On the Point Odyssey and TUM-Dynamics benchmarks, Genie 4D retains the linear time complexity O(T) of feed-forward baselines while improving 3D tracking accuracy (APD) and reconstruction completeness, and it runs interactively on a single consumer GPU (RTX 5090) from iPhone, Mac, Windows, and Linux capture clients. Genie 4D offers a practical, semantic-prior-guided path toward physically grounded world models.