NeoWorld-Pro: 単眼画像から対話型シーンをプログラミングし具現化シミュレーションを実現
NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation
単眼RGB画像を、物理特性や関節構造を持つ対話型3Dシーンとして再構成するフレームワークを提案。MLLMによるコード生成と物理エンジンでの検証を組み合わせ、安定した積み重ねや細かい操作などの複雑なタスクを可能にした。
著者: Yumeng He, Yichen Song, Xiaotian Yang, Weijia Zhang, Zanwei Zhou, Junru Gong, Xiaokang Yang, Yunbo Wang
分類: cs.CV
原文アブストラクト
The advancement of Embodied AI necessitates high-quality simulation assets that faithfully mirror the real world. However, transforming raw visual observations into simulation-ready scenes remains challenging due to the lack of physical grounding and scene-level interactivity in current image-to-URDF methods. We propose NeoWorld-Pro, a framework that reformulates monocular scene reconstruction as procedural programming for interactive 3D environments. Leveraging the zero-shot reasoning and code synthesis capabilities of MLLMs, NeoWorld-Pro converts a single RGB image into executable programs specifying object geometry, articulation, and physical properties. A physics-in-the-loop mechanism then iteratively refines the generated programs by validating their execution in a physics engine, enforcing physically plausible articulations, valid object compositions and interactions, and accurate spatial relationships. Experiments show that NeoWorld-Pro outperforms open-loop and prior monocular reconstruction methods, while enabling complex downstream tasks such as stable stacking and fine-grained manipulation.