Flex-π: 計算柔軟性を備えたマルチストリーム世界行動モデル
Flex-$π$: A Multi-Stream World-Action Model with Compute Flexibility
凍結したビデオ生成VAEが3D点群も符号化できることを利用し、RGBに加えて3D幾何学と物体意味論を共同で学習する世界行動モデルを提案。ストリームのドロップアウトにより、単一のチェックポイントで高速な行動のみから完全な生成まで柔軟に動作し、実世界の両腕操作タスクで高い性能を達成した。
著者: Ge Yan, Jinghao Liu, Yuzhi Fan, Lei Cai, Minwen Liao, Jesse Zhang, Dieter Fox
分類: cs.RO, cs.CV
原文アブストラクト
World-action models (WAMs) predict the future to act better, but nearly all of them predict only RGB latents, trained purely for pixel reconstruction, with no explicit signal for the 3D geometry or object semantics manipulation needs. We find a surprising free lunch: the same frozen video-generation VAE that encodes RGB also encodes 3D pointmaps almost losslessly, with no pointmap-specific training at all. This lets us supervise Flex-$π$, a 6B-parameter WAM, on 3D geometry and object-centric DINO semantics alongside RGB, at no cost in new sensors, new pre-training, or inference latency. Every visual signal is projected into this shared latent space and denoised jointly with actions inside a Mixture-of-Transformers backbone; per-stream dropout with cross-modality forcing then lets a single trained checkpoint run on any subset of these streams, from a fast action-only mode to full joint generation. The result is a policy that is exceptionally demonstration-efficient and generalizes well, beating the strongest baselines by up to 2-7$\times$ on dexterous, precise, real-world bimanual manipulation tasks both in and out of distribution, all while running faster than $π_{0.5}$. Our project website: https://flex-pi.github.io/