ViSMoE: 身体化された指示表現グラウンディングのための視覚認識スパース混合エキスパート
ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Grounding
エージェントが自然言語指示に基づいて遠隔の物体を特定するタスクにおいて、視覚情報の種類に応じて専門化したスパース混合エキスパートを用いることで、ビューと物体の識別表現を改善し、既存手法を上回る性能を達成した。
分類: cs.CV
原文アブストラクト
Embodied Referring Expression Grounding is the task of enabling an agent to navigate in real environments and to localize a remote object based on natural language instructions. In this scenario, the agent needs to select one view for navigation at each step and identify a specific object among all candidate objects at the destination. However, most of the previous approaches fail to distinguish between views and objects, instead processing them using the vanilla vision encoder, which results in ambiguous representations of both views and objects. To address the above issues, we propose ViSMoE, which equips sparse Mixture-of-Experts with a visual-aware routing policy for the embodied agent. This framework processes different types of visual information specifically, resulting in discriminative visual representations for both views and objects. Experimental results on REVERIE and SOON datasets demonstrate that ViSMoE outperforms the previous state-of-the-art methods, showing the superiority of our proposed method.