移動ロボットのための物理・地理空間に基づくシーン解釈
Physico-Geospatial Grounded Scene Interpretation for Mobile Robotics
事前学習済みのVLMの出力に、OpenStreetMapの建物・道路情報とセンサから得た位置・時間・距離情報をLLMで統合し、物理・地理空間に基づく自然言語のシーン記述を生成する手法を提案した。大学構内での実験で建物接地F1=0.83、路面接地F1=0.64を達成した。
著者: Nicolas Schuler, Janik Kurtz, Lea Dewald, Marcel Sauber, Félicia Teferle, Jürgen Graf
分類: cs.CV, cs.RO
原文アブストラクト
Recent advancements in deep learning allow robotic agents to interact with dynamic and unstructured environments. Of special interest is the integration of physico-geospatial world knowledge into such systems, either by using physics-aware machine learning models, knowledge graphs to model relationships or spatio-temporal and logical reasoning. In the present work, we introduce an approach to augment the output of pre-trained, unmodified VLMs used for scene interpretation by integrating semantic descriptions, OpenStreetMap building data and street information with positional, temporal and metric information obtained from our sensory systems, fusing this information using LLMs. We apply this concept to an outdoor recording within a university campus, achieving an F1-Score of 0.83 in the task of grounding buildings and 0.64 for path surface grounding on our pilot evaluation set. The results demonstrate the conceptual capability of the proposed solution to deliver physico-geospatial grounded natural language descriptions. Code and results are available at https://datahub.rz.rptu.de/hstr-csrl-public/publications/physico-geospatial-grounded-scene-interpretation