EgoExoMem: 同期した自己中心・他者中心ビデオにわたるクロスビュー記憶推論
EgoExoMem: Cross-View Memory Reasoning over Synchronized Egocentric and Exocentric Videos
自己中心視点と他者中心視点の同期ビデオを用いたクロスビュー記憶推論のベンチマークを新たに構築し、既存のマルチモーダルLLMでは困難であることを示した。また、学習不要のフレーム選択手法E^2-Selectを提案し、ベースラインを上回る性能を達成した。
著者: Ruiping Liu, Junwei Zheng, Yufan Chen, Di Wen, Shaofang Quan, Chengzhi Wu, Jiaming Zhang, Kailun Yang, Kunyu Peng, Rainer Stiefelhagen
分類: cs.CV
原文アブストラクト
Egocentric memory is widely used in embodied intelligence, but it may be insufficient for comprehensive spatial-temporal reasoning. Inspired by human recall from both field and observer perspectives, we introduce EgoExoMem, the first benchmark for cross-view memory reasoning over synchronized egocentric and exocentric videos. EgoExoMem contains $2.6K$ high-quality MCQs across eight temporal, spatial, and cross-view QA types. To support dual-view retrieval, we propose E$^2$-Select, a training-free frame selection method for synchronized ego-exo videos. It combines relevance-based budget allocation with per-view k-DPP sampling to handle view asymmetry and cross-view temporal consistency. Experiments show that ego and exo views provide complementary memory cues, while existing MLLMs remain far from solving the benchmark: the best model reaches only $55.3\%$. E$^2$-Select achieves state-of-the-art performance of $58.2\%$ over frame-selection and RAG-based memory baselines. Further analysis reveals systematic view-preference conflicts between question framing and answer grounding, underscoring the novelty and challenge of cross-view memory reasoning.