TartanMatch: モダリティを超えた普遍的密対応の実現
TartanMatch: Towards Universal Dense Correspondence Across Modalities
RGB・イベント・熱・深度・LiDARの25通りのモダリティペア間で密対応を取る単一モデルを提案し、ゼロショットベンチマークで大幅な精度向上を達成した。
著者: Hyeokjoon Kwon, Jiting Cai, Ruogu Li, Kritan Bhandari, Geethika Hemkumar, Parv Maheshwari, Yuheng Qiu, Yuchen Zhang, Sebastian Scherer, Wenshan Wang
分類: cs.CV, cs.LG, cs.RO
原文アブストラクト
Robotic platforms combine multi-modal sensors to operate across diverse conditions, but most dense correspondence methods are limited to RGB-RGB or a small set of modality pairs. Training a model for each modality pair scales poorly with the number of sensors and forgoes potential positive transfer across modality pairs. We introduce TartanMatch, a unified model for dense correspondence across all 25 ordered pairs among RGB, event, thermal, depth, and projected LiDAR. TartanMatch adapts an RGB-pretrained correspondence model to all modality pairs within one shared network using lightweight modality adaptation and a mixture of real and synthetic data. On zero-shot real-world benchmarks, TartanMatch reduces Endpoint Error (EPE) over the strongest baselines by 61.9% for cross-modal matching and 49.6% for same-modal matching. We further show that joint training across all modality pairs improves average matching accuracy over pair-specific training, especially on challenging cross-modal pairs. These results demonstrate that a single model achieves broad multi-modal coverage and state-of-the-art matching accuracy.