Time-JEPA：物理時系列で動画ワールドモデルを接地する
Time-JEPA: Grounding Video World Models with Physical Time Series
低次元で時間的に整合した物理時系列を補助監督として動画ワールドモデルを微調整し、物理状態の時間変化を保持させる手法を提案。自動運転・ロボット操作・人体動作で現在状態復号や将来予測が改善。
著者: Shenyi Li, Ziyang Zhang, Wen Wu, Ming Liu, Chao Zhang
分類: cs.CV
原文アブストラクト
Video world models learn how visual environments evolve by predicting missing or future content. However, physical-state information can provide useful context for predicting how environments evolve. Prior work incorporates this information through different mechanisms: domain-specific physical laws can complicate multi-domain training, while supervision on individual frames or frame pairs gives limited attention to how physical state evolves over time. We introduce Time-JEPA, which fine-tunes a video world model using low-dimensional, temporally aligned physical time series as auxiliary supervision. Physical time series connect video representations with physical state at corresponding temporal positions, encouraging the model to preserve how physical state evolves. In multiple domains including autonomous driving, robot manipulation, and human motion, Time-JEPA improves current-state decoding, multi-horizon future-state prediction, and kinematic reconstruction over matched video-fine-tuned controls across both condition and state-variable distribution shifts. Multi-domain training provides aggregate gains over matched single-domain training in all three tasks. External evaluations further suggest that time-series supervision improves sensitivity to physical anomalies and the representation of physical interactions, including in unseen scenarios. Aggregate gains on a second video backbone indicate that the method can be applied to more backbones. These results suggest that physical time series help video world models better understand the physical world.