強化学習のための因子分解スペクトル表現
Factorized Spectral Representations for Reinforcement Learning
遷移カーネルを3モードテンソルとみなし、CP分解とノイズ対比学習で状態・行動・次状態のエンコーダを学習する手法FaStRを提案。表現学習に必要なサンプル数を削減し、高次元の移動タスクで効果を発揮する。
分類: cs.LG
原文アブストラクト
Learning a compact model of the world from interaction data is central to sample-efficient deep reinforcement learning. Spectral representation methods have become the leading paradigm for representation learning in continuous control by taking a matrix view of the transition kernel, with state-action pairs on one side and next states on the other, and learning a low-rank factorization through self-supervised contrastive objectives. We take this view one step further. The transition kernel is naturally a three-mode tensor over states, actions, and next states, and a CP decomposition gives one feature map per mode. We propose FaStR, which fits this decomposition with a noise contrastive objective, producing separate state, action, and next-state encoders that together form a single spectral representation. The factored form yields a smaller hypothesis class, and the sample size needed for representation learning shrinks by a factor that scales with the smaller of the state and action dimensions. Empirically, FaStR delivers its largest gains on high-dimensional locomotion tasks whose dynamics align with the factored structure, and the learned state encoder transfers intact across actuator shift while only the action encoder is retrained.