D2AC：拡散方策と分布型クリティックを融合した強化学習
D2 Actor Critic: Diffusion Actor Meets Distributional Critic
拡散モデルベースの方策を安定してオンライン学習させるため、分布型RLとクリップ付き二重Q学習を組み合わせた頑健なクリティックを提案し、多数の難タスクで最高性能を達成した。
著者: Lunjun Zhang, Shuo Han, Hanrui Lyu, Bradly C Stadie
分類: cs.LG
原文アブストラクト
We introduce D2AC, a new model-free reinforcement learning (RL) algorithm designed to train expressive diffusion policies online effectively. At its core is a policy improvement objective that avoids the high variance of typical policy gradients and the complexity of backpropagation through time. This stable learning process is critically enabled by our second contribution: a robust distributional critic, which we design through a fusion of distributional RL and clipped double Q-learning. The resulting algorithm is highly effective, achieving state-of-the-art performance on a benchmark of eighteen hard RL tasks, including Humanoid, Dog, and Shadow Hand domains, spanning both dense-reward and goal-conditioned RL scenarios. Beyond standard benchmarks, we also evaluate a biologically motivated predator-prey task to examine the behavioral robustness and generalization capacity of our approach. Code: https://github.com/d2ac-actor-critic/d2ac-public