D2C-HRHR: 高リスク高リターンタスクのための離散行動と二重分布クリティック
D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks
高リスク高リターンなタスクでは行動分布が多峰的で報酬が確率的になるため、連続行動を離散化しエントロピー正則化と二重クリティックで価値分布を推定する強化学習フレームワークを提案した。
著者: Jundong Zhang, Yuhui Situ, Fanji Zhang, Rongji Deng, Tianqi Wei
分類: cs.LG, cs.AI
原文アブストラクト
Tasks involving high-risk-high-return (HRHR) actions, such as obstacle crossing, often exhibit multimodal action distributions and stochastic returns. Most reinforcement learning (RL) methods assume unimodal Gaussian policies and rely on scalar-valued critics, which limits their effectiveness in HRHR settings. We formally define HRHR tasks and theoretically show that Gaussian policies cannot guarantee convergence to the optimal solution. To address this, we propose a reinforcement learning framework that (i) discretizes continuous action spaces to approximate multimodal distributions, (ii) employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and (iii) introduces a dual-critic architecture for more accurate discrete value distribution estimation. The framework scales to high-dimensional action spaces, supporting complex control domains. Experiments on locomotion and manipulation benchmarks with high risks of failure demonstrate that our method outperforms baselines, underscoring the importance of explicitly modeling multimodality and risk in RL.