深層Expected Sarsaと非線形時間差分学習を用いた収束型NMPCベース強化学習
Convergent NMPC-based Reinforcement Learning Using Deep Expected Sarsa and Nonlinear Temporal Difference Learning
NMPCの最適重みを強化学習で学習する手法を提案し、ニューラルネットで行動価値関数を近似して計算負荷を半減、さらに勾配時間差分法でパラメータの発散を防ぎ安定収束を実現した。
著者: Amine Salaje, Thomas Chevet, Nicolas Langlois
分類: eess.SY, cs.RO, cs.SY
原文アブストラクト
In this paper, we present a learning-based nonlinear model predictive controller (NMPC) using an original reinforcement learning (RL) method to learn the optimal weights of the NMPC scheme, for which two methods are proposed. Firstly, the controller is used as the current action-value function of a deep Expected Sarsa where the subsequent action-value function, usually obtained with a secondary NMPC, is approximated with a neural network (NN). With respect to existing methods, we add to the NN's input the current value of the NMPC's learned parameters so that the network is able to approximate the action-value function and stabilize the learning performance. Additionally, with the use of the NN, the real-time computational burden is approximately halved without affecting the closed-loop performance. Secondly, we combine gradient temporal difference methods with a parametrized NMPC as a function approximator of the Expected Sarsa RL method to overcome the potential parameters' divergence and instability issues when nonlinearities are present in the function approximation. The simulation results show that the proposed approach converges to a locally optimal solution without instability problems.