ランダムネットワーク蒸留を用いた効率的な能動的模倣学習
Efficient Active Imitation Learning with Random Network Distillation
学習済みの状態分布外検出器で専門家への問い合わせを必要な時だけに絞る能動的模倣学習手法RND-DAggerを提案し、ゲームやロボット歩行で問い合わせ回数を削減しつつ性能を向上させた。
著者: Emilien Biré, Anthony Kobanda, Ludovic Denoyer, Rémy Portelas
分類: cs.LG
原文アブストラクト
Developing agents for complex and underspecified tasks, where no clear objective exists, remains challenging but offers many opportunities. This is especially true in video games, where simulated players (bots) need to play realistically, and there is no clear reward to evaluate them. While imitation learning has shown promise in such domains, these methods often fail when agents encounter out-of-distribution scenarios during deployment. Expanding the training dataset is a common solution, but it becomes impractical or costly when relying on human demonstrations. This article addresses active imitation learning, aiming to trigger expert intervention only when necessary, reducing the need for constant expert input along training. We introduce Random Network Distillation DAgger (RND-DAgger), a new active imitation learning method that limits expert querying by using a learned state-based out-of-distribution measure to trigger interventions. This approach avoids frequent expert-agent action comparisons, thus making the expert intervene only when it is useful. We evaluate RND-DAgger against traditional imitation learning and other active approaches in 3D video games (racing and third-person navigation) and in a robotic locomotion task and show that RND-DAgger surpasses previous methods by reducing expert queries. https://sites.google.com/view/rnd-dagger