階層的組合せ計画のためのマルチタイムスケール抽象化の学習
Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planning
確率的組合せ最適化問題に対して、可変時間の意思決定を扱うSMDP対応の世界モデルと潜在空間ツリー探索を組み合わせたモデルベース階層強化学習手法を提案し、ベンチマークで優位性を示した。
著者: Vivienne Huiling Wang, Tinghuai Wang, Joni Pajarinen
分類: cs.LG
原文アブストラクト
The combination of exponentially large action spaces, stochastic dynamics, and long-horizon decision-making under limited resources makes Sequential Stochastic Combinatorial Optimization (SSCO) particularly challenging for reinforcement learning. Hierarchical Reinforcement Learning (HRL) offers a natural decomposition, but it places the high-level policy in a Semi-Markov Decision Process (SMDP) where actions have variable durations, making it difficult to learn a world model that is suitable for planning. We introduce a model-based hierarchical framework for sequential stochastic combinatorial decision-making that directly addresses this issue. Our method combines a latent-space tree-search planner with an SMDP-aware world model for variable-duration decisions. A multi-timescale objective structures the latent dynamics so that transition magnitudes reflect the effective temporal scales of abstract actions, enabling efficient lookahead under adaptive temporal abstraction. We further learn a subgoal-conditioned budget policy jointly with the world model to support context-aware resource allocation. Across challenging SSCO benchmarks, our method outperforms strong baselines.