1つのポリシーで多様なスタイルを：操縦可能なヒューマノイド制御のための選好集約の最適な場所
One Policy, Many Styles: Where to Aggregate Preferences for Steerable Humanoid Control?
ヒューマノイド制御において、選好条件付き強化学習の選好集約を報酬・アドバンテージ・損失・PCGradの各レベルで比較し、応答性と制御精度のトレードオフを明らかにした。
著者: Ziqian Huang, Yuzhuo Chen, Shimin Zhang, Junyang He, Xinyue Annie Yang, Animesh Garg
分類: cs.RO, cs.LG
原文アブストラクト
Modern robotic agents are increasingly expected to support steerable control, where a single policy can adjust its behavioral style at runtime under changing user preferences. Preference-conditioned reinforcement learning provides this interface, but conditioning alone does not ensure that a requested behavior change is strong, does not leak into unrelated behaviors, and does not compromise task performance. Our evaluation combines Preference Reward Responsiveness (PRR) with coverage and utility metrics. We use these measures to compare reward-level, advantage-level, loss-level, and PCGrad-based aggregation across two humanoid-control frameworks. No aggregation level performs best across all environments and criteria. Experiments show that aggregating objectives early produces a wider range of behavioral responses but greater unintended changes in other objectives, while aggregating them later improves control precision at some cost to responsiveness.