匿名クラウドフィードバックから操縦可能な多元的方針整合
Steerable Pluralistic Policy Alignment from Anonymized Crowd Feedback
注釈者IDを必要とせず、異種の群衆フィードバックからPCA分解で選好方向の共有基底と軌道埋め込みを同時学習し、テスト時に個別ユーザーへ迅速にパーソナライズ整合する枠組みP3Lを提案。
著者: Akansha Kalra, Ellen R. Novoseller, Vernon J. Lawhern, Daniel S. Brown
分類: cs.LG, cs.AI, cs.RO
原文アブストラクト
Pluralistic alignment requires AI systems to learn from diverse populations whose preferences may reflect heterogeneous or even conflicting objectives, and to personalize their behavior to individual users. Existing approaches in pluralistic alignment often assume access to annotator identities and rely on powerful pretrained representations to model individual preference variations. We propose Preference-PCA Pluralistic Learning (P3L), a novel framework for pluralistic alignment that jointly learns trajectory embeddings and the shared basis of preference directions via PCA decomposition from heterogeneous crowd feedback without requiring annotator identities. At test time, our approach enables rapid and accurate personalized reward learning and downstream policy optimization. Experiments across multiple embodied control tasks demonstrate that P3L enables steerable personal alignment that generalizes beyond the offline dataset to out-of-distribution new users and attains competitive RL and test performance, while substantially reducing computational cost and supporting variable adaptation query budgets.