管理された自律性としての知能：エージェント型AIシステムにおける失敗、エスカレーション、ガバナンス
Intelligence as Managed Autonomy: Failure, Escalation, and Governance for Agentic AI Systems
エージェント型AIの自律性が高まる中、不確実性が増した際に動作を継続する「無制限な自律性」の脆弱性を指摘し、エピステミックな逸脱を検出して推論を中断・回復・制御委譲する「管理された自律性」理論を提案。SMARtモデルという4層フレームワークをペトリネットで形式化し、安全性とガバナンス到達可能性を理論的に保証する。
著者: Srini Ramaswamy
分類: cs.AI, cs.CY, cs.ET, cs.MA, eess.SY
原文アブストラクト
As autonomous and agentic AI systems scale in robotic and human-machine environments, managing hallucination and persistent but unjustified action remains an open challenge. Rather than attributing these failures solely to model or alignment limitations, this paper explores the architectural vulnerability of unbounded autonomy - the presumption that an agent should continue operating regardless of rising uncertainty. It introduces a theory of managed autonomy that defines intelligent behavior through the formal capacity to detect epistemic drift, suspend reasoning, attempt recovery, and ultimately surrender control when reliability diminishes. We instantiate this theory via the SMARt (Self-Managing Multi-tier Autonomous Reasoning with Regulated/Revoked transitions) model, a four-layer framework featuring Stable, Meta-cognitive, Assisted, and Regulated states. By developing a timed, guarded Petri net formulation, we establish theoretically bounded properties for the system, demonstrating how architecture can formally mandate escalation, constrain invalid outputs, and ensure governance reachability under specified conditions. We further analyze how incorporating domain-specific trigger sets across varied operational settings (e.g., healthcare, robotics, etc.) can systematically preserve safety, assuming completeness and soundness criteria are met. Because these triggers are designed to be adaptive, the SMARt model accommodates the safe, controlled expansion of an agent's operational scope over time. We conclude that formalizing failure management within the autonomy lifecycle is a crucial step toward realizing reliable and governed artificial intelligence.