大規模言語モデルによるロボティクスシステムの障害診断
Diagnosing Robotics Systems Issues with Large Language Models
ロボットシステムの障害原因を特定するため、2500件超の診断ベンチマークSYSDIAGBENCHを構築し、LLMの性能を評価した。QLoRA微調整した7BモデルがGPT-4を上回る診断精度を低コストで達成した。
著者: Jordis Emilia Herrmann, Aswath Mandakath Gopinath, Mikael Norrlof, Mark Niklas Müller
分類: cs.CL, cs.AI, cs.LG, cs.RO
原文アブストラクト
Quickly resolving issues reported in industrial applications is crucial to minimize economic impact. However, the required data analysis makes diagnosing the underlying root causes a challenging and time-consuming task, even for experts. In contrast, large language models (LLMs) excel at analyzing large amounts of data. Indeed, prior work in AI-Ops demonstrates their effectiveness in analyzing IT systems. Here, we extend this work to the challenging and largely unexplored domain of robotics systems. To this end, we create SYSDIAGBENCH, a proprietary system diagnostics benchmark for robotics, containing over 2500 reported issues. We leverage SYSDIAGBENCH to investigate the performance of LLMs for root cause analysis, considering a range of model sizes and adaptation techniques. Our results show that QLoRA finetuning can be sufficient to let a 7B-parameter model outperform GPT-4 in terms of diagnostic accuracy while being significantly more cost-effective. We validate our LLM-as-a-judge results with a human expert study and find that our best model achieves similar approval ratings as our reference labels.