MEGA Hub

RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

Authors

Do you know Yunqi Liu?You can claim authorship or link another user.Do you know Peiyu Wang?You can claim authorship or link another user.Do you know Bingli Wang?You can claim authorship or link another user.Do you know Leo Luo?You can claim authorship or link another user.Do you know Xuanwu Song?You can claim authorship or link another user.Do you know Xudong Ma?You can claim authorship or link another user.Do you know Hongxin Li?You can claim authorship or link another user.Do you know Yaxin Du?You can claim authorship or link another user.Do you know Weicheng Gu?You can claim authorship or link another user.Do you know Fanglin Xu?You can claim authorship or link another user.Do you know Yi Xing?You can claim authorship or link another user.Do you know XiangLong Liu?You can claim authorship or link another user.Do you know Jujie He?You can claim authorship or link another user.Do you know Zhange Zhang?You can claim authorship or link another user.Do you know Huatong Song?You can claim authorship or link another user.Do you know Zhaoyi Li?You can claim authorship or link another user.Do you know Deyang Kong?You can claim authorship or link another user.Do you know Chuan Hao?You can claim authorship or link another user.Do you know Peihao Wu?You can claim authorship or link another user.Do you know Mingjie Tang?You can claim authorship or link another user.Do you know Ran Tao?You can claim authorship or link another user.Do you know Jian Yang?You can claim authorship or link another user.Do you know Wayne Xin Zhao?You can claim authorship or link another user.Do you know Bryan Dai?You can claim authorship or link another user.

Abstract

Command-line interface (CLI) agents increasingly execute longer-horizon tasks, yet existing benchmarks either assume that user instructions remain fixed throughout execution or evaluate task outcomes without capturing runtime collaboration quality. In this paper, we present REALCLI, a paired benchmark that unifies the evaluation of task execution and human-agent collaboration quality on realistic CLI tasks. To comprehensively measure collaboration, REALCLI builds a five-dimensional framework, operationalized into 22 sub-dimensions and 51 interaction scenario templates. REALCLI provides two modes for paired comparison: REALCLI-EXEC supplies complete specifications upfront, whereas REALCLI-COLLAB introduces a scripted user that delivers progressive information, corrections, constraint revisions, and authorization decisions. Evaluations reveal profound capability gaps in frontier agents: the best configuration attains an average score of only 49.2%, and when evaluated by the joint collaboration score (J_{collab}), the peak performance reaches merely 37.6%. These results show that task effectiveness diverges sharply from collaboration quality, with models consistently struggling in steerability, trust calibration, and safety boundary management. REALCLI provides a rigorous, reproducible foundation for measuring and advancing collaborative CLI agents.

Community

00