MEGA Hub

Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

Authors

Do you know Yifei Yao?You can claim authorship or link another user.Do you know Yude Zou?You can claim authorship or link another user.Do you know Guanbo Jiao?You can claim authorship or link another user.Do you know Xing Gao?You can claim authorship or link another user.Do you know Ning Gao?You can claim authorship or link another user.Do you know Yukai Wang?You can claim authorship or link another user.Do you know Jiantong Chen?You can claim authorship or link another user.Do you know Zihou Zhu?You can claim authorship or link another user.Do you know Xinyu Li?You can claim authorship or link another user.Do you know Ningyu Yan?You can claim authorship or link another user.Do you know Shuai Wang?You can claim authorship or link another user.Do you know Jianming Ma?You can claim authorship or link another user.Do you know Haoyu Guo?You can claim authorship or link another user.Do you know Weinan Zhang?You can claim authorship or link another user.Do you know Chunhua Shen?You can claim authorship or link another user.

Abstract

Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated. In this report, we evaluate GPT-6-Astra on EBench as the core of an agent that controls a robot through an agent harness under the same visual and proprioceptive observation protocol as post-trained policies. We study two agent configurations: direct control, in which the agent generates waypoint commands directly, and harness with policy, in which it can also invoke a post-trained VLA or WAM policy for action execution. Under direct control, the agent achieves competitive overall performance relative to post-trained policies. It leads on short-horizon tasks and exhibits the least performance variation across generalization dimensions, whereas it underperforms post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks. In the harness with policy, invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost. However, performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control. Overall, our study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design. Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/

Community

00

Publication notes

Author note
Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/