MEGA Hub

BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

Authors

Do you know Fei Lin?You can claim authorship or link another user.Do you know Tengchao Zhang?You can claim authorship or link another user.Do you know Ziyang Gong?You can claim authorship or link another user.Do you know Xiaotong Yu?You can claim authorship or link another user.Do you know Bohan Zhang?You can claim authorship or link another user.Do you know Yifan Zhou?You can claim authorship or link another user.Do you know Qihao Yang?You can claim authorship or link another user.Do you know Ji Dai?You can claim authorship or link another user.Do you know Dong Li?You can claim authorship or link another user.Do you know Yining Jiang?You can claim authorship or link another user.Do you know Qinghua Ni?You can claim authorship or link another user.Do you know Jun Huang?You can claim authorship or link another user.Do you know Qiang Zhang?You can claim authorship or link another user.Do you know Yue Zhou?You can claim authorship or link another user.Do you know Zhihang Zhong?You can claim authorship or link another user.Do you know Yonglin Tian?You can claim authorship or link another user.Do you know Fei-Yue Wang?You can claim authorship or link another user.Do you know Xue Yang?You can claim authorship or link another user.

Abstract

Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.

Community

00

Publication notes

Author note
Accepted by NeurIPS 2026 (Track on Evaluations and Datasets, Spotlight)