MEGA Hub

Reinforcement Learning Enables Scaling Generalist Robot Policy Improvement

Authors

Do you know Michal Korniak?You can claim authorship or link another user.Do you know Marco Bagatella?You can claim authorship or link another user.Do you know Ralf Römer?You can claim authorship or link another user.Do you know Michal Nauman?You can claim authorship or link another user.Do you know Yarden As?You can claim authorship or link another user.Do you know Pranav Vignesh?You can claim authorship or link another user.Do you know Manan Agarwal?You can claim authorship or link another user.Do you know Leander Diaz-Bone?You can claim authorship or link another user.Do you know Piotr Zalewski?You can claim authorship or link another user.Do you know Mikołaj Czarnecki?You can claim authorship or link another user.Do you know Lenart Treven?You can claim authorship or link another user.Do you know Shashwat Saxena?You can claim authorship or link another user.Do you know Sreyas Venkataraman?You can claim authorship or link another user.Do you know Sarvesh Patil?You can claim authorship or link another user.Do you know Zhiyuan Zhou?You can claim authorship or link another user.Do you know Daniel Marta?You can claim authorship or link another user.Do you know Oier Mees?You can claim authorship or link another user.Do you know Nur Muhammad Mahi Shafiullah?You can claim authorship or link another user.Do you know Carmelo Sferrazza?You can claim authorship or link another user.Do you know Bhavya Sukhija?You can claim authorship or link another user.Do you know Angela P. Schoellig?You can claim authorship or link another user.Do you know Andreas Krause?You can claim authorship or link another user.Do you know Max Simchowitz?You can claim authorship or link another user.Do you know Andrew Wagenmaker?You can claim authorship or link another user.

Abstract

The introduction of generalist robot policies has enabled a single, large model to learn to perform a variety of tasks from a vast pre-training corpus. While these models often exhibit powerful zero-shot generalization capabilities, they typically require an adaptation phase to achieve sufficient performance in deployment. Iterated self-improvement via reinforcement learning (RL) has demonstrated promise in enabling this adaptation, yet such approaches have been largely explored in single-task settings - where the goal is to simply improve on a narrow deployment task - which lies in stark contrast to the multi-task capabilities of the base policy. In this work we ask whether generalist robot policies are capable of multi-task, generalist improvement in deployment - targeting many diverse tasks simultaneously - and what algorithmic recipe enables this. To answer this question, we first develop a set of structured evaluations to test generalist improvement capabilities, and systematically evaluate algorithmic and practical design choices regarding value estimation, policy extraction and resource allocation. We show that not only can generalist policies be improved online across diverse tasks, but that the right recipe can enable generalization and increase performance on tasks not considered during online training. We characterize when this generalization occurs and show that both the training task distribution and the use of value-based RL are critical. To the best of our knowledge, this is among the first evidence for online generalist improvement for robot foundation models.

Community

00