A new large-scale simulation platform to train robots on everyday tasks

The performance of artificial intelligence (AI) tools, including large computational models for natural language processing (NLP) and computer vision algorithms, has been rapidly improving over the past decades. One reason for this is that datasets to train these algorithms have exponentially grown, collecting hundreds of thousands of images and texts often collected from the internet.

Training data for robot control and planning algorithms, on the other hand, remains far less abundant, in part because acquiring it is not as straightforward. Some computer scientists have thus been trying to create larger datasets and platforms that could be used to train computational models for a wide range of robotics applications.

In a recent paper, pre-published on the server arXiv and set to be presented at the Robotics: Science and Systems 2024 conference, researchers at the University of Texas at Austin and NVIDIA Research introduced one of these platforms, called RoboCasa.

RoboCasa is a large-scale simulation framework that can be used to train generalist robots to complete various tasks in everyday settings.

“Recent progress in AI has been largely propelled by training large models on massive data sources,” Yuke Zhu, lead author of the paper, told Tech Xplore.

“Inspired by these advancements, we seek to develop foundation models for generally capable robots that can perform various everyday tasks. RoboCasa is designed to supply the high-quality simulation data required for training such robotics foundation models.”

The primary objective of the recent work by Zhu, Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi and Ajay Mandlekar was to develop a new open-source simulation platform that would facilitate the training of robotics algorithms.

Their efforts ultimately led to the development of RoboCasa, which is an extension of RoboSuite, a simulation framework they introduced a few years ago. RoboSuite serves as the simulation infrastructure that the team used to create RoboCasa’s simulated environments.

“We utilized generative AI tools to create diverse object assets, scenes and tasks,” Zhu explained. “These AI tools significantly improved the diversity and realism of the simulated world. In addition, RoboCasa supports various robot hardware platforms and provides large datasets with over 100k trajectories for model training.”

The RoboCasa platform includes thousands of 3D scenes containing over 150 different types of everyday objects and dozens of furniture items and electrical appliances. RoboCasa features highly realistic simulations, which were enriched using generative AI tools.

Zhu and his colleagues designed 100 tasks that robotics algorithms can be trained on and compiled high-quality human demonstrations for these tasks. Their platform also includes methods to generate effective trajectories and motions that would allow robots to complete these tasks.

“Two key findings excited me the most,” Zhu said. “First, we demonstrated a scaling trend: as we increased the size of the (machine-generated) training datasets, the model’s performance grew steadily. Second, by combining the simulation data with real-world data, we found the augmented dataset enhanced the robot’s performance in real-world tasks.”

In initial experiments, the new simulation platform proved to be a valuable resource for generating synthetic training data that can then be used to train imitation learning algorithms. Overall, this study demonstrates that simulation data can be highly effective in training AI models for robotics applications.

In the future, other teams could experiment with RoboCasa, which is open-source and can thus be readily accessed on GitHub. Meanwhile, Zhu and his colleagues plan to continue expanding and improving their platform, to facilitate its widespread use within the robotics community.

“First, we aim to incorporate more advanced generative AI methods to expand our simulations further, capturing the variety and richness of human-centered environments, from homes and factories to offices,” Zhu added.

“Second, we plan to develop better algorithms to harness simulation data for building robotics systems that are more robust and generalizable in the real world.”

© 2024 Science X Network