Intrinsic uses NVIDIA foundation models to improve robotic grasping

Listen to this article

Intrinsic, a software and AI robotics company that spun out of Alphabet, now integrates with NVIDIA AI and Isaac platform technologies in hopes of advancing the state of autonomous robotic manipulation. Under the collaboration, NVIDIA and Intrinsic plan to bring state-of-the-art dexterity and modular AI capabilities for robotic arms, with a robust collection of foundation models and GPU-accelerated libraries to accelerate a greater number of new robotics tasks.

NVIDIA unveiled Isaac Manipulator in March. It is a collection of foundation models and modular GPU-accelerated libraries that help industrial automation companies build scalable and repeatable workflows for dynamic manipulation tasks by accelerating AI model training and task reprogramming.
NVIDIA claims Isaac Manipulator can accelerate path planning by up to 80x.

Foundation models are based on a transformer deep learning architecture that allows a neural network to learn by tracking relationships in data. They’re generally trained on huge datasets and can be used to process and understand sensor and robot information as magically as ChatGPT for text. This enables robot perception and decision-making like never before and provides zero-shot learning — the ability to perform tasks without prior examples.

NVIDIA recently introduced a foundation model for humanoids called Project GROOT. GR00T stands for “Generalist Robot 00 Technology,” and with the race for humanoid robotics heating up, this new technology is intended to help accelerate development. GR00T is a large multimodal model (LMM) providing robotics developers with a generative AI platform to begin the implementation of large language models (LLMs). Other robotics companies building foundation models include Covariant for its industrial picking robots and Electric Sheep for its outdoor landscaping robots.

“For the broader industry, our work with NVIDIA shows how foundation models can have a profound impact, including making today’s processing challenges easier to manage at scale, creating previously infeasible applications, reducing development costs, and increasing flexibility for end users,” said Wendy Tan White, CEO at Intrinsic, who will be discussing the new partnership during Automate 2024.

Grasping demo of sheet metal parts

Grasping has been a long sought after robotics skill. However, Intrinsic and NVIDIA said so far it’s been time-consuming, expensive to program and difficult to scale. As a result, many repetitive pick-and-place conditions haven’t been seamlessly handled to date by robots.
Simulation is changing that. Enlisting NVIDIA Isaac Sim on the NVIDIA Omniverse platform, Intrinsic generated synthetic data for vacuum grasping using computer-aided design models of sheet metal and suction grippers. This allowed Intrinsic to create a prototype for its customer Trumpf Machine Tools, a leading maker of industrial machine tools.
The prototype uses Intrinsic Flowstate, a developer environment for AI-based robotics solutions, for visualizing processes, associated perception and motion planning. With a workflow that includes Isaac Manipulator, one can generate grasp poses and CUDA-accelerated robot motions, which can first be evaluated in simulation with Isaac Sim — a cost-saving step — before deployment in the real world with the Intrinsic platform. You can watch a video of the demo in the video above.
Intrinsic first talked about Flowstate in May 2023. Flowstate is starting with support for industrial robots as this class of robotics operates in semi-structured settings, which simplifies many of the system variables. The product roadmap is to build a set of software skills that can be extended to other classes of robots.
In December 2022, Intrinsic acquired the Open Source Robotics Corporation (OSRC), the for-profit arm of the Open Source Robotics Foundation, which is the developer of the Robot Operating System (ROS).