Intrinsic uses NVIDIA foundation models to improve robotic grasping

Companies unveil collaboration to enhance robotic grasping capabilities using NVIDIA AI, the Isaac Manipulator and Intrinsic Flowstate.

The post Intrinsic uses NVIDIA foundation models to improve robotic grasping appeared first on The Robot Report.

Intrinsic, a software and AI robotics company that spun out of Alphabet, now integrates with NVIDIA AI and Isaac platform technologies in hopes of advancing the state of autonomous robotic manipulation. Under the collaboration, NVIDIA and Intrinsic plan to bring state-of-the-art dexterity and modular AI capabilities for robotic arms, with a robust collection of foundation models and GPU-accelerated libraries to accelerate a greater number of new robotics tasks.

NVIDIA unveiled Isaac Manipulator in March. It is a collection of foundation models and modular GPU-accelerated libraries that help industrial automation companies build scalable and repeatable workflows for dynamic manipulation tasks by accelerating AI model training and task reprogramming.
NVIDIA claims Isaac Manipulator can accelerate path planning by up to 80x.

Foundation models are based on a transformer deep learning architecture that allows a neural network to learn by tracking relationships in data. They’re generally trained on huge datasets and can be used to process and understand sensor and robot information as magically as ChatGPT for text. This enables robot perception and decision-making like never before and provides zero-shot learning — the ability to perform tasks without prior examples.

NVIDIA recently introduced a foundation model for humanoids called Project GROOT. GR00T stands for “Generalist Robot 00 Technology,” and with the race for humanoid robotics heating up, this new technology is intended to help accelerate development. GR00T is a large multimodal model (LMM) providing robotics developers with a generative AI platform to begin the implementation of large language models (LLMs). Other robotics companies building foundation models include Covariant for its industrial picking robots and Electric Sheep for its outdoor landscaping robots.

“For the broader industry, our work with NVIDIA shows how foundation models can have a profound impact, including making today’s processing challenges easier to manage at scale, creating previously infeasible applications, reducing development costs, and increasing flexibility for end users,” said Wendy Tan White, CEO at Intrinsic, who will be discussing the new partnership during Automate 2024.

Grasping demo of sheet metal parts

Grasping has been a long sought after robotics skill. However, Intrinsic and NVIDIA said so far it’s been time-consuming, expensive to program and difficult to scale. As a result, many repetitive pick-and-place conditions haven’t been seamlessly handled to date by robots.

Simulation is changing that. Enlisting NVIDIA Isaac Sim on the NVIDIA Omniverse platform, Intrinsic generated synthetic data for vacuum grasping using computer-aided design models of sheet metal and suction grippers. This allowed Intrinsic to create a prototype for its customer Trumpf Machine Tools, a leading maker of industrial machine tools.

The prototype uses Intrinsic Flowstate, a developer environment for AI-based robotics solutions, for visualizing processes, associated perception and motion planning. With a workflow that includes Isaac Manipulator, one can generate grasp poses and CUDA-accelerated robot motions, which can first be evaluated in simulation with Isaac Sim — a cost-saving step — before deployment in the real world with the Intrinsic platform. You can watch a video of the demo in the video above.

“With the latest AI foundation models, companies can program a variety of robot configurations that are able to generalize and interact with diverse objects inside real-world environments,” said Deepu Talla, vice president of robotics and edge computing, NVIDIA. “As the collaboration between Intrinsic and NVIDIA deepens we will be able to help companies scale and automate their industrial manufacturing operations.”

Intrinsic first talked about Flowstate in May 2023. Flowstate is starting with support for industrial robots as this class of robotics operates in semi-structured settings, which simplifies many of the system variables. The product roadmap is to build a set of software skills that can be extended to other classes of robots.

In December 2022, Intrinsic acquired the Open Source Robotics Corporation (OSRC), the for-profit arm of the Open Source Robotics Foundation, which is the developer of the Robot Operating System (ROS).

The post Intrinsic uses NVIDIA foundation models to improve robotic grasping appeared first on The Robot Report.