These robots learned tennis and boxing after observing people

If Boston Dynamics-style large, bipedal robots ever do end up replacing humans in workspaces, they may do so after being trained by their flesh-and-bone predecessors. In a new paper, researchers from Stanford explain how they taught a 5 ‘9 humanoid-looking robot equipped with a single RGB camera to play piano, tennis, and even learn to box simply by imitating or “shadowing” human movements. The novel learning method could help speed up training time and reduce costs associated with future humanoid robot development. That could prove useful, especially as companies like Figure and Tesla sprint to release bi-pedal robots capable of performing factory and home work.  

Introduce HumanPlus – Shadowing part

Humanoids are born for using human data. We build a real-time shadowing system using a single RGB camera and a whole-body policy for cloning human motion. Examples:
– boxing🥊
– playing the piano🎹/ping pong
– tossing
– typing

Open-sourced! pic.twitter.com/DQgVDPiNnS

— Zipeng Fu (@zipengfu) June
13, 2024

How are these mimicking robots different? 

Robots shaped like people have existed in some form for decades but they often struggle to replicate the same fluid motions many humans learn naturally. Training robots to perform movements that can seem relatively simple to humans involves large swathes of often complex, and multi-faceted human motion training data. In the past, robotics researchers have tried to decouple different elements of that data—such as data related to visual perception or control for arms and legs—but researchers say that approach is time consuming and not suited well for scaling up. 

The Stanford researchers took a different approach. They first used a reinforcement AI model to train a custom designed robot, called “HumanPlus” on 40 hours of various human motion data. They could then take the base lessons learned by training the robot on that data in a simulation environment and then apply it in the physical world. Armed with that knowledge and a webcam attached to its head, the robot was able to “follow” a human operator’s body and hand movements and eventually mimic them. This process, referred to as “shadowing” resulted in the humanoid robots replicating human movements more naturally.

“By mimicking humans, humanoids can potentially tap into the rich repertoire of skills and motion exhibited by humans, offering a promising avenue towards achieving general robot intelligence,” the authors write. 

The various tasks and movements the robot was asked to mimic ran the gamut of human motion. In one example, the robot was tasked with putting on a shoe and walking, which tested both its hand dexterity and overall locomotion. Other tasks like playing ping pong or learning to throw a solid left jab, meanwhile, placed more of an emphasis on visual perception and timing. Another activity, which involved the robot using a keyboard to type the coding phrase “Hello Word” demonstrated more precise finger movements. Once fully trained, the researchers claim HumanPlus was successful in its movement 60-100% of the time, depending on the task. 

Introduce HumanPlus – Autonomous Skills part

Humanoids are born for using human data. Imitating humans, our humanoid learns:
– fold sweatshirts
– unload objects from warehouse racks
– diverse locomotion skills (squatting, jumping, standing)
– greet another robot

Open-sourced! pic.twitter.com/jFzfES6mMf

— Zipeng Fu (@zipengfu) June
13, 2024

Physically, HumanPlus is a Frakenstein’s monster of various robot parts. The researchers used a Unitree’s Robotics H1 robot as the base body but then attached mechanical hands and wrists from the companies Inspire-Robots and Robotis. A simple Razer webcam attached to the robot’s eyes served as its main way to view the world around it. All told, the final price tag for the robot flocked in at roughly $107,945. Anyone with access to that kind of dough can learn to build their own HumanPlus robot by following instructions published by the researchers on this GitHub repository.

Which hardware platform should HumanPlus be embodied on?

We build our own 33-DoF humanoid with two dexterous hands using components:
– Inspire-Robots RH56DFX hands
– @UnitreeRobotics
H1 robot
– @ROBOTIS
Dynamixel motors
– @Razer
webcams

We open-source our hardware design. pic.twitter.com/AkY9MPEzyd

— Zipeng Fu (@zipengfu) June
13, 2024

Mimicry learning could make commercial robots more adaptable 

The researcher’s more fluid training methods come amid a surge of commercial interest in humanoid robots. Figure and Agility Robots, two leading names in the space, have already begun testing their products in automobile and logistics manufacturing facilities. Tesla, who’s Optimus robot has evolved from a man in a body suit to a real-world machine capable of fondling eggs, envisions a reality where these walking talking robots someday wash dishes and perform other household chores. Though still (very) nascent, all that movement could result in a sizable humanoid robot industry. A 2022 report from Goldman Sachs predicts the global humanoid robot market could reach $154 billion by 2035.

It’s not hard to imagine how a robot capable of learning through mimicry could be useful in these commercial settings. Like new human workers training on a job, managers or trained robot operators could teach humanoid robots how to perform specific tasks specific to one particular industry of business. And unlike other pre-programmed robots, these more arable machines could similarly be updated to shadow new tasks. This more organic approach to movement learning could also potentially increase the functionality of a growing segment of accessibility focused robots  aimed at improving the lives of people with disabilities.