Students test their robotic interface, living with a man with quadriplegia

PITTSBURGH — It started as a theoretical class project to improve the use of robots in a health care setting. It ended in a trip to California for two Carnegie Mellon students, who lived with a man with quadriplegia for a week as they tested and perfected their robotic interface.

Akhil Padmanabha, a third-year robotics Ph.D. student, and Janavi Gupta, a sophomore computer science major, began working on the technology in mid-2022 and early 2023, respectively. The project, called Head-Worn Assisted Teleoperation, or HAT, allows users to control a robot using the tilt of their head or voice recognition, rather than strictly handheld controls.

In California, in the Los Altos Hills home of Henry and Jane Evans, they watched the interface come to life. Evans, who lost the ability to move his limbs and speak after a stroke nearly 20 years ago, used their technology to feed himself, close the blinds and even grab a Red Bull.

They practiced some tasks every day, such as “blanket, tissue, trash” — in which Evans needed to move a blanket, grab a tissue, wipe his face with it, and throw the tissue away. Evans improved his speed on that particular task from taking more than 16 minutes the first day to less than nine minutes at the end of the week.

Other parts of the day were more open-ended. They picked strawberries with Jane Evans in her garden and played poker with Henry, who used the HAT interface to control his cards.

Evans volunteers frequently to assist with research testing new adaptive devices, and regularly invites researchers to stay with him in his home. Padmanabha offered to stay with his parents, who live nearby, but Evans suggested that it would be more effective for the researchers to spend time living with him.

“You hear about the struggles of someone with quadriplegia but until you actually live with them, you don’t see all the details,” said Padmanabha. “It’s very interesting to see that and think about how you can use robotics or technology to address those issues.”

Without being there, it would be hard to predict which tasks had the most impact, said Gupta.

“He really enjoyed being able to scratch his itches,” she said. “I didn’t realize that was so important — it feels like such a small thing.”

Evans has some movement in his fingers, and was able to use the interface robot partially with his head movement and partially with his hand and a computer mouse.

Gupta and Padmanabha used existing hardware components, but had to solder them together to create the prototype and developed the code from scratch. The HAT interface controlled a commercially available robot called Stretch, which is more often used via a computer screen.

Evans appreciated that he was able to perform tasks, such as eating, without a computer right in front of his face. His favorite feature of the HAT device was the “assisted driving” feature, in which the technology is able to figure out how to grasp objects without explicit direction from the user. It’s a feature that Evans hopes will be incorporated into other assistive devices.

“Technology in this area is moving very quickly and it is exciting to be a part of it,” he said in an email.

Evans also provided them with feedback to improve the performance of HAT, such as allowing the device to record sequences of movements and play them back. Such technologies are essential for his well-being, he said.

“My quality of life is definitely positively correlated with increased independence,” he said via email. “Without technology, I would be lying in bed waiting to die.”

Gupta and Padmanabha don’t have active plans to commercialize the HAT device, but have heard from other researchers who are interested in incorporating some of their work.

Both of them plan to focus their careers in the development of health care technology.

“This helped me cement that interest,” said Gupta. “A lot of times, when you are taking classes, you don’t know how it could actually impact people.”