Developing a Robotic System to Improve Autonomy for People with Mobility Issues

When Amisha Bhaskar was an undergraduate studying engineering in Delhi, India, she took a field trip to a facility for disabled war veterans where she encountered a man with amputated hands. Bhaskar asked the man what technologies she could design that would help improve his quality of life.

His reply—assistance with the ability to take care of himself so that he would not have to rely upon others—left an indelible impression.

Now a second-year doctoral student at the University of Maryland studying computer science, Bhaskar is intently focused on this area of study. Working with others in the Robotics Algorithms & Autonomous Systems Lab, she is developing innovative technology that addresses a fundamental need for individuals with mobility impairments to regain autonomy in feeding themselves.

The team’s work was recently recognized at the IEEE International Conference on Robotics and Automation (ICRA 2024) in Yokohama, Japan, where Bhaskar took part in a poster session and a paper she was lead co-author on received top honors in a specialized workshop on cooking and robotics.

While robotic-assisted feeding technology already exists, it is very limited in its current form, the UMD researchers say. Commercial robotic arms have a fixed, pre-programmed motion that only allows them to pick up food in a specific spot on a plate. Furthermore, the robotic arms currently available do not have the ability to detect whether it’s successfully picked up the food, meaning they sometimes aimlessly repeat an unsuccessful motion.

“They are not learning on the go, so it will just keep doing this motion no matter if you want to eat it or not, or if the food is picked up or not,” says Bhaskar.

Robotic-assisted feeding can be divided into two steps, Bhaskar explains: acquisition and transfer. The acquisition step involves a utensil picking up the food, while the transfer step is the process of the food successfully entering a person’s mouth without being dropped.

Bhaskar and the UMD team are currently working on the acquisition step, with a lofty goal. While other research groups working on assisted feeding describe success as a utensil picking up food a single time, the target for the UMD team is for their robotic arm to be able to clear the plate.

They aim to build a system that can recognize and transport a variety of foods that are often given to people in an assisted-care setting—from granular foods such as cereals, to semi-solid foods such as yogurt and tofu, to liquid foods.

One of the most significant challenges in the team’s research has been developing a robotic system capable of handling foods with varied textures and consistencies within a single dish. Ramen, for example, presents a complex scenario that includes a liquid broth, deformable tofu, solid vegetables, and irregularly shaped noodles.

Despite these obstacles, the UMD team is making steady progress, says Bhaskar.

“Every single element requires different strategies, some of which have to be combined,” she explains. “While our system has successfully managed to pick up each element—ranging from the soft tofu to the crisp vegetables—we continue to refine our approach to effectively manage the noodles, which remain a challenging aspect due to their unique shape and texture.”

The team’s interdisciplinary approach has played a key role in the project’s success, says Pratap Tokekar, an associate professor of computer science with an appointment in the University of Maryland Institute for Advanced Computer Studies.

“The technology we’re working on involves computer vision, artificial intelligence, deep neural networks, mechanical engineering and more—it all needs to come together seamlessly so that the robotic system is both safe for users and efficient in accomplishing the task at hand,” he explains.

Tokekar is academic adviser to Bhaskar and another graduate student working on the project, Rui Liu, who is a third-year doctoral student in computer science.

Robotic-assisted feeding is a relatively new area of research for Liu, who had previously focused much of his work on computer vision and human-robot interaction.

But like Bhaskar, Liu sees the potential in his current research to greatly improve people’s lives, particularly the elderly or those with mobility issues that make feeding themselves difficult.

Additional team members include Vishnu D. Sharma, who is graduating this year with a doctoral degree in computer science, and Guangyao Shi, who graduated last year with his doctoral degree in electrical and computer engineering and is now doing postdoctoral research at the University of Southern California.

While the project is probably several years away from any real-world application, Tokekar is confident in the progress that’s being made, largely because of Bhaskar’s and Riu’s eagerness and intense focus.

“The best part of this project is that every time we meet, they have 10 new ideas since the last time that we met,” Tokekar says. “Instead of me telling them what to do, they already know what to do. I’m just helping shape their ideas.”

—Story by Shaun Chornobroff, UMIACS communications group