People walking by a fifth-floor lab in the Brendan Iribe Center for Computer Science and Engineering often do a double-take when peering through the lab’s glass doorway.
Lush green ferns populate the space, which is filled to the brim with hand-held power tools and physical measuring devices, oscilloscopes for testing digital signals, 3D printers to help build multiple exoskeletal prototypes, and a pair of very lifelike prosthetic hands reaching skyward from a desktop work area.
Known as the Embodied Dynamics Laboratory, this unique space is dedicated to exploring connections between the digital world and human embodiment and cognition. Ongoing science and scholarship is focused on how our bodies, perceptions and somatic interactions—the interplay between bodily systems or functions—contribute to the formation of physical skills, knowledge, subjectivity and behavior.
“By bridging our embodiment and digital worlds, we can unlock the immense potential of human-machine collaboration, forging connections that transcend the boundaries of traditional interaction and bringing people closer together in ways we’ve yet to imagine,” says Jun Nishida, an assistant professor of computer science who is the lead principal investigator of the lab.
Nishida, who has a joint appointment in the University of Maryland Institute for Advanced Computer Studies (UMIACS) and is active in the university’s Immersive Media Design program, says the ongoing work by him and almost a dozen graduate students in the lab has numerous real-life applications. This includes novel ways to promote physical rehabilitation, educational tools for musical instruction, and prototypes that allow for engineers designing new systems to better understand the special needs of people with mobility issues.
One project involves an exoskeleton device attached to one’s hand that is used for piano instruction. Traditional piano learning methods often rely heavily on visual and audio cues, Nishida says, which can be inefficient for mastering intricate skills quickly.
But the exoskeleton under development in Nishida’s lab provides essential physical cues, such as force and body sensations. By incorporating tactile and body awareness experiences into the musical pedagogy, the researchers aim to create a new computing paradigm that allows students a more effective and immersive learning process.
Another project—which Nishida’s team presented last October at the ACM Symposium on User Interface Software and Technology—involves an exoskeleton device called DexteriSync.
It features flexible silicone-copper tube segments, 3D-printed finger sockets, and an advanced water temperature control system that utilizes both copper conduction and water cooling to regulate skin temperature.
By dynamically adjusting finger dexterity and modulating thermal perceptions, DexteriSync provides a tactile experience that helps product designers understand the difficulties faced by individuals with compromised dexterity, such as elderly and neurodivergent users.
Nishida says his vision of better connecting people through their physical experiences was influenced by a childhood fascination with anime while growing up in Japan—especially titles like “Ghost in the Shell,” which explores identity in a cybernetic future where technology and human biology seamlessly integrate, and “Paprika,” a story that examines the intersection of dreams and reality.
These same themes have driven his commitment to integrating humans and machines, while also addressing potential crises of self-identity that may arise from such integration.
“This has been the focus of my research since graduate school and continues to guide my work as a faculty member,” says Nishida, who received his Ph.D. in human informatics from the University of Tsukuba in 2019.
His subsequent journey to the University of Maryland—his lab began operations in early 2024—included a postdoctoral fellowship at the University of Chicago’s Human Computer Integration Lab, as well as internships at Microsoft Research Asia in Beijing and Sony Computer Science Laboratories in Tokyo.
Nishida believes all these prior work experiences shaped his understanding of how technology and the human sciences are both needed to successfully acquire new skills.
“Effective skill acquisition goes beyond observation—it’s about feeling and experiencing movements,” he says.
But Nishida acknowledges a critical challenge remains in his work: the risk of diminishing users’ sense of agency. He says that while technology can improve our performance, it is crucial for users to feel a sense of control over their bodies.
To help address this challenge, Nishida says he relies on the interdisciplinary collaboration and creativity of the students in his lab. This includes future recruits that may want to explore fully functioning on-body devices that interact with human embodiment and human cognition.
“If you’re interested in getting your hands dirty, dusty or oily, my lab is the place to be,” he says.
—Story by Melissa Brachfeld, UMIACS communications group