Researchers at the University of Maryland are advancing the frontiers of robotics with a new initiative designed to enable humanoid systems to perform complex, real-world household tasks with greater autonomy and reliability. Built on NVIDIA AI infrastructure through its Academic Grant Program, the project integrates breakthroughs in trustworthy machine learning, sequential decision-making and generative AI to create robotic systems that can reason, adapt and act in dynamic home environments.
The effort is led by Furong Huang, an associate professor of computer science, and Tom Goldstein, a professor of computer science. Both researchers hold appointments in the University of Maryland Institute for Advanced Computer Studies (UMIACS), which will install and maintain the new computing infrastructure in its high-performance data center.
At the core of the project is the development of “foundation models” for robotics—general-purpose AI systems that unify perception, planning and control. These models are designed to allow robots to transfer knowledge across tasks, environments and even different physical embodiments, a critical step toward building adaptable, general-purpose machines.
“By integrating advanced AI with scalable computing infrastructure, we aim to accelerate progress toward generalist household robots—systems that can adapt to new environments and tasks rather than rely on narrowly programmed behaviors,” Huang said.
While robotics has made significant progress in controlled environments, real-world household settings remain a major challenge. Tasks such as tidying a cluttered room or preparing a simple meal require robots to interpret incomplete or ambiguous sensory input, track objects over extended time horizons and make context-aware decisions. Even routine activities like loading a dishwasher involve recognizing objects of varying shapes and materials, understanding spatial relationships and adjusting actions when conditions change.
To address these challenges, the Maryland team plan to develop HomeGraph, a novel framework that structures a robot’s understanding of its environment. HomeGraph will combine functional scene graphs—capturing spatial relationships such as “on,” “inside” and “next to”—with skill and tool graphs derived from motion trajectories and large-scale video demonstrations. This hybrid representation enables robots to generate multi-step plans, monitor execution and adapt in real time. If a robot encounters an unexpected obstacle or error, it can update its internal model and revise its strategy without restarting the task.
Large-scale simulation and synthetic data generation are also central to the project. Using NVIDIA Isaac open robotics platform, researchers can create photorealistic virtual home environments populated with diverse objects and layouts. These simulations allow robots to practice millions of task variations and safely test rare or complex scenarios. The resulting datasets are used to train foundation models that generalize more effectively to new, unseen environments.
The collaboration is further strengthened by industry expertise. NVIDIA RTX PRO 6000 Blackwell GPUs for training large models and NVIDIA Jetson AGX Thor developer kits for efficient deployment on physical robots, bridges the gap between research and real-world applications.
Researchers are also exploring how generative AI techniques—such as large language and vision-language models—can enhance instruction following and human-robot interaction. By enabling users to issue natural language commands like “clean up the kitchen after dinner,” robots can translate high-level goals into executable action plans grounded in the HomeGraph framework.
Beyond household assistance, the implications of this work extend to elder care, rehabilitation and disaster response—domains where robots must operate in complex, unpredictable environments. The long-term vision is to develop versatile robotic assistants capable of seamlessly supporting everyday life.
—Story by UMIACS communications group