Detection of Manipulation Action Consequences (MAC)

Abstract

The problem of action recognition and human activity has been an active research area in Computer Vision and Robotics. While full-body motions can be characterized by movement and change of posture, no characterization, that holds invariance, has yet been proposed for the description of manipulation actions. We propose that a fundamental concept in understanding such actions, are the \textbf{consequences of actions}. There is a small set of fundamental primitive action consequences that provides a systematic high-level classification of manipulation actions. In this paper a technique is developed to recognize these action consequences. At the heart of the technique lies a novel active tracking and segmentation method that monitors the changes in appearance and topological structure of the manipulated object. These are then used in a visual semantic graph (VSG) based procedure applied to the time sequence of the monitored object to recognize the action consequence. We provide a new dataset, called \textbf{M}anipulation \textbf{A}ction \textbf{C}onsequences (MAC 1.0), which can serve as testbed for other studies on this topic. Several experiments on this dataset demonstrates that our method can robustly track objects and detect their deformations and division during the manipulation. Quantitative tests prove the effectiveness and efficiency of the method.

The First Manipulation Action Consequences dataset (MAC 1.0)

24 sequences are available for download, 6 for each consequence. Please refer to the readme file for the details.

ZIP file: MAC 1.0

NEW!!! A sample implementation as a ROS (Robot Operating System) package can be downloaded HERE.

Special thanks to given for sharing their manipulation action dataset,

E. Aksoy, A. Abramov, J. Dorr, K. Ning, B. Dellen, and F. Worgotter. Learning the semantics of object–action relations by observation. The International Journal of Robotics Research, 30(10):1229–1249, 2011.

Questions? Please contact yzyang "at" cs dot umd dot edu