Capturing Hands in Action using Discriminative Salient Points and Physics Simulation

Dimitrios Tzionas Luca Ballan Abhilash Srikantha Pablo Aponte Marc Pollefeys Juergen Gall

Abstract

Hand motion capture is a popular research field, recently gaining more attention due to the ubiquity of RGB-D sensors. However, even most recent approaches focus on the case of a single isolated hand. In this work, we focus on hands that interact with other hands or objects and present a framework that successfully captures motion in such interaction scenarios for both rigid and articulated objects. Our framework combines a generative model with discriminatively trained salient points to achieve a low tracking error and with collision detection and physics simulation to achieve physically plausible estimates even in case of occlusions and missing visual data. Since all components are unified in a single objective function which is almost everywhere differentiable, it can be optimized with standard optimization techniques. Our approach works for monocular RGB-D sequences as well as setups with multiple synchronized RGB cameras. For a qualitative and quantitative evaluation, we captured 29 sequences with a large variety of interactions and up to 150 degrees of freedom.

Publications

Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M. and Gall, J.
Capturing Hands in Action using Discriminative Salient Points and Physics Simulation [PDF] [arXiv] [Springer] [BibTex]
International Journal of Computer Vision (IJCV)
Special issue "Human Activity Understanding from 2D and 3D data" (link)
(Submitted on 17.10.14 / Accepted on 16.02.2016)

Tzionas, D., Srikantha, A., Aponte, P. and Gall, J.
Capturing Hand Motion with an RGB-D Sensor, Fusing a Generative Model with Salient Points [PDF] [Web] [BibTex] [Sup1] [Sup2]
German Conference on Pattern Recognition (GCPR'14)

Ballan, L., Taneja, A., Gall, J., Van Gool, L. and Pollefeys, M.
Motion Capture of Hands in Action using Discriminative Salient Points [PDF] [Web] [BibTex] [Suppl.]
European Conference on Computer Vision (ECCV'12)

Videos

Related Projects

In chronological order:

3D Object Reconstruction from Hand-Object Interactions, ICCV 2015 [Web]

Capturing Hand Motion with an RGB-D Sensor, Fusing a Generative Model with Salient Points, GCPR 2014 [Web]

A Comparison of Directional Distances for Hand Pose Estimation, GCPR 2013 [Web]

Motion Capture of Hands in Action using Discriminative Salient Points, ECCV 2012 [Web]

Marker-less Motion Capture of Skinned Models in a Four Camera Set-up using Optical Flow and Silhouettes, 3DPVT 2008 [Web]

Citation

@article{Tzionas:IJCV:2016,
  title = {Capturing Hands in Action using Discriminative Salient Points and Physics Simulation},
  author = {Tzionas, Dimitrios and Ballan, Luca and Srikantha, Abhilash and Aponte, Pablo and Pollefeys, Marc and Gall, Juergen},
  journal = {International Journal of Computer Vision (IJCV)},
  year = {2016},
  url = {http://files.is.tue.mpg.de/dtzionas/Hand-Object-Capture}
}

Contact

If you have questions concerning the data, please contact:

	Dimitrios Tzionas	for monocular RGB-D data/experiments
	Luca Ballan	for multicamera RGB data/experiments

If you have general questions/comments concerning the paper or the website, please contact Dimitrios Tzionas