Real-time 3-D Motion Estimation
People
 A. Chiuso, P. Favaro, X. Feng, H. Jin, P. Perona, S. Soatto
References
  • Mfm: 3D motion from 2D motion causally integrated over time (submitted, 2000)
  • A real-time system for 3D motion estimation (to appear IEEE CVPR 2000)
Code
  Real-time code for shape and motion estimation (zipped Visual C++6.0 workspace)
 For academic purpose only.
Synopsis
 We are in the process of building a real-time vision system for use as a flexible sensor for intelligent control applications in complex and unknown environments. The system is envisioned to support a variety of applications in different disciplines ranging from vehicle navigation and exploration in unknown and unstructured environments to the realization of autonomous intelligent robotic agents and real-time special effects editing; it will track a large number of point and line features in the 2D image plane; it will include a recursive estimator for calculating the position of these features in 3D space, and feedback loops from this estimate to the 2D image trackers. We can currently track about 30 point-features and estimate their position and motion in 3D at nearly frame-rate. In order to achieve a considerably higher performance  much attention will be devoted to the design of hardware-implementable robust optical flow and feature tracking algorithms. The appearance/disappearance of features and correspondence across frames will be handled via feedback loops from the 3D estimator. 

Despite the wealth of algorithms to estimate 3-D structure from image motion (SFM), none of them has proven effective on real-world sequences without user intervention. While the geometry of SFM is by now fairly well understood, the role of noise has been studied only superficially, and the interplay between geometry and
noise almost completely ignored. Therefore, there is an urgent need to study the fundamental limitations on the quality of the reconstruction from the point of view of noise, with the aid of both analysis and experimentation.  Analysis will involve studying the local sensitivity of the optimization underlying the problem of SFM as well as the
global characterization of extrema. The analysis will then guide a thorough experimental assessment of the performance, robustness, sensitivity and domain of convergence of SFM algorithms.  The real-time implementation will, for the first time, allow extensive experimental validation of SFM algorithms.

Real scenes are rarely static. Therefore, the problem of segmenting a visual scene into portions which move according to the same motion model is extremely important towards the feasibility of vision as a sensor for control systems operating in non-trivial environments.  At this stage, various consistency criteria will be tested: (i) metric (norm of the estimation residual), (ii) graph-theoretic (partitioning and normalized cuttings) and (iii) statistical.  The interplay between image-plane 2D segmentation, and 3D motion segmentation will also be investigated.

The integration of the feature tracking, structure from motion, and segmentation modules into the complete system presents some novel issues that have both practical and theoretical relevance. The feed-forward of local inconsistency in the feature tracking provides hypotheses for the segmentation. Predictions in the segmentation (high-level) are fed back to the feature selection (low-level) to both validate the measurements, and to guide the selection of new features for robustness and speed. Information is also fed to the structure from motion module. The overall system will thus implement a complex control system where information is represented at different levels of granularity.