How to estimate/determine surface normals and tangent planes at points of a depth image?

Question

I have a depth image, that I\'ve generated using 3D CAD data. This depth image can also be taken from a depth imaging sensor such as Microsoft Kinect or any other stereo cam

Answer 1

I will describe what I think you have to do conceptually and provide links to the relevant parts of opencv,

To determine normal of a given (3d) point in a pointcloud:

1. Create a kd-tree or (balltree?) representation of your point cloud so that you can efficiently compute the k nearest neighbors. Your choice of k should depend on the density of your data. http://docs.opencv.org/trunk/modules/flann/doc/flann_fast_approximate_nearest_neighbor_search http://physics.nyu.edu/grierlab/manuals/opencv/classcv_1_1KDTree.html

2. After querying for the k-nearest neighbors of a given point p, use them to find a best fit plane. You can use PCA to do this. Set maxComponents=2. http://physics.nyu.edu/grierlab/manuals/opencv/classcv_1_1PCA.html https://github.com/Itseez/opencv/blob/master/samples/cpp/pca.cpp

3. Step 2 should return two eigenvectors which define the plane you are interested in. The cross product of these two vectors should be (an estimation of) your desired normal vector. You can find info how to calculate this in opencv (Mat::cross) http://docs.opencv.org/modules/core/doc/basic_structures.html