How to align RGB and Depth image from Kinect in Matlab

问题

I am trying to align RGB and Depth image from Kinect using Matlab. I am trying to do it using the algorithm from this page.

Here is the code I have written so far

depth = imread('depth_00500.png');
color = imread('rgb_00500.png');

rotationMat=[9.9984628826577793e-01 1.2635359098409581e-03 -1.7487233004436643e-02;
 -1.4779096108364480e-03 9.9992385683542895e-01 -1.2251380107679535e-02;
1.7470421412464927e-02 1.2275341476520762e-02 9.9977202419716948e-01 ];

 translationMat=[1.9985242312092553e-02, -7.4423738761617583e-04, -1.0916736334336222e-02 ];

%parameters for color matrix
fx_rgb= 5.2921508098293293e+02;
fy_rgb= 5.2556393630057437e+02;
cx_rgb= 3.2894272028759258e+02;
cy_rgb= 2.6748068171871557e+02;
k1_rgb= 2.6451622333009589e-01;
k2_rgb= -8.3990749424620825e-01;
p1_rgb= -1.9922302173693159e-03;
p2_rgb= 1.4371995932897616e-03;
k3_rgb= 9.1192465078713847e-01;

%parameters for depth matrix
fx_d= 5.9421434211923247e+02;
fy_d= 5.9104053696870778e+02;
cx_d= 3.3930780975300314e+02;
cy_d= 2.4273913761751615e+02;
k1_d= -2.6386489753128833e-01;
k2_d =9.9966832163729757e-01;
p1_d =-7.6275862143610667e-04;
p2_d =5.0350940090814270e-03;
k3_d =-1.3053628089976321e+00;

row_num=480;
col_num=640;

for row=1:row_num
    for col=1:col_num

pixel3D(row,col,1) = (row - cx_d) * depth(row,col) / fx_d;
pixel3D(row,col,2) = (col - cy_d) * depth(row,col) / fy_d;
pixel3D(row,col,3) = depth(row,col);

    end
end

pixel3D(:,:,1)=rotationMat*pixel3D(:,:,1)+translationMat;
pixel3D(:,:,2)=rotationMat*pixel3D(:,:,2)+translationMat;
pixel3D(:,:,3)=rotationMat*pixel3D(:,:,3)+translationMat;

P2Drgb_x = fx_rgb*pixel3D(:,:,1)/pixel3D(:,:,3)+cx_rgb;
P2Drgb_y = fy_rgb*pixel3D(:,:,2)/pixel3D(:,:,3)+cy_rgb;

I am especially failing to understand why we're assigning value of depth pixel to dimension x,y and z of 3-dimensional space, shouldn't we assign (x,y,z) dimension to the depth pixel value?

I mean this part:

P3D.x = (x_d - cx_d) * depth(x_d,y_d) / fx_d
P3D.y = (y_d - cy_d) * depth(x_d,y_d) / fy_d
P3D.z = depth(x_d,y_d)

Also I'm not sure If I can represent 3d space using matrix. I am trying to use it in my code but for sure it has wrong size as multiplication by 3x3 rotation matrix is impossible.

Thank you for very much for every suggestion and help!

回答1:

This is a quite complex topic to explain in a short answer. As per me, the code is correct. Please read about intrinsic and extrinsic camera matrices. And reading about perspective projection will also help you to understand 2D to 3D projection.

P3D.x = (x_d - cx_d) * depth(x_d,y_d) / fx_d

In above line, depth(x_d, y_d) gives you the depth value at a pixel from the depth image. Then it is multiplied by the (x_d - cx_d), which is the difference along x-axis with the x coordinate of the centre point of depth map to the current pixel. Then finally this is divided by the fx_d, which is the focal length of the depth camera.

Following two references will help you to understand this mathematically well if you are interested in.

Mueller, K., Smolic, A., Dix, K., Merkle, P., Kauff, P., & Wiegand, T. (2008). View synthesis for advanced 3D video systems. EURASIP Journal on Image and Video Processing, 2008(1), 1-11.
Daribo, I., & Saito, H. (2011). A novel inpainting-based layered depth video for 3DTV. Broadcasting, IEEE Transactions on, 57(2), 533-541.

来源：https://stackoverflow.com/questions/30003005/how-to-align-rgb-and-depth-image-from-kinect-in-matlab

标签

matlab

kinect