3D Screenspace Raycasting/Picking DirectX9

Question

I need to convert a 2D mouse coordinate to a 3D world coordinate based on a specific depth (it\'s for raycasting). I\'m not directly using DirectX in C++, I\'m using a propr

Answer 1

To convert you mouse to ray, you do this process:

Convert your mouse coordinates from pixel coordinate to -1/1 coordinates (-1,-1 being bottom left).

ray origin will be (vector at near plane)

vector3 origin = vector3(mousex,mousey,0);

ray end is (vector at far plane)

vector3 far = vector3(mousex,mousey,1);

now you need to apply transform to those vectors, first you need to create transformation matrix for it:

matrix4x4 inverseviewproj = invertmatrix(view * proj)

Apply this transformation to both vectors:

vector3 rayorigin = transform(origin, inverseviewproj);
vector3 rayend = transform(far, inverseviewproj);

Your ray direction is :

vector3 raydirection = normalize(rayend-rayorigin);

That's about it, now you can use raycast functions.

In case you only have access to vector4 to transform vector by a matrix, the w component needs to be 1 in eg:

vector4 origin = vector3(mousex,mousey,0,1);
vector4 far = vector3(mousex,mousey,1,1);

to extract direction make sure to first convert your vector4 into vector3 (since it will affect normalization otherwise).