3D Screenspace Raycasting/Picking DirectX9

Question

I need to convert a 2D mouse coordinate to a 3D world coordinate based on a specific depth (it's for raycasting). I'm not directly using DirectX in C++, I'm using a proprietary language, but if you give me an answer in C++ or pseudocode (C++ preferred) I can convert it.

I have access to the world matrix, view matrix and projection matrix and a variety of matrix manipulation functions.

If it's necessary to multiply a vector4 by a matrix4, the only function I have available that takes both a vector4 and a matrix4 is transformVector4(vector4Source,matrix4Source). I'm not sure which order it multiplies them in, if that matters.

Any help would be much appreciated :) I've been working on this for hours and I just don't get 3D maths...

Answer 1

To convert you mouse to ray, you do this process:

Convert your mouse coordinates from pixel coordinate to -1/1 coordinates (-1,-1 being bottom left).

ray origin will be (vector at near plane)

vector3 origin = vector3(mousex,mousey,0);

ray end is (vector at far plane)

vector3 far = vector3(mousex,mousey,1);

now you need to apply transform to those vectors, first you need to create transformation matrix for it:

matrix4x4 inverseviewproj = invertmatrix(view * proj)

Apply this transformation to both vectors:

vector3 rayorigin = transform(origin, inverseviewproj);
vector3 rayend = transform(far, inverseviewproj);

Your ray direction is :

vector3 raydirection = normalize(rayend-rayorigin);

That's about it, now you can use raycast functions.

In case you only have access to vector4 to transform vector by a matrix, the w component needs to be 1 in eg:

vector4 origin = vector3(mousex,mousey,0,1);
vector4 far = vector3(mousex,mousey,1,1);

to extract direction make sure to first convert your vector4 into vector3 (since it will affect normalization otherwise).