3D Scene transformations not quite working

Question

In an attempt to convert screen space coordinates into world space coordinates, I've been doing the following calculation:

WorldSpace Vector = inverse(Projection Matrix) * inverse(View Matrix) * ScreenSpace vector

Up to this point, I believe I have most of my calculations right, however I'm unsure of how to perform the last step needed in order to have my vector transformed to world coordinates.

Problem:

• I've been told that the last step of this process is to divide my results by the variable "w" because we are using homogeneous coordinates. However I have no idea what "w" represents.

I've defined the following variables (x and y are integers of the mouse coordinates):

GLint viewport[4];
GLfloat winX, winY;
glGetIntegerv(GL_VIEWPORT, viewport);

winX = ((float)x/viewport[2]*2)-1;
winY = ((float)(viewport[3]-y)/viewport[3]*2)-1;

glm::mat4x4 mMatrix;
glm::mat4x4 vMatrix;

glm::mat4x4 cameraTransformation;

And I performed the following transformations:

cameraTransformation = glm::rotate(cameraTransformation, (float)alpha*(float)M_PI/180, glm::vec3(0, 1, 0));
cameraTransformation = glm::rotate(cameraTransformation, (float)beta*(float)M_PI/180, glm::vec3(1, 0, 0));

glm::vec4 cameraPosition = (cameraTransformation * glm::vec4(camX, camY, distance, 0));
glm::vec4 cameraUpDirection = cameraTransformation * glm::vec4(0, 1, 0, 0);

vMatrix = glm::lookAt(glm::vec3(cameraPosition[0],cameraPosition[1],cameraPosition[2]), glm::vec3((float)camX, (float)camY, 0.0), glm::vec3(cameraUpDirection[0],cameraUpDirection[1],cameraUpDirection[2]));

glm::mat4x4 mat =  glm::inverse(vMatrix) * glm::inverse(mMatrix) * glm::inverse(pMatrix);
glm::vec4 wSC = (mat) * glm::vec4(winX,winY,0,0);

In my resize event, my pMatrix is projected as such:

pMatrix = glm::mat4x4(1.0);//sets identity
pMatrix = glm::perspective( (float)fov*(float)M_PI/180, (float) width / (float) height, (float)0.001, (float)10000 );

Note: I've had issues in the past with using the GLU library, and apparently with using unProject functions overall, so I've elected to perform the calculations myself. At this point its almost a justification of effort, but I'm not going to use preexisting unprojection functions period.

Answer 1

In this last stage of my ray picking problem, I was informed that I was doing several things wrong:

1. My z coordinate for a window space near plane should be -1, not 0
2. Reminder that I need 2 separate vectors at the opposing planes, don't just use 1 for testing, run it through the actual ray picker to see if it works.
3. Use a "w" value of 1, not 0. This is the last spot of the glm::vec4 wSC = (mat) * glm::vec4(winX,winY,0,0); line.
4. That I need to convert my mouse coordinates into NDC coordinates before performing any calculations, so this is:
   • divide mouse x by screen width, times 2, minus 1.
   • subtract mouse y from screen height, divide by screen height, times 2, minus 1: winX = ((float)x/viewport[2]*2)-1; winY = ((float)(viewport[3]-y)/viewport[3]*2)-1;

Since my problem extended over quite a great length of time, and took up several questions, I owe credit to a few individuals who helped me along the way:

• srobins of facepunch, in this thread
• derhass from here, in this question, and this discussion