converting 2D mouse coordinates to 3D space in OpenGL ES

Question

I want to convert mouse's current X and Y coordinates into the 3D space I have drawn in the viewport. I need to do this on the OpenGL ES platform. I found following possible solutions implemented in OpenGL, but none fits what I am looking for.

1. I found NeHe's tutorial on doing exactly this, but in traditional OpenGL way. It uses gluUnProject. http://nehe.gamedev.net/data/articles/article.asp?article=13 Although gluUnProject is not available in OpenGL ES, its implementation seems simple enough to port back. But before calling it, we need to call glReadPixels with GL_DEPTH_COMPONENT and that is not possible in OpenGL ES. (The reason I found in this thread: http://www.khronos.org/message_boards/viewtopic.php?f=4&t=771)

2. What I want to do is similar to picking, except that I don't want to select the object but I want exact coordinates so that I can recognize particular portion of the object that is currently under mouse cursor. I went through the Picking tutorials in this answer. https://stackoverflow.com/posts/2211312/revisions But they need glRenderMode, which I believe is absent in OpenGL ES.

If you know how to solve this problem in OpenGL ES, please let me know.

Thanks.

Answer 1

I think the general solution is to figure out where in world space the clicked coordinate falls, assuming the screen is a plane in the world (at the camera's location). Then you shoot a ray perpendicular to the plane, into your scene.

This requires "world-space" code to figure out which object(s) the ray intersects with; the solutions you mention as being unsuitable for OpenGL ES seem to be image-based, i.e. depend on the pixels generated when rendering the scene.

Answer 2

With OpenGL ES 2.0 you could use a FBO and render the depth values to a texture. Obviously, this wouldn't be exactly cheap (just a way around the restriction of glReadPixels)...

Further, since - as I understand it - you want to pick certain parts of your object you might want to do some sort of color-picking where each selectable portion of the object has an unique color (note that the Lighthouse 3D tutorial only shows the general idea behind color-picking, your implementation would probably be different). You could optimize a little by performing a ray/bounding-box intersection beforehand and only rendering the relevant candidates to the texture used for picking.