How do I determine what is touched in 3D space from the screen?

Question

How do I use gl.gluUnproject in my OpenGL ES 1.1 android app to determine what is selected when the user touches the screen?

My understanding is that the touch event

Answer 1

It completely depends on what it is that you're rendering. You probably shouldn't use the picking approach because

a) It sounds like it will be slow, especially on android. b) You might want to have a larger 'touch area' for small objects so you don't have to touch precisely where they are. c) It doesn't really answer the right question - it gives you the result "What is the top-most graphical item that has been rendered exactly where I touched?" whereas you want to know "What game entity did the user touch or touch near?"

As I said, it completely depends on your game. If it is a 2D or nearly 2D game then it's simple and you just feed the touch coordinates into your game model. For 3D games I would suggest this simple algorithm that I just came up with on the spot:

1. Make a list of all the touchable game objects that might have been touched.
2. Transform their centre coordinates into 3D screen coordinates. (2D screen coordinates described here: http://www.flipcode.com/archives/Plotting_A_3D_Point_On_A_2D_Screen.shtml )
3. Find the 2D distance to each object, discard objects with a distance greater your touch threshold.
4. Find the closest object according to some distance metric. You'll have to experiment at this point and again, it depends on the nature of the game.

For exact results the line intersection thing could be used. There are algorithms for that (search for 'plane line intersection raycasting').