Android OpenGL ES 2.0 screen coordinates to world coordinates

Question

I'm building an Android application that uses OpenGL ES 2.0 and I've run into a wall. I'm trying to convert screen coordinates (where the user touches) to world coordinates. I've tried reading and playing around with GLU.gluUnProject but I'm either doing it wrong or just don't understand it.

This is my attempt....

public void getWorldFromScreen(float x, float y) {
    int viewport[] = { 0, 0, width , height};

    float startY = ((float) (height) - y);
    float[] near = { 0.0f, 0.0f, 0.0f, 0.0f };
    float[] far = { 0.0f, 0.0f, 0.0f, 0.0f };

    float[] mv = new float[16];
    Matrix.multiplyMM(mv, 0, mViewMatrix, 0, mModelMatrix, 0);

    GLU.gluUnProject(x, startY, 0, mv, 0, mProjectionMatrix, 0, viewport, 0, near, 0);
    GLU.gluUnProject(x, startY, 1, mv, 0, mProjectionMatrix, 0, viewport, 0, far, 0);

    float nearX = near[0] / near[3];
    float nearY = near[1] / near[3];
    float nearZ = near[2] / near[3];

    float farX = far[0] / far[3];
    float farY = far[1] / far[3];
    float farZ = far[2] / far[3];
}

The numbers I am getting don't seem right, is this the right way to utilize this method? Does it work for OpenGL ES 2.0? Should I make the Model Matrix an identity matrix before these calculations (Matrix.setIdentityM(mModelMatix, 0))?

As a follow up, if this is correct, how do I pick the output Z? Basically, I always know at what distance I want the world coordinates to be at, but the Z parameter in GLU.gluUnProject appears to be some kind of interpolation between the near and far plane. Is it just a linear interpolation?

Thanks in advance

Answer 1

/**
    * Calculates the transform from screen coordinate
    * system to world coordinate system coordinates
    * for a specific point, given a camera position.
    *
    * @param touch Vec2 point of screen touch, the
      actual position on physical screen (ej: 160, 240)
    * @param cam camera object with x,y,z of the
      camera and screenWidth and screenHeight of
      the device.
    * @return position in WCS.
    */
   public Vec2 GetWorldCoords( Vec2 touch, Camera cam)
   {  
       // Initialize auxiliary variables.
       Vec2 worldPos = new Vec2();

       // SCREEN height & width (ej: 320 x 480)
       float screenW = cam.GetScreenWidth();
       float screenH = cam.GetScreenHeight();

       // Auxiliary matrix and vectors
       // to deal with ogl.
       float[] invertedMatrix, transformMatrix,
           normalizedInPoint, outPoint;
       invertedMatrix = new float[16];
       transformMatrix = new float[16];
       normalizedInPoint = new float[4];
       outPoint = new float[4];

       // Invert y coordinate, as android uses
       // top-left, and ogl bottom-left.
       int oglTouchY = (int) (screenH - touch.Y());

       /* Transform the screen point to clip
       space in ogl (-1,1) */       
       normalizedInPoint[0] =
        (float) ((touch.X()) * 2.0f / screenW - 1.0);
       normalizedInPoint[1] =
        (float) ((oglTouchY) * 2.0f / screenH - 1.0);
       normalizedInPoint[2] = - 1.0f;
       normalizedInPoint[3] = 1.0f;

       /* Obtain the transform matrix and
       then the inverse. */
       Print("Proj", getCurrentProjection(gl));
       Print("Model", getCurrentModelView(gl));
       Matrix.multiplyMM(
           transformMatrix, 0,
           getCurrentProjection(gl), 0,
           getCurrentModelView(gl), 0);
       Matrix.invertM(invertedMatrix, 0,
           transformMatrix, 0);       

       /* Apply the inverse to the point
       in clip space */
       Matrix.multiplyMV(
           outPoint, 0,
           invertedMatrix, 0,
           normalizedInPoint, 0);

       if (outPoint[3] == 0.0)
       {
           // Avoid /0 error.
           Log.e("World coords", "ERROR!");
           return worldPos;
       }

       // Divide by the 3rd component to find
       // out the real position.
       worldPos.Set(
           outPoint[0] / outPoint[3],
           outPoint[1] / outPoint[3]);

       return worldPos;       
   }

Algorithm is further explained here.

Answer 2

Hopefully my question (and answer) should help you out:

How to find absolute position of click while zoomed in

It has not only the code but also diagrams and diagrams and diagrams explaining it :) Took me ages to figure it out as well.

Answer 3

IMHO one doesn't need to re-implement this function... I experimented with Erol's solution and it worked, so thanks a lot for it Erol. Furthermore, I played with

        Matrix.orthoM(mtrxProjection, 0, left, right, bottom, top, near, far);

and it works fine as well in my tiny noob example 2D OpenGL ES 2.0 project:

• public void onSurfaceChanged(GL10 unused, int width, int height) {...