How do I apply proper perspective to this OpenGL ES texture?

Question

I would like to describe my objective and get some feedback of a proposed solution. I am quite new to OpenGL ES (or OpenGL at all), so please be gentle. I would also like to mention that the platform is iPad (iOS) with it's belonging restrictions.

Objective: To let the user place vertices in which a texture should be drawn properly in the correct perspective (or at least adjustable perspective). To explain what I mean, consider the following example.

Let's say that we have a photograph of a house with a corresponding lawn in front of it. Now, I want to pave a (user) specified path with tiles (texture) such that the perspective looks right. If the view is just orthogonal, the tiles will be completely square which is not realistic when having some sort of depth in the image.

Now, one approach would be to set up an perspective view with a frustum on OpenGL ES. Then put a plane (like below) and put the image texture on it for starters.

Now as you expect from the look of the image above, the image will be distorted. But it's fine because if we tilt the plane backwards (so that it fit the bounds of the display perfectly) the image will appear square.

If the user now defines the area to be paved, we can put those points in the same angle as the tilted plane, and then just put a texture with square tiles in it, and they will appear with a perspective. If our tilt angle is incorrect, the user can adjust it manually, what then happens is that the corner angles of the plane in the above figure will be adjusted and the plane will tilt differently to fit the display. The tiles texture will then appear from a different angle.

Is this a workable approach or is there some other, better way to do this? I think it's an ugly hack that might have to do with my limited skills in 3D programming.

Answer 1

If all you want to do is apply perspective to an image, you may not even need OpenGL ES for that. You could simply create a proper CATransform3D and set that to a UIView or CALayer in the manner that I describe in this answer. The key is the m34 component of the transform matrix.

If you still want to do this with OpenGL ES and a texture, you could apply a transformation to the vertices of your textured quad to create a perspective effect. An example of this can be seen here:

where I transform a texture displaying camera input so that it has a perspective effect applied to it. This was done using the following vertex shader:

 attribute vec4 position;
 attribute vec4 inputTextureCoordinate;

 uniform mat4 transformMatrix;
 uniform mat4 orthographicMatrix;

 varying vec2 textureCoordinate;

 void main()
 {
     gl_Position = transformMatrix * vec4(position.xyz, 1.0) * orthographicMatrix;
     textureCoordinate = inputTextureCoordinate.xy;
 }

where I'd generated a matrix based on a CATransform3D in the following manner:

 CATransform3D perspectiveTransform = CATransform3DIdentity;
 perspectiveTransform.m34 = 0.4;
 perspectiveTransform.m33 = 0.4;
 perspectiveTransform = CATransform3DScale(perspectiveTransform, 0.75, 0.75, 0.75);
 perspectiveTransform = CATransform3DRotate(perspectiveTransform, rotation, 0.0, 1.0, 0.0);

and then converted the CATransform3D to a mat4 (it has a 4x4 matrix internal structure, so this is fairly easy to do). If you want to see this in action, grab the code for the FilterShowcase sample application in my GPUImage framework and try the 3-D Transform entry. The code is all in the GPUImageTransformFilter and should be reasonably easy to follow.

Answer 2

That is the easiest way to do it. You're going to have to display a plane angled to the correct vanishing point regardless, so having the user rotate the plane themselves is a workable solution. Otherwise it's an image processing problem. If you're interested I would look into Vanishing point Determination in Computer Vision.