Trying to triangulate a set of simple 2d polygons, I\'ve come up with this algorithm:
You are doing a version of "ear clipping" approach to triangulation, see: http://en.wikipedia.org/wiki/Polygon_triangulation
Your algorithm fails if another polygon vertex (say from the other side of the polygon) ends up inside the triangle 'ear' you form. Consider this example:

Your algorithm will choose A first, and make a triangle with the two adjacent edges (connected with the dashed line). But this triangle includes another vertex (B) and is clearly incorrect.
The generalized ear clipping approach depends on finding ears you can clip off that do not have any included vertices.