I wrote this code before a final attempt with Google and finding this page, so I thought I'd share it. It is basically an optimized version of Kisielewicz answer. I looked into the Barycentric method also but judging from the Wikipedia article I have a hard time seeing how it is more efficient (I'm guessing there is some deeper equivalence). Anyway, this algorithm has the advantage of not using division; a potential problem is the behavior of the edge detection depending on orientation.
bool intpoint_inside_trigon(intPoint s, intPoint a, intPoint b, intPoint c)
{
int as_x = s.x-a.x;
int as_y = s.y-a.y;
bool s_ab = (b.x-a.x)*as_y-(b.y-a.y)*as_x > 0;
if((c.x-a.x)*as_y-(c.y-a.y)*as_x > 0 == s_ab) return false;
if((c.x-b.x)*(s.y-b.y)-(c.y-b.y)*(s.x-b.x) > 0 != s_ab) return false;
return true;
}
In words, the idea is this: Is the point s to the left of or to the right of both the lines AB and AC? If true, it can't be inside. If false, it is at least inside the "cones" that satisfy the condition. Now since we know that a point inside a trigon (triangle) must be to the same side of AB as BC (and also CA), we check if they differ. If they do, s can't possibly be inside, otherwise s must be inside.
Some keywords in the calculations are line half-planes and the determinant (2x2 cross product). Perhaps a more pedagogical way is probably to think of it as a point being inside iff it's to the same side (left or right) to each of the lines AB, BC and CA. The above way seemed a better fit for some optimization however.
