Intersection of polygons

Question

There are two polygons given. how can one determine whether one polygon is inside, outside or intersecting the other polygon? Polygons can be Concave or convex.

Answer 1

There is an easyish method to check whether a point lies in a polygon. According to this Wikipedia article it is called the ray casting algorithm.

The basic idea of the algorithm is that you cast a ray in some arbitrary direction from the point you are testing and count with how many of the edges of the polygon it intersects. If this number is even then the point lies outside the polygon, otherwise if it is odd the point lies inside the polygon.

There are a number of issues with this algorithm that I won't delve into (they're discussed in the Wikipedia article I linked earlier), but they are the reason I call this algorithm easyish. But to give you an idea you have to handle corner cases involving the ray intersecting vertices, the ray running parallel and intersecting an edge and numeric stability issues with the point lying close to an edge.

You can then use this method in the way Thomas described in his answer to test whether two polygons intersect. This should give you an O(NM) algorithm where the two polygons have N and M vertices respectively.