Below are 2 rectangles. Given the coordinates of the rectangle vertices - (x1, y1)...(x8, y8), how can the area of the overlapping region (white in the figure below
You can calculate the intersection points by solving intersection equations for all pairs of sides of the figures: /frac{x - a}{b - a} = /frac{x - c}{d - c}
The points that you obtain in such a fashion can lie on the sides of the paralelepide, though they must not. You have to check whether the intersection points you obtained by solving the equations lie on the sides of the figure or not. If they do, you can calculate the length of the sides of the figure that stretch out into the inside of the both figures, and calculate the square of the intersection by taking their multiple.
I guess my method sounds a bit over-complicated, but this is the first thought that came to my mind.