Below are 2 rectangles. Given the coordinates of the rectangle vertices - (x1, y1)...(x8, y8), how can the area of the overlapping region (white in the figure below
Maybe you need to use opencv. Use the fillPoly() function to generate a rectangle. Make sure the fill rectangle is white (255, 255, 255). Then use the copyTo() function and you will get the overlap area. Then check the value of each pixel , if it is white then +1.