Finding location of rectangles in an image with OpenCV

Question

I\'m trying to use OpenCV to \"parse\" screenshots from the iPhone game Blocked. The screenshots are cropped to look like this:

Answer 1

Since your problem is the small rectangles I would start by removing them. Since those lines are much thinner than the borders of the rectangles I would start by applying morphological operations on the image.

Using a structural element that looks like this:

 element = [ 1 1
             1 1 ]

should remove lines that are less than two pixels wide. After the small lines are removed the rectangle finding algorithm of OpenCV will most likely do the rest of the job for you. The erosion can be done in OpenCV by the function cvErode

Answer 2

Here's a complete Python solution. The main idea is:

• Apply pyramid mean shift filtering to help threshold accuracy
• Otsu's threshold to get a binary image
• Find contours and filter using contour approximation

---

Here's a visualization of each detected rectangle contour

Results

import cv2

image = cv2.imread('1.png')
blur = cv2.pyrMeanShiftFiltering(image, 11, 21)
gray = cv2.cvtColor(blur, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
for c in cnts:
    peri = cv2.arcLength(c, True)
    approx = cv2.approxPolyDP(c, 0.015 * peri, True)
    if len(approx) == 4:
        x,y,w,h = cv2.boundingRect(approx)
        cv2.rectangle(image,(x,y),(x+w,y+h),(36,255,12),2)

cv2.imshow('thresh', thresh)
cv2.imshow('image', image)
cv2.waitKey()

Answer 3

I wound up just building on my original method and doing as Robert suggested in his comment on my question. After I get my list of rectangles, I then run through and calculate the average color over each rectangle. I check to see if the red, green, and blue components of the average color are each within 10% of the gray and blue rectangle colors, and if they are I save the rectangle, if they aren't I discard it. This process gives me something like this:

From this, it's trivial to get the information I need (orientation, starting point, and length of each rectangle, considering the game window as a 6x6 grid).

Answer 4

The blocks look like bitmaps - why don't you use simple template matching with different templates for each block size/color/orientation?

Answer 5

Try one of the many corner detectors like harris corner detector. also it is in general a good idea to try that at multiple resolutions : so do some preprocessing of of varying magnification. It appears that you want some sort of color dominated square then you can suppress the other colors, by first using something like cvsplit .....and then thresholding the color...so only that region remains....follow that with a cropping operation ...I think that could work as well ....

Answer 6

The similar issue has already been discussed: How to recognize rectangles in this image?

As for your data, rectangles you are trying to find are the only black objects. So you can try to do a threshold binarization: black pixels are those ones which have ALL three RGB values less than 40 (I've found it empirically). This simple operation makes your picture look like this:

After that you could apply Hough transform to find lines (discussed in the topic I referred to), or you can do it easier. Compute integral projections of the black pixels to X and Y axes. (The projection to X is a vector of x_i - numbers of black pixels such that it has the first coordinate equal to x_i). So, you get possible x and y values as the peaks of the projections. Then look through all the possible segments restricted by the found x and y (if there are a lot of black pixels between (x_i, y_j) and (x_i, y_k), there probably is a line probably). Finally, compose line segments to rectangles!