Using PIL (Python Image Library) to detect image on screen

Question

I am trying to understand how I can use PIL in Python 2.7 to search the whole screen for a certain image and click on it. I\'ve been searching around and haven\'t been able

Answer 1

PIL is the wrong tool for this job. Instead you should look into openCV (open source computer vision), which has fantastic python bindings. Here is a link to an example (in C but should be easy to redo with the python bindings) that does what you are looking for, but even allows the image to be rotated, scaled, etc.

http://docs.opencv.org/doc/tutorials/features2d/feature_homography/feature_homography.html http://docs.opencv.org/doc/tutorials/features2d/detection_of_planar_objects/detection_of_planar_objects.html

Edit:

I assume you are using windows, as your example image looks like window. In this case you can use:

from PIL import ImageGrab
pil_img = ImageGrab.grab()
opencv_img = numpy.array(pil_img)

then use opencv to process the image to find sub image you are looking for.

If you want to do this cross platform, then you will need to use wxWidgets to do the screengrab: https://stackoverflow.com/a/10089645/455532

Answer 2

Even I wanted to do the same but using different module - pyautogui. I finally found the solution for my problem and I am sure this solution will also help you. You have to just go to this webpage and read the locate function topic completely and you'll be able to solve your problem.