Recognize recurring images into a larger one

Question

Edit: this is not a duplicate of Determine if an image exists within a larger image, and if so, find it, using Python since I do not know the pattern beforehand<

Answer 1

The goal is to find several equal or very similar patterns which are not known before in a picture. As it is this problem is still a bit ill posed.

• Are the patterns exactly equal or only similar (added noise maybe)?
• Do you want to have the largest possible patterns or are smaller subpatterns okay too or are all possible patterns needed? Reason is that of course each pattern could consist of equal patterns too.
• Is the background always that simple (completely white) or can it be much more difficult? What do we know about it?
• Are the patterns always equally oriented, equally scaled, non-overlapping?

For the simple case of non-overlapping patterns with simple background, the answer of Yves Daoust using segmentation is well performing but fails if patterns are very close or overlapping.

For other cases the idea of the keypoints by Micka will help but might not perform well if there is noise or might be slow.

I have one alternative: look at correlations of subblocks of the image.

In pseudocode:

• Divide the image in overlapping areas of size MxN for a suitable M,N (pixel width and height chosen to be approximately the size of the desired pattern)
• Correlate each subblock with the whole image. Look for local maxima in the correlation. The position of these maxima denotes the position of similar regions.
• Choose a global threshold on all correlations (smartly somehow) and find sets of equal patterns.
• Determine the fine structure of these patterns by shanging the shape from rectangular (bounding box) to a more sophisticaed shape (maybe by looking at the shape of the peaks in the correlation)
• In case the approximate size of the desired patterns is not known before, try with large values of M, N and go down to smaller ones.
• To speed up the whole process start on a coarse scale (downscaled version of the image) and then process finer scales only where needed. Needs balancing of zooming in and performing correlations.

Sorry, I cannot make this a full Matlab project right now, but I hope this helps you.

Answer 2

You can try to use described keypoints (Sift/SURF/ORB/etc.) to find features in the image and try to detect the same features in the image. You can see such a result in How to find euclidean distance between keypoints of a single image in opencv where 3x the same image is present and features are detected and linked between those subimages automatically.

In your image the result looks like

so you can see that the different occurances of the same pattern is indeed automatically detected and linked.

Next steps would be to group features to objects, so that the "whole" pattern can be extracted. Once you have a candidate for a pattern, you can extract a homography for each occurance of the pattern (with one reference candidate pattern) to verify that it is a pattern. One open problem is how to find such candidates. Maybe it is worth trying to find "parallel features", so keypoint matches that have parallel lines and/or same length lines (see image). Or maybe there is some graph theory approach.

All in all, this whole approach will have some advantages and disadvantes:

Advantages:

• real world applicability - Sift and other keypoints are working quite well even with noise and some perspective effects, so chances are increased to find such patterns.

Disadvantages

• slow
• parametric (define what it means that two features are successfully matched)
• not suitable for all kind of patterns - your pattern must have some extractable keypoints

Those are some thoughts and probably not complete ;)

Unfortunately no full code yet for your concrete task, but I hope the idea is clear.

Answer 3

For such a clean image, it suffices to segment the patterns by blob analysis and to compare the segments or ROI that contain them. The size is a first matching criterion. The SAD, SSD or correlation similarity scores can do finer comparison.

In practice you will face more difficulties such as

• not possible to segment the patterns

• geometric variations in size/orientation

• partial occlusion

• ...

Handling these is out of the scope of this answer; it makes things much harder than in the "toy" case.