Removing square objects

Question

I have the image includes circular, elipsoidal, square objects and somethings

Answer 1

You can compare the area of the mask to its perimeter using the following formula

ratio = 4 * pi * Area / ( Perimeter^2 )

For circles this ration should be very close to one, for other shapes it should be significantly lower.
See this tutorial for an example.

The rationale behind this formula: circles are optimal in their perimeter-area ratio - max area for given perimeter. Given Perimeter, you can estimate radius of equivalent circle by Perimeter = 2*pi*R, using this estimated R you can compute the "equivalent circle area" using eqArea = pi*R^2. Now you only need to check the ratio between the actual area of the shape and the "equivalent area" computed.

Note: since Area and Perimeter of objects in mask are estimated based on the pixel-level discretization these estimates may be quite crude especially for small shapes. Consider working with higher resolution masks if you notice quantization/discretization errors.

Answer 2

There exists a Hough transform (imfindcircles) in order to find circles within an image which is what you needed in the first place.