OpenCV RotatedRect with specified angle

Question

I have the situation that I have a small binary image that has one shape, around which I want to find the best fitting rotated rectangle (not bounding rectangle). I know that there is cv::minAreaRect() that you apply on the result found by cv::findContours(), but this has delivered poor results in my case, because the data is noisy (coming from MS Kinect, see example picture

where rotation changes due to the input data (contour) being slightly different). What I did instead was to calculate the principal axis using PCA on my binary image (which is less sensitive to noise), which yields angle "a", and now I want to create a RotatedRect around my shape, given the angle of the principal axis, a).

I have an illustration, made with my superb Paint skills!

So then my question is: do you guys have code snippets or concrete suggestions to solve this? I'm afraid that I have to do many Bresenham iterations, hoping that there is a clever approach.

Btw, for those who are not too familiar with the RotatedRect data structure of openCV: it is defined by height, width, angle, and center point, assuming that center point is actually, well, in the center of the rectangle.

Cheers!

Answer 1

OK, my solution: Approach:

1. PCA, gives the angle and a first approximation for the rotatedRect's center
2. Get the contour of the binary shape, rotate it into upright position, get min/max of X and Y coordinates to get the width and height of the bounding rect
3. Subtract half the width (height) from maximum X (Y) to get the center point in the "upright space"

4. Rotate this center point back by the inverse rotation matrix

   cv::RotatedRect Utilities::getBoundingRectPCA( cv::Mat& binaryImg ) {
   cv::RotatedRect result;
   
   //1. convert to matrix that contains point coordinates as column vectors
   int count = cv::countNonZero(binaryImg);
   if (count == 0) {
       std::cout << "Utilities::getBoundingRectPCA() encountered 0 pixels in binary image!" << std::endl;
       return cv::RotatedRect();
   }
   
   cv::Mat data(2, count, CV_32FC1);
   int dataColumnIndex = 0;
   for (int row = 0; row < binaryImg.rows; row++) {
       for (int col = 0; col < binaryImg.cols; col++) {
           if (binaryImg.at<unsigned char>(row, col) != 0) {
               data.at<float>(0, dataColumnIndex) = (float) col; //x coordinate
               data.at<float>(1, dataColumnIndex) = (float) (binaryImg.rows - row); //y coordinate, such that y axis goes up
               ++dataColumnIndex;
           }
       }
   }
   
   //2. perform PCA
   const int maxComponents = 1;
   cv::PCA pca(data, cv::Mat() /*mean*/, CV_PCA_DATA_AS_COL, maxComponents);
   //result is contained in pca.eigenvectors (as row vectors)
   //std::cout << pca.eigenvectors << std::endl;
   
   //3. get angle of principal axis
   float dx = pca.eigenvectors.at<float>(0, 0);
   float dy = pca.eigenvectors.at<float>(0, 1);
   float angle = atan2f(dy, dx)  / (float)CV_PI*180.0f;
   
   //find the bounding rectangle with the given angle, by rotating the contour around the mean so that it is up-right
   //easily finding the bounding box then
   cv::Point2f center(pca.mean.at<float>(0,0), binaryImg.rows - pca.mean.at<float>(1,0));
   cv::Mat rotationMatrix = cv::getRotationMatrix2D(center, -angle, 1);
   cv::Mat rotationMatrixInverse = cv::getRotationMatrix2D(center, angle, 1);
   
   std::vector<std::vector<cv::Point> > contours;
   cv::findContours(binaryImg, contours, CV_RETR_EXTERNAL, CV_CHAIN_APPROX_SIMPLE);
   if (contours.size() != 1) {
       std::cout << "Warning: found " << contours.size() << " contours in binaryImg (expected one)" << std::endl;
       return result;
   }
   
   //turn vector of points into matrix (with points as column vectors, with a 3rd row full of 1's, i.e. points are converted to extended coords)
   cv::Mat contourMat(3, contours[0].size(), CV_64FC1);
   double* row0 = contourMat.ptr<double>(0);
   double* row1 = contourMat.ptr<double>(1);
   double* row2 = contourMat.ptr<double>(2);
   for (int i = 0; i < (int) contours[0].size(); i++) {
       row0[i] = (double) (contours[0])[i].x;
       row1[i] = (double) (contours[0])[i].y;
       row2[i] = 1;
   }
   
   cv::Mat uprightContour = rotationMatrix*contourMat;
   
   //get min/max in order to determine width and height
   double minX, minY, maxX, maxY;
   cv::minMaxLoc(cv::Mat(uprightContour, cv::Rect(0, 0, contours[0].size(), 1)), &minX, &maxX); //get minimum/maximum of first row
   cv::minMaxLoc(cv::Mat(uprightContour, cv::Rect(0, 1, contours[0].size(), 1)), &minY, &maxY); //get minimum/maximum of second row
   
   int minXi = cvFloor(minX);
   int minYi = cvFloor(minY);
   int maxXi = cvCeil(maxX);
   int maxYi = cvCeil(maxY);
   
   //fill result
   result.angle = angle;
   result.size.width = (float) (maxXi - minXi);
   result.size.height = (float) (maxYi - minYi);
   
   //Find the correct center:
   cv::Mat correctCenterUpright(3, 1, CV_64FC1);
   correctCenterUpright.at<double>(0, 0) = maxX - result.size.width/2;
   correctCenterUpright.at<double>(1,0) = maxY - result.size.height/2;
   correctCenterUpright.at<double>(2,0) = 1;
   cv::Mat correctCenterMat = rotationMatrixInverse*correctCenterUpright;
   cv::Point correctCenter = cv::Point(cvRound(correctCenterMat.at<double>(0,0)), cvRound(correctCenterMat.at<double>(1,0)));
   
   result.center = correctCenter;
   
   return result;
   

   }

Answer 2

If understand the problem correctly, you're saying the method of using findContours and minAreaRectsuffers from jitter/wobbling due to the noisy input data. PCA is not more robust against this noise, so I don't see why you think finding the orientation of the pattern this way won't be as bad as your current code.

If you need temporal smoothness a commonly used and simple solution is to use a filter, even a very simple filter like an alpha-beta filter probably gives you the smoothness you want. Say at frame n you estimate the parameters of the rotated rectangle A, and in frame n+1 you have the rectangle with the estimated parameters B. Instead of drawing the rectangle with B you find C which is between A and B, and then draw a rectangle with C in frame n+1.

Answer 3

Here's another approach (just a guess)

Wikipedia page on Principal Component Analysis says:

PCA can be thought of as fitting an n-dimensional ellipsoid to the data ...

And as your data is 2D, you can use the cv::fitEllipse function to fit an ellipse to your data and use the coordinates of the generated RotatedRect to calculate the angle. This gives better results as compared to cv::minAreaRect.