open cv ios video processing

Question

I\'m trying to do the tutorial found here for ios video processing with openCv framework.

I\'ve successfully loaded the ios openCv framework to my project -

Answer 1

Here is the conversion that I use. You lock the pixel buffer, create a cv::Mat, process with the cv::Mat, then unlock the pixel buffer.

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

CVPixelBufferLockBaseAddress( pixelBuffer, 0 );

int bufferWidth = CVPixelBufferGetWidth(pixelBuffer);
int bufferHeight = CVPixelBufferGetHeight(pixelBuffer);
int bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
unsigned char *pixel = (unsigned char *)CVPixelBufferGetBaseAddress(pixelBuffer);
cv::Mat image = cv::Mat(bufferHeight,bufferWidth,CV_8UC4,pixel, bytesPerRow); //put buffer in open cv, no memory copied
//Processing here

//End processing
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );
}

The above method does not copy any memory and as such you do not own the memory, pixelBuffer will free it for you. If you want your own copy of the buffer, just do

cv::Mat copied_image = image.clone();

Answer 2

This is the updated version of the code in the previous accepted answer which should work with any iOS device.

Since bufferWidth is not equal to bytePerRow at least on iPhone 6 and iPhone 6+, we need to specify the number of byte in each rows as the last argument to the cv::Mat constructor.

CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

CVPixelBufferLockBaseAddress(pixelBuffer, 0);

int bufferWidth = CVPixelBufferGetWidth(pixelBuffer);
int bufferHeight = CVPixelBufferGetHeight(pixelBuffer);
int bytePerRow = CVPixelBufferGetBytesPerRow(pixelBuffer);
unsigned char *pixel = (unsigned char *) CVPixelBufferGetBaseAddress(pixelBuffer);
cv::Mat image = cv::Mat(bufferHeight, bufferWidth, CV_8UC4, pixel, bytePerRow); 

// Process you cv::Mat here

CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);

The code has been tested on my iPhone5, iPhone6 and iPhone6+ running iOS 10.