What is the best/fastest way to convert CMSampleBufferRef to OpenCV IplImage?

问题

I am writing an iPhone app that does some sort of real-time image detection with OpenCV. What is the best way to convert a CMSampleBufferRef image from the camera (I'm using AVCaptureVideoDataOutputSampleBufferDelegate of AVFoundation) into an IplImage that OpenCV understands? The conversion needs to be fast enough so it can run realtime.

- (void)captureOutput:(AVCaptureOutput *)captureOutput
    didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
    fromConnection:(AVCaptureConnection *)connection
{
  NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

  // Convert CMSampleBufferRef into IplImage
  IplImage *openCVImage = ???(sampleBuffer);

  // Do OpenCV computations realtime
  // ...

  [pool release];
}

Thanks in advance.

回答1:

This sample code is based on Apple's sample to manage CMSampleBuffer's pointer:

- (IplImage *)createIplImageFromSampleBuffer:(CMSampleBufferRef)sampleBuffer {
    IplImage *iplimage = 0;
    if (sampleBuffer) {
        CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
        CVPixelBufferLockBaseAddress(imageBuffer, 0);

        // get information of the image in the buffer
        uint8_t *bufferBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
        size_t bufferWidth = CVPixelBufferGetWidth(imageBuffer);
        size_t bufferHeight = CVPixelBufferGetHeight(imageBuffer);

        // create IplImage
        if (bufferBaseAddress) {
            iplimage = cvCreateImage(cvSize(bufferWidth, bufferHeight), IPL_DEPTH_8U, 4);
            iplimage->imageData = (char*)bufferBaseAddress;
        }

        // release memory
        CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
    }
    else
        DLog(@"No sampleBuffer!!");

    return iplimage;
}

You need to create a 4-channel IplImage because the Phone's camera buffer is in BGRA.

To my experience, this conversion is fast enough to be done in a real-time application, but of course, anything you will add to it will cost time, especially with OpenCV.

回答2:

"iplimage->imageData = (char*)bufferBaseAddress;" will lead to memory leak.

It should be "memcpy(iplimage->imageData, (char*)bufferBaseAddress, iplimage->imageSize);"

so the complete coded is:

-(IplImage *)createIplImageFromSampleBuffer:(CMSampleBufferRef)sampleBuffer {
  IplImage *iplimage = 0;

  if (sampleBuffer) {
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, 0);

    // get information of the image in the buffer
    uint8_t *bufferBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
    size_t bufferWidth = CVPixelBufferGetWidth(imageBuffer);
    size_t bufferHeight = CVPixelBufferGetHeight(imageBuffer);

    // create IplImage
    if (bufferBaseAddress) {
        iplimage = cvCreateImage(cvSize(bufferWidth, bufferHeight), IPL_DEPTH_8U, 4);

        //iplimage->imageData = (char*)bufferBaseAddress; 
        memcpy(iplimage->imageData, (char*)bufferBaseAddress, iplimage->imageSize);
    }

    // release memory
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
}
else
    DLog(@"No sampleBuffer!!");

return iplimage;

}

来源：https://stackoverflow.com/questions/5176209/what-is-the-best-fastest-way-to-convert-cmsamplebufferref-to-opencv-iplimage

标签

iphone

OpenCV

real-time

avfoundation